Mailing-List: contact chukwa-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-dev@hadoop.apache.org
Message-ID: <1877420291.1245343687337.JavaMail.jira@brutus>
Date: Thu, 18 Jun 2009 09:48:07 -0700 (PDT)
From: "Jerome Boulon (JIRA)" <jira@apache.org>
To: chukwa-dev@hadoop.apache.org
Subject: [jira] Commented: (CHUKWA-306) Standalone (non-daemon) Chukwa
 operation
In-Reply-To: <516723908.1245276847477.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CHUKWA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721338#action_12721338 ] 

Jerome Boulon commented on CHUKWA-306:
--------------------------------------

The solution to this is to use the backfilling tool. 
This tool will produce your dataSink files (on per input file) then you just have to run demux like a standard M/R job.
By configuring chukwa-collector-conf.xml to use the local filesystem then you don't have to worry about hadoop config at all.


> Standalone (non-daemon) Chukwa operation
> ----------------------------------------
>
>                 Key: CHUKWA-306
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-306
>             Project: Hadoop Chukwa
>          Issue Type: Wish
>            Reporter: Jiaqi Tan
>            Priority: Critical
>
> This is an articulation of a possible alternative use of Chukwa as a standalone log analysis pipeline. This would enable users to read in existing logs from files, process (Demux) and perform analysis (e.g. current SALSA/Mochi toolchain) on them, and visualize them, without requiring the user to setup or run any daemons, nor database servers. 
> This can be presented as an alternative interface to Chukwa for the user, where the main architectural parts (Chunks, post-Demux SequenceFiles of ChukwaRecords, post-Demux-processing SequenceFiles of ChukwaRecords, and finally time-aggregated database entries for fast visualization) remain unchanged, and Chukwa is manifest as a set of files in HDFS. The main value that Chukwa then provides to users is 1. centralized one-stop-shop for log processing+analysis+anomaly detection, 2. the ability to use MapReduce to process logs, regardless of whether they had used Chukwa to collect the logs. 
> That way, the ability to process logs and analyze/do diagnosis is not tied to having to run the entire Chukwa daemon infrastructure, since many users who use Hadoop clusters may not have superuser access to those machines, e.g. users at universities using shared clusters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.