incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerome Boulon <>
Subject Re: Directly create chukwa records?
Date Thu, 25 Feb 2010 18:14:50 GMT
I don't see why you would ever want todo that since like Eric said you're
going to have a scaling issue but if you still want to do this then a quick
an easy way todo it, will be to use some code I wrote for the
validationframework, using the fact that M/R can run as a single java app.

But again, if you thing that one machine will be more than you ever need
then maybe Chukwa is not the right tools.

On 2/25/10 9:44 AM, "Eric Yang" <> wrote:

> This may be time consuming to write, but here are the steps:
> - Implement a File reader which extends BackfillingLoader
> - Implement another QueueToRecord which extends QueueToWriterConnector.
> - In QueueToRecord, pass each chunk to another wrapper class of Demux, which
> pass and collect Chukwa Records from mapper and reducer parsers, then write
> the output as sequence file.
> I don't recommend to attempt this because it will take longer to back fill
> the data with single machine to do the parsing and output, and it defeats
> the purpose of using Chukwa.
> Regards,
> Eric
> On 2/25/10 3:01 AM, "Guillermo PĂ©rez" <> wrote:
>> I'm considering the possibility of directly creating chukwa records as
>> part of the backfilling script, to speed up the process. Any clue on
>> how could I start with this?

View raw message