incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <asrab...@gmail.com>
Subject Re: Problem in ChukwaRecord file contents
Date Thu, 10 Jun 2010 18:45:05 GMT
Howdy --

The .done files hold a sequence of Chunks. Each chunk will include
several lines, plus a table of record endpoints, indicating where the
linebreaks are.

The demux job, not the collector, uses this line break table in order
to output a set of ChukwaRecords, one line per record.

--Ari

On Thu, Jun 10, 2010 at 11:40 AM, Gerrit van Vuuren
<gvanvuuren@specificmedia.com> wrote:
> Hi,
>
> Ok then this is a bug.
>
> I have some general questions:
> Does this mean that each record in the done file written by the collector will only contain
one line? Or will the collector batch these up?
>
> Cheers,
> ------Original Message------
> From: Stuti Awasthi
> To: chukwa-user@hadoop.apache.org
> Cc: asrabkin@gmail.com
> Cc: corbin@tynt.com
> Cc: Gerrit van Vuuren
> Subject: RE: Problem in ChukwaRecord file contents
> Sent: 10 Jun 2010 06:02
>
> Currently I am using FileTailing Adapter. I will try with CharFileTailingAdaptorUTF8.
>
> -----Original Message-----
> From: Ariel Rabkin [mailto:asrabkin@gmail.com]
> Sent: Thursday, June 10, 2010 3:13 AM
> To: chukwa-user@hadoop.apache.org
> Subject: Re: Problem in ChukwaRecord file contents
>
> Yes, CharFileTailingAdaptorUTF8 is designed to do one line per record.
> If it does more than that, it's a bug.
>
> --Ari
>
> On Wed, Jun 9, 2010 at 8:00 AM, Corbin Hoenes <corbin@tynt.com> wrote:
>> Gerrit -
>> It's my understanding that the CharFileTailingAdaptorUTF8 sends only 1 line
>> per record.  Why can't Stuti just use this?
>> On Jun 8, 2010, at 7:55 AM, Gerrit Jansen van Vuuren wrote:
>>
>> Have a look at:
>>
>>
>> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.LWFTAdaptor
>>
>> protected int extractRecords(ChunkReceiver eq, long buffOffsetInFile,
>>       byte[] buf) throws InterruptedException {
>>
>>
>> org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8
>> protected int extractRecords(ChunkReceiver eq, long buffOffsetInFile,
>>       byte[] buf) throws InterruptedException {
>>
>> If you need one line per record you'd have to write your own adaptor. Maybe
>> subclass CharFileTailingAdaptorUTF8
>>
>>
>>
>> From: Stuti Awasthi [mailto:Stuti_Awasthi@persistent.co.in]
>> Sent: Tuesday, June 08, 2010 1:31 PM
>> To: chukwa-user@hadoop.apache.org; Gerrit van Vuuren
>> Subject: RE: Problem in ChukwaRecord file contents
>>
>> So is  that means
>
> ------Original Message Truncated------
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Mime
View raw message