chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <asrab...@gmail.com>
Subject Re: Increase size of Demux Output
Date Wed, 26 May 2010 19:24:41 GMT
The chunk size isn't relevant here -- that controls the size of chunks
inside the file, not files themselves.  You probably want to interval
the rotation interval, which tells collectors to bundle more data into
a file before marking it for processing.

Do you need large files for efficiency or to reduce namenode load?
The Chukwa philosophy is to do hourly and daily consolidation
mapReduce jobs to keep namenode load bounded.

--Ari

On Wed, May 26, 2010 at 10:24 AM, Stuti Awasthi
<Stuti_Awasthi@persistent.co.in> wrote:
> Hi all,
>
>
>
> I am new to chukwa and was trying to analyze chukwa output. I noticed that
> the demuxer output i.e files present in /repos/ *.evt files are generally of
> small sizes. In my case the maximum file size 1.2MB when agent was
> continuously running.
>
>
>
> I require demux output file to be 1GB or more in size. Is there any
> collector configuration setting through which I can increase the chunk size
> to greater than 64MB so that we can generate larger *.evt files.
>
>
>
> Any pointers will be helpful.
>
>
>
> Thanks in advance
>
> Stuti
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Mime
View raw message