incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Himanshu Gahlot <himanshu.gahlo...@gmail.com>
Subject Re: collector not closing files larger than 64MB
Date Tue, 12 Jul 2011 23:57:31 GMT
Hi Ari,

The chukwa dataflow documentation
(http://incubator.apache.org/chukwa/docs/r0.4.0/dataflow.html)
mentions -  "Collectors write chunks to logs/*.chukwa files until a
64MB chunk size is reached or a given time interval has passed." I
think this should be corrected.

Thanks,
Himanshu



On Tue, Jul 12, 2011 at 4:18 PM, Ariel Rabkin <asrabkin@gmail.com> wrote:
> Howdy.
>
> There isn't such a check; if there's documentation somewhere that
> suggests there is, let us know where and we can fix it. In general,
> the goal is to have .done files as large as possible while compatible
> with SLAs; there wasn't any intent to have them only be one block
> long.
>
> --Ari
>
> On Tue, Jul 12, 2011 at 3:57 PM, Himanshu Gahlot
> <himanshu.gahlot86@gmail.com> wrote:
>> Hi,
>>
>> We have a system where the .chukwa files produced are larger than
>> 64MB, but I do not see them being closed by collector and convert to
>> .done file. They are closing only at the scheduled time and hence are
>> greater than the block size (64MB). I do not see a check to close
>> files larger than a block size in SeqFileWriter class. Where is this
>> check made in the code ?
>>
>> Thanks,
>> Himanshu
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

Mime
View raw message