incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Fiala <e...@fiala.ca>
Subject Supercharging Chukwa
Date Fri, 13 Aug 2010 16:11:56 GMT
Hello all,
We would like to bring our production Chukwa (0.3.0) infrastructure to the
next level.

Currently, we have 5 machines generating 400GB per day (80GB in single log,
per machine).
These are using chukwa-agent CharFileTailingAdaptorUTF8.  Of
note, chukwaAgent.fileTailingAdaptor.maxReadSize has been upped to 4000000.
 We've left httpConnector.maxPostSize to default.

The agents are sending to 3 chukwa-collectors which are simply gateways into
HDFS (one also handles demux/processing - but this doesn't appear to be the
wall... yet).  The agents have all three collectors listed in their conf.

We are hitting walls somewhere, the whole 400GB is worked all the way into
our repos over the course of the day, but during peeks we are falling
upwards of 1-2 hours behind between being written to the tailed log and
hitting hdfs://chukwa/logs as a .chukwa.
Further we have observed that hdfs://chukwa/logs in our setup does not fill
faster than 2GB per 5 minute period.  This is whether we use 2 chukwa
collectors or 3.  This is further discouragement once foreseeable growth
takes us to over ~ 575GB per day.

All the machines are definitely not load bound, have noticed that chukwa was
built with low resource utilization in mind - one thought is if this could
be tweaked we could probably get more data through quicker.

We have toyed with changing default Xmx or like value but don't want to
start turning too many knobs before consulting the experts, considering all
the pieces involved it's probably wise.  Scaling out is also an option, but
I'm determined to squeeze x10 or more than current out of these multicore
machines.

Any suggestions are welcome,
Thanks.
EF

Mime
View raw message