chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: speeding up demux
Date Fri, 27 May 2011 15:58:21 GMT
I would recommend to skip Chukwa 0.4 and go to the trunk.  In addition, use HBaseWriter to
stream data into HBase in parallel, hence, the data can be processed in near real time for
demux.

Regards,
Eric

On 5/26/11 8:30 PM, "Bill Graham" <billgraham@gmail.com> wrote:

This seems possible, but one thing that would need to be changed is the directories that demux
uses. For example:
demuxProcessing/mrInput
demuxProcessing/mrOutput

These would need to dynamic directories with the timestamp or something else in them to keep
two jobs from interfering with each other.

On Thu, May 26, 2011 at 8:23 PM, Corbin Hoenes <corbin@tynt.com> wrote:
Finding demux to be a bit too slow for our needs.  It seems like only 1 runs at a time; is
there some technical reason why we couldn't run a couple in parallel?  If so any hints on
how difficult it would be to run multiple demuxers at a time?





Mime
View raw message