incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oded Rosen <>
Subject Chukwa customized ReduceProcessor keys
Date Sun, 28 Mar 2010 13:52:25 GMT
Hey everyone,

Thanks to your help (especially by Eric & Jerome), I've managed to write my
own little demux processor, including a customized mapper & reducer, for my
data type.
For now, all of my map output is sent to only reduce process (although
Chukwa opens 8 different reduce processes in each demux run).

I would like to exploit the whole cluster, and to have multiple reduce
processes (same reducer class, of course, just many instances of them).
I've tried to do it by setting different values to ChukwaRecordKey.setKey()
in my mapper:

protected void parse(String recordEntry,
            OutputCollector<ChukwaRecordKey, ChukwaRecord> output,
            Reporter reporter) throws Throwable {

key = new ChukwaRecordKey();
String *keyStr* = DATA_TYPE + Math.floor((NUM_OF_REDUCERS*Math.random()))+1;
ChukwaRecord record = new ChukwaRecord();
this.buildGenericRecord(record, null, timestamp, *keyStr*);

.... (record logic)....

output.collect(key, record);


Although I have multiple keys, all of the records are still sent to the same
reducer process.
How can I send records to different processes?

Thanks a lot,

View raw message