chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <>
Subject Re: Extend demux aliases to reducers?
Date Thu, 11 Mar 2010 04:34:19 GMT
Having config file to map out the output filename template is nice.
However, I think it is probably secondary.  I think having parser decide
where to put the filename is a good start.


On 3/10/10 10:02 AM, "Guillermo PĂ©rez" <> wrote:

> Right now the conf/chukwa-demux-conf.xml file defines a set of aliases
> for mappers, so you just specify a Name of the record, and the
> appropriate mapper class is used. I would like to do more or less the
> same for reducers.
> My mapper generates different chukwa records with different keys. For
> example record "network" and record "disk". But the reducer class is
> the same, just makes some calculations. Right now if I specify the
> same reducer class in the mapper, the output file is the name of the
> reducer, so both types of records end in the same file.
> I would like to just make key.setReduceType(ReduceAlias). That
> ReduceAlias will be the name of the output file, but the actual class
> for the reduce will be defined in the config, the same way we do for
> mappers. Or maybe -again- setting the output file name in the key as I
> suggested for the cluster destination, but I think the first will be
> more general, but may tie code to config (a mapper sets reduce type
> "foo" in code, and needs "foo" to be defined in the config) . What do
> you think? What will be the best approach?

View raw message