chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: Chukwa trunk does not write to HBase
Date Wed, 26 Oct 2011 17:00:50 GMT
Yes, write a mapper class which extends org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcessor.java.
 There are several extraction class in org/apache/hadoop/chukwa/extraction/demux/processor/mapper
as examples.  Once the extraction class is written, configure chukwa-demux-conf.xml to map
the data type to the new extraction class.

regards,
Eric

On Oct 26, 2011, at 5:48 AM, AD wrote:

> interesting.  So demux is now part of the collector and does not need to be run as another
job?
> 
> Since demux is doing basic ETL into hbase, is there a way to actually parse the fields
of the log record and insert those into sep fields into Hbase to run mapreduce there (instead
of the whole body as a field is what I think is happening)
> 
> On Wed, Oct 26, 2011 at 12:59 AM, SALAMI Patrick <salami@systransoft.com> wrote:
> Ok, thanks!
> 
> -----Original Message-----
> From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric Yang
> Sent: Tuesday, October 25, 2011 6:19 PM
> To: chukwa-user@incubator.apache.org
> Subject: Re: Chukwa trunk does not write to HBase
> 
> Demuxer for Chukwa 0.4 is a mapreduce job acting as ETL process to convert the data into
semi structure format for further processing.  For Chukwa trunk with HBase, demux is running
as part of collector for ETL.  Hence, there is no need to run demux process.
> 
> regards,
> Eric
> 
> On Oct 25, 2011, at 3:49 PM, SALAMI Patrick wrote:
> 
> > Also, while we are chatting, I was hoping to understand the role of the demuxer.
I am assuming that HICC pulls all of its data from HBase. If using HBase, is it still necessary
to run the demuxer? I didn't see any mention of it in the latest quick start guide.
> >
> > Thanks!
> >
> > Patrick
> >
> > -----Original Message-----
> > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric Yang
> > Sent: Tuesday, October 25, 2011 2:45 PM
> > To: chukwa-user@incubator.apache.org
> > Subject: Re: Chukwa trunk does not write to HBase
> >
> > Hadoop trunk will require a different configuration than the one described in the
Quick_Start guide.
> >
> > 1. Apply this patch: HADOOP-7436, and rebuild hadoop.
> > 2. Copy hadoop-metrics2.properties enclosed in this message to HADOOP_CONF_DIR
> >
> > Restart hadoop.  You might need to match the HBase table schema with the metrics
emitted by Hadoop metrics 2 framework.
> > Hope this helps.
> >
> >
> > This message contains confidential information and is intended only for the individual(s)
named. If you are not the named addressee you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by
mistake and delete this e-mail from your system.
> 
> 
> This message contains confidential information and is intended only for the individual(s)
named. If you are not the named addressee you should not disseminate, distribute or copy this
e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by
mistake and delete this e-mail from your system.
> 


Mime
View raw message