Return-Path: X-Original-To: apmail-incubator-chukwa-user-archive@www.apache.org Delivered-To: apmail-incubator-chukwa-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B5DCE9F4B for ; Thu, 27 Oct 2011 05:26:58 +0000 (UTC) Received: (qmail 5360 invoked by uid 500); 27 Oct 2011 05:26:58 -0000 Delivered-To: apmail-incubator-chukwa-user-archive@incubator.apache.org Received: (qmail 5336 invoked by uid 500); 27 Oct 2011 05:26:57 -0000 Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@incubator.apache.org Delivered-To: mailing list chukwa-user@incubator.apache.org Delivered-To: moderator for chukwa-user@incubator.apache.org Received: (qmail 5228 invoked by uid 99); 27 Oct 2011 05:25:31 -0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Sender: Eric Yang Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Apple Message framework v1251.1) Subject: Re: Chukwa trunk does not write to HBase From: Eric Yang In-Reply-To: Date: Wed, 26 Oct 2011 22:25:00 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References:

<914FD9DE-036F-4F89-A170-91B53B760C9D@gmail.com>

<56F189B0-0F4C-4792-9208-DFC89775112A@gmail.com>

To: chukwa-user@incubator.apache.org X-Mailer: Apple Mail (2.1251.1) Do you have both chukwa-core.jar and chukwa-core-0.4.0.jar? Run jar tf = chukwa-core-0.4.0.jar | grep TsProcessor2, does chukwa-core-0.4.0.jar = file have TsProcessor2.class? regards, Eric On Oct 26, 2011, at 7:59 PM, AD wrote: > Hey Eric, >=20 > So as a test, i copied TsProcessor.java to TsProcessor2.java. I = changed the references for TsProcessor to be TsProcessor2 and updated my = chukwa-demux.conf to be=20 >=20 > > TsProcessor > = org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProces= sor2 > Parser class for > >=20 > I then ran ant in the root and copied build/collector-0.4.0.war and = build/chukwa-core.jar to the root and started collector and demux. >=20 > I am now getting the following errors, any ideas ? >=20 > = org.apache.hadoop.chukwa.extraction.demux.processor.mapper.UnknownRecordTy= peException: Unknown = parserClass:org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsP= rocessor2 >=20 > Caused by: java.lang.ClassNotFoundException: = org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2 >=20 > On Wed, Oct 26, 2011 at 1:00 PM, Eric Yang wrote: > Yes, write a mapper class which extends = org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcess= or.java. There are several extraction class in = org/apache/hadoop/chukwa/extraction/demux/processor/mapper as examples. = Once the extraction class is written, configure chukwa-demux-conf.xml to = map the data type to the new extraction class. >=20 > regards, > Eric >=20 > On Oct 26, 2011, at 5:48 AM, AD wrote: >=20 > > interesting. So demux is now part of the collector and does not = need to be run as another job? > > > > Since demux is doing basic ETL into hbase, is there a way to = actually parse the fields of the log record and insert those into sep = fields into Hbase to run mapreduce there (instead of the whole body as a = field is what I think is happening) > > > > On Wed, Oct 26, 2011 at 12:59 AM, SALAMI Patrick = wrote: > > Ok, thanks! > > > > -----Original Message----- > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric = Yang > > Sent: Tuesday, October 25, 2011 6:19 PM > > To: chukwa-user@incubator.apache.org > > Subject: Re: Chukwa trunk does not write to HBase > > > > Demuxer for Chukwa 0.4 is a mapreduce job acting as ETL process to = convert the data into semi structure format for further processing. For = Chukwa trunk with HBase, demux is running as part of collector for ETL. = Hence, there is no need to run demux process. > > > > regards, > > Eric > > > > On Oct 25, 2011, at 3:49 PM, SALAMI Patrick wrote: > > > > > Also, while we are chatting, I was hoping to understand the role = of the demuxer. I am assuming that HICC pulls all of its data from = HBase. If using HBase, is it still necessary to run the demuxer? I = didn't see any mention of it in the latest quick start guide. > > > > > > Thanks! > > > > > > Patrick > > > > > > -----Original Message----- > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric = Yang > > > Sent: Tuesday, October 25, 2011 2:45 PM > > > To: chukwa-user@incubator.apache.org > > > Subject: Re: Chukwa trunk does not write to HBase > > > > > > Hadoop trunk will require a different configuration than the one = described in the Quick_Start guide. > > > > > > 1. Apply this patch: HADOOP-7436, and rebuild hadoop. > > > 2. Copy hadoop-metrics2.properties enclosed in this message to = HADOOP_CONF_DIR > > > > > > Restart hadoop. You might need to match the HBase table schema = with the metrics emitted by Hadoop metrics 2 framework. > > > Hope this helps. > > > > > > > > > This message contains confidential information and is intended = only for the individual(s) named. If you are not the named addressee you = should not disseminate, distribute or copy this e-mail. Please notify = the sender immediately by e-mail if you have received this e-mail by = mistake and delete this e-mail from your system. > > > > > > This message contains confidential information and is intended only = for the individual(s) named. If you are not the named addressee you = should not disseminate, distribute or copy this e-mail. Please notify = the sender immediately by e-mail if you have received this e-mail by = mistake and delete this e-mail from your system. > > >=20 >=20