Return-Path: X-Original-To: apmail-incubator-chukwa-user-archive@www.apache.org Delivered-To: apmail-incubator-chukwa-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9DC019E8A for ; Thu, 27 Oct 2011 19:01:25 +0000 (UTC) Received: (qmail 65148 invoked by uid 500); 27 Oct 2011 19:01:25 -0000 Delivered-To: apmail-incubator-chukwa-user-archive@incubator.apache.org Received: (qmail 65122 invoked by uid 500); 27 Oct 2011 19:01:25 -0000 Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@incubator.apache.org Delivered-To: mailing list chukwa-user@incubator.apache.org Received: (qmail 65115 invoked by uid 99); 27 Oct 2011 19:01:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Oct 2011 19:01:25 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of straightflush@gmail.com designates 209.85.215.175 as permitted sender) Received: from [209.85.215.175] (HELO mail-ey0-f175.google.com) (209.85.215.175) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Oct 2011 19:01:17 +0000 Received: by eyd9 with SMTP id 9so2924686eyd.6 for ; Thu, 27 Oct 2011 12:00:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ZSNXBWmn+4OVgZDJ38ZARcR0XYF35AoedDzoCqBw+yQ=; b=e0HIdDnhCkix5iMf2UrYyY61xe7Lc3jnQvZMxg45JE4hox4Ph3EOVRMzKPRnVMM1Ku jaJlSOHDs902T9MRIhWMXeiZXZhRDwyE1LYqDYVtvuRDxilthmbfOO3w0MhxqLHZjKlV vWADJHET9hfIkTG0pvnABCDoRCmp0DClQlHas= Received: by 10.14.4.142 with SMTP id 14mr4990722eej.38.1319742056778; Thu, 27 Oct 2011 12:00:56 -0700 (PDT) Received: from mail-bw0-f47.google.com (mail-bw0-f47.google.com [209.85.214.47]) by mx.google.com with ESMTPS id z58sm17525902eea.3.2011.10.27.12.00.54 (version=SSLv3 cipher=OTHER); Thu, 27 Oct 2011 12:00:55 -0700 (PDT) Received: by bkat8 with SMTP id t8so2431622bka.6 for ; Thu, 27 Oct 2011 12:00:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.59.5 with SMTP id v5mr3927626obq.78.1319742053615; Thu, 27 Oct 2011 12:00:53 -0700 (PDT) Received: by 10.182.121.5 with HTTP; Thu, 27 Oct 2011 12:00:53 -0700 (PDT) In-Reply-To: References: <914FD9DE-036F-4F89-A170-91B53B760C9D@gmail.com> <56F189B0-0F4C-4792-9208-DFC89775112A@gmail.com> Date: Thu, 27 Oct 2011 15:00:53 -0400 Message-ID: Subject: Re: Chukwa trunk does not write to HBase From: AD To: chukwa-user@incubator.apache.org Content-Type: multipart/alternative; boundary=14dae93a151b4d085e04b04c641e X-Virus-Checked: Checked by ClamAV on apache.org --14dae93a151b4d085e04b04c641e Content-Type: text/plain; charset=ISO-8859-1 Hmm, even after removing i am still seeing 2 tails being started with lsof. I am only using 1 filetailer.CharFileTailingAdaptorUTF8 in initial_adaptors. On Thu, Oct 27, 2011 at 1:07 PM, AD wrote: > argh. yes. I guess # is not a comment in the initial_adaptors file :-) i > commented out the old filetailer.FileTailingAdaptor for the > new filetailer.CharFileTailingAdaptorUTF8 > > anyway thanks for the help. > > > > On Thu, Oct 27, 2011 at 12:19 PM, Eric Yang wrote: > >> Do you have two data stream that are tailing the same file? To verify: >> >> telnet agent-host 9093 >> list >> >> Make sure you are not streaming the same data twice. >> >> regards, >> Eric >> >> On Oct 27, 2011, at 7:25 AM, AD wrote: >> >> > I migrated to the filetailer.CharFileTailingAdaptorUTF8 which ensured >> newline breaks but i am now getting 2 of every entry. Put the below debug >> code into the demux parser, any ideas why i would be seeing 2 of every event >> in the log? >> > >> > throws Throwable { >> > try { >> > String lines[] = recordEntry.split("\\r?\\n"); >> > FileWriter out = new FileWriter("/tmp/demux.test",true); >> > PrintWriter p = new PrintWriter(out); >> > for(int i = 0; i < lines.length; i++) { >> > log.warn("*** TRYING TO PARSE **** " + lines[i]); >> > p.println(lines[i]); >> > } >> > p.close(); >> > >> > On Thu, Oct 27, 2011 at 6:39 AM, AD wrote: >> > yep that was it thanks. For the "recordEntry" variable, is there any >> way to guarantee how this is structured? I am testing tailing a file and i >> notice recordEntry in the parser is around 20-30 lines from the logfile, is >> this expected? can we safely assume newline termination and just loop >> through? I was expecting 1 recordEntry for each line of the logfile but i >> think this has to do with it being a chukwaRecord not a log entry. >> > >> > >> > On Thu, Oct 27, 2011 at 1:25 AM, Eric Yang wrote: >> > Do you have both chukwa-core.jar and chukwa-core-0.4.0.jar? Run jar tf >> chukwa-core-0.4.0.jar | grep TsProcessor2, does chukwa-core-0.4.0.jar file >> have TsProcessor2.class? >> > >> > regards, >> > Eric >> > >> > On Oct 26, 2011, at 7:59 PM, AD wrote: >> > >> > > Hey Eric, >> > > >> > > So as a test, i copied TsProcessor.java to TsProcessor2.java. I >> changed the references for TsProcessor to be TsProcessor2 and updated my >> chukwa-demux.conf to be >> > > >> > > >> > > TsProcessor >> > > >> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2 >> > > Parser class for >> > > >> > > >> > > I then ran ant in the root and copied build/collector-0.4.0.war and >> build/chukwa-core.jar to the root and started collector and demux. >> > > >> > > I am now getting the following errors, any ideas ? >> > > >> > > >> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.UnknownRecordTypeException: >> Unknown >> parserClass:org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2 >> > > >> > > Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2 >> > > >> > > On Wed, Oct 26, 2011 at 1:00 PM, Eric Yang wrote: >> > > Yes, write a mapper class which extends >> org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcessor.java. >> There are several extraction class in >> org/apache/hadoop/chukwa/extraction/demux/processor/mapper as examples. >> Once the extraction class is written, configure chukwa-demux-conf.xml to >> map the data type to the new extraction class. >> > > >> > > regards, >> > > Eric >> > > >> > > On Oct 26, 2011, at 5:48 AM, AD wrote: >> > > >> > > > interesting. So demux is now part of the collector and does not >> need to be run as another job? >> > > > >> > > > Since demux is doing basic ETL into hbase, is there a way to >> actually parse the fields of the log record and insert those into sep fields >> into Hbase to run mapreduce there (instead of the whole body as a field is >> what I think is happening) >> > > > >> > > > On Wed, Oct 26, 2011 at 12:59 AM, SALAMI Patrick < >> salami@systransoft.com> wrote: >> > > > Ok, thanks! >> > > > >> > > > -----Original Message----- >> > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric >> Yang >> > > > Sent: Tuesday, October 25, 2011 6:19 PM >> > > > To: chukwa-user@incubator.apache.org >> > > > Subject: Re: Chukwa trunk does not write to HBase >> > > > >> > > > Demuxer for Chukwa 0.4 is a mapreduce job acting as ETL process to >> convert the data into semi structure format for further processing. For >> Chukwa trunk with HBase, demux is running as part of collector for ETL. >> Hence, there is no need to run demux process. >> > > > >> > > > regards, >> > > > Eric >> > > > >> > > > On Oct 25, 2011, at 3:49 PM, SALAMI Patrick wrote: >> > > > >> > > > > Also, while we are chatting, I was hoping to understand the role >> of the demuxer. I am assuming that HICC pulls all of its data from HBase. If >> using HBase, is it still necessary to run the demuxer? I didn't see any >> mention of it in the latest quick start guide. >> > > > > >> > > > > Thanks! >> > > > > >> > > > > Patrick >> > > > > >> > > > > -----Original Message----- >> > > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric >> Yang >> > > > > Sent: Tuesday, October 25, 2011 2:45 PM >> > > > > To: chukwa-user@incubator.apache.org >> > > > > Subject: Re: Chukwa trunk does not write to HBase >> > > > > >> > > > > Hadoop trunk will require a different configuration than the one >> described in the Quick_Start guide. >> > > > > >> > > > > 1. Apply this patch: HADOOP-7436, and rebuild hadoop. >> > > > > 2. Copy hadoop-metrics2.properties enclosed in this message to >> HADOOP_CONF_DIR >> > > > > >> > > > > Restart hadoop. You might need to match the HBase table schema >> with the metrics emitted by Hadoop metrics 2 framework. >> > > > > Hope this helps. >> > > > > >> > > > > >> > > > > This message contains confidential information and is intended >> only for the individual(s) named. If you are not the named addressee you >> should not disseminate, distribute or copy this e-mail. Please notify the >> sender immediately by e-mail if you have received this e-mail by mistake and >> delete this e-mail from your system. >> > > > >> > > > >> > > > This message contains confidential information and is intended only >> for the individual(s) named. If you are not the named addressee you should >> not disseminate, distribute or copy this e-mail. Please notify the sender >> immediately by e-mail if you have received this e-mail by mistake and delete >> this e-mail from your system. >> > > > >> > > >> > > >> > >> > >> > >> >> > --14dae93a151b4d085e04b04c641e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hmm, even after removing i am still seeing 2 tails being started with lsof.= =A0I am only using 1=A0filetailer.CharFileTailingAdaptorUTF8 in initial_ad= aptors.



On Thu, Oct 2= 7, 2011 at 1:07 PM, AD <straightflush@gmail.com> wrote:
argh. yes. =A0I guess # is not a comment in= the initial_adaptors file :-) i commented out the old=A0filetailer.FileTai= lingAdaptor for the new=A0filetailer.CharFileTailingAdaptorUTF8

anyway thanks for the help.



On Thu, Oct 2= 7, 2011 at 12:19 PM, Eric Yang <eric818@gmail.com> wrote:
Do you have two data stream that are tailing the same file? =A0To verify:
telnet agent-host 9093
list

Make sure you are not streaming the same data twice.

regards,
Eric

On Oct 27, 2011, at 7:25 AM, AD wrote:

> I migrated to the =A0filetailer.CharFileTailingAdaptorUTF8 which ensur= ed newline breaks but i am now getting 2 of every entry. =A0Put the below d= ebug code into the demux parser, any ideas why i would be seeing 2 of every= event in the log?
>
> =A0throws Throwable {
> =A0 =A0 try {
> =A0 =A0 =A0 =A0 =A0 String lines[] =3D recordEntry.split("\\r?\\n= ");
> =A0 =A0 =A0 =A0 =A0 FileWriter out =3D new FileWriter("/tmp/demux= .test",true);
> =A0 =A0 =A0 =A0 =A0 PrintWriter p =3D new PrintWriter(out);
> =A0 =A0 =A0 =A0 =A0 for(int i =3D 0; i < lines.length; i++) {
> =A0 =A0 =A0 =A0 =A0 =A0log.warn("*** TRYING TO PARSE **** " = + lines[i]);
> =A0 =A0 =A0 =A0 =A0 =A0p.println(lines[i]);
> =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 =A0 p.close();
>
> On Thu, Oct 27, 2011 at 6:39 AM, AD <straightflush@gmail.com> wrote:
> yep that was it thanks. =A0For the "recordEntry" variable, i= s there any way to guarantee how this is structured? =A0I am testing tailin= g a file and i notice recordEntry in the parser is around 20-30 lines from = the logfile, is this expected? =A0can we safely assume newline termination = and just loop through? =A0I was expecting 1 recordEntry for each line of th= e logfile but i think this has to do with it being a chukwaRecord not a log= entry.
>
>
> On Thu, Oct 27, 2011 at 1:25 AM, Eric Yang <eric818@gmail.com> wrote:
> Do you have both chukwa-core.jar and chukwa-core-0.4.0.jar? =A0Run jar= tf chukwa-core-0.4.0.jar | grep TsProcessor2, does chukwa-core-0.4.0.jar f= ile have TsProcessor2.class?
>
> regards,
> Eric
>
> On Oct 26, 2011, at 7:59 PM, AD wrote:
>
> > Hey Eric,
> >
> > =A0So as a test, i copied TsProcessor.java to TsProcessor2.java. = =A0I changed the references for TsProcessor to be TsProcessor2 and updated = my chukwa-demux.conf to be
> >
> > <property>
> > =A0 =A0 <name>TsProcessor</name>
> > =A0 =A0 <value>org.apache.hadoop.chukwa.extraction.demux.pr= ocessor.mapper.TsProcessor2</value>
> > =A0 =A0 <description>Parser class for </description><= br> > > =A0 =A0</property>
> >
> > I then ran ant in the root and copied build/collector-0.4.0.war a= nd build/chukwa-core.jar to the root and started collector and demux.
> >
> > I am now getting the following errors, any ideas ?
> >
> > org.apache.hadoop.chukwa.extraction.demux.processor.mapper.Unknow= nRecordTypeException: Unknown parserClass:org.apache.hadoop.chukwa.extracti= on.demux.processor.mapper.TsProcessor2
> >
> > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.ch= ukwa.extraction.demux.processor.mapper.TsProcessor2
> >
> > On Wed, Oct 26, 2011 at 1:00 PM, Eric Yang <eric818@gmail.com> wrote:
> > Yes, write a mapper class which extends org/apache/hadoop/chukwa/= extraction/demux/processor/mapper/AbstractProcessor.java. =A0There are seve= ral extraction class in org/apache/hadoop/chukwa/extraction/demux/processor= /mapper as examples. =A0Once the extraction class is written, configure chu= kwa-demux-conf.xml to map the data type to the new extraction class.
> >
> > regards,
> > Eric
> >
> > On Oct 26, 2011, at 5:48 AM, AD wrote:
> >
> > > interesting. =A0So demux is now part of the collector and do= es not need to be run as another job?
> > >
> > > Since demux is doing basic ETL into hbase, is there a way to= actually parse the fields of the log record and insert those into sep fiel= ds into Hbase to run mapreduce there (instead of the whole body as a field = is what I think is happening)
> > >
> > > On Wed, Oct 26, 2011 at 12:59 AM, SALAMI Patrick <salami@systransoft.com= > wrote:
> > > Ok, thanks!
> > >
> > > -----Original Message-----
> > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric Yang > > > Sent: Tuesday, October 25, 2011 6:19 PM
> > > To: chukwa-user@incubator.apache.org
> > > Subject: Re: Chukwa trunk does not write to HBase
> > >
> > > Demuxer for Chukwa 0.4 is a mapreduce job acting as ETL proc= ess to convert the data into semi structure format for further processing. = =A0For Chukwa trunk with HBase, demux is running as part of collector for E= TL. =A0Hence, there is no need to run demux process.
> > >
> > > regards,
> > > Eric
> > >
> > > On Oct 25, 2011, at 3:49 PM, SALAMI Patrick wrote:
> > >
> > > > Also, while we are chatting, I was hoping to understand= the role of the demuxer. I am assuming that HICC pulls all of its data fro= m HBase. If using HBase, is it still necessary to run the demuxer? I didn&#= 39;t see any mention of it in the latest quick start guide.
> > > >
> > > > Thanks!
> > > >
> > > > Patrick
> > > >
> > > > -----Original Message-----
> > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric Ya= ng
> > > > Sent: Tuesday, October 25, 2011 2:45 PM
> > > > To: chukwa-user@incubator.apache.org
> > > > Subject: Re: Chukwa trunk does not write to HBase
> > > >
> > > > Hadoop trunk will require a different configuration tha= n the one described in the Quick_Start guide.
> > > >
> > > > 1. Apply this patch: HADOOP-7436, and rebuild hadoop. > > > > 2. Copy hadoop-metrics2.properties enclosed in this mes= sage to HADOOP_CONF_DIR
> > > >
> > > > Restart hadoop. =A0You might need to match the HBase ta= ble schema with the metrics emitted by Hadoop metrics 2 framework.
> > > > Hope this helps.
> > > >
> > > >
> > > > This message contains confidential information and is i= ntended only for the individual(s) named. If you are not the named addresse= e you should not disseminate, distribute or copy this e-mail. Please notify= the sender immediately by e-mail if you have received this e-mail by mista= ke and delete this e-mail from your system.
> > >
> > >
> > > This message contains confidential information and is intend= ed only for the individual(s) named. If you are not the named addressee you= should not disseminate, distribute or copy this e-mail. Please notify the = sender immediately by e-mail if you have received this e-mail by mistake an= d delete this e-mail from your system.
> > >
> >
> >
>
>
>



--14dae93a151b4d085e04b04c641e--