incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <eric...@gmail.com>
Subject Re: Chukwa trunk does not write to HBase
Date Fri, 28 Oct 2011 01:51:48 GMT
Glad it works.  Check point file should be located in CHUKWA_LOG_DIR.

regards,
Eric

On Oct 27, 2011, at 6:28 PM, AD wrote:

> that worked thanks.  Out of curiosity where is this persisted (i restarted and it came
back apparently).
> 
> 
> On Thu, Oct 27, 2011 at 4:11 PM, Eric Yang <eric818@gmail.com> wrote:
> Agent check point file may be storing the multiple adaptors, make sure you use port 9093
to stop the existing running adaptors.
> 
> regards,
> Eric
> 
> On Oct 27, 2011, at 12:00 PM, AD wrote:
> 
> > Hmm, even after removing i am still seeing 2 tails being started with lsof.  I am
only using 1 filetailer.CharFileTailingAdaptorUTF8 in initial_adaptors.
> >
> >
> >
> > On Thu, Oct 27, 2011 at 1:07 PM, AD <straightflush@gmail.com> wrote:
> > argh. yes.  I guess # is not a comment in the initial_adaptors file :-) i commented
out the old filetailer.FileTailingAdaptor for the new filetailer.CharFileTailingAdaptorUTF8
> >
> > anyway thanks for the help.
> >
> >
> >
> > On Thu, Oct 27, 2011 at 12:19 PM, Eric Yang <eric818@gmail.com> wrote:
> > Do you have two data stream that are tailing the same file?  To verify:
> >
> > telnet agent-host 9093
> > list
> >
> > Make sure you are not streaming the same data twice.
> >
> > regards,
> > Eric
> >
> > On Oct 27, 2011, at 7:25 AM, AD wrote:
> >
> > > I migrated to the  filetailer.CharFileTailingAdaptorUTF8 which ensured newline
breaks but i am now getting 2 of every entry.  Put the below debug code into the demux parser,
any ideas why i would be seeing 2 of every event in the log?
> > >
> > >  throws Throwable {
> > >     try {
> > >           String lines[] = recordEntry.split("\\r?\\n");
> > >           FileWriter out = new FileWriter("/tmp/demux.test",true);
> > >           PrintWriter p = new PrintWriter(out);
> > >           for(int i = 0; i < lines.length; i++) {
> > >            log.warn("*** TRYING TO PARSE **** " + lines[i]);
> > >            p.println(lines[i]);
> > >           }
> > >           p.close();
> > >
> > > On Thu, Oct 27, 2011 at 6:39 AM, AD <straightflush@gmail.com> wrote:
> > > yep that was it thanks.  For the "recordEntry" variable, is there any way to
guarantee how this is structured?  I am testing tailing a file and i notice recordEntry in
the parser is around 20-30 lines from the logfile, is this expected?  can we safely assume
newline termination and just loop through?  I was expecting 1 recordEntry for each line of
the logfile but i think this has to do with it being a chukwaRecord not a log entry.
> > >
> > >
> > > On Thu, Oct 27, 2011 at 1:25 AM, Eric Yang <eric818@gmail.com> wrote:
> > > Do you have both chukwa-core.jar and chukwa-core-0.4.0.jar?  Run jar tf chukwa-core-0.4.0.jar
| grep TsProcessor2, does chukwa-core-0.4.0.jar file have TsProcessor2.class?
> > >
> > > regards,
> > > Eric
> > >
> > > On Oct 26, 2011, at 7:59 PM, AD wrote:
> > >
> > > > Hey Eric,
> > > >
> > > >  So as a test, i copied TsProcessor.java to TsProcessor2.java.  I changed
the references for TsProcessor to be TsProcessor2 and updated my chukwa-demux.conf to be
> > > >
> > > > <property>
> > > >     <name>TsProcessor</name>
> > > >     <value>org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2</value>
> > > >     <description>Parser class for </description>
> > > >    </property>
> > > >
> > > > I then ran ant in the root and copied build/collector-0.4.0.war and build/chukwa-core.jar
to the root and started collector and demux.
> > > >
> > > > I am now getting the following errors, any ideas ?
> > > >
> > > > org.apache.hadoop.chukwa.extraction.demux.processor.mapper.UnknownRecordTypeException:
Unknown parserClass:org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2
> > > >
> > > > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2
> > > >
> > > > On Wed, Oct 26, 2011 at 1:00 PM, Eric Yang <eric818@gmail.com> wrote:
> > > > Yes, write a mapper class which extends org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcessor.java.
 There are several extraction class in org/apache/hadoop/chukwa/extraction/demux/processor/mapper
as examples.  Once the extraction class is written, configure chukwa-demux-conf.xml to map
the data type to the new extraction class.
> > > >
> > > > regards,
> > > > Eric
> > > >
> > > > On Oct 26, 2011, at 5:48 AM, AD wrote:
> > > >
> > > > > interesting.  So demux is now part of the collector and does not
need to be run as another job?
> > > > >
> > > > > Since demux is doing basic ETL into hbase, is there a way to actually
parse the fields of the log record and insert those into sep fields into Hbase to run mapreduce
there (instead of the whole body as a field is what I think is happening)
> > > > >
> > > > > On Wed, Oct 26, 2011 at 12:59 AM, SALAMI Patrick <salami@systransoft.com>
wrote:
> > > > > Ok, thanks!
> > > > >
> > > > > -----Original Message-----
> > > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric
Yang
> > > > > Sent: Tuesday, October 25, 2011 6:19 PM
> > > > > To: chukwa-user@incubator.apache.org
> > > > > Subject: Re: Chukwa trunk does not write to HBase
> > > > >
> > > > > Demuxer for Chukwa 0.4 is a mapreduce job acting as ETL process to
convert the data into semi structure format for further processing.  For Chukwa trunk with
HBase, demux is running as part of collector for ETL.  Hence, there is no need to run demux
process.
> > > > >
> > > > > regards,
> > > > > Eric
> > > > >
> > > > > On Oct 25, 2011, at 3:49 PM, SALAMI Patrick wrote:
> > > > >
> > > > > > Also, while we are chatting, I was hoping to understand the
role of the demuxer. I am assuming that HICC pulls all of its data from HBase. If using HBase,
is it still necessary to run the demuxer? I didn't see any mention of it in the latest quick
start guide.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Patrick
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of
Eric Yang
> > > > > > Sent: Tuesday, October 25, 2011 2:45 PM
> > > > > > To: chukwa-user@incubator.apache.org
> > > > > > Subject: Re: Chukwa trunk does not write to HBase
> > > > > >
> > > > > > Hadoop trunk will require a different configuration than the
one described in the Quick_Start guide.
> > > > > >
> > > > > > 1. Apply this patch: HADOOP-7436, and rebuild hadoop.
> > > > > > 2. Copy hadoop-metrics2.properties enclosed in this message
to HADOOP_CONF_DIR
> > > > > >
> > > > > > Restart hadoop.  You might need to match the HBase table schema
with the metrics emitted by Hadoop metrics 2 framework.
> > > > > > Hope this helps.
> > > > > >
> > > > > >
> > > > > > This message contains confidential information and is intended
only for the individual(s) named. If you are not the named addressee you should not disseminate,
distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have
received this e-mail by mistake and delete this e-mail from your system.
> > > > >
> > > > >
> > > > > This message contains confidential information and is intended only
for the individual(s) named. If you are not the named addressee you should not disseminate,
distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have
received this e-mail by mistake and delete this e-mail from your system.
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
> 
> 


Mime
View raw message