incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AD <straightfl...@gmail.com>
Subject Re: Chukwa trunk does not write to HBase
Date Thu, 27 Oct 2011 19:00:53 GMT
Hmm, even after removing i am still seeing 2 tails being started with lsof.
 I am only using 1 filetailer.CharFileTailingAdaptorUTF8 in
initial_adaptors.



On Thu, Oct 27, 2011 at 1:07 PM, AD <straightflush@gmail.com> wrote:

> argh. yes.  I guess # is not a comment in the initial_adaptors file :-) i
> commented out the old filetailer.FileTailingAdaptor for the
> new filetailer.CharFileTailingAdaptorUTF8
>
> anyway thanks for the help.
>
>
>
> On Thu, Oct 27, 2011 at 12:19 PM, Eric Yang <eric818@gmail.com> wrote:
>
>> Do you have two data stream that are tailing the same file?  To verify:
>>
>> telnet agent-host 9093
>> list
>>
>> Make sure you are not streaming the same data twice.
>>
>> regards,
>> Eric
>>
>> On Oct 27, 2011, at 7:25 AM, AD wrote:
>>
>> > I migrated to the  filetailer.CharFileTailingAdaptorUTF8 which ensured
>> newline breaks but i am now getting 2 of every entry.  Put the below debug
>> code into the demux parser, any ideas why i would be seeing 2 of every event
>> in the log?
>> >
>> >  throws Throwable {
>> >     try {
>> >           String lines[] = recordEntry.split("\\r?\\n");
>> >           FileWriter out = new FileWriter("/tmp/demux.test",true);
>> >           PrintWriter p = new PrintWriter(out);
>> >           for(int i = 0; i < lines.length; i++) {
>> >            log.warn("*** TRYING TO PARSE **** " + lines[i]);
>> >            p.println(lines[i]);
>> >           }
>> >           p.close();
>> >
>> > On Thu, Oct 27, 2011 at 6:39 AM, AD <straightflush@gmail.com> wrote:
>> > yep that was it thanks.  For the "recordEntry" variable, is there any
>> way to guarantee how this is structured?  I am testing tailing a file and i
>> notice recordEntry in the parser is around 20-30 lines from the logfile, is
>> this expected?  can we safely assume newline termination and just loop
>> through?  I was expecting 1 recordEntry for each line of the logfile but i
>> think this has to do with it being a chukwaRecord not a log entry.
>> >
>> >
>> > On Thu, Oct 27, 2011 at 1:25 AM, Eric Yang <eric818@gmail.com> wrote:
>> > Do you have both chukwa-core.jar and chukwa-core-0.4.0.jar?  Run jar tf
>> chukwa-core-0.4.0.jar | grep TsProcessor2, does chukwa-core-0.4.0.jar file
>> have TsProcessor2.class?
>> >
>> > regards,
>> > Eric
>> >
>> > On Oct 26, 2011, at 7:59 PM, AD wrote:
>> >
>> > > Hey Eric,
>> > >
>> > >  So as a test, i copied TsProcessor.java to TsProcessor2.java.  I
>> changed the references for TsProcessor to be TsProcessor2 and updated my
>> chukwa-demux.conf to be
>> > >
>> > > <property>
>> > >     <name>TsProcessor</name>
>> > >
>> <value>org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2</value>
>> > >     <description>Parser class for </description>
>> > >    </property>
>> > >
>> > > I then ran ant in the root and copied build/collector-0.4.0.war and
>> build/chukwa-core.jar to the root and started collector and demux.
>> > >
>> > > I am now getting the following errors, any ideas ?
>> > >
>> > >
>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.UnknownRecordTypeException:
>> Unknown
>> parserClass:org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2
>> > >
>> > > Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2
>> > >
>> > > On Wed, Oct 26, 2011 at 1:00 PM, Eric Yang <eric818@gmail.com> wrote:
>> > > Yes, write a mapper class which extends
>> org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcessor.java.
>>  There are several extraction class in
>> org/apache/hadoop/chukwa/extraction/demux/processor/mapper as examples.
>>  Once the extraction class is written, configure chukwa-demux-conf.xml to
>> map the data type to the new extraction class.
>> > >
>> > > regards,
>> > > Eric
>> > >
>> > > On Oct 26, 2011, at 5:48 AM, AD wrote:
>> > >
>> > > > interesting.  So demux is now part of the collector and does not
>> need to be run as another job?
>> > > >
>> > > > Since demux is doing basic ETL into hbase, is there a way to
>> actually parse the fields of the log record and insert those into sep fields
>> into Hbase to run mapreduce there (instead of the whole body as a field is
>> what I think is happening)
>> > > >
>> > > > On Wed, Oct 26, 2011 at 12:59 AM, SALAMI Patrick <
>> salami@systransoft.com> wrote:
>> > > > Ok, thanks!
>> > > >
>> > > > -----Original Message-----
>> > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric
>> Yang
>> > > > Sent: Tuesday, October 25, 2011 6:19 PM
>> > > > To: chukwa-user@incubator.apache.org
>> > > > Subject: Re: Chukwa trunk does not write to HBase
>> > > >
>> > > > Demuxer for Chukwa 0.4 is a mapreduce job acting as ETL process to
>> convert the data into semi structure format for further processing.  For
>> Chukwa trunk with HBase, demux is running as part of collector for ETL.
>>  Hence, there is no need to run demux process.
>> > > >
>> > > > regards,
>> > > > Eric
>> > > >
>> > > > On Oct 25, 2011, at 3:49 PM, SALAMI Patrick wrote:
>> > > >
>> > > > > Also, while we are chatting, I was hoping to understand the role
>> of the demuxer. I am assuming that HICC pulls all of its data from HBase. If
>> using HBase, is it still necessary to run the demuxer? I didn't see any
>> mention of it in the latest quick start guide.
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > Patrick
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: Eric Yang [mailto:eyang@hortonworks.com] On Behalf Of Eric
>> Yang
>> > > > > Sent: Tuesday, October 25, 2011 2:45 PM
>> > > > > To: chukwa-user@incubator.apache.org
>> > > > > Subject: Re: Chukwa trunk does not write to HBase
>> > > > >
>> > > > > Hadoop trunk will require a different configuration than the
one
>> described in the Quick_Start guide.
>> > > > >
>> > > > > 1. Apply this patch: HADOOP-7436, and rebuild hadoop.
>> > > > > 2. Copy hadoop-metrics2.properties enclosed in this message to
>> HADOOP_CONF_DIR
>> > > > >
>> > > > > Restart hadoop.  You might need to match the HBase table schema
>> with the metrics emitted by Hadoop metrics 2 framework.
>> > > > > Hope this helps.
>> > > > >
>> > > > >
>> > > > > This message contains confidential information and is intended
>> only for the individual(s) named. If you are not the named addressee you
>> should not disseminate, distribute or copy this e-mail. Please notify the
>> sender immediately by e-mail if you have received this e-mail by mistake and
>> delete this e-mail from your system.
>> > > >
>> > > >
>> > > > This message contains confidential information and is intended only
>> for the individual(s) named. If you are not the named addressee you should
>> not disseminate, distribute or copy this e-mail. Please notify the sender
>> immediately by e-mail if you have received this e-mail by mistake and delete
>> this e-mail from your system.
>> > > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>

Mime
View raw message