Mailing-List: contact user-help@chukwa.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@chukwa.apache.org
Received-SPF: pass (nike.apache.org: domain of eric818@gmail.com designates
 209.85.214.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGaRif3fhONkJJTkOTyz79Tmun2R0TQWUjE0dxKRiGay9hmPSg@mail.gmail.com>
References: 
 <CAGaRif3fhONkJJTkOTyz79Tmun2R0TQWUjE0dxKRiGay9hmPSg@mail.gmail.com>
Date: Sat, 14 Feb 2015 09:30:14 -0800
Message-ID: 
 <CADfbTpHknsi=-rkyXzkyis9DNNc6SqkgUshaL=hfb9v0dhjo3w@mail.gmail.com>
Subject: Re: Using Chuckwa for Nutch Log Analysis and Monitoring
From: Eric Yang <eric818@gmail.com>
To: user@chukwa.apache.org
Content-Type: multipart/alternative; boundary=001a11c21066c097ed050f0fb425

--001a11c21066c097ed050f0fb425
Content-Type: text/plain; charset=UTF-8

Hi Lewis,

Parse error can be captured and store errors to another HDFS location.  In
Chukwa 0.4 and earlier, we have demux map reduce job which does the
extraction and store structured data in HDFS, and errors are channel to
another HDFS folder called InError, with the cause of the parsing error.
This is still a batch oriented operation.  In Chukwa 0.6, we can setup
multiple pipeline writer.  The pipeline writers can be configured to
provide parsing and channel error to somewhere else, if data parse
properly, then write it to HBase or HDFS.  However, you will need to write
the pipeline writer class to extend this functionality.  We currently only
have a couple pipeline writers, LocalWriter, HBaseWriter, and
SeqFileWriter.  SeqFileWriter needs to be the last one in the pipeline, if
you choose to write data to HDFS.  See this page for how to configure
pipeline writer to achieve partially of what you are looking for:

http://chukwa.apache.org/docs/r0.6.0/pipeline.html

Hope this helps.

regards,
Eric

On Thu, Feb 12, 2015 at 11:12 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Hi Folks,
> For some time I have been meaning to get in touch to get advice on
> developing a tool for log analysis of Apache Nutch [0] logs.
> What I am referring to particularly is monitoring of logs in a bid to
> identify particular errors which we may anticipate.
> Nutch jobs are batch oriented in architecture which are inherited from
> Hadoop, we typically see errors in the parse phase of a crawl so it is
> events like this that I would like to anticipate, monitor and report on,
> possibly through email.
> So I am therefore thinking about building a Chuckwa-powered tool for Nutch
> which would become part of our codebase.
> Is Chukwa the right tool for this? Any information about similar efforts
> would be very much appreciated.
> best
> Lewis
>
> [0] http://nutch.apache.org
>
> --
> *Lewis*
>

--001a11c21066c097ed050f0fb425
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Lewis,<div><br></div><div>Parse error can be captured a=
nd store errors to another HDFS location.=C2=A0 In Chukwa 0.4 and earlier, =
we have demux map reduce job which does the extraction and store structured=
 data in HDFS, and errors are channel to another HDFS folder called InError=
, with the cause of the parsing error.=C2=A0 This is still a batch oriented=
 operation.=C2=A0 In Chukwa 0.6, we can setup multiple pipeline writer.=C2=
=A0 The pipeline writers can be configured to provide parsing and channel e=
rror to somewhere else, if data parse properly, then write it to HBase or H=
DFS.=C2=A0 However, you will need to write the pipeline writer class to ext=
end this functionality.=C2=A0 We currently only have a couple pipeline writ=
ers, LocalWriter, HBaseWriter, and SeqFileWriter.=C2=A0 SeqFileWriter needs=
 to be the last one in the pipeline, if you choose to write data to HDFS.=
=C2=A0 See this page for how to configure pipeline writer to achieve partia=
lly of what you are looking for:</div><div><br></div><div><a href=3D"http:/=
/chukwa.apache.org/docs/r0.6.0/pipeline.html">http://chukwa.apache.org/docs=
/r0.6.0/pipeline.html</a><br></div><div><br></div><div>Hope this helps.</di=
v><div><br></div><div>regards,</div><div>Eric</div></div><div class=3D"gmai=
l_extra"><br><div class=3D"gmail_quote">On Thu, Feb 12, 2015 at 11:12 PM, L=
ewis John Mcgibbney <span dir=3D"ltr">&lt;<a href=3D"mailto:lewis.mcgibbney=
@gmail.com" target=3D"_blank">lewis.mcgibbney@gmail.com</a>&gt;</span> wrot=
e:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l=
eft:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div><div>Hi Folks,<b=
r></div>For some time I have been meaning to get in touch to get advice on =
developing a tool for log analysis of Apache Nutch [0] logs.<br>What I am r=
eferring to particularly is monitoring of logs in a bid to identify particu=
lar errors which we may anticipate.<br>Nutch jobs are batch oriented in arc=
hitecture which are inherited from Hadoop, we typically see errors in the p=
arse phase of a crawl so it is events like this that I would like to antici=
pate, monitor and report on, possibly through email.<br></div><div>So I am =
therefore thinking about building a Chuckwa-powered tool for Nutch which wo=
uld become part of our codebase.<br></div><div>Is Chukwa the right tool for=
 this? Any information about similar efforts would be very much appreciated=
.<br></div><div>best<br>Lewis<br></div><div><br></div>[0] <a href=3D"http:/=
/nutch.apache.org" target=3D"_blank">http://nutch.apache.org</a><span class=
=3D"HOEnZb"><font color=3D"#888888"><br clear=3D"all"><div><div><div><div><=
br>-- <br><div><font style=3D"font-family:courier new,monospace;background-=
color:rgb(255,255,255);color:rgb(51,102,255)" size=3D"4"><i>Lewis</i></font=
><span style=3D"background-color:rgb(255,255,255);color:rgb(51,102,255)"></=
span> <br></div>
</div></div></div></div></font></span></div>
</blockquote></div><br></div>

--001a11c21066c097ed050f0fb425--