Return-Path: X-Original-To: apmail-chukwa-user-archive@www.apache.org Delivered-To: apmail-chukwa-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1DD581054B for ; Sat, 21 Feb 2015 22:56:37 +0000 (UTC) Received: (qmail 34444 invoked by uid 500); 21 Feb 2015 22:56:37 -0000 Delivered-To: apmail-chukwa-user-archive@chukwa.apache.org Received: (qmail 34408 invoked by uid 500); 21 Feb 2015 22:56:37 -0000 Mailing-List: contact user-help@chukwa.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@chukwa.apache.org Delivered-To: mailing list user@chukwa.apache.org Received: (qmail 34396 invoked by uid 99); 21 Feb 2015 22:56:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Feb 2015 22:56:36 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lewis.mcgibbney@gmail.com designates 209.85.223.182 as permitted sender) Received: from [209.85.223.182] (HELO mail-ie0-f182.google.com) (209.85.223.182) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Feb 2015 22:56:09 +0000 Received: by iecrl12 with SMTP id rl12so15826075iec.2 for ; Sat, 21 Feb 2015 14:55:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Ea2KHAccGQitegmkTd+PF+vQRMJ4hMJY+PUXmVxotHI=; b=nxB64OW/4OHGII37y983TTbToscxWDtmj4iBHD1xEGSWMOD8nk9yLvFSwEoSMKMCRY LoJSVU6i5ogUSlbar7LtYRGduhe6gMJ2pgpx8GwY1+v4Zcuq/Oau9W6JAge/pTwjFpto g9wI9RM7hm4CkKsuJ7FGRfiRQWPOuD5Z8HYHm2sM/F5GNavKFO0KHirskSsETCnYGDg/ gjRLnMFrjJisyRiR28I7DzbCzPx6ykIGeyqmD5r1Euwd8OocdLXlVmPUCZYTjz0JB8Il ghUfXaepqTApZOXeXg71xsdaN5QFwn8khEEMFSZ46SvGqGARqgcbM6eSAfk48hxe1eqV jJVw== MIME-Version: 1.0 X-Received: by 10.107.14.141 with SMTP id 135mr5601648ioo.4.1424559322672; Sat, 21 Feb 2015 14:55:22 -0800 (PST) Received: by 10.36.95.212 with HTTP; Sat, 21 Feb 2015 14:55:22 -0800 (PST) In-Reply-To: References: Date: Sat, 21 Feb 2015 14:55:22 -0800 Message-ID: Subject: Re: Using Chuckwa for Nutch Log Analysis and Monitoring From: Lewis John Mcgibbney To: "user@chukwa.apache.org" Content-Type: multipart/alternative; boundary=001a113fefc263b531050fa11023 X-Virus-Checked: Checked by ClamAV on apache.org --001a113fefc263b531050fa11023 Content-Type: text/plain; charset=UTF-8 Yeah your absolutely right Eric this is Dev now. I'll sign up to dev right now and participate over there. Thanks Lewis On Saturday, February 21, 2015, Eric Yang wrote: > Hi Lewis, > > Thank you for the patch. This looks interesting. I think this thread has > turned into a development thread. I replied your questions via Chukwa > JIRA, and our discussions will automatically post to dev mailing list for > people who are interested to join the development discussion. Thanks > > regards, > Eric > > On Fri, Feb 20, 2015 at 6:43 AM, Lewis John Mcgibbney < > lewis.mcgibbney@gmail.com > > wrote: > >> Hi Eric, >> I submitted an initial patch for CHUKWA-734 with a bunch of issues in my >> comment as well. >> I've also taken time to read through as much documentation as possible >> and have a much better idea of data adaptors which essentially provide data >> to the agent(s). >> The Nagios work is very nice and I'll look at implementing this with my >> monitoring of Nutch logs. >> Thanks >> Lewis >> >> >> On Sun, Feb 15, 2015 at 10:03 AM, Lewis John Mcgibbney < >> lewis.mcgibbney@gmail.com >> > wrote: >> >>> This is dynamite I checked out the source from svn and I have undertaken >>> initial analysis. Your hBase package looks like it inherits feom a nice >>> interface face plus your documentation is very helpful. >>> Thanks >>> >>> >>> On Sunday, February 15, 2015, Eric Yang >> > wrote: >>> >>>> Hi Lewis, >>>> >>>> There is an example in Chukwa to write errors out map reduce error to >>>> InError file, and send out Nagios alert. >>>> The example is in: >>>> src/main/java/org/apache/hadoop/chukwa/extraction/demux/DemuxManager.java >>>> >>>> Nagios support is updated in CHUKWA-735 to support better encryption >>>> for Nagios communication. >>>> >>>> Log4j Nagios appender maybe useful in CHUKWA-734. >>>> >>>> regards, >>>> Eric >>>> >>>> On Sat, Feb 14, 2015 at 1:31 PM, Lewis John Mcgibbney < >>>> lewis.mcgibbney@gmail.com> wrote: >>>> >>>>> https://issues.apache.org/jira/browse/CHUKWA-734 >>>>> >>>>> On Sat, Feb 14, 2015 at 12:13 PM, Lewis John Mcgibbney < >>>>> lewis.mcgibbney@gmail.com> wrote: >>>>> >>>>>> Hi Eric, >>>>>> Thank you for the feedback. >>>>>> This is more than helpful. >>>>>> I am going to write a Gora module for Chuckwa. >>>>>> I am going to progress on basis of implementing log monitor for >>>>>> Nutch. >>>>>> Can Chuckwa currently write to file and email response? >>>>>> Thanks >>>>>> Lewis >>>>>> >>>>>> [0] http://gora.apache.org >>>>>> >>>>>> On Sat, Feb 14, 2015 at 9:30 AM, Eric Yang wrote: >>>>>> >>>>>>> Hi Lewis, >>>>>>> >>>>>>> Parse error can be captured and store errors to another HDFS >>>>>>> location. In Chukwa 0.4 and earlier, we have demux map reduce job which >>>>>>> does the extraction and store structured data in HDFS, and errors are >>>>>>> channel to another HDFS folder called InError, with the cause of the >>>>>>> parsing error. This is still a batch oriented operation. In Chukwa 0.6, >>>>>>> we can setup multiple pipeline writer. The pipeline writers can be >>>>>>> configured to provide parsing and channel error to somewhere else, if data >>>>>>> parse properly, then write it to HBase or HDFS. However, you will need to >>>>>>> write the pipeline writer class to extend this functionality. We currently >>>>>>> only have a couple pipeline writers, LocalWriter, HBaseWriter, and >>>>>>> SeqFileWriter. SeqFileWriter needs to be the last one in the pipeline, if >>>>>>> you choose to write data to HDFS. See this page for how to configure >>>>>>> pipeline writer to achieve partially of what you are looking for: >>>>>>> >>>>>>> http://chukwa.apache.org/docs/r0.6.0/pipeline.html >>>>>>> >>>>>>> Hope this helps. >>>>>>> >>>>>>> regards, >>>>>>> Eric >>>>>>> >>>>>>> On Thu, Feb 12, 2015 at 11:12 PM, Lewis John Mcgibbney < >>>>>>> lewis.mcgibbney@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Folks, >>>>>>>> For some time I have been meaning to get in touch to get advice on >>>>>>>> developing a tool for log analysis of Apache Nutch [0] logs. >>>>>>>> What I am referring to particularly is monitoring of logs in a bid >>>>>>>> to identify particular errors which we may anticipate. >>>>>>>> Nutch jobs are batch oriented in architecture which are inherited >>>>>>>> from Hadoop, we typically see errors in the parse phase of a crawl so it is >>>>>>>> events like this that I would like to anticipate, monitor and report on, >>>>>>>> possibly through email. >>>>>>>> So I am therefore thinking about building a Chuckwa-powered tool >>>>>>>> for Nutch which would become part of our codebase. >>>>>>>> Is Chukwa the right tool for this? Any information about similar >>>>>>>> efforts would be very much appreciated. >>>>>>>> best >>>>>>>> Lewis >>>>>>>> >>>>>>>> [0] http://nutch.apache.org >>>>>>>> >>>>>>>> -- >>>>>>>> *Lewis* >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Lewis* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Lewis* >>>>> >>>> >>>> >>> >>> -- >>> *Lewis* >>> >>> >> >> >> -- >> *Lewis* >> > > -- *Lewis* --001a113fefc263b531050fa11023 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Yeah your absolutely right Eric this is Dev now.
I'll sign up to de= v right now and participate over there.
Thanks
Lewis=C2= =A0

On Saturday, February 21, 2015, Eric Yang <eric818@gmail.com> wrote:
Hi Lewis,

Thank you for the = patch.=C2=A0 This looks interesting.=C2=A0 I think this thread has turned i= nto a development thread.=C2=A0 I replied your questions via Chukwa JIRA, a= nd our discussions will automatically post to dev mailing list for people w= ho are interested to join the development discussion.=C2=A0 Thanks

regards,
Eric

On Fri, Feb 20, 2015 at 6:43 AM, Lewis J= ohn Mcgibbney <lewis.mcgi= bbney@gmail.com> wrote:
Hi Eric,
I submitted an ini= tial patch for CHUKWA-734 with a bunch of issues in my comment as well.
=
I've also taken time to read through as much documentation as pos= sible and have a much better idea of data adaptors which essentially provid= e data to the agent(s).
The Nagios work is very nice and I'll = look at implementing this with my monitoring of Nutch logs.
Thanks=
Lewis
=C2=A0
On Sun, Feb 15, 2015 at 10:03 AM, Lewis John Mc= gibbney <lewis.mcgibbney@= gmail.com> wrote:
This is = dynamite I checked out the source from svn and I have undertaken initial an= alysis. Your hBase package looks like it inherits feom a nice interface fac= e plus your documentation is very helpful.
Thanks


On S= unday, February 15, 2015, Eric Yang <eric818@gmail.= com> wrote:
Hi Le= wis,

There is an example in Chukwa to write errors out m= ap reduce error to InError file, and send out Nagios alert.
The e= xample is in:=C2=A0src/mai= n/java/org/apache/hadoop/chukwa/extraction/demux/DemuxManager.java

=
Nagios support is upd= ated in CHUKWA-735 to support better encryption for Nagios communication.

Log4j Nagios appender maybe useful in CHUKWA-734.

regards,
= Eric

On Sat, Feb 14, 2015 at 1:31 PM, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com> wrote:

On Sat, Feb 14, 2015 at 12:13 PM, Lewis John Mcgib= bney <lewis.mcgibbney@gmail.com> wrot= e:
H= i Eric,
Thank you for the feedback.
This is more than hel= pful.
I am going to write a Gora module for Chuckwa.
I am going to progress on basis of implementing log monitor for=C2=A0 Nutc= h.
Can Chuckwa currently write to file and email response?
Thanks
Lewis

[0] http://gora.apache.org

On Sat, Feb = 14, 2015 at 9:30 AM, Eric Yang <eric818@gmail.com> wrote:
Hi = Lewis,

Parse error can be captured and store errors to a= nother HDFS location.=C2=A0 In Chukwa 0.4 and earlier, we have demux map re= duce job which does the extraction and store structured data in HDFS, and e= rrors are channel to another HDFS folder called InError, with the cause of = the parsing error.=C2=A0 This is still a batch oriented operation.=C2=A0 In= Chukwa 0.6, we can setup multiple pipeline writer.=C2=A0 The pipeline writ= ers can be configured to provide parsing and channel error to somewhere els= e, if data parse properly, then write it to HBase or HDFS.=C2=A0 However, y= ou will need to write the pipeline writer class to extend this functionalit= y.=C2=A0 We currently only have a couple pipeline writers, LocalWriter, HBa= seWriter, and SeqFileWriter.=C2=A0 SeqFileWriter needs to be the last one i= n the pipeline, if you choose to write data to HDFS.=C2=A0 See this page fo= r how to configure pipeline writer to achieve partially of what you are loo= king for:


Hope this helps.

regards,
Eric

On Thu, Feb 12, 2015 at 11:12 = PM, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com= > wrote:
<= div>
Hi Folks,
For some time I have been meaning to get in tou= ch to get advice on developing a tool for log analysis of Apache Nutch [0] = logs.
What I am referring to particularly is monitoring of logs in a bid= to identify particular errors which we may anticipate.
Nutch jobs are b= atch oriented in architecture which are inherited from Hadoop, we typically= see errors in the parse phase of a crawl so it is events like this that I = would like to anticipate, monitor and report on, possibly through email.
So I am therefore thinking about building a Chuckwa-powered too= l for Nutch which would become part of our codebase.
Is Chukw= a the right tool for this? Any information about similar efforts would be v= ery much appreciated.
best
Lewis

[= 0] http://nutch.apach= e.org

--
Lewis




= --
Lewis



--
Lewis=



-- Lewis



--
Lewis



--
Lewis

--001a113fefc263b531050fa11023--