chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Using Chuckwa for Nutch Log Analysis and Monitoring
Date Sat, 21 Feb 2015 23:00:11 GMT
Just to wrap this thread up.
Chukwa is exactly the tool I need for analysis of my Nutch logs. We are
looking to identify specific errors and send email notifications when their
presence is identified.
Thanks

On Saturday, February 21, 2015, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Yeah your absolutely right Eric this is Dev now.
> I'll sign up to dev right now and participate over there.
> Thanks
> Lewis
>
> On Saturday, February 21, 2015, Eric Yang <eric818@gmail.com
> <javascript:_e(%7B%7D,'cvml','eric818@gmail.com');>> wrote:
>
>> Hi Lewis,
>>
>> Thank you for the patch.  This looks interesting.  I think this thread
>> has turned into a development thread.  I replied your questions via Chukwa
>> JIRA, and our discussions will automatically post to dev mailing list for
>> people who are interested to join the development discussion.  Thanks
>>
>> regards,
>> Eric
>>
>> On Fri, Feb 20, 2015 at 6:43 AM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>>> Hi Eric,
>>> I submitted an initial patch for CHUKWA-734 with a bunch of issues in my
>>> comment as well.
>>> I've also taken time to read through as much documentation as possible
>>> and have a much better idea of data adaptors which essentially provide data
>>> to the agent(s).
>>> The Nagios work is very nice and I'll look at implementing this with my
>>> monitoring of Nutch logs.
>>> Thanks
>>> Lewis
>>>
>>>
>>> On Sun, Feb 15, 2015 at 10:03 AM, Lewis John Mcgibbney <
>>> lewis.mcgibbney@gmail.com> wrote:
>>>
>>>> This is dynamite I checked out the source from svn and I have
>>>> undertaken initial analysis. Your hBase package looks like it inherits feom
>>>> a nice interface face plus your documentation is very helpful.
>>>> Thanks
>>>>
>>>>
>>>> On Sunday, February 15, 2015, Eric Yang <eric818@gmail.com> wrote:
>>>>
>>>>> Hi Lewis,
>>>>>
>>>>> There is an example in Chukwa to write errors out map reduce error to
>>>>> InError file, and send out Nagios alert.
>>>>> The example is in:
>>>>> src/main/java/org/apache/hadoop/chukwa/extraction/demux/DemuxManager.java
>>>>>
>>>>> Nagios support is updated in CHUKWA-735 to support better encryption
>>>>> for Nagios communication.
>>>>>
>>>>> Log4j Nagios appender maybe useful in CHUKWA-734.
>>>>>
>>>>> regards,
>>>>> Eric
>>>>>
>>>>> On Sat, Feb 14, 2015 at 1:31 PM, Lewis John Mcgibbney <
>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>
>>>>>> https://issues.apache.org/jira/browse/CHUKWA-734
>>>>>>
>>>>>> On Sat, Feb 14, 2015 at 12:13 PM, Lewis John Mcgibbney <
>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Eric,
>>>>>>> Thank you for the feedback.
>>>>>>> This is more than helpful.
>>>>>>> I am going to write a Gora module for Chuckwa.
>>>>>>> I am going to progress on basis of implementing log monitor for
>>>>>>> Nutch.
>>>>>>> Can Chuckwa currently write to file and email response?
>>>>>>> Thanks
>>>>>>> Lewis
>>>>>>>
>>>>>>> [0] http://gora.apache.org
>>>>>>>
>>>>>>> On Sat, Feb 14, 2015 at 9:30 AM, Eric Yang <eric818@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Lewis,
>>>>>>>>
>>>>>>>> Parse error can be captured and store errors to another HDFS
>>>>>>>> location.  In Chukwa 0.4 and earlier, we have demux map reduce
job which
>>>>>>>> does the extraction and store structured data in HDFS, and
errors are
>>>>>>>> channel to another HDFS folder called InError, with the cause
of the
>>>>>>>> parsing error.  This is still a batch oriented operation.
 In Chukwa 0.6,
>>>>>>>> we can setup multiple pipeline writer.  The pipeline writers
can be
>>>>>>>> configured to provide parsing and channel error to somewhere
else, if data
>>>>>>>> parse properly, then write it to HBase or HDFS.  However,
you will need to
>>>>>>>> write the pipeline writer class to extend this functionality.
 We currently
>>>>>>>> only have a couple pipeline writers, LocalWriter, HBaseWriter,
and
>>>>>>>> SeqFileWriter.  SeqFileWriter needs to be the last one in
the pipeline, if
>>>>>>>> you choose to write data to HDFS.  See this page for how
to configure
>>>>>>>> pipeline writer to achieve partially of what you are looking
for:
>>>>>>>>
>>>>>>>> http://chukwa.apache.org/docs/r0.6.0/pipeline.html
>>>>>>>>
>>>>>>>> Hope this helps.
>>>>>>>>
>>>>>>>> regards,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>> On Thu, Feb 12, 2015 at 11:12 PM, Lewis John Mcgibbney <
>>>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Folks,
>>>>>>>>> For some time I have been meaning to get in touch to
get advice on
>>>>>>>>> developing a tool for log analysis of Apache Nutch [0]
logs.
>>>>>>>>> What I am referring to particularly is monitoring of
logs in a bid
>>>>>>>>> to identify particular errors which we may anticipate.
>>>>>>>>> Nutch jobs are batch oriented in architecture which are
inherited
>>>>>>>>> from Hadoop, we typically see errors in the parse phase
of a crawl so it is
>>>>>>>>> events like this that I would like to anticipate, monitor
and report on,
>>>>>>>>> possibly through email.
>>>>>>>>> So I am therefore thinking about building a Chuckwa-powered
tool
>>>>>>>>> for Nutch which would become part of our codebase.
>>>>>>>>> Is Chukwa the right tool for this? Any information about
similar
>>>>>>>>> efforts would be very much appreciated.
>>>>>>>>> best
>>>>>>>>> Lewis
>>>>>>>>>
>>>>>>>>> [0] http://nutch.apache.org
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Lewis*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Lewis*
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Lewis*
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Lewis*
>>>>
>>>>
>>>
>>>
>>> --
>>> *Lewis*
>>>
>>
>>
>
> --
> *Lewis*
>
>

-- 
*Lewis*

Mime
View raw message