chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Using Chuckwa for Nutch Log Analysis and Monitoring
Date Sat, 21 Feb 2015 22:55:22 GMT
Yeah your absolutely right Eric this is Dev now.
I'll sign up to dev right now and participate over there.
Thanks
Lewis

On Saturday, February 21, 2015, Eric Yang <eric818@gmail.com> wrote:

> Hi Lewis,
>
> Thank you for the patch.  This looks interesting.  I think this thread has
> turned into a development thread.  I replied your questions via Chukwa
> JIRA, and our discussions will automatically post to dev mailing list for
> people who are interested to join the development discussion.  Thanks
>
> regards,
> Eric
>
> On Fri, Feb 20, 2015 at 6:43 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com
> <javascript:_e(%7B%7D,'cvml','lewis.mcgibbney@gmail.com');>> wrote:
>
>> Hi Eric,
>> I submitted an initial patch for CHUKWA-734 with a bunch of issues in my
>> comment as well.
>> I've also taken time to read through as much documentation as possible
>> and have a much better idea of data adaptors which essentially provide data
>> to the agent(s).
>> The Nagios work is very nice and I'll look at implementing this with my
>> monitoring of Nutch logs.
>> Thanks
>> Lewis
>>
>>
>> On Sun, Feb 15, 2015 at 10:03 AM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com
>> <javascript:_e(%7B%7D,'cvml','lewis.mcgibbney@gmail.com');>> wrote:
>>
>>> This is dynamite I checked out the source from svn and I have undertaken
>>> initial analysis. Your hBase package looks like it inherits feom a nice
>>> interface face plus your documentation is very helpful.
>>> Thanks
>>>
>>>
>>> On Sunday, February 15, 2015, Eric Yang <eric818@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','eric818@gmail.com');>> wrote:
>>>
>>>> Hi Lewis,
>>>>
>>>> There is an example in Chukwa to write errors out map reduce error to
>>>> InError file, and send out Nagios alert.
>>>> The example is in:
>>>> src/main/java/org/apache/hadoop/chukwa/extraction/demux/DemuxManager.java
>>>>
>>>> Nagios support is updated in CHUKWA-735 to support better encryption
>>>> for Nagios communication.
>>>>
>>>> Log4j Nagios appender maybe useful in CHUKWA-734.
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>> On Sat, Feb 14, 2015 at 1:31 PM, Lewis John Mcgibbney <
>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>
>>>>> https://issues.apache.org/jira/browse/CHUKWA-734
>>>>>
>>>>> On Sat, Feb 14, 2015 at 12:13 PM, Lewis John Mcgibbney <
>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>
>>>>>> Hi Eric,
>>>>>> Thank you for the feedback.
>>>>>> This is more than helpful.
>>>>>> I am going to write a Gora module for Chuckwa.
>>>>>> I am going to progress on basis of implementing log monitor for
>>>>>> Nutch.
>>>>>> Can Chuckwa currently write to file and email response?
>>>>>> Thanks
>>>>>> Lewis
>>>>>>
>>>>>> [0] http://gora.apache.org
>>>>>>
>>>>>> On Sat, Feb 14, 2015 at 9:30 AM, Eric Yang <eric818@gmail.com>
wrote:
>>>>>>
>>>>>>> Hi Lewis,
>>>>>>>
>>>>>>> Parse error can be captured and store errors to another HDFS
>>>>>>> location.  In Chukwa 0.4 and earlier, we have demux map reduce
job which
>>>>>>> does the extraction and store structured data in HDFS, and errors
are
>>>>>>> channel to another HDFS folder called InError, with the cause
of the
>>>>>>> parsing error.  This is still a batch oriented operation.  In
Chukwa 0.6,
>>>>>>> we can setup multiple pipeline writer.  The pipeline writers
can be
>>>>>>> configured to provide parsing and channel error to somewhere
else, if data
>>>>>>> parse properly, then write it to HBase or HDFS.  However, you
will need to
>>>>>>> write the pipeline writer class to extend this functionality.
 We currently
>>>>>>> only have a couple pipeline writers, LocalWriter, HBaseWriter,
and
>>>>>>> SeqFileWriter.  SeqFileWriter needs to be the last one in the
pipeline, if
>>>>>>> you choose to write data to HDFS.  See this page for how to configure
>>>>>>> pipeline writer to achieve partially of what you are looking
for:
>>>>>>>
>>>>>>> http://chukwa.apache.org/docs/r0.6.0/pipeline.html
>>>>>>>
>>>>>>> Hope this helps.
>>>>>>>
>>>>>>> regards,
>>>>>>> Eric
>>>>>>>
>>>>>>> On Thu, Feb 12, 2015 at 11:12 PM, Lewis John Mcgibbney <
>>>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Folks,
>>>>>>>> For some time I have been meaning to get in touch to get
advice on
>>>>>>>> developing a tool for log analysis of Apache Nutch [0] logs.
>>>>>>>> What I am referring to particularly is monitoring of logs
in a bid
>>>>>>>> to identify particular errors which we may anticipate.
>>>>>>>> Nutch jobs are batch oriented in architecture which are inherited
>>>>>>>> from Hadoop, we typically see errors in the parse phase of
a crawl so it is
>>>>>>>> events like this that I would like to anticipate, monitor
and report on,
>>>>>>>> possibly through email.
>>>>>>>> So I am therefore thinking about building a Chuckwa-powered
tool
>>>>>>>> for Nutch which would become part of our codebase.
>>>>>>>> Is Chukwa the right tool for this? Any information about
similar
>>>>>>>> efforts would be very much appreciated.
>>>>>>>> best
>>>>>>>> Lewis
>>>>>>>>
>>>>>>>> [0] http://nutch.apache.org
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Lewis*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Lewis*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Lewis*
>>>>>
>>>>
>>>>
>>>
>>> --
>>> *Lewis*
>>>
>>>
>>
>>
>> --
>> *Lewis*
>>
>
>

-- 
*Lewis*

Mime
View raw message