chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shouvanik Haldar <shouvanik.hal...@gmail.com>
Subject Re: Writing logs collected by chukwa to hbase
Date Tue, 08 Jan 2013 14:50:37 GMT
A little SUCCESS !!!

I can see that logs are read from Hadoop Logs and data written in
hdfs..evident from the following command
hadoop fs -ls /chukwa/logs

-rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09
/chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
-rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14
/chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
-rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19
/chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
-rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24
/chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
-rw-r--r--   1 ec2-user supergroup     386337 2013-01-08 11:36
/chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
-rw-r--r--   1 ec2-user supergroup     479656 2013-01-08 11:41
/chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
-rw-r--r--   1 ec2-user supergroup     487748 2013-01-08 11:46
/chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
-rw-r--r--   1 ec2-user supergroup     479696 2013-01-08 11:51
/chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
-rw-r--r--   1 ec2-user supergroup     488030 2013-01-08 11:56
/chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done
-rw-r--r--   1 ec2-user supergroup     479754 2013-01-08 12:01
/chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done
-rw-r--r--   1 ec2-user supergroup     488067 2013-01-08 12:06
/chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done
-rw-r--r--   1 ec2-user supergroup     479961 2013-01-08 12:11
/chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done
-rw-r--r--   1 ec2-user supergroup     487976 2013-01-08 12:16
/chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done
-rw-r--r--   1 ec2-user supergroup     479863 2013-01-08 12:21
/chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done
-rw-r--r--   1 ec2-user supergroup     649168 2013-01-08 12:42
/chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done
-rw-r--r--   1 ec2-user supergroup     483972 2013-01-08 12:47
/chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done
-rw-r--r--   1 ec2-user supergroup     483417 2013-01-08 12:52
/chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done
-rw-r--r--   1 ec2-user supergroup     576711 2013-01-08 13:06
/chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done
-rw-r--r--   1 ec2-user supergroup     483877 2013-01-08 13:11
/chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done
-rw-r--r--   1 ec2-user supergroup     484352 2013-01-08 13:16
/chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done
-rw-r--r--   1 ec2-user supergroup     486158 2013-01-08 13:21
/chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done
-rw-r--r--   1 ec2-user supergroup     483925 2013-01-08 13:26
/chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done
-rw-r--r--   1 ec2-user supergroup     629788 2013-01-08 13:37
/chukwa/logs/201308133253308_domU12313907193B_7382697b13c1a5d14db8000.done
-rw-r--r--   1 ec2-user supergroup     485716 2013-01-08 13:42
/chukwa/logs/201308133753394_domU12313907193B_7382697b13c1a5d14db7ffb.done
-rw-r--r--   1 ec2-user supergroup     486475 2013-01-08 13:47
/chukwa/logs/201308134253394_domU12313907193B_7382697b13c1a5d14db7ff5.done


.done tells us that


On Tue, Jan 8, 2013 at 6:34 PM, Shouvanik Haldar <shouvanik.haldar@gmail.com
> wrote:

> I am not able to see any thing on hdfs:
> when I browse using the url ->
> http://ec2-54-234-57-44.compute-1.amazonaws.com:50070/nn_browsedfscontent.jsp
> I see nothing..but
>
> " Network Error (tcp_error)
>
>   A communication error occurred: "Operation timed out"   The Web Server
> may be down, too busy, or experiencing other problems preventing it from
> responding to requests. You may wish to try again at a later time.
> For assistance, contact your network support team. "
>
>
>
>
> On Tue, Jan 8, 2013 at 6:22 PM, Shouvanik Haldar <
> shouvanik.haldar@gmail.com> wrote:
>
>> How I was sure data getting collected at collector, is by seeing the
>> below statistics
>>
>>
>> Date:1357649516006
>> Now:1357649518803
>> numberHTTPConnection in time window:2
>> numberchunks in time window:7
>> lifetimechunks:2612
>>
>>
>>
>>
>> On Tue, Jan 8, 2013 at 6:20 PM, Shouvanik Haldar <
>> shouvanik.haldar@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I was following the post -> http://www.undercloud.org/?p=429. I did the
>>> steps as was mentioned here. But, when I was going to check in HDFS whether
>>> data has come from chukwa, I cannot see the page. I get the exception as
>>>
>>>
>>>
>>> Also, I want to know what is the default location of chukwa logs. I
>>> found out a location ->
>>> "/home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/logs". Is this the
>>> location? Previously, I had some logs inside this directory. I have deleted
>>> these logs. Now, no logs are getting created.
>>>
>>> Previously, when logs were existing, I checked one log entry. It says
>>> "nice: /home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/bin/chukwa: No
>>> such file or directory".
>>>
>>>
>>> I have edited bash_profile to give the path of CHUKWA_HOME. Is this
>>> required?
>>>
>>> I am using "chukwa-incubating-0.5.0" and "hadoop-1.0.4". I can see data
>>> getting collected at the collector by browsing the url =>
>>> http://ec2-54-234-57-44.compute-1.amazonaws.com:8080/chukwa?ping=true
>>>
>>>
>>>
>>>
>>>
>>> But somehow, data not getting inside hdfs. Please help.
>>>
>>>
>>> Regards,
>>> Shouvanik
>>>
>>>
>>> On Tue, Jan 8, 2013 at 10:22 AM, Eric Yang <eric818@gmail.com> wrote:
>>>
>>>> The quick start guide may help:
>>>>
>>>> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>>
>>>> On Mon, Jan 7, 2013 at 1:17 AM, Shouvanik Haldar <
>>>> shouvanik.haldar@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I want to install chukwa on a single node and test it. Where can I
>>>>> find a suitable document?
>>>>>
>>>>>
>>>>> On Mon, Jan 7, 2013 at 2:47 AM, Eric Yang <eric818@gmail.com> wrote:
>>>>>
>>>>>> There are a few things that you need to do:
>>>>>>
>>>>>> 1. Configure the collector to use HBaseWriter.
>>>>>>
>>>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html
>>>>>>
>>>>>> Make sure that hadoop and hbase configuration are in the classpath
of
>>>>>> the collector.
>>>>>>
>>>>>> 2. Write a demux parser for your log file to store on HBase, and
this
>>>>>> runs in chukwa collector.
>>>>>>
>>>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/programming.html
>>>>>>
>>>>>> 3. What log4j adaptors and appenders are you using to write the log
>>>>>> files?
>>>>>>   I would recommend to use socket appender from Log4j and use socket
>>>>>> adaptor to listen for the incoming data stream.  This will ensure
the
>>>>>> records are delimited properly.
>>>>>>
>>>>>> Hope this helps.
>>>>>>
>>>>>> regards,
>>>>>> Eric
>>>>>>
>>>>>> On Sun, Jan 6, 2013 at 6:00 AM, NehaS Singh <
>>>>>> NehaS.Singh@lntinfotech.com> wrote:
>>>>>>
>>>>>>> I am using chukwa 0.5. My probleam is want to store the logs
from
>>>>>>> log4j or other party which is not hadoop.Also i want to  write
the logs to
>>>>>>> hbase.
>>>>>>> But i am not ale to do so. I have configuerd log4j.properties
file
>>>>>>> so i am able to see th logs in hdfs.
>>>>>>> what are the configuration or steps using which i can write the
logs
>>>>>>> to hbase.
>>>>>>> Please help.
>>>>>>> ________________________________________
>>>>>>> From: Eric Yang [eric818@gmail.com]
>>>>>>> Sent: Saturday, January 05, 2013 11:55 PM
>>>>>>> To: chukwa-user@incubator.apache.org
>>>>>>> Subject: Re: Writing logs collected by chukwa to hbase
>>>>>>>
>>>>>>> What version of Chukwa are you running?
>>>>>>>
>>>>>>> regards,
>>>>>>> Eric
>>>>>>>
>>>>>>> On Fri, Dec 28, 2012 at 2:56 AM, NehaS Singh <
>>>>>>> NehaS.Singh@lntinfotech.com<mailto:NehaS.Singh@lntinfotech.com>>
>>>>>>> wrote:
>>>>>>> Hi,
>>>>>>>                 I am using Chukwa to collect log4j logs which
I can
>>>>>>> see in HDFS. But I am not able to write it to HBASE. Can anyone
help ?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Neha Singh
>>>>>>>
>>>>>>>
>>>>>>> ________________________________
>>>>>>> The contents of this e-mail and any attachment(s) may contain
>>>>>>> confidential or privileged information for the intended recipient(s).
>>>>>>> Unintended recipients are prohibited from taking action on the
basis of
>>>>>>> information in this e-mail and using or disseminating the information,
and
>>>>>>> must notify the sender and delete it from their system. L&T
Infotech will
>>>>>>> not accept responsibility or liability for the accuracy or completeness
of,
>>>>>>> or the presence of any virus or disabling code in this e-mail"
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> *Shouvanik*
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> *Shouvanik*
>>>
>>
>>
>>
>> --
>> Thanks,
>> *Shouvanik*
>>
>
>
>
> --
> Thanks,
> *Shouvanik*
>



-- 
Thanks,
*Shouvanik*

Mime
View raw message