hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: State of Art in Hadoop Log aggregation
Date Fri, 11 Oct 2013 13:54:54 GMT
Hi,

http://flume.apache.org

- Alex

On Oct 11, 2013, at 7:36 AM, Sagar Mehta <sagarmehta@gmail.com> wrote:

> Hi Guys,
> 
> We have fairly decent sized Hadoop cluster of about 200 nodes and was wondering what
is the state of art if I want to aggregate and visualize Hadoop ecosystem logs, particularly
> Tasktracker logs
> Datanode logs
> Hbase RegionServer logs
> One way is to use something like a Flume on each node to aggregate the logs and then
use something like Kibana - http://www.elasticsearch.org/overview/kibana/ to visualize the
logs and make them searchable.
> 
> However I don't want to write another ETL for the hadoop/hbase logs  themselves. We currently
log in to each machine individually to 'tail -F logs' when there is an hadoop problem on a
particular node.
> 
> We want a better way to look at the hadoop logs themselves in a centralized way when
there is an issue without having to login to 100 different machines and was wondering what
is the state of are in this regard.
> 
> Suggestions/Pointers are very welcome!!
> 
> Sagar

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message