Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of wget.null@gmail.com designates
 209.85.215.177 as permitted sender)
From: Alexander Alten-Lorenz <wget.null@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_2C8EC0CC-2461-4F30-839D-9AAB8FE6CADA"
Message-Id: <F4C6B533-5307-49CE-930E-948DA251C235@gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: State of Art in Hadoop Log aggregation
Date: Fri, 11 Oct 2013 15:54:54 +0200
References: 
 <CAMq4vAEbcKFH7s47pzvDzUFAbFLVJfhbMZB9XfSh+XL9Kwg_hA@mail.gmail.com>
To: user@hadoop.apache.org
In-Reply-To: 
 <CAMq4vAEbcKFH7s47pzvDzUFAbFLVJfhbMZB9XfSh+XL9Kwg_hA@mail.gmail.com>


--Apple-Mail=_2C8EC0CC-2461-4F30-839D-9AAB8FE6CADA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

Hi,

http://flume.apache.org

- Alex

On Oct 11, 2013, at 7:36 AM, Sagar Mehta <sagarmehta@gmail.com> wrote:

> Hi Guys,
>=20
> We have fairly decent sized Hadoop cluster of about 200 nodes and was =
wondering what is the state of art if I want to aggregate and visualize =
Hadoop ecosystem logs, particularly
> Tasktracker logs
> Datanode logs
> Hbase RegionServer logs
> One way is to use something like a Flume on each node to aggregate the =
logs and then use something like Kibana - =
http://www.elasticsearch.org/overview/kibana/ to visualize the logs and =
make them searchable.
>=20
> However I don't want to write another ETL for the hadoop/hbase logs  =
themselves. We currently log in to each machine individually to 'tail -F =
logs' when there is an hadoop problem on a particular node.
>=20
> We want a better way to look at the hadoop logs themselves in a =
centralized way when there is an issue without having to login to 100 =
different machines and was wondering what is the state of are in this =
regard.
>=20
> Suggestions/Pointers are very welcome!!
>=20
> Sagar

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


--Apple-Mail=_2C8EC0CC-2461-4F30-839D-9AAB8FE6CADA
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=iso-8859-1

<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hi,</div><div><br></div><div><a href="http://flume.apache.org">http://flume.apache.org</a></div><div><br></div><div>- Alex</div><br><div><div>On Oct 11, 2013, at 7:36 AM, Sagar Mehta &lt;<a href="mailto:sagarmehta@gmail.com">sagarmehta@gmail.com</a>&gt; wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Hi Guys,<div><br></div><div>We have fairly decent sized Hadoop cluster of about 200 nodes and was wondering what is the state of art if I want to aggregate and visualize Hadoop ecosystem logs, particularly</div>
<div><ol><li>Tasktracker logs<br></li><li>Datanode logs<br></li><li>Hbase RegionServer logs<br></li></ol><div>One way is to use something like a Flume on each node to aggregate the logs and then use something like Kibana -&nbsp;<a href="http://www.elasticsearch.org/overview/kibana/">http://www.elasticsearch.org/overview/kibana/</a> to visualize the logs and make them searchable.</div>
</div><div><br></div><div>However I don't want to write another ETL for the hadoop/hbase logs &nbsp;themselves. We currently log in to each machine individually to 'tail -F logs' when there is an hadoop problem on a particular node.</div>
<div><br></div><div>We want a better way to look at the hadoop logs themselves in a centralized way when there is an issue without having to login to 100 different machines and was wondering what is the state of are in this regard.</div>
<div><br></div><div>Suggestions/Pointers are very welcome!!</div><div><br></div><div>Sagar</div></div>
</blockquote></div><br><div apple-content-edited="true">
--<br>Alexander Alten-Lorenz<br><a href="http://mapredit.blogspot.com">http://mapredit.blogspot.com</a><br>German Hadoop LinkedIn Group:&nbsp;http://goo.gl/N8pCF

</div>
<br></body></html>
--Apple-Mail=_2C8EC0CC-2461-4F30-839D-9AAB8FE6CADA--