Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of dsuiter@rdx.com designates
 74.125.82.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <F4C6B533-5307-49CE-930E-948DA251C235@gmail.com>
References: 
 <CAMq4vAEbcKFH7s47pzvDzUFAbFLVJfhbMZB9XfSh+XL9Kwg_hA@mail.gmail.com>
	<F4C6B533-5307-49CE-930E-948DA251C235@gmail.com>
Date: Fri, 11 Oct 2013 10:05:30 -0400
Message-ID: 
 <CAE_UNJWJE8WYULnSHV=5ZYuuhhezD+4jy6HYwZdiVY-jr1FBOg@mail.gmail.com>
Subject: Re: State of Art in Hadoop Log aggregation
From: DSuiter RDX <dsuiter@rdx.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11c37ba27800f204e8779c06

--001a11c37ba27800f204e8779c06
Content-Type: text/plain; charset=ISO-8859-1

Sagar,

It sounds like you want a management console. We are using Cloudera
Manager, but for 200 nodes you would need to license it, it is only free up
to 50 nodes.

The FOSS version of this is Ambari, iirc.
http://incubator.apache.org/ambari/

Flume will provide a Hadoop-integrated pipeline for ingesting data. The
data will still need to be analyzed and visualized if you use Flume. Kafka
is a newer project for collecting and aggregating logs, but is a separate
project and will need a server of its own to manage.

We use Splunk also, since it is approved by our auditing compliance agency.

Thanks,
*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com


On Fri, Oct 11, 2013 at 9:54 AM, Alexander Alten-Lorenz <wget.null@gmail.com
> wrote:

> Hi,
>
> http://flume.apache.org
>
> - Alex
>
> On Oct 11, 2013, at 7:36 AM, Sagar Mehta <sagarmehta@gmail.com> wrote:
>
> Hi Guys,
>
> We have fairly decent sized Hadoop cluster of about 200 nodes and was
> wondering what is the state of art if I want to aggregate and visualize
> Hadoop ecosystem logs, particularly
>
>    1. Tasktracker logs
>    2. Datanode logs
>    3. Hbase RegionServer logs
>
> One way is to use something like a Flume on each node to aggregate the
> logs and then use something like Kibana -
> http://www.elasticsearch.org/overview/kibana/ to visualize the logs and
> make them searchable.
>
> However I don't want to write another ETL for the hadoop/hbase logs
>  themselves. We currently log in to each machine individually to 'tail -F
> logs' when there is an hadoop problem on a particular node.
>
> We want a better way to look at the hadoop logs themselves in a
> centralized way when there is an issue without having to login to 100
> different machines and was wondering what is the state of are in this
> regard.
>
> Suggestions/Pointers are very welcome!!
>
> Sagar
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>

--001a11c37ba27800f204e8779c06
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Sagar,<div><br></div><div>It sounds like you want a manage=
ment console. We are using Cloudera Manager, but for 200 nodes you would ne=
ed to license it, it is only free up to 50 nodes.</div><div><br></div><div>
The FOSS version of this is Ambari, iirc.=A0<a href=3D"http://incubator.apa=
che.org/ambari/">http://incubator.apache.org/ambari/</a></div><div><br></di=
v><div>Flume will provide a Hadoop-integrated pipeline for ingesting data. =
The data will still need to be analyzed and visualized if you use Flume. Ka=
fka is a newer project for collecting and aggregating logs, but is a separa=
te project and will need a server of its own to manage.</div>
<div><br></div><div>We use Splunk also, since it is approved by our auditin=
g compliance agency.</div><div class=3D"gmail_extra"><br></div><div class=
=3D"gmail_extra">Thanks,<br clear=3D"all"><div><div dir=3D"ltr"><b>Devin Su=
iter</b><div>
<div>Jr. Data Solutions Software Engineer</div><div><div><img src=3D"http:/=
/i76.servimg.com/u/f76/12/40/55/53/untitl10.png"></div><div>100 Sandusky St=
reet | 2nd Floor | Pittsburgh, PA 15212<br><span>Google Voice: <span id=3D"=
gc-number-0" class=3D"gc-cs-link" title=3D"Call with Google Voice">412-256-=
8556</span> |=A0</span><a href=3D"http://www.rdx.com/" target=3D"_blank">ww=
w.rdx.com</a></div>
</div></div></div></div>
<br><br><div class=3D"gmail_quote">On Fri, Oct 11, 2013 at 9:54 AM, Alexand=
er Alten-Lorenz <span dir=3D"ltr">&lt;<a href=3D"mailto:wget.null@gmail.com=
" target=3D"_blank">wget.null@gmail.com</a>&gt;</span> wrote:<br><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex">
<div style=3D"word-wrap:break-word"><div>Hi,</div><div><br></div><div><a hr=
ef=3D"http://flume.apache.org" target=3D"_blank">http://flume.apache.org</a=
></div><div><br></div><div>- Alex</div><br><div><div>On Oct 11, 2013, at 7:=
36 AM, Sagar Mehta &lt;<a href=3D"mailto:sagarmehta@gmail.com" target=3D"_b=
lank">sagarmehta@gmail.com</a>&gt; wrote:</div>
<br><blockquote type=3D"cite"><div dir=3D"ltr">Hi Guys,<div><br></div><div>=
We have fairly decent sized Hadoop cluster of about 200 nodes and was wonde=
ring what is the state of art if I want to aggregate and visualize Hadoop e=
cosystem logs, particularly</div>

<div><ol><li>Tasktracker logs<br></li><li>Datanode logs<br></li><li>Hbase R=
egionServer logs<br></li></ol><div>One way is to use something like a Flume=
 on each node to aggregate the logs and then use something like Kibana -=A0=
<a href=3D"http://www.elasticsearch.org/overview/kibana/" target=3D"_blank"=
>http://www.elasticsearch.org/overview/kibana/</a> to visualize the logs an=
d make them searchable.</div>

</div><div><br></div><div>However I don&#39;t want to write another ETL for=
 the hadoop/hbase logs =A0themselves. We currently log in to each machine i=
ndividually to &#39;tail -F logs&#39; when there is an hadoop problem on a =
particular node.</div>

<div><br></div><div>We want a better way to look at the hadoop logs themsel=
ves in a centralized way when there is an issue without having to login to =
100 different machines and was wondering what is the state of are in this r=
egard.</div>

<div><br></div><div>Suggestions/Pointers are very welcome!!</div><div><br><=
/div><div>Sagar</div></div>
</blockquote></div><br><div>
--<br>Alexander Alten-Lorenz<br><a href=3D"http://mapredit.blogspot.com" ta=
rget=3D"_blank">http://mapredit.blogspot.com</a><br>German Hadoop LinkedIn =
Group:=A0<a href=3D"http://goo.gl/N8pCF" target=3D"_blank">http://goo.gl/N8=
pCF</a>

</div>
<br></div></blockquote></div><br></div></div>

--001a11c37ba27800f204e8779c06--