Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of smarty.juice@gmail.com
 designates 209.85.219.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BAY177-W379BBB818D13469704A4CB14E0@phx.gbl>
References: 
 <CAJpToqdiK4cd-51RkoD8ieMbJmcuTsZjiDy+yhPML6R-W0CK+A@mail.gmail.com>
	<BAY177-W179353C542747B10E710D9B1430@phx.gbl>
	<BAY177-W352001DEC51F30E5F50BEBB1430@phx.gbl>
	<11564200.Tc0LUDiuXJ@f6-dat6-306-27>
	<BAY177-W379BBB818D13469704A4CB14E0@phx.gbl>
Date: Thu, 8 May 2014 21:25:08 -0400
Message-ID: 
 <CAMOW4CN5N_R-3y-no5uqjNX3UqHC=vWV4e4qKH-U9jpdwp=cWg@mail.gmail.com>
Subject: Re: Realtime sensor's tcpip data to hadoop
From: Hardik Pandya <smarty.juice@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b4727dce0fd5e04f8ed77dd

--047d7b4727dce0fd5e04f8ed77dd
Content-Type: text/plain; charset=UTF-8

If I were you I would ask following questions to get the answer

> forget about for a minute and ask yourself how tcpip data are currently
being stored - in fs/rdbmbs?
> hadoop is for offiline batch processing - if you are looking for real
time streaming solution - there is a storm (from linkedin) that can go well
with kafka (messaging queue) or spark streaming (which is in memory
map-reduce) and takes real time streams - has in built twitter api but you
need to write your own service to poll data every few seconds and send it
in RDD format
> storm is complementary to hadoop - spark in conjuction with hadoop will
allow you to do both offline and real time data analytics


On Tue, May 6, 2014 at 10:48 PM, Alex Lee <eliyart@hotmail.com> wrote:

> Sensors' may send tcpip data to server. Each sensor may send tcpip data
> like a stream to the server, the quatity of the sensors and the data rate
> of the data is high.
>
> Firstly, how the data from tcpip can be put into hadoop. It need to do
> some process and store in hbase. Does it need through save to data files
> and put into hadoop or can be done in some direct ways from tcpip. Is there
> any software module can take care of this. Searched that Ganglia Nagios and
> Flume may do it. But when looking into details, ganglia and nagios are
> more for monitoring hadoop cluster itself. Flume is for log files.
>
> Secondly, if the total network traffic from sensors are over the limit of
> one lan port, how to share the loads, is there any component in hadoop to
> make this done automatically.
>
> Any suggestions, thanks.
>

--047d7b4727dce0fd5e04f8ed77dd
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">If I were you I would ask following questions to get the a=
nswer<div><br></div><div>&gt; forget about for a minute and ask yourself ho=
w tcpip data are currently being stored - in fs/rdbmbs?<div>&gt; hadoop is =
for offiline batch processing - if you are looking for real time streaming =
solution - there is a storm (from linkedin) that can go well with kafka (me=
ssaging queue) or spark streaming (which is in memory map-reduce) and takes=
 real time streams - has in built twitter api but you need to write your ow=
n service to poll data every few seconds and send it in RDD format=C2=A0</d=
iv>
</div><div>&gt; storm is complementary to hadoop - spark in conjuction with=
 hadoop will allow you to do both offline and real time data analytics</div=
><div><br></div><div><br></div></div><div class=3D"gmail_extra"><br><br><di=
v class=3D"gmail_quote">
On Tue, May 6, 2014 at 10:48 PM, Alex Lee <span dir=3D"ltr">&lt;<a href=3D"=
mailto:eliyart@hotmail.com" target=3D"_blank">eliyart@hotmail.com</a>&gt;</=
span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8e=
x;border-left:1px #ccc solid;padding-left:1ex">


<div><div dir=3D"ltr">Sensors&#39;=C2=A0may send tcpip data to server. Each=
 sensor may send tcpip=C2=A0data like a stream to the server, the quatity o=
f the sensors and the data rate of the data is high.<br>=C2=A0<br>Firstly, =
how the=C2=A0data from tcpip can be put into hadoop. It need to do some pro=
cess and store in hbase. Does it need through save to=C2=A0data files and p=
ut into hadoop=C2=A0or can be done in some=C2=A0direct ways from tcpip. Is =
there any software module can take care of this.=C2=A0Searched that=C2=A0Ga=
nglia=C2=A0Nagios and Flume may do it. But when looking into details, gangl=
ia and nagios are more=C2=A0for monitoring hadoop cluster itself. Flume is =
for log files.<br>
=C2=A0<br>Secondly, if=C2=A0the=C2=A0total network traffic from sensors are=
 over the limit of one lan port, how to share the loads, is there any compo=
nent in hadoop to make this done automatically.<br>=C2=A0<br>Any suggestion=
s, thanks.<br> 		 	   		  </div>
</div>
</blockquote></div><br></div>

--047d7b4727dce0fd5e04f8ed77dd--