Mailing-List: contact user-help@storm.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@storm.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAELSTOKF3-1TV9MXPOF2FkErV0OAEHEsDapTEHkcorDXf+R5GQ@mail.gmail.com>
References: 
 <CAELSTOKF3-1TV9MXPOF2FkErV0OAEHEsDapTEHkcorDXf+R5GQ@mail.gmail.com>
Date: Wed, 3 Feb 2016 19:34:58 +0800
Message-ID: 
 <CAELSTOJQnrCSc0gLTQfZTBbsLzoepbPZgiZ3FKE6K17Hx=P5zQ@mail.gmail.com>
Subject: Re: Storm + HDFS
From: K Zharas <kgzharas@gmail.com>
To: user@storm.apache.org
Content-Type: multipart/alternative; boundary=001a11c00ce40526dd052adc0253

--001a11c00ce40526dd052adc0253
Content-Type: text/plain; charset=UTF-8

Can anyone help to create a Spout which reads a file from HDFS?
I have tried with the code below, but it is not working.

public void nextTuple() {
      Path pt=new Path("hdfs://localhost:50070/user/BCpredict.txt");
      FileSystem fs = FileSystem.get(new Configuration());
      BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(pt)));
      String line = br.readLine();
      while (line != null){
         System.out.println(line);
         line=br.readLine();
         _collector.emit(new Values(line));
      }
}

On Tue, Feb 2, 2016 at 1:19 PM, K Zharas <kgzharas@gmail.com> wrote:

> Hi.
>
> I have a project I'm currently working on. The idea is to implement
> "scikit-learn" into Storm and integrate it with HDFS.
>
> I've already implemented "scikit-learn". But, currently I'm using a text
> file to read and write. However, I need to use HDFS, but finding it hard to
> integrate with HDFS.
>
> Here is the link to github <https://github.com/kgzharas/StormTopologyTest>.
> (I only included files that I used, not whole project)
>
> Basically, I have a few questions if you don't mint to answer them
> 1) How to use HDFS to read and write?
> 2) Is my "scikit-learn" implementation correct?
> 3) How to create a Storm project? (Currently working in "storm-starter")
>
> These questions may sound a bit silly, but I really can't find a proper
> solution.
>
> Thank you for your attention to this matter.
> Sincerely, Zharas.
>


-- 
Best regards,
Zharas

--001a11c00ce40526dd052adc0253
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Can anyone help to create a Spout which reads a file from =
HDFS?<div>I have tried with the code below, but it is not working.</div><di=
v><br></div><div><div>public void nextTuple() {</div><div><div>=C2=A0 =C2=
=A0 =C2=A0 Path pt=3Dnew Path(&quot;hdfs://localhost:50070/user/BCpredict.t=
xt&quot;);</div><div>=C2=A0 =C2=A0 =C2=A0 FileSystem fs =3D FileSystem.get(=
new Configuration());</div><div>=C2=A0 =C2=A0 =C2=A0 BufferedReader br =3D =
new BufferedReader(new InputStreamReader(fs.open(pt)));</div><div>=C2=A0 =
=C2=A0 =C2=A0 String line =3D br.readLine();</div><div>=C2=A0 =C2=A0 =C2=A0=
 while (line !=3D null){</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0System=
.out.println(line);</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0line=3Dbr.r=
eadLine();</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0_collector.emit(new =
Values(line));</div><div>=C2=A0 =C2=A0 =C2=A0 }</div></div></div><div>}</di=
v></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, F=
eb 2, 2016 at 1:19 PM, K Zharas <span dir=3D"ltr">&lt;<a href=3D"mailto:kgz=
haras@gmail.com" target=3D"_blank">kgzharas@gmail.com</a>&gt;</span> wrote:=
<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi.=C2=A0<div><br></div=
><div>I have a project I&#39;m currently working on. The idea is to impleme=
nt &quot;scikit-learn&quot; into Storm and integrate it with HDFS.=C2=A0</d=
iv><div><br></div><div>I&#39;ve already implemented &quot;scikit-learn&quot=
;. But, currently I&#39;m using a text file to read and write. However, I n=
eed to use HDFS, but finding it hard to integrate with HDFS.=C2=A0</div><di=
v><br></div><div>Here is the link to <a href=3D"https://github.com/kgzharas=
/StormTopologyTest" target=3D"_blank">github</a>. (I only included files th=
at I used, not whole project)</div><div><br></div><div>Basically, I have a =
few questions if you don&#39;t mint to answer them</div><div>1) How=C2=A0to=
 use HDFS to read and write?</div><div>2) Is my &quot;scikit-learn&quot; im=
plementation correct?</div><div>3) How to create a Storm project? (Currentl=
y working in &quot;storm-starter&quot;)</div><div><br></div><div>These ques=
tions may sound a bit silly, but I really can&#39;t find a proper solution.=
</div><div><br></div><div>Thank you for your attention to this matter.</div=
><div>Sincerely, Zharas.</div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div style=3D"c=
olor:rgb(136,136,136);font-size:12.8000001907349px">Best regards,</div><div=
 style=3D"color:rgb(136,136,136);font-size:12.8000001907349px">Zharas</div>=
</div></div></div></div>
</div>

--001a11c00ce40526dd052adc0253--