storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From K Zharas <kgzha...@gmail.com>
Subject Re: Storm + HDFS
Date Wed, 03 Feb 2016 11:34:58 GMT
Can anyone help to create a Spout which reads a file from HDFS?
I have tried with the code below, but it is not working.

public void nextTuple() {
      Path pt=new Path("hdfs://localhost:50070/user/BCpredict.txt");
      FileSystem fs = FileSystem.get(new Configuration());
      BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(pt)));
      String line = br.readLine();
      while (line != null){
         System.out.println(line);
         line=br.readLine();
         _collector.emit(new Values(line));
      }
}

On Tue, Feb 2, 2016 at 1:19 PM, K Zharas <kgzharas@gmail.com> wrote:

> Hi.
>
> I have a project I'm currently working on. The idea is to implement
> "scikit-learn" into Storm and integrate it with HDFS.
>
> I've already implemented "scikit-learn". But, currently I'm using a text
> file to read and write. However, I need to use HDFS, but finding it hard to
> integrate with HDFS.
>
> Here is the link to github <https://github.com/kgzharas/StormTopologyTest>.
> (I only included files that I used, not whole project)
>
> Basically, I have a few questions if you don't mint to answer them
> 1) How to use HDFS to read and write?
> 2) Is my "scikit-learn" implementation correct?
> 3) How to create a Storm project? (Currently working in "storm-starter")
>
> These questions may sound a bit silly, but I really can't find a proper
> solution.
>
> Thank you for your attention to this matter.
> Sincerely, Zharas.
>



-- 
Best regards,
Zharas

Mime
View raw message