Can anyone help to create a Spout which reads a file from HDFS?
I have tried with the code below, but it is not working.
public void nextTuple() {
Path pt=new Path("hdfs://localhost:50070/user/BCpredict.txt");
FileSystem fs = FileSystem.get(new Configuration());
BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(pt)));
String line = br.readLine();
while (line != null){
System.out.println(line);
line=br.readLine();
_collector.emit(new Values(line));
}
}
On Tue, Feb 2, 2016 at 1:19 PM, K Zharas <kgzharas@gmail.com> wrote:
> Hi.
>
> I have a project I'm currently working on. The idea is to implement
> "scikit-learn" into Storm and integrate it with HDFS.
>
> I've already implemented "scikit-learn". But, currently I'm using a text
> file to read and write. However, I need to use HDFS, but finding it hard to
> integrate with HDFS.
>
> Here is the link to github <https://github.com/kgzharas/StormTopologyTest>.
> (I only included files that I used, not whole project)
>
> Basically, I have a few questions if you don't mint to answer them
> 1) How to use HDFS to read and write?
> 2) Is my "scikit-learn" implementation correct?
> 3) How to create a Storm project? (Currently working in "storm-starter")
>
> These questions may sound a bit silly, but I really can't find a proper
> solution.
>
> Thank you for your attention to this matter.
> Sincerely, Zharas.
>
--
Best regards,
Zharas
|