Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0554B1892D for ; Wed, 3 Feb 2016 19:41:39 +0000 (UTC) Received: (qmail 35699 invoked by uid 500); 3 Feb 2016 19:41:38 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 35651 invoked by uid 500); 3 Feb 2016 19:41:38 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 35641 invoked by uid 99); 3 Feb 2016 19:41:38 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2016 19:41:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id BBC291A0517 for ; Wed, 3 Feb 2016 19:41:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id rZxCr5b6v7AJ for ; Wed, 3 Feb 2016 19:41:35 +0000 (UTC) Received: from mail-yw0-f169.google.com (mail-yw0-f169.google.com [209.85.161.169]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 744BF207E1 for ; Wed, 3 Feb 2016 19:41:35 +0000 (UTC) Received: by mail-yw0-f169.google.com with SMTP id h129so11778724ywb.1 for ; Wed, 03 Feb 2016 11:41:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=SW3Ugukp7Qo5hjETk9EKjr8VDQd23fhWoXy6rbQcZWE=; b=E2PLd5xxHU8XQjBo7xCHg/WwRbMkdliRCMQHiXq7Qszb3vD0sSBOiVOcevmyoUoWuY l+OspGbyt09yMoaGTsDU3flsxZc7qFCKPu3kYDCGPraB3zVktJq0pSWw8GBPXPTg5clm XjG6LSjidX68WfBUHIMOvTwW9c2Bt22+YNqcxXyeePco71mu7Tq8JL/2d/Onu5k9qjlv ri+jDyTby7ISsfqA5LaL9UPq+0r4H1nSZN7GmX905Qc+i1j35qK8vZnvs5vXqhNGuPoI z1KMMFKoxLr0/TEvT4G40xy5VbyBEMRIhEoLik6sAxeE7s+x27IEeHHDboT4rn8FuIGC FjBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=SW3Ugukp7Qo5hjETk9EKjr8VDQd23fhWoXy6rbQcZWE=; b=SZOJJCFuPDsftURwI0ByShs9agQe+9P0/vUG/EqM/OqX7o21KH51Ma1o0Mrx43HBE6 ZEPAIZ+ob5IAzrBcoJ9IGtpELgeywvnopzhvTk3Wu0SCK7GKpucmviGMoeOaQEl4It4X HwhxD1EFMGeYZNIdcYkroKBIpnqNjMoHJnGVhl3MfnXtaGNSbJdDcrzWsJliaKBN2LZ1 sIr1qvB/ak2TvuJOvo7fWUhb5ve22EPlex1BBg61PUHmzVpPFN0zhYlUKtP3iSRXQ6kF ml0w12Zkmb1oEy9uj7INsavrqSh+09+iKCayK86lRv2jU0oociN9uTuY4hdNEMeWt9lN EOTQ== X-Gm-Message-State: AG10YORxy117rUJ+BkngAbyKpKjaC36sRtlUKLo24ve3l52ChF4SVJoQhl1wh3FWAVlV85NutOasKoMW9Pg5SQ== MIME-Version: 1.0 X-Received: by 10.13.201.131 with SMTP id l125mr2124233ywd.150.1454528489382; Wed, 03 Feb 2016 11:41:29 -0800 (PST) Received: by 10.37.10.5 with HTTP; Wed, 3 Feb 2016 11:41:29 -0800 (PST) In-Reply-To: References: <56B1F91D.5010007@apache.org> Date: Thu, 4 Feb 2016 03:41:29 +0800 Message-ID: Subject: Re: Storm + HDFS From: K Zharas To: user@storm.apache.org Content-Type: multipart/alternative; boundary=94eb2c0545dcecea2e052ae2cd6a --94eb2c0545dcecea2e052ae2cd6a Content-Type: text/plain; charset=UTF-8 Yes, looks like it is. But, I have added dependencies required by storm-hdfs as stated in a guide. On Thu, Feb 4, 2016 at 3:33 AM, Nick R. Katsipoulakis wrote: > Well, > > those errors look like a problem with the way you build your jar file. > Please, make sure that you build your jar with the proper storm maven > dependency). > > Cheers, > Nick > > On Wed, Feb 3, 2016 at 2:31 PM, K Zharas wrote: > >> It throws and error that packages does not exist. I have also tried >> changing org.apache to backtype, still got an error but only for >> storm.hdfs.spout. Btw, I use Storm-0.10.0 and Hadoop-2.7.1 >> >> package org.apache.storm does not exist >> package org.apache.storm does not exist >> package org.apache.storm.generated does not exist >> package org.apache.storm.metric does not exist >> package org.apache.storm.topology does not exist >> package org.apache.storm.utils does not exist >> package org.apache.storm.utils does not exist >> package org.apache.storm.hdfs.spout does not exist >> package org.apache.storm.hdfs.spout does not exist >> package org.apache.storm.topology.base does not exist >> package org.apache.storm.topology does not exist >> package org.apache.storm.tuple does not exist >> package org.apache.storm.task does not exist >> >> On Wed, Feb 3, 2016 at 8:57 PM, Matthias J. Sax wrote: >> >>> Storm does provide HdfsSpout and HdfsBolt already. Just use those, >>> instead of writing your own spout/bolt: >>> >>> https://github.com/apache/storm/tree/master/external/storm-hdfs >>> >>> -Matthias >>> >>> >>> On 02/03/2016 12:34 PM, K Zharas wrote: >>> > Can anyone help to create a Spout which reads a file from HDFS? >>> > I have tried with the code below, but it is not working. >>> > >>> > public void nextTuple() { >>> > Path pt=new Path("hdfs://localhost:50070/user/BCpredict.txt"); >>> > FileSystem fs = FileSystem.get(new Configuration()); >>> > BufferedReader br = new BufferedReader(new >>> > InputStreamReader(fs.open(pt))); >>> > String line = br.readLine(); >>> > while (line != null){ >>> > System.out.println(line); >>> > line=br.readLine(); >>> > _collector.emit(new Values(line)); >>> > } >>> > } >>> > >>> > On Tue, Feb 2, 2016 at 1:19 PM, K Zharas >> > > wrote: >>> > >>> > Hi. >>> > >>> > I have a project I'm currently working on. The idea is to implement >>> > "scikit-learn" into Storm and integrate it with HDFS. >>> > >>> > I've already implemented "scikit-learn". But, currently I'm using a >>> > text file to read and write. However, I need to use HDFS, but >>> > finding it hard to integrate with HDFS. >>> > >>> > Here is the link to github >>> > . (I only included >>> > files that I used, not whole project) >>> > >>> > Basically, I have a few questions if you don't mint to answer them >>> > 1) How to use HDFS to read and write? >>> > 2) Is my "scikit-learn" implementation correct? >>> > 3) How to create a Storm project? (Currently working in >>> "storm-starter") >>> > >>> > These questions may sound a bit silly, but I really can't find a >>> > proper solution. >>> > >>> > Thank you for your attention to this matter. >>> > Sincerely, Zharas. >>> > >>> > >>> > >>> > >>> > -- >>> > Best regards, >>> > Zharas >>> >>> >> >> >> -- >> Best regards, >> Zharas >> > > > > -- > Nick R. Katsipoulakis, > Department of Computer Science > University of Pittsburgh > -- Best regards, Zharas --94eb2c0545dcecea2e052ae2cd6a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes, looks like it is. But, I have added dependencies requ= ired by storm-hdfs as stated in a guide.
On Thu, Feb 4, 2016 at 3:33 AM, Nick R. Katsip= oulakis <nick.katsip@gmail.com> wrote:
Well,=C2=A0

those erro= rs look like a problem with the way you build your jar file.=C2=A0
Plea= se, make sure that you build your jar with the proper storm maven dependenc= y).

Cheers,=C2=A0
Nick
=

On Wed, Feb 3, 2016 at 2:31 PM, K Zharas <kgzharas@gmail.com>= ; wrote:
It = throws and error that packages does not exist. I have also tried changing o= rg.apache to backtype, still got an error but only for storm.hdfs.spout. Bt= w, I use Storm-0.10.0 and Hadoop-2.7.1

=C2=A0 =C2= =A0package org.apache.storm does not exist
=C2=A0 =C2=A0package o= rg.apache.storm does not exist
=C2=A0 =C2=A0package org.apache.st= orm.generated does not exist
=C2=A0 =C2=A0package org.apache.stor= m.metric does not exist
=C2=A0 =C2=A0package org.apache.storm.top= ology does not exist
=C2=A0 =C2=A0package org.apache.storm.utils = does not exist
=C2=A0 =C2=A0package org.apache.storm.utils does n= ot exist=C2=A0
=C2=A0 =C2=A0package org.apache.storm.hdfs.spout d= oes not exist
=C2=A0 =C2=A0package org.apache.storm.hdfs.spout do= es not exist
=C2=A0 =C2=A0package org.apache.storm.topology.base = does not exist
=C2=A0 =C2=A0package org.apache.storm.topology doe= s not exist
=C2=A0 =C2=A0package org.apache.storm.tuple does = not exist
=C2=A0 =C2=A0package org.apache.storm.task does not exi= st

On Wed, Feb 3, 2016 at 8:57 PM, Matthias J. Sax &l= t;mjsax@apache.org> wrote:
Storm does provide Hd= fsSpout and HdfsBolt already. Just use those,
instead of writing your own spout/bolt:

https://github.com/apache/storm/tree/= master/external/storm-hdfs

-Matthias


On 02/03/2016 12:34 PM, K Zharas wrote:
> Can anyone help to create a Spout which reads a file from HDFS?
> I have tried with the code below, but it is not working.
>
> public void nextTuple() {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0Path pt=3Dnew Path("hdfs://localhost:50= 070/user/BCpredict.txt");
>=C2=A0 =C2=A0 =C2=A0 =C2=A0FileSystem fs =3D FileSystem.get(new Configu= ration());
>=C2=A0 =C2=A0 =C2=A0 =C2=A0BufferedReader br =3D new BufferedReader(new=
> InputStreamReader(fs.open(pt)));
>=C2=A0 =C2=A0 =C2=A0 =C2=A0String line =3D br.readLine();
>=C2=A0 =C2=A0 =C2=A0 =C2=A0while (line !=3D null){
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println(line);
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 line=3Dbr.readLine();
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _collector.emit(new Values(line)); >=C2=A0 =C2=A0 =C2=A0 =C2=A0}
> }
>
> On Tue, Feb 2, 2016 at 1:19 PM, K Zharas <kgzharas@gmail.com
> <mailto:kgzharas@gmail.com>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Hi.
>
>=C2=A0 =C2=A0 =C2=A0I have a project I'm currently working on. The = idea is to implement
>=C2=A0 =C2=A0 =C2=A0"scikit-learn" into Storm and integrate i= t with HDFS.
>
>=C2=A0 =C2=A0 =C2=A0I've already implemented "scikit-learn&quo= t;. But, currently I'm using a
>=C2=A0 =C2=A0 =C2=A0text file to read and write. However, I need to use= HDFS, but
>=C2=A0 =C2=A0 =C2=A0finding it hard to integrate with HDFS.
>
>=C2=A0 =C2=A0 =C2=A0Here is the link to github
>=C2=A0 =C2=A0 =C2=A0<https://github.com/k= gzharas/StormTopologyTest>. (I only included
>=C2=A0 =C2=A0 =C2=A0files that I used, not whole project)
>
>=C2=A0 =C2=A0 =C2=A0Basically, I have a few questions if you don't = mint to answer them
>=C2=A0 =C2=A0 =C2=A01) How to use HDFS to read and write?
>=C2=A0 =C2=A0 =C2=A02) Is my "scikit-learn" implementation co= rrect?
>=C2=A0 =C2=A0 =C2=A03) How to create a Storm project? (Currently workin= g in "storm-starter")
>
>=C2=A0 =C2=A0 =C2=A0These questions may sound a bit silly, but I really= can't find a
>=C2=A0 =C2=A0 =C2=A0proper solution.
>
>=C2=A0 =C2=A0 =C2=A0Thank you for your attention to this matter.
>=C2=A0 =C2=A0 =C2=A0Sincerely, Zharas.
>
>
>
>
> --
> Best regards,
> Zharas




<= /div>--
Best regards,
Zharas



--
<= div dir=3D"ltr">
Nick R. Katsipoulakis,=C2= =A0
Department of Computer Science=C2=A0
University of P= ittsburgh



--
Best regards,
Zharas
=
--94eb2c0545dcecea2e052ae2cd6a--