Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 42BB91801C for ; Wed, 3 Feb 2016 21:04:18 +0000 (UTC) Received: (qmail 39988 invoked by uid 500); 3 Feb 2016 21:04:15 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 39940 invoked by uid 500); 3 Feb 2016 21:04:15 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 39930 invoked by uid 99); 3 Feb 2016 21:04:14 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2016 21:04:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 86FCFC01A8 for ; Wed, 3 Feb 2016 21:04:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id sG4wCarH2rop for ; Wed, 3 Feb 2016 21:04:12 +0000 (UTC) Received: from mail-yk0-f174.google.com (mail-yk0-f174.google.com [209.85.160.174]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 35EB8428ED for ; Wed, 3 Feb 2016 21:04:12 +0000 (UTC) Received: by mail-yk0-f174.google.com with SMTP id r207so30632982ykd.2 for ; Wed, 03 Feb 2016 13:04:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:message-id :references:to; bh=2MahCayu5+ZPEohrHCQVbTlswrWYkT5h/kOaH25hzXs=; b=MSTxyGhp+tsTZpSI94O86rDTktdXLhFWtrFvm3wd5zYi8XpvSBRhxFXNUXyAYugtwe RTfEmfKsryiM3HV/ZBZoAzVYbKyp/RvBdFPlPC2XFrlNX8pL7LKexYO1hS6LEHrOGDP6 bGADqli93wf7rjJFv1zkTiWQ94rRHyLS0IaaZZANARuHODEScIxqR36xRcqoGCMdto98 PAxSwTa79eJ56alOqD5dEV1bYfxezbkLhicz+GI3OZdet8o/iojSHLOqXvQKEKF82IZY V2pp9lRlXItx6qCF3bGqS/9cQnwJnzfTD+hKFhzoAqazrPmMh20acn0ukOfysey8XPCe 5HkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:message-id:references:to; bh=2MahCayu5+ZPEohrHCQVbTlswrWYkT5h/kOaH25hzXs=; b=LSK22FodZUdJZUjC4oil+6+CaOKxei4X9qaDKwMxp8EqrVE+bdJ8ttiyC3kNcfPKdx OZVJzEKvOyhHU++TVv8Yi8yYtfyHchkCwv7peIcg03io9F6ctLZSx2ALWy1+2jKMQ8Rw dXL+yGD7z4zRDeUwiDwO7FX+uvJ3Ps90129g+aGpoczlko8l+DjZH3gXcQAH7xfjxGUf qKIdYYaLiXOh8ZzUzmw9U1knyCDKGV3bqE74mo3/2vFbt7Vg4UgA50UB+9JuysD7NQmq dvlDhb5wDgdatkvq+wrGiXs1Ke5OUXUtvdQwpQv+yepMSZNGWhXbo8HfkRn7RlQBowph zU/w== X-Gm-Message-State: AG10YOQGS+jnaKDBoKiGR1BGXjzOKxQ4YvSPyHDn7M3Qp/U1ZHKwzWVpYPCCGHATj1dMDw== X-Received: by 10.13.206.2 with SMTP id q2mr2273525ywd.76.1454533445878; Wed, 03 Feb 2016 13:04:05 -0800 (PST) Received: from new-host-2.local (pool-98-115-92-11.phlapa.fios.verizon.net. [98.115.92.11]) by smtp.gmail.com with ESMTPSA id e4sm4565538ywb.0.2016.02.03.13.04.04 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Feb 2016 13:04:04 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_2CCE97EB-A772-46ED-B226-65E81E54F0EB"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Storm + HDFS X-Pgp-Agent: GPGMail 2.5.2 From: "P. Taylor Goetz" In-Reply-To: Date: Wed, 3 Feb 2016 16:04:03 -0500 Message-Id: <943BBA23-7AAD-454C-B608-540A0F754434@gmail.com> References: <56B1F91D.5010007@apache.org> To: user@storm.apache.org X-Mailer: Apple Mail (2.2104) --Apple-Mail=_2CCE97EB-A772-46ED-B226-65E81E54F0EB Content-Type: multipart/alternative; boundary="Apple-Mail=_9F4FDF3B-92B0-4A8C-82B1-8403141AA573" --Apple-Mail=_9F4FDF3B-92B0-4A8C-82B1-8403141AA573 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Assuming you have git and maven installed: git clone git@github.com:apache/storm.git cd storm git checkout -b 1.x origin/1.x-branch mvn install -DskipTests That third step checks out the 1.x-branch branch which is the base for = the upcoming 1.0 release. You can then include the storm-hdfs dependency in your project: org.apache.storm storm-hdfs 1.0.0-SNAPSHOT You can find more information on using the spout and other HDFS = components here: = https://github.com/apache/storm/tree/1.x-branch/external/storm-hdfs#hdfs-s= pout = -Taylor > On Feb 3, 2016, at 2:54 PM, K Zharas wrote: >=20 > Oh ok. Can you plz give me an idea how can I do it manually? I'm quite = beginner :) >=20 > On Thu, Feb 4, 2016 at 3:43 AM, Parth Brahmbhatt = > = wrote: > Storm-hdfs spout is not yet published in maven. You will have to = checkout storm locally and build it to make it available for = development. >=20 > From: K Zharas > > Reply-To: "user@storm.apache.org " = > > Date: Wednesday, February 3, 2016 at 11:41 AM > To: "user@storm.apache.org " = > > Subject: Re: Storm + HDFS >=20 > Yes, looks like it is. But, I have added dependencies required by = storm-hdfs as stated in a guide. >=20 > On Thu, Feb 4, 2016 at 3:33 AM, Nick R. Katsipoulakis = > wrote: > Well, >=20 > those errors look like a problem with the way you build your jar file. > Please, make sure that you build your jar with the proper storm maven = dependency). >=20 > Cheers, > Nick >=20 > On Wed, Feb 3, 2016 at 2:31 PM, K Zharas > wrote: > It throws and error that packages does not exist. I have also tried = changing org.apache to backtype, still got an error but only for = storm.hdfs.spout. Btw, I use Storm-0.10.0 and Hadoop-2.7.1 >=20 > package org.apache.storm does not exist > package org.apache.storm does not exist > package org.apache.storm.generated does not exist > package org.apache.storm.metric does not exist > package org.apache.storm.topology does not exist > package org.apache.storm.utils does not exist > package org.apache.storm.utils does not exist > package org.apache.storm.hdfs.spout does not exist > package org.apache.storm.hdfs.spout does not exist > package org.apache.storm.topology.base does not exist > package org.apache.storm.topology does not exist > package org.apache.storm.tuple does not exist > package org.apache.storm.task does not exist >=20 > On Wed, Feb 3, 2016 at 8:57 PM, Matthias J. Sax > wrote: > Storm does provide HdfsSpout and HdfsBolt already. Just use those, > instead of writing your own spout/bolt: >=20 > https://github.com/apache/storm/tree/master/external/storm-hdfs = >=20 > -Matthias >=20 >=20 > On 02/03/2016 12:34 PM, K Zharas wrote: > > Can anyone help to create a Spout which reads a file from HDFS? > > I have tried with the code below, but it is not working. > > > > public void nextTuple() { > > Path pt=3Dnew = Path("hdfs://localhost:50070/user/BCpredict.txt"); > > FileSystem fs =3D FileSystem.get(new Configuration()); > > BufferedReader br =3D new BufferedReader(new > > InputStreamReader(fs.open(pt))); > > String line =3D br.readLine(); > > while (line !=3D null){ > > System.out.println(line); > > line=3Dbr.readLine(); > > _collector.emit(new Values(line)); > > } > > } > > > > On Tue, Feb 2, 2016 at 1:19 PM, K Zharas > > >> wrote: > > > > Hi. > > > > I have a project I'm currently working on. The idea is to = implement > > "scikit-learn" into Storm and integrate it with HDFS. > > > > I've already implemented "scikit-learn". But, currently I'm = using a > > text file to read and write. However, I need to use HDFS, but > > finding it hard to integrate with HDFS. > > > > Here is the link to github > > >. (I only included > > files that I used, not whole project) > > > > Basically, I have a few questions if you don't mint to answer = them > > 1) How to use HDFS to read and write? > > 2) Is my "scikit-learn" implementation correct? > > 3) How to create a Storm project? (Currently working in = "storm-starter") > > > > These questions may sound a bit silly, but I really can't find a > > proper solution. > > > > Thank you for your attention to this matter. > > Sincerely, Zharas. > > > > > > > > > > -- > > Best regards, > > Zharas >=20 >=20 >=20 >=20 > -- > Best regards, > Zharas >=20 >=20 >=20 > -- > Nick R. Katsipoulakis, > Department of Computer Science > University of Pittsburgh >=20 >=20 >=20 > -- > Best regards, > Zharas >=20 >=20 >=20 > -- > Best regards, > Zharas --Apple-Mail=_9F4FDF3B-92B0-4A8C-82B1-8403141AA573 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Assuming you have git and maven = installed:

git = clone git@github.com:apache/storm.git
cd = storm
git checkout -b 1.x = origin/1.x-branch
mvn install -DskipTests

That third step checks = out the 1.x-branch branch which is the base for the upcoming 1.0 = release.

You = can then include the storm-hdfs dependency in your project:

<dependency>
= <groupId>org.apache.storm</groupId>
= <artifactId>storm-hdfs</artifactId>
= <version>1.0.0-SNAPSHOT</version>
</dependency>

You can find more information on using = the spout and other HDFS components here:

https://github.com/apache/storm/tree/1.x-branch/external/storm-= hdfs#hdfs-spout

-Taylor

On Feb 3, 2016, at 2:54 PM, K Zharas <kgzharas@gmail.com> = wrote:

Oh ok. Can you plz give me an idea how can I do = it manually? I'm quite beginner :)

On Thu, Feb 4, 2016 at 3:43 AM, = Parth Brahmbhatt <pbrahmbhatt@hortonworks.com> wrote:
Storm-hdfs spout is not yet published in maven. You will = have to checkout storm locally and build it to make it available for = development. 

From: K Zharas <kgzharas@gmail.com>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>
Date: Wednesday, = February 3, 2016 at 11:41 AM
To: "user@storm.apache.org" <user@storm.apache.org>
Subject: Re: Storm + = HDFS

Yes, looks like it is. But, I have added = dependencies required by storm-hdfs as stated in a guide.

On Thu, Feb 4, 2016 at 3:33 AM, Nick R. = Katsipoulakis <nick.katsip@gmail.com> wrote:
Well, 

those errors look like a problem with the way you build = your jar file. 
Please, make sure that you build your jar with the = proper storm maven dependency).

Cheers, 
Nick

On Wed, Feb 3, 2016 at 2:31 PM, K Zharas = <kgzharas@gmail.com> wrote:
It throws and error that packages does not exist. I have = also tried changing org.apache to backtype, still got an error but only = for storm.hdfs.spout. Btw, I use Storm-0.10.0 and Hadoop-2.7.1

   package org.apache.storm does not = exist
   package org.apache.storm does not = exist
   package org.apache.storm.generated does not = exist
   package org.apache.storm.metric does not = exist
   package org.apache.storm.topology does not = exist
   package org.apache.storm.utils does not = exist
   package org.apache.storm.utils does not = exist 
   package org.apache.storm.hdfs.spout does = not exist
   package org.apache.storm.hdfs.spout does = not exist
   package org.apache.storm.topology.base does = not exist
   package org.apache.storm.topology does not = exist
   package org.apache.storm.tuple does not = exist
   package org.apache.storm.task does not = exist

On Wed, Feb 3, 2016 at 8:57 PM, Matthias J. = Sax <mjsax@apache.org> wrote:
Storm does provide HdfsSpout and HdfsBolt already. Just use those,
instead of writing your own spout/bolt:

https://github.com/apache/storm/tree/master/external/storm-hdfs=

-Matthias


On 02/03/2016 12:34 PM, K Zharas wrote:
> Can anyone help to create a Spout which reads a file from HDFS?
> I have tried with the code below, but it is not working.
>
> public void nextTuple() {
>       Path pt=3Dnew Path("hdfs://localhost:50070/user/BCpredict.txt");
= >       FileSystem fs =3D FileSystem.get(new = Configuration());
>       BufferedReader br =3D new = BufferedReader(new
> InputStreamReader(fs.open(pt)));
>       String line =3D br.readLine();
>       while (line !=3D null){
>          System.out.println(line);
>          line=3Dbr.readLine();
>          _collector.emit(new = Values(line));
>       }
> }
>
> On Tue, Feb 2, 2016 at 1:19 PM, K Zharas <kgzharas@gmail.com
> <mailto:kgzharas@gmail.com>> wrote:
>
>     Hi.
>
>     I have a project I'm currently working on. The = idea is to implement
>     "scikit-learn" into Storm and integrate it with = HDFS.
>
>     I've already implemented "scikit-learn". But, = currently I'm using a
>     text file to read and write. However, I need to = use HDFS, but
>     finding it hard to integrate with HDFS.
>
>     Here is the link to github
>     <https://github.com/kgzharas/StormTopologyTest>. (I = only included
>     files that I used, not whole = project)
>
>     Basically, I have a few questions if you don't = mint to answer them
>     1) How to use HDFS to read and write?
>     2) Is my "scikit-learn" implementation = correct?
>     3) How to create a Storm project? (Currently = working in "storm-starter")
>
>     These questions may sound a bit silly, but I = really can't find a
>     proper solution.
>
>     Thank you for your attention to this matter.
>     Sincerely, Zharas.
>
>
>
>
> --
> Best regards,
> Zharas




--
Best regards,
Zharas



--
Nick R. Katsipoulakis, 
Department of Computer Science 
University of Pittsburgh



--
Best regards,
Zharas



--
Best regards,
Zharas

= --Apple-Mail=_9F4FDF3B-92B0-4A8C-82B1-8403141AA573-- --Apple-Mail=_2CCE97EB-A772-46ED-B226-65E81E54F0EB Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJWsmtDAAoJEI3gOWLoC4/9iVUIANufkA1bKtiqcdpYt+gMxsTE Lw7MaG21pWRoEHG1pDuTnckTGN884Mt4Yvp0pwpSwPE7DX0W741fajB37g0mD61E g8FNWHVWrvlNWm7VK1VWXHQZCvb2U/f67GyU1RlKvjkWZiSsTdcMGwe5N1rbW0WS 04ZLdzcduAClagDWMvoMt5VVIXCGAB4eeWkzxoWLeUxdxNFOR+bW4c7Ro9LgTRH7 iWsDH6p9PdpV2aCn+QOFRlzT88czCh2Itj0O3olaNl+CD4n4z3GksEQhxmialtYp nnfZaEJvcg+dOM1T3zvylFrHT4I4xOhSWe6Pr1Bs4MNeMu5WehRDZ1iMMoExWF0= =gYZz -----END PGP SIGNATURE----- --Apple-Mail=_2CCE97EB-A772-46ED-B226-65E81E54F0EB--