Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
Date: Fri, 14 Oct 2016 12:01:06 -0500
From: sblackmon <sblackmon@apache.org>
To: Trevor Grant <trevor.d.grant@gmail.com>
Cc: dev@streams.incubator.apache.org
Message-ID: <etPan.58010f52.56be32db.d81@apache.org>
In-Reply-To: <CAN2fLa-FTgiyvg4LeEdpN-Ag3v_L-Hp0GigPWEi+6bC9oQfxjA@mail.gmail.com>
References: <etPan.57f69e39.37c16be2.283@apache.org>
 <CAENpzYU6hxBvJ4pmg96qszdXKQyASUxpxAbS5F=1e6rc38dWaQ@mail.gmail.com>
 <etPan.57fbb40e.7def0c24.4d07@apache.org>
 <CAENpzYWb+mh=XnpEYCXcCO1bex=MhRo9KyHmNND-q206iw_H4Q@mail.gmail.com>
 <CAN2fLa-FTgiyvg4LeEdpN-Ag3v_L-Hp0GigPWEi+6bC9oQfxjA@mail.gmail.com>
Subject: Re: Ease-of-use : minimizing TTHW (time-to-hello-world)
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="58010f52_41e7d84b_d81"
archived-at: Fri, 14 Oct 2016 17:01:17 -0000

--58010f52_41e7d84b_d81
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Trevor,

Awesome, thanks for giving it a shot. =C2=A0

With some recent changes we=E2=80=99re quite close to making data collect=
ion with streams providers turnkey for new users

I ran the following through my deployment of zeppelin - it should work fo=
r you too. =C2=A0Please confirm :)

Cheers,
Steve

=E2=80=94=E2=80=94

%dep
z.reset()
z.addRepo(=22apache-snapshots=22).url(=22https://repository.apache.org/co=
ntent/repositories/snapshots=22).snapshot()
z.load(=22org.apache.streams:streams-provider-twitter:0.4-incubating-SNAP=
SHOT=22)

import com.typesafe.config.=5F
import org.apache.streams.config.=5F
import org.apache.streams.core.=5F
import java.util.Iterator
import org.apache.streams.twitter.pojo.=5F
import org.apache.streams.twitter.provider.=5F

val hocon =3D s=22=22=22
=C2=A0 =C2=A0 twitter =7B
=C2=A0 =C2=A0 =C2=A0 oauth =7B
=C2=A0 =C2=A0 =C2=A0 =C2=A0consumerKey =3D =22=22
=C2=A0 =C2=A0 consumerSecret =3D =22=22
=C2=A0 =C2=A0 accessToken =3D =22=22
=C2=A0 =C2=A0 accessTokenSecret =3D =22=22
=C2=A0 =C2=A0 =C2=A0 =7D
=C2=A0 =C2=A0 =C2=A0 retrySleepMs =3D 5000
=C2=A0 retryMax =3D 250
=C2=A0 info =3D =5B
=C2=A0 =C2=A0 18055613
=C2=A0 =5D
=C2=A0 =C2=A0 =7D
=22=22=22

val typesafe =3D Config=46actory.parseString(hocon)
val config =3D new ComponentConfigurator(classOf=5BTwitterUserInformation=
Configuration=5D).detectConfiguration(typesafe, =22twitter=22);
val provider =3D new TwitterTimelineProvider(config);
provider.prepare(null)
provider.startStream()
while(provider.isRunning())

val resultSet =3D provider.readCurrent()
resultSet.size()
val iterator =3D resultSet.iterator();
while(iterator.hasNext()) =7B
=C2=A0 =C2=A0 val datum =3D iterator.next();
=C2=A0 =C2=A0 println(datum.getDocument)
=7D
On October 14, 2016 at 8:33:55 AM, Trevor Grant (trevor.d.grant=40gmail.c=
om) wrote:

I agree a minimal TTHW would be good- esp a user who is trying to create =
a =20
hello world. =20

I am a big fan of Apache Zeppelin notebooks for this sort of thing- easy =
to =20
host and include Markdown. =20

If I could get some community assistance getting myself started, I'd be =20
happy to write it up. =20

I need to know: =20
Minimum dependencies- =20
=46rom the little work I have done so far I know this can be a murky =20
subject as we migrate version. I'd prefer to do the minimal example in =20
what ever version can be ran based on artifacts sitting in maven now. Hap=
py =20
to update when new version is pushed. =20

Scala- =20
Zeppelin is for all intents and purposes like running in the Spark/=46lin=
k =20
shell. I'll need some help getting things going in this sort of env. =20

If someone reading this is like =22oh that's easy, here's your dependenci=
es, =20
and then run this code=22, that would be very helpful, I can get to writi=
ng =20
right away. Otherwise I can hack it out, but again will need some support=
. =20

tg =20


Trevor Grant =20
Data Scientist =20
https://github.com/rawkintrevo =20
http://stackexchange.com/users/3002022/rawkintrevo =20
http://trevorgrant.org =20

*=22=46ortunate is he, who is able to know the causes of things.=22 -Virg=
il* =20


On Tue, Oct 11, 2016 at 11:00 AM, Matt =46ranklin <m.ben.franklin=40gmail=
.com> =20
wrote: =20

> On Mon, Oct 10, 2016 at 11:30 AM sblackmon <sblackmon=40apache.org> wro=
te: =20
> =20
> > Some other projects are currently looking at publishing docker contai=
ners =20
> > that people can easily extend. I am totally in favor of this approach=
. =20
> > =20
> > =20
> > Docker distribution would open up a lot of cool options for this proj=
ect. =20
> > =20
> > Which projects are farthest along this road=3F =20
> > =20
> =20
> https://hub.docker.com/r/apache/ =20
> =20
> =20
> > =20
> > I think even publishing this as a Docker file example on the website =
=20
> would =20
> > be a good start. =20
> > =20
> > These PRs use a maven docker plugin during verify phase. =20
> > https://github.com/apache/incubator-streams-examples/pull/14 =20
> > https://github.com/apache/incubator-streams/pull/288 =20
> > =20
> > The same plugin can build tag and deploy images with goals docker:bui=
ld =20
> > and docker:push . =20
> > =20
> =20
> Per policy, the only thing that should make it to repositories like Doc=
ker =20
> hub and Maven Central should be released convenience binaries. =20
> =20
> =20
> > =20
> > Once these merge I=E2=80=99ll take another pass through the examples =
=20
> documentation =20
> > and for each describe a few alternative processes (STREAMS-428) =20
> > =20
> > 1) Build from source, run stream from *nix shell with dist uber-jar. =
=20
> > 2) Run stream with sbt interactive shell using artifacts from maven =20
> central =20
> > 3) Run stream with docker using artifacts from docker hub =20
> > =20
> > On October 10, 2016 at 8:09:45 AM, Matt =46ranklin ( =20
> m.ben.franklin=40gmail.com) =20
> > wrote: =20
> > =20
> > On Thu, Oct 6, 2016 at 2:56 PM sblackmon <sblackmon=40apache.org> wro=
te: =20
> > =20
> > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > TL;DR I=E2=80=99ve found a way to dramatically reduce barriers to u=
sing streams =20
> > as =20
> > =20
> > > a beginner. =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > Using the streams 0.3 release, it=E2=80=99s quite a headache for a =
novice to =20
> use =20
> > =20
> > > streams. We have a tutorial on the website, but it=E2=80=99s quite =
a journey. =20
> You =20
> > =20
> > > have to check out all three repos and install them each in order be=
fore =20
> > you =20
> > =20
> > > get a jar file you could use to get data, then you can run a few =20
> > pre-canned =20
> > =20
> > > streams, and those are intermediate not beginner level. =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > In an ideal world, anyone would be able to yum or apt-get (or docke=
r =20
> > pull) =20
> > =20
> > > individual providers or processors and run them on their own withou=
t =20
> > =20
> > > building from source or composing them into multi-step streams. =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > We'd have increase our build and compliance complexity significantl=
y to =20
> > =20
> > > publish official binaries. So what can we do to drop the learning c=
urve =20
> > =20
> > > precipitously without doing that=3F =20
> > =20
> > > =20
> > =20
> > =20
> > =20
> > Some other projects are currently looking at publishing docker contai=
ners =20
> > =20
> > that people can easily extend. I am totally in favor of this approach=
. =20
> > =20
> > =20
> > =20
> > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > Providers are really simple to run. The hard part is getting all of=
 the =20
> > =20
> > > right classes and configuration properties into a JVM. Inspired by =
how =20
> > =20
> > > zeppelin=E2=80=99s %dep interpreter reduces the friction in composi=
ng and =20
> > running a =20
> > =20
> > > scala notebook, I wanted to find a way to get the same ability from=
 a =20
> > linux =20
> > =20
> > > shell. =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > The commands below go from just a java installation to flat files o=
f =20
> > =20
> > > twitter data in just a few minutes. =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > I think until we have binary distributions, this is how our tutoria=
ls =20
> > =20
> > > should tell the world to get started with streams. =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > Thoughts=3F =20
> > =20
> > > =20
> > =20
> > =20
> > =20
> > I think even publishing this as a Docker file example on the website =
=20
> would =20
> > =20
> > be a good start. =20
> > =20
> > =20
> > =20
> > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > ----- =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =23 install sbtx =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > curl -s https://raw.githubusercontent.com/paulp/sbt-extras/master/s=
bt =20
> > =20
> > =20
> > > /usr/bin/sbtx && chmod 0755 /usr/bin/sbtx =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =23 create a workspace =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > mkdir twitter-test; cd twitter-test; =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =23 supply a config file with credentials =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > cat > application.conf << EO=46 =20
> > =20
> > > =20
> > =20
> > > twitter =7B =20
> > =20
> > > =20
> > =20
> > > oauth =7B =20
> > =20
> > > =20
> > =20
> > > consumerKey =3D =22=22 =20
> > =20
> > > =20
> > =20
> > > consumerSecret =3D =22=22 =20
> > =20
> > > =20
> > =20
> > > accessToken =3D =22=22 =20
> > =20
> > > =20
> > =20
> > > accessTokenSecret =3D =22=22 =20
> > =20
> > > =20
> > =20
> > > =7D =20
> > =20
> > > =20
> > =20
> > > retrySleepMs =3D 5000 =20
> > =20
> > > =20
> > =20
> > > retryMax =3D 250 =20
> > =20
> > > =20
> > =20
> > > info =3D =5B =20
> > =20
> > > =20
> > =20
> > > 18055613 =20
> > =20
> > > =20
> > =20
> > > =5D =20
> > =20
> > > =20
> > =20
> > > =7D =20
> > =20
> > > =20
> > =20
> > > EO=46 =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > sbtx -210 -sbt-create =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > set resolvers +=3D =22Local Maven Repository=22 at =20
> > =20
> > > =22file://=22+Path.userHome.absolutePath+=22/.m2/repository=22 =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > set libraryDependencies +=3D =22org.apache.streams=22 % =20
> > =20
> > > =22streams-provider-twitter=22 % =220.4-incubating-SNAPSHOT=22 =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > set fork :=3D true =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > run-main =20
> > =20
> > > org.apache.streams.twitter.provider.TwitterUserInformationProvider =
=20
> > =20
> > > application.conf users.txt =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > run-main org.apache.streams.twitter.provider.TwitterTimelineProvide=
r =20
> > =20
> > > application.conf statuses.txt =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > set javaOptions +=3D =22-Dtwitter.endpoint=3Dfriends=22 =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > run-main org.apache.streams.twitter.provider.Twitter=46ollowingProv=
ider =20
> > =20
> > > application.conf friends.txt =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > set javaOptions +=3D =22-Dtwitter.endpoint=3Dfollowers=22 =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > exit =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > ls -l =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > Steves-MacBook-Pro-3:twitter sblackmon=24 ls -l =20
> > =20
> > > =20
> > =20
> > > -rw-r--r--=40 1 sblackmon staff 356 Oct 6 11:54 application.conf =20
> > =20
> > > =20
> > =20
> > > -rw-r--r-- 1 sblackmon staff 293780 Oct 6 13:42 followers.txt =20
> > =20
> > > =20
> > =20
> > > -rw-r--r-- 1 sblackmon staff 6260 Oct 6 13:43 friends.txt =20
> > =20
> > > =20
> > =20
> > > drwxr-xr-x 3 sblackmon staff 102 Oct 6 10:17 project =20
> > =20
> > > =20
> > =20
> > > -rw-r--r-- 1 sblackmon staff 3339460 Oct 6 13:43 statuses.txt =20
> > =20
> > > =20
> > =20
> > > drwxr-xr-x 6 sblackmon staff 204 Oct 6 10:19 target =20
> > =20
> > > =20
> > =20
> > > -rw-r--r-- 1 sblackmon staff 3321 Oct 6 13:43 users.txt =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > > =20
> > =20
> > =20
> =20

--58010f52_41e7d84b_d81--