streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Franklin <m.ben.frank...@gmail.com>
Subject Re: Ease-of-use : minimizing TTHW (time-to-hello-world)
Date Tue, 11 Oct 2016 16:00:59 GMT
On Mon, Oct 10, 2016 at 11:30 AM sblackmon <sblackmon@apache.org> wrote:

> Some other projects are currently looking at publishing docker containers
> that people can easily extend. I am totally in favor of this approach.
>
>
> Docker distribution would open up a lot of cool options for this project.
>
> Which projects are farthest along this road?
>

https://hub.docker.com/r/apache/


>
> I think even publishing this as a Docker file example on the website would
> be a good start.
>
> These PRs use a maven docker plugin during verify phase.
> https://github.com/apache/incubator-streams-examples/pull/14
> https://github.com/apache/incubator-streams/pull/288
>
> The same plugin can build tag and deploy images with goals docker:build
> and docker:push .
>

Per policy, the only thing that should make it to repositories like Docker
hub and Maven Central should be released convenience binaries.


>
> Once these merge I’ll take another pass through the examples documentation
> and for each describe a few alternative processes (STREAMS-428)
>
> 1) Build from source, run stream from *nix shell with dist uber-jar.
> 2) Run stream with sbt interactive shell using artifacts from maven central
> 3) Run stream with docker using artifacts from docker hub
>
> On October 10, 2016 at 8:09:45 AM, Matt Franklin (m.ben.franklin@gmail.com)
> wrote:
>
> On Thu, Oct 6, 2016 at 2:56 PM sblackmon <sblackmon@apache.org> wrote:
>
>
>
> >
>
> >
>
> > TL;DR I’ve found a way to dramatically reduce barriers to using streams
> as
>
> > a beginner.
>
> >
>
> >
>
> >
>
> > Using the streams 0.3 release, it’s quite a headache for a novice to use
>
> > streams. We have a tutorial on the website, but it’s quite a journey. You
>
> > have to check out all three repos and install them each in order before
> you
>
> > get a jar file you could use to get data, then you can run a few
> pre-canned
>
> > streams, and those are intermediate not beginner level.
>
> >
>
> >
>
> >
>
> > In an ideal world, anyone would be able to yum or apt-get (or docker
> pull)
>
> > individual providers or processors and run them on their own without
>
> > building from source or composing them into multi-step streams.
>
> >
>
> >
>
> >
>
> > We'd have increase our build and compliance complexity significantly to
>
> > publish official binaries. So what can we do to drop the learning curve
>
> > precipitously without doing that?
>
> >
>
>
>
> Some other projects are currently looking at publishing docker containers
>
> that people can easily extend.  I am totally in favor of this approach.
>
>
>
>
>
> >
>
> >
>
> >
>
> > Providers are really simple to run. The hard part is getting all of the
>
> > right classes and configuration properties into a JVM. Inspired by how
>
> > zeppelin’s %dep interpreter reduces the friction in composing and
> running a
>
> > scala notebook, I wanted to find a way to get the same ability from a
> linux
>
> > shell.
>
> >
>
> >
>
> >
>
> > The commands below go from just a java installation to flat files of
>
> > twitter data in just a few minutes.
>
> >
>
> >
>
> >
>
> > I think until we have binary distributions, this is how our tutorials
>
> > should tell the world to get started with streams.
>
> >
>
> >
>
> >
>
> > Thoughts?
>
> >
>
>
>
> I think even publishing this as a Docker file example on the website would
>
> be a good start.
>
>
>
>
>
> >
>
> >
>
> >
>
> > -----
>
> >
>
> >
>
> >
>
> > # install sbtx
>
> >
>
> >
>
> >
>
> > curl -s https://raw.githubusercontent.com/paulp/sbt-extras/master/sbt >
>
> > /usr/bin/sbtx && chmod 0755 /usr/bin/sbtx
>
> >
>
> >
>
> >
>
> > # create a workspace
>
> >
>
> >
>
> >
>
> > mkdir twitter-test; cd twitter-test;
>
> >
>
> >
>
> >
>
> > # supply a config file with credentials
>
> >
>
> >
>
> >
>
> > cat > application.conf << EOF
>
> >
>
> > twitter {
>
> >
>
> >   oauth {
>
> >
>
> >     consumerKey = ""
>
> >
>
> >     consumerSecret = ""
>
> >
>
> >     accessToken = ""
>
> >
>
> >     accessTokenSecret = ""
>
> >
>
> >   }
>
> >
>
> >   retrySleepMs = 5000
>
> >
>
> >   retryMax = 250
>
> >
>
> >   info = [
>
> >
>
> >     18055613
>
> >
>
> >   ]
>
> >
>
> > }
>
> >
>
> > EOF
>
> >
>
> >
>
> >
>
> > sbtx -210 -sbt-create
>
> >
>
> >
>
> >
>
> > set resolvers += "Local Maven Repository" at
>
> > "file://"+Path.userHome.absolutePath+"/.m2/repository"
>
> >
>
> >
>
> >
>
> > set libraryDependencies += "org.apache.streams" %
>
> > "streams-provider-twitter" % "0.4-incubating-SNAPSHOT"
>
> >
>
> >
>
> >
>
> > set fork := true
>
> >
>
> >
>
> >
>
> > run-main
>
> > org.apache.streams.twitter.provider.TwitterUserInformationProvider
>
> > application.conf users.txt
>
> >
>
> >
>
> >
>
> > run-main org.apache.streams.twitter.provider.TwitterTimelineProvider
>
> > application.conf statuses.txt
>
> >
>
> >
>
> >
>
> > set javaOptions += "-Dtwitter.endpoint=friends"
>
> >
>
> >
>
> >
>
> > run-main org.apache.streams.twitter.provider.TwitterFollowingProvider
>
> > application.conf friends.txt
>
> >
>
> >
>
> >
>
> > set javaOptions += "-Dtwitter.endpoint=followers"
>
> >
>
> >
>
> >
>
> > exit
>
> >
>
> >
>
> >
>
> > ls -l
>
> >
>
> >
>
> >
>
> > Steves-MacBook-Pro-3:twitter sblackmon$ ls -l
>
> >
>
> > -rw-r--r--@ 1 sblackmon staff 356 Oct 6 11:54 application.conf
>
> >
>
> > -rw-r--r-- 1 sblackmon staff 293780 Oct 6 13:42 followers.txt
>
> >
>
> > -rw-r--r-- 1 sblackmon staff 6260 Oct 6 13:43 friends.txt
>
> >
>
> > drwxr-xr-x 3 sblackmon staff 102 Oct 6 10:17 project
>
> >
>
> > -rw-r--r-- 1 sblackmon staff 3339460 Oct 6 13:43 statuses.txt
>
> >
>
> > drwxr-xr-x 6 sblackmon staff 204 Oct 6 10:19 target
>
> >
>
> > -rw-r--r-- 1 sblackmon staff 3321 Oct 6 13:43 users.txt
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message