streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sblackmon <sblack...@apache.org>
Subject Distribution / Docker next steps
Date Mon, 17 Oct 2016 15:36:37 GMT

On October 11, 2016 at 11:01:18 AM, Matt Franklin (m.ben.franklin@gmail.com) wrote:
On Mon, Oct 10, 2016 at 11:30 AM sblackmon <sblackmon@apache.org> wrote:  

> Some other projects are currently looking at publishing docker containers  
> that people can easily extend. I am totally in favor of this approach.  
>  
>  
> Docker distribution would open up a lot of cool options for this project.  
>  
> Which projects are farthest along this road?  
>  

https://hub.docker.com/r/apache/  


I had been thinking more along the lines of publishing a distribution for each provider, processor,
and persister module containing a minimal uber-jar.  Going this route would probably warrant
a dedicated organization for streams.  OTOH, if we get to the point of having a binary distribution
containing all of the classes in streams-project, that could be published to a top-level /apache
repository and perform all of the same work (probably with a much larger docker image)


>  
> I think even publishing this as a Docker file example on the website would  
> be a good start.  
>  
> These PRs use a maven docker plugin during verify phase.  
> https://github.com/apache/incubator-streams-examples/pull/14  
> https://github.com/apache/incubator-streams/pull/288  
>  
> The same plugin can build tag and deploy images with goals docker:build  
> and docker:push .  
>  

Per policy, the only thing that should make it to repositories like Docker  
hub and Maven Central should be released convenience binaries.  


I think the next step is to figure out what would need to happen to build, certify, and publish
a convenience binary and docker image for (initially) just one one individual provider module
in an upcoming releases.  The dependency tree for a single provider will be more tractable
than for the whole project and there’s a clear user benefit - greatly simplified project
tutorial.


>  
> Once these merge I’ll take another pass through the examples documentation  
> and for each describe a few alternative processes (STREAMS-428)  
>  
> 1) Build from source, run stream from *nix shell with dist uber-jar.  
> 2) Run stream with sbt interactive shell using artifacts from maven central  
> 3) Run stream with docker using artifacts from docker hub  
>  
> On October 10, 2016 at 8:09:45 AM, Matt Franklin (m.ben.franklin@gmail.com)  
> wrote:  
>  
> On Thu, Oct 6, 2016 at 2:56 PM sblackmon <sblackmon@apache.org> wrote:  
>  
>  
>  
> >  
>  
> >  
>  
> > TL;DR I’ve found a way to dramatically reduce barriers to using streams  
> as  
>  
> > a beginner.  
>  
> >  
>  
> >  
>  
> >  
>  
> > Using the streams 0.3 release, it’s quite a headache for a novice to use  
>  
> > streams. We have a tutorial on the website, but it’s quite a journey. You  
>  
> > have to check out all three repos and install them each in order before  
> you  
>  
> > get a jar file you could use to get data, then you can run a few  
> pre-canned  
>  
> > streams, and those are intermediate not beginner level.  
>  
> >  
>  
> >  
>  
> >  
>  
> > In an ideal world, anyone would be able to yum or apt-get (or docker  
> pull)  
>  
> > individual providers or processors and run them on their own without  
>  
> > building from source or composing them into multi-step streams.  
>  
> >  
>  
> >  
>  
> >  
>  
> > We'd have increase our build and compliance complexity significantly to  
>  
> > publish official binaries. So what can we do to drop the learning curve  
>  
> > precipitously without doing that?  
>  
> >  
>  
>  
>  
> Some other projects are currently looking at publishing docker containers  
>  
> that people can easily extend. I am totally in favor of this approach.  
>  
>  
>  
>  
>  
> >  
>  
> >  
>  
> >  
>  
> > Providers are really simple to run. The hard part is getting all of the  
>  
> > right classes and configuration properties into a JVM. Inspired by how  
>  
> > zeppelin’s %dep interpreter reduces the friction in composing and  
> running a  
>  
> > scala notebook, I wanted to find a way to get the same ability from a  
> linux  
>  
> > shell.  
>  
> >  
>  
> >  
>  
> >  
>  
> > The commands below go from just a java installation to flat files of  
>  
> > twitter data in just a few minutes.  
>  
> >  
>  
> >  
>  
> >  
>  
> > I think until we have binary distributions, this is how our tutorials  
>  
> > should tell the world to get started with streams.  
>  
> >  
>  
> >  
>  
> >  
>  
> > Thoughts?  
>  
> >  
>  
>  
>  
> I think even publishing this as a Docker file example on the website would  
>  
> be a good start.  
>  
>  
>  
>  
>  
> >  
>  
> >  
>  
> >  
>  
> > -----  
>  
> >  
>  
> >  
>  
> >  
>  
> > # install sbtx  
>  
> >  
>  
> >  
>  
> >  
>  
> > curl -s https://raw.githubusercontent.com/paulp/sbt-extras/master/sbt >  
>  
> > /usr/bin/sbtx && chmod 0755 /usr/bin/sbtx  
>  
> >  
>  
> >  
>  
> >  
>  
> > # create a workspace  
>  
> >  
>  
> >  
>  
> >  
>  
> > mkdir twitter-test; cd twitter-test;  
>  
> >  
>  
> >  
>  
> >  
>  
> > # supply a config file with credentials  
>  
> >  
>  
> >  
>  
> >  
>  
> > cat > application.conf << EOF  
>  
> >  
>  
> > twitter {  
>  
> >  
>  
> > oauth {  
>  
> >  
>  
> > consumerKey = ""  
>  
> >  
>  
> > consumerSecret = ""  
>  
> >  
>  
> > accessToken = ""  
>  
> >  
>  
> > accessTokenSecret = ""  
>  
> >  
>  
> > }  
>  
> >  
>  
> > retrySleepMs = 5000  
>  
> >  
>  
> > retryMax = 250  
>  
> >  
>  
> > info = [  
>  
> >  
>  
> > 18055613  
>  
> >  
>  
> > ]  
>  
> >  
>  
> > }  
>  
> >  
>  
> > EOF  
>  
> >  
>  
> >  
>  
> >  
>  
> > sbtx -210 -sbt-create  
>  
> >  
>  
> >  
>  
> >  
>  
> > set resolvers += "Local Maven Repository" at  
>  
> > "file://"+Path.userHome.absolutePath+"/.m2/repository"  
>  
> >  
>  
> >  
>  
> >  
>  
> > set libraryDependencies += "org.apache.streams" %  
>  
> > "streams-provider-twitter" % "0.4-incubating-SNAPSHOT"  
>  
> >  
>  
> >  
>  
> >  
>  
> > set fork := true  
>  
> >  
>  
> >  
>  
> >  
>  
> > run-main  
>  
> > org.apache.streams.twitter.provider.TwitterUserInformationProvider  
>  
> > application.conf users.txt  
>  
> >  
>  
> >  
>  
> >  
>  
> > run-main org.apache.streams.twitter.provider.TwitterTimelineProvider  
>  
> > application.conf statuses.txt  
>  
> >  
>  
> >  
>  
> >  
>  
> > set javaOptions += "-Dtwitter.endpoint=friends"  
>  
> >  
>  
> >  
>  
> >  
>  
> > run-main org.apache.streams.twitter.provider.TwitterFollowingProvider  
>  
> > application.conf friends.txt  
>  
> >  
>  
> >  
>  
> >  
>  
> > set javaOptions += "-Dtwitter.endpoint=followers"  
>  
> >  
>  
> >  
>  
> >  
>  
> > exit  
>  
> >  
>  
> >  
>  
> >  
>  
> > ls -l  
>  
> >  
>  
> >  
>  
> >  
>  
> > Steves-MacBook-Pro-3:twitter sblackmon$ ls -l  
>  
> >  
>  
> > -rw-r--r--@ 1 sblackmon staff 356 Oct 6 11:54 application.conf  
>  
> >  
>  
> > -rw-r--r-- 1 sblackmon staff 293780 Oct 6 13:42 followers.txt  
>  
> >  
>  
> > -rw-r--r-- 1 sblackmon staff 6260 Oct 6 13:43 friends.txt  
>  
> >  
>  
> > drwxr-xr-x 3 sblackmon staff 102 Oct 6 10:17 project  
>  
> >  
>  
> > -rw-r--r-- 1 sblackmon staff 3339460 Oct 6 13:43 statuses.txt  
>  
> >  
>  
> > drwxr-xr-x 6 sblackmon staff 204 Oct 6 10:19 target  
>  
> >  
>  
> > -rw-r--r-- 1 sblackmon staff 3321 Oct 6 13:43 users.txt  
>  
> >  
>  
> >  
>  
> >  
>  
> >  
>  
> >  
>  
> >  
>  
>  

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message