streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Franklin <m.ben.frank...@gmail.com>
Subject Re: Distribution / Docker next steps
Date Mon, 17 Oct 2016 15:40:08 GMT
On Mon, Oct 17, 2016 at 11:36 AM sblackmon <sblackmon@apache.org> wrote:

>
> On October 11, 2016 at 11:01:18 AM, Matt Franklin (
> m.ben.franklin@gmail.com) wrote:
>
> On Mon, Oct 10, 2016 at 11:30 AM sblackmon <sblackmon@apache.org> wrote:
>
> > Some other projects are currently looking at publishing docker
> containers
> > that people can easily extend. I am totally in favor of this approach.
> >
> >
> > Docker distribution would open up a lot of cool options for this
> project.
> >
> > Which projects are farthest along this road?
> >
>
> https://hub.docker.com/r/apache/
>
>
> I had been thinking more along the lines of publishing a distribution for
> each provider, processor, and persister module containing a minimal
> uber-jar.  Going this route would probably warrant a dedicated organization
> for streams.  OTOH, if we get to the point of having a binary distribution
> containing all of the classes in streams-project, that could be published
> to a top-level /apache repository and perform all of the same work
> (probably with a much larger docker image)
>

Tomcat (and I think a few others) have their own organization on Docker
Hub, so it is definitely a possibility.


>
>
> >
> > I think even publishing this as a Docker file example on the website
> would
> > be a good start.
> >
> > These PRs use a maven docker plugin during verify phase.
> > https://github.com/apache/incubator-streams-examples/pull/14
> > https://github.com/apache/incubator-streams/pull/288
> >
> > The same plugin can build tag and deploy images with goals docker:build
> > and docker:push .
> >
>
> Per policy, the only thing that should make it to repositories like Docker
> hub and Maven Central should be released convenience binaries.
>
>
> I think the next step is to figure out what would need to happen to build,
> certify, and publish a convenience binary and docker image for (initially)
> just one one individual provider module in an upcoming releases.  The
> dependency tree for a single provider will be more tractable than for the
> whole project and there’s a clear user benefit - greatly simplified project
> tutorial.
>

I would submit an Infra ticket


>
>
> >
> > Once these merge I’ll take another pass through the examples
> documentation
> > and for each describe a few alternative processes (STREAMS-428)
> >
> > 1) Build from source, run stream from *nix shell with dist uber-jar.
> > 2) Run stream with sbt interactive shell using artifacts from maven
> central
> > 3) Run stream with docker using artifacts from docker hub
> >
> > On October 10, 2016 at 8:09:45 AM, Matt Franklin (
> m.ben.franklin@gmail.com)
> > wrote:
> >
> > On Thu, Oct 6, 2016 at 2:56 PM sblackmon <sblackmon@apache.org> wrote:
> >
> >
> >
> > >
> >
> > >
> >
> > > TL;DR I’ve found a way to dramatically reduce barriers to using
> streams
> > as
> >
> > > a beginner.
> >
> > >
> >
> > >
> >
> > >
> >
> > > Using the streams 0.3 release, it’s quite a headache for a novice to
> use
> >
> > > streams. We have a tutorial on the website, but it’s quite a journey.
> You
> >
> > > have to check out all three repos and install them each in order
> before
> > you
> >
> > > get a jar file you could use to get data, then you can run a few
> > pre-canned
> >
> > > streams, and those are intermediate not beginner level.
> >
> > >
> >
> > >
> >
> > >
> >
> > > In an ideal world, anyone would be able to yum or apt-get (or docker
> > pull)
> >
> > > individual providers or processors and run them on their own without
> >
> > > building from source or composing them into multi-step streams.
> >
> > >
> >
> > >
> >
> > >
> >
> > > We'd have increase our build and compliance complexity significantly
> to
> >
> > > publish official binaries. So what can we do to drop the learning
> curve
> >
> > > precipitously without doing that?
> >
> > >
> >
> >
> >
> > Some other projects are currently looking at publishing docker
> containers
> >
> > that people can easily extend. I am totally in favor of this approach.
> >
> >
> >
> >
> >
> > >
> >
> > >
> >
> > >
> >
> > > Providers are really simple to run. The hard part is getting all of
> the
> >
> > > right classes and configuration properties into a JVM. Inspired by how
> >
> > > zeppelin’s %dep interpreter reduces the friction in composing and
> > running a
> >
> > > scala notebook, I wanted to find a way to get the same ability from a
> > linux
> >
> > > shell.
> >
> > >
> >
> > >
> >
> > >
> >
> > > The commands below go from just a java installation to flat files of
> >
> > > twitter data in just a few minutes.
> >
> > >
> >
> > >
> >
> > >
> >
> > > I think until we have binary distributions, this is how our tutorials
> >
> > > should tell the world to get started with streams.
> >
> > >
> >
> > >
> >
> > >
> >
> > > Thoughts?
> >
> > >
> >
> >
> >
> > I think even publishing this as a Docker file example on the website
> would
> >
> > be a good start.
> >
> >
> >
> >
> >
> > >
> >
> > >
> >
> > >
> >
> > > -----
> >
> > >
> >
> > >
> >
> > >
> >
> > > # install sbtx
> >
> > >
> >
> > >
> >
> > >
> >
> > > curl -s https://raw.githubusercontent.com/paulp/sbt-extras/master/sbt
> >
> >
> > > /usr/bin/sbtx && chmod 0755 /usr/bin/sbtx
> >
> > >
> >
> > >
> >
> > >
> >
> > > # create a workspace
> >
> > >
> >
> > >
> >
> > >
> >
> > > mkdir twitter-test; cd twitter-test;
> >
> > >
> >
> > >
> >
> > >
> >
> > > # supply a config file with credentials
> >
> > >
> >
> > >
> >
> > >
> >
> > > cat > application.conf << EOF
> >
> > >
> >
> > > twitter {
> >
> > >
> >
> > > oauth {
> >
> > >
> >
> > > consumerKey = ""
> >
> > >
> >
> > > consumerSecret = ""
> >
> > >
> >
> > > accessToken = ""
> >
> > >
> >
> > > accessTokenSecret = ""
> >
> > >
> >
> > > }
> >
> > >
> >
> > > retrySleepMs = 5000
> >
> > >
> >
> > > retryMax = 250
> >
> > >
> >
> > > info = [
> >
> > >
> >
> > > 18055613
> >
> > >
> >
> > > ]
> >
> > >
> >
> > > }
> >
> > >
> >
> > > EOF
> >
> > >
> >
> > >
> >
> > >
> >
> > > sbtx -210 -sbt-create
> >
> > >
> >
> > >
> >
> > >
> >
> > > set resolvers += "Local Maven Repository" at
> >
> > > "file://"+Path.userHome.absolutePath+"/.m2/repository"
> >
> > >
> >
> > >
> >
> > >
> >
> > > set libraryDependencies += "org.apache.streams" %
> >
> > > "streams-provider-twitter" % "0.4-incubating-SNAPSHOT"
> >
> > >
> >
> > >
> >
> > >
> >
> > > set fork := true
> >
> > >
> >
> > >
> >
> > >
> >
> > > run-main
> >
> > > org.apache.streams.twitter.provider.TwitterUserInformationProvider
> >
> > > application.conf users.txt
> >
> > >
> >
> > >
> >
> > >
> >
> > > run-main org.apache.streams.twitter.provider.TwitterTimelineProvider
> >
> > > application.conf statuses.txt
> >
> > >
> >
> > >
> >
> > >
> >
> > > set javaOptions += "-Dtwitter.endpoint=friends"
> >
> > >
> >
> > >
> >
> > >
> >
> > > run-main org.apache.streams.twitter.provider.TwitterFollowingProvider
> >
> > > application.conf friends.txt
> >
> > >
> >
> > >
> >
> > >
> >
> > > set javaOptions += "-Dtwitter.endpoint=followers"
> >
> > >
> >
> > >
> >
> > >
> >
> > > exit
> >
> > >
> >
> > >
> >
> > >
> >
> > > ls -l
> >
> > >
> >
> > >
> >
> > >
> >
> > > Steves-MacBook-Pro-3:twitter sblackmon$ ls -l
> >
> > >
> >
> > > -rw-r--r--@ 1 sblackmon staff 356 Oct 6 11:54 application.conf
> >
> > >
> >
> > > -rw-r--r-- 1 sblackmon staff 293780 Oct 6 13:42 followers.txt
> >
> > >
> >
> > > -rw-r--r-- 1 sblackmon staff 6260 Oct 6 13:43 friends.txt
> >
> > >
> >
> > > drwxr-xr-x 3 sblackmon staff 102 Oct 6 10:17 project
> >
> > >
> >
> > > -rw-r--r-- 1 sblackmon staff 3339460 Oct 6 13:43 statuses.txt
> >
> > >
> >
> > > drwxr-xr-x 6 sblackmon staff 204 Oct 6 10:19 target
> >
> > >
> >
> > > -rw-r--r-- 1 sblackmon staff 3321 Oct 6 13:43 users.txt
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message