storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bobby Evans <>
Subject Re: [DISCUSS] Ideas for resolving storm-drpc-server compilation issue on IDE
Date Mon, 27 Mar 2017 14:10:07 GMT
+1 for fixing dependency/IDE issues, but I am not sure it is as simple as what you describe.

The issue is that there is no clean way to get local mode without pulling in almost all of
the daemons too.  If we are going to go through the pain of separating them out, I would
prefer to do it once and do it right.  I am happy to help out with this, as it is something
I have been thinking about for a while, but just haven't found the time to tackle on my own.
First we need a good way to give a control to our users about the base classpath of the worker,
ideally the JVM version too.  We have been doing a really good job with rolling upgrades
and I think it would be great if we could have multiple versions of storm/JVM installed on
the worker nodes and the end user can pick what JVM and what version of storm they want their
worker to run with.  We can argue over details of how that would work later. The point is
that it lets us make changes to the classpath in very drastic ways without breaking end users.

Second we need a better way to hide local mode.  Every example we have supports local mode
which means we will ship a copy of the storm daemons in each topology jar if we pull them
out of the default classpath.  We need to be able to run existing topologies that do not
have "local mode support" in local mode.  We should be able to make storm-submitter work,
there are already stubs for this kind of thing, but we may need to play around with DRPC and
a few other APIs to make it transparent.

We then create new jars from the existing storm-core and storm-drpc-server.

storm-client - Just what the client and worker needs.  The only external dependencies are
logging and possibly metrics.
storm-local - This would pull in local mode dependencies (almost everything in storm core). 
We might even make it a test jar.

storm-daemon - all of our daemon processes (most if not all shading removed).  We can subdivide
this more if we want to.

storm-core would go away or just pull in storm-client.
The storm jar command would by default only pull in storm-client and its dependencies.  If
you wanted local mode you could add in a flag that would adjust the classpath, boot up a local
mode cluster, change the client to transparently interact with that instead of a regular cluster,
and jump to the end users main.  There could also be an option to just include everything
on the classpath without the local mode cluster.  Ideally if we include everything on the
classpath with storm jar, that would also add a flag that would make the supervisor include
everything on the classpath when launching the worker.

- Bobby

On Monday, March 27, 2017, 12:11:44 AM CDT, Jungtaek Lim <> wrote:Hi

I took a first step of finalizing port work via resolving dependency issue
with DRPC.

Here's what I'm giving a try:
- rename 'storm-drpc-server' to 'storm-webapp'
- remove 'storm-core' from 'storm-drpc-server'
-- 'storm-drpc-server' will have its own library directory or shaded jar
- create 'storm-common' and extract all the things used for both
'storm-core' and 'storm-webapp'

It requires numerous files to be moved to, and huge code block should be
moved / modified. A bit painful to work on.

Other approach would be separating 'storm-worker' (or 'storm-client') and
'storm-daemon', and link to different libraries directory.
(Maybe we could make uber jar for 'storm-daemon'.)
This also requires similar work and maybe introduce more big effect to

Other than above ideas I don't have any other ideas. We're shading
libraries which are both needed from 'storm-core' and 'storm-drpc-server'
which in turn makes known issue - able to build with maven but IDE can't
compile 'storm-drpc-server' project.

Please share other ideas if you have one.

Jungtaek Lim (HeartSaVioR)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message