apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Sharing jars among different Apex apps on cluster
Date Mon, 16 Jan 2017 18:08:23 GMT
Bhupesh,
If stram changes the logical dag (runs only the sub-dag) on each runs, it
should be ok.

Thks
Amol


On Mon, Jan 16, 2017 at 9:18 AM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> Yes, I thought of that. That way it will be a single application
> throughout.
> Just wanted to see if there can be other options.
>
> ~ Bhupesh
>
> On Mon, Jan 16, 2017 at 9:17 PM, Thomas Weise <thw@apache.org> wrote:
>
> > Sounds like a fit for https://github.com/apache/apex-core/pull/410 ?
> >
> > On Mon, Jan 16, 2017 at 3:27 AM, Bhupesh Chawda <bhupesh@datatorrent.com
> >
> > wrote:
> >
> > > Hi All,
> > >
> > > We have a use case where I need to launch a number of DAGs on the
> cluster
> > > one after the other in sequence programatically.
> > >
> > > We are using the StramAppLauncher and StramAppFactory classes to
> launch a
> > > DAG programatically on the cluster and adding any third party
> > dependencies
> > > as part of the configuration.
> > >
> > > It is working fine except for the following issue:
> > > Every time a DAG is launched, it copies the dependencies to the
> > application
> > > folder and hence spends a good amount of time before the app actually
> > > starts running. All of the apps I run belong to the same project and
> > hence
> > > don't actually need separate set of jars.
> > >
> > > Is there any way I can make all the applications "share" the jars which
> > are
> > > uploaded when the first application is run?
> > >
> > > Thanks.
> > >
> > > ~ Bhupesh
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message