oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Starch <starc...@umich.edu>
Subject Re: Multiple Processing Paradigms at Once
Date Wed, 06 Aug 2014 16:50:10 GMT
Chris,

This makes sense to me.  Do you see any problems with a BatchMgr
implementation that submits to children BatchMgr implementations?  This
would allow for users to mux to any BatchMgrs that they choose.

-Michael





On Wed, Aug 6, 2014 at 9:35 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Gotcha, thanks Mike, great job and glad you are using the trunk.
> I think it may make sense to do this as a BatchMgr implementation
> rather than a Runner.
>
> The big reason is that the whole intent of obfuscating the where/how
> something is run is architecturally supposed to be the job of the
> Resource Manager. In Wengine this was blurred a bit, and the Runner
> framework is great that was developed there, but I'd like to get to
> pushing this functionality into the Resource Manager. IOW, to me
> it makes sense to update the Resource Manager to talk to Mesos, not
> the Workflow Manager (which shouldn't care). Does that make sense?
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Michael Starch <starchmd@umich.edu>
> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Date: Wednesday, August 6, 2014 9:22 AM
> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Subject: Re: Multiple Processing Paradigms at Once
>
> >Chris,
> >
> >I did find the Runners in the trunk, and used them for the prototype piece
> >to submit to Spark.  Specifically, "
> >
> http://svn.apache.org/repos/asf/oodt/branches/wengine-branch/wengine/src/m
> >ain/java/org/apache/oodt/cas/workflow/engine/runner/MappedMultiRunner.java
> >has yet to be ported over to the trunk, and I am wondering if there is a
> >better solution than to using a Runner to submit to multiple Runners.
> >
> >
> >-Michael
> >
> >
> >On Wed, Aug 6, 2014 at 9:08 AM, Mattmann, Chris A (3980) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> See OODT-215 and OODT-491 the functionality is not absent it is present
> >> and those issues are the  current status
> >>
> >> Sent from my iPhone
> >>
> >> > On Aug 6, 2014, at 8:53 AM, "Michael Starch" <starchmd@umich.edu>
> >>wrote:
> >> >
> >> > Chris,
> >> >
> >> > I believe I am working on the truck of OODT.  The WEngine branch
> >>already
> >> > does what I need it to, however; the absence of this functionality in
> >> trunk
> >> > led me to wonder if this approach is undesired...hence this question.
> >> >
> >> > -Michael
> >> >
> >> >
> >> > On Wed, Aug 6, 2014 at 8:50 AM, Chris Mattmann
> >><chris.mattmann@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> Hi Mike,
> >> >>
> >> >> I think you are using the wengine branch of Apache OODT.
> >> >> That is unmaintained. I would sincerely urge you to get
> >> >> this working in trunk, that's where the developers are working
> >> >> right now.
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >> P.S. Let me think more about the below I have some ideas there.
> >> >>
> >> >> ------------------------
> >> >> Chris Mattmann
> >> >> chris.mattmann@gmail.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Michael Starch <starchmd@umich.edu>
> >> >> Reply-To: <dev@oodt.apache.org>
> >> >> Date: Wednesday, August 6, 2014 8:29 AM
> >> >> To: <dev@oodt.apache.org>
> >> >> Subject: Multiple Processing Paradigms at Once
> >> >>
> >> >>> All,
> >> >>>
> >> >>> I am working on upgrading OODT to allow it to process streaming
> >>data,
> >> >>> alongside traditional non-streaming jobs.  This means that some
jobs
> >> need
> >> >>> to be run by the resource manager, and other jobs need to be
> >>submitted
> >> to
> >> >>> the stream-processing.  Therefore, processing needs to be forked
or
> >> >>> multiplexed at some point in the life-cycle.
> >> >>>
> >> >>> There are two places where this can be done: workflow manager
> >>runners,
> >> and
> >> >>> the resource manager.  Currently, I am  working on building workflow
> >> >>> runners, and doing the job-multiplexing there because this cuts
out
> >>one
> >> >>> superfluous step for the streaming jobs (namely going to the
> >>resource
> >> >>> manager before being routed).
> >> >>>
> >> >>> Are there any comments on this approach or does this approach make
> >> sense?
> >> >>>
> >> >>> -Michael Starch
> >> >>
> >> >>
> >> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message