Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@oodt.apache.org
Received-SPF: pass (athena.apache.org: domain of mdstarch@gmail.com designates
 209.85.218.52 as permitted sender)
MIME-Version: 1.0
Sender: mdstarch@gmail.com
In-Reply-To: <D007A70E.1700BF%chris.a.mattmann@jpl.nasa.gov>
References: 
 <CAFC9u93mtGO0eoH7o37vT8pdxzjwpYKPg_4swHokFGGxsn2ZsA@mail.gmail.com>
	<D0079C92.170044%chris.mattmann@gmail.com>
	<CAFC9u92tOOxaj2u-fCJx_jFBe_qHXdXWJagG797mbkaBH8ZaYQ@mail.gmail.com>
	<C09B9EC7-8AC7-4E59-B105-93D85AEA8623@jpl.nasa.gov>
	<CAFC9u90jDwUKURniGWTD79q+1gATEz1ZuAVdjB1n0rEK9=Yy=A@mail.gmail.com>
	<D007A70E.1700BF%chris.a.mattmann@jpl.nasa.gov>
Date: Wed, 6 Aug 2014 09:50:10 -0700
Message-ID: 
 <CAFC9u914m7e7nJ94YcXzhdygFCvOGhY45K-8Yv--jhFnjCoByQ@mail.gmail.com>
Subject: Re: Multiple Processing Paradigms at Once
From: Michael Starch <starchmd@umich.edu>
To: dev@oodt.apache.org
Content-Type: multipart/alternative; boundary=001a11c1fe1eec2d5d04fff8c306

--001a11c1fe1eec2d5d04fff8c306
Content-Type: text/plain; charset=UTF-8

Chris,

This makes sense to me.  Do you see any problems with a BatchMgr
implementation that submits to children BatchMgr implementations?  This
would allow for users to mux to any BatchMgrs that they choose.

-Michael


On Wed, Aug 6, 2014 at 9:35 AM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Gotcha, thanks Mike, great job and glad you are using the trunk.
> I think it may make sense to do this as a BatchMgr implementation
> rather than a Runner.
>
> The big reason is that the whole intent of obfuscating the where/how
> something is run is architecturally supposed to be the job of the
> Resource Manager. In Wengine this was blurred a bit, and the Runner
> framework is great that was developed there, but I'd like to get to
> pushing this functionality into the Resource Manager. IOW, to me
> it makes sense to update the Resource Manager to talk to Mesos, not
> the Workflow Manager (which shouldn't care). Does that make sense?
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Michael Starch <starchmd@umich.edu>
> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Date: Wednesday, August 6, 2014 9:22 AM
> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Subject: Re: Multiple Processing Paradigms at Once
>
> >Chris,
> >
> >I did find the Runners in the trunk, and used them for the prototype piece
> >to submit to Spark.  Specifically, "
> >
> http://svn.apache.org/repos/asf/oodt/branches/wengine-branch/wengine/src/m
> >ain/java/org/apache/oodt/cas/workflow/engine/runner/MappedMultiRunner.java
> >has yet to be ported over to the trunk, and I am wondering if there is a
> >better solution than to using a Runner to submit to multiple Runners.
> >
> >
> >-Michael
> >
> >
> >On Wed, Aug 6, 2014 at 9:08 AM, Mattmann, Chris A (3980) <
> >chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> >> See OODT-215 and OODT-491 the functionality is not absent it is present
> >> and those issues are the  current status
> >>
> >> Sent from my iPhone
> >>
> >> > On Aug 6, 2014, at 8:53 AM, "Michael Starch" <starchmd@umich.edu>
> >>wrote:
> >> >
> >> > Chris,
> >> >
> >> > I believe I am working on the truck of OODT.  The WEngine branch
> >>already
> >> > does what I need it to, however; the absence of this functionality in
> >> trunk
> >> > led me to wonder if this approach is undesired...hence this question.
> >> >
> >> > -Michael
> >> >
> >> >
> >> > On Wed, Aug 6, 2014 at 8:50 AM, Chris Mattmann
> >><chris.mattmann@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> Hi Mike,
> >> >>
> >> >> I think you are using the wengine branch of Apache OODT.
> >> >> That is unmaintained. I would sincerely urge you to get
> >> >> this working in trunk, that's where the developers are working
> >> >> right now.
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >> P.S. Let me think more about the below I have some ideas there.
> >> >>
> >> >> ------------------------
> >> >> Chris Mattmann
> >> >> chris.mattmann@gmail.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Michael Starch <starchmd@umich.edu>
> >> >> Reply-To: <dev@oodt.apache.org>
> >> >> Date: Wednesday, August 6, 2014 8:29 AM
> >> >> To: <dev@oodt.apache.org>
> >> >> Subject: Multiple Processing Paradigms at Once
> >> >>
> >> >>> All,
> >> >>>
> >> >>> I am working on upgrading OODT to allow it to process streaming
> >>data,
> >> >>> alongside traditional non-streaming jobs.  This means that some jobs
> >> need
> >> >>> to be run by the resource manager, and other jobs need to be
> >>submitted
> >> to
> >> >>> the stream-processing.  Therefore, processing needs to be forked or
> >> >>> multiplexed at some point in the life-cycle.
> >> >>>
> >> >>> There are two places where this can be done: workflow manager
> >>runners,
> >> and
> >> >>> the resource manager.  Currently, I am  working on building workflow
> >> >>> runners, and doing the job-multiplexing there because this cuts out
> >>one
> >> >>> superfluous step for the streaming jobs (namely going to the
> >>resource
> >> >>> manager before being routed).
> >> >>>
> >> >>> Are there any comments on this approach or does this approach make
> >> sense?
> >> >>>
> >> >>> -Michael Starch
> >> >>
> >> >>
> >> >>
> >>
>
>

--001a11c1fe1eec2d5d04fff8c306--