oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mallder, Valerie" <Valerie.Mall...@jhuapl.edu>
Subject RE: workflow/lib and resource/lib jar files.
Date Tue, 15 Sep 2015 15:30:50 GMT
Hi Chris,

Ok, this all makes sense. So, I’m glad I asked the question before actually making any changes.
 I should probably withdraw the JIRA issue that I submitted for this, but let me check on
which libraries RADiX provides before I do that.

This is kind of an odd problem to try to solve.  It’s impossible to think of all the permeations
of configurations that people might run the system in. But, I have to wonder if there isn’t
some way we can make this sort of thing easier for the user.  Aside from just “remembering”
to manually copy some jar files from one place to another after I install a new release, is
there an easier way to do it?  Can maven help us out? For instance, I don’t know maven that
well, but, is it possible to create a goal, or goals, to copy a jar file from one place to
another?  Lets say, for example, I am building worfkow.  Typically I use the command “mvn
package” to build it. But, is there a way a goal could be set up to do something like “mvn
package include-jars:crawler,pge,filemgr” and have maven copy crawler, pge, filemgr jar
files to the workflow/lib directory?  I’m just thinking out loud here.  I'm dreaming of
having a totally automated OODT installation for myself.  :)

Val



Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory

> -----Original Message-----
> From: Mattmann, Chris A (3980) [mailto:chris.a.mattmann@jpl.nasa.gov]
> Sent: Friday, September 11, 2015 1:56 AM
> To: dev@oodt.apache.org
> Subject: Re: workflow/lib and resource/lib jar files.
> 
> To clarify this, since I was dictating on my cell while sending this.
> 
> 1. The design of the workflow manager is so that it doesn’t need CAS-PGE. You
> need it Val, b/c you are using CAS-PGE and combining workflow, resmgr, filemgr,
> and crawler. That’s the point of CAS-PGE.
> However, if that came standard in workflow manager, it would be a cyclic
> dependency nightmare.
> 
> 2. Workflow doesn’t require CAS-PGE, nor file manager, nor crawler, etc. It only
> requires Resource Manager (as an optional dependency if you submit jobs to it).
> The 3 service components of OODT were carefully designed to have this
> independence from one another (e.g., file manager doesn’t depend on workflow;
> workflow only [optionally] depends on resource manager; resource manager
> doesn’t depend on FM or WM).
> 
> 3. You can use Workflow manager to manage control flow and data flow (send
> first run this script, then send an email) and it works fine.
> 
> 4. RADIX should DEFINITELY have these dependencies right. It’s meant to give
> you a full stack OODT deployment including CAS-PGE. If it doesn’t have the deps
> right, we can fix RADIX.
> 
> Cheers,
> Chris
> 
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398) NASA Jet
> Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +++++
> Adjunct Associate Professor, Computer Science Department University of
> Southern California, Los Angeles, CA 90089 USA
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +++++
> 
> 
> 
> 
> 
> -----Original Message-----
> From: jpluser <chris.a.mattmann@jpl.nasa.gov>
> Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Date: Thursday, September 10, 2015 at 3:31 PM
> To: "<dev@oodt.apache.org>" <dev@oodt.apache.org>
> Subject: Re: workflow/lib and resource/lib jar files.
> 
> >I guys I would strongly advise against making hash workflow is
> >separated from PGE for reason it has to do with the click dependencies
> >and also because the standard work let manager out of the box still can
> >manage controlling dataflow it doesn't need a file management or
> >crawler or things like that
> >
> >Sent from my iPhone
> >
> >> On Sep 10, 2015, at 3:27 PM, Mallder, Valerie
> >><Valerie.Mallder@jhuapl.edu> wrote:
> >>
> >> Sure, I'll do it next week. I'm off tomorrow.  It will be nice to
> >>have this fixed!
> >>
> >> Val
> >>
> >>
> >> Valerie A. Mallder
> >> New Horizons Deputy Mission System Engineer Johns Hopkins
> >> University/Applied Physics Laboratory
> >>
> >>
> >>> -----Original Message-----
> >>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> >>> Sent: Thursday, September 10, 2015 5:48 PM
> >>> To: dev@oodt.apache.org
> >>> Subject: Re: workflow/lib and resource/lib jar files.
> >>>
> >>> Hi Val,
> >>> I think your right!
> >>>
> >>> lmcgibbn@LMC-032857
> >>>
> >>>/usr/local/oodt_trunk/workflow/target/cas-workflow-0.11-SNAPSHOT/lib(
> >>>mas
> >>>ter)
> >>> $ ls
> >>> antlr-2.7.6.jar                commons-collections-2.1.jar
> >>> geronimo-activation_1.1_spec-1.1.jar    junit-3.8.2.jar
> >>> tika-core-1.10.jar
> >>> aopalliance-1.0.jar            commons-dbcp-1.2.1.jar
> >>> geronimo-javamail_1.4_mail-1.8.1.jar    lucene-core-2.0.0.jar
> >>> xercesImpl-2.9.1.jar
> >>> asm-1.5.3.jar                commons-httpclient-3.0.jar
> >>> guava-10.0.1.jar            oodt-commons-0.11-SNAPSHOT.jar
> >>> xml-apis-1.0.b2.jar
> >>> asm-attrs-1.5.3.jar            commons-io-1.4.jar
> >>> hibernate-3.2.5.ga.jar            pcs-input-0.11-SNAPSHOT.jar
> >>> xmlrpc-2.0.1.jar
> >>> cas-cli-0.11-SNAPSHOT.jar        commons-lang-2.4.jar
> >>> hsqldb-1.8.0.7.jar            spring-beans-2.0.8.jar
> >>> xpp3_min-1.1.4c.jar
> >>> cas-metadata-0.11-SNAPSHOT.jar        commons-logging-1.0.3.jar
> >>> httpclient-4.2.1.jar            spring-context-2.0.8.jar
> >>> xstream-1.3.1.jar
> >>> cas-resource-0.11-SNAPSHOT.jar        commons-pool-1.2.jar
> >>> httpcore-4.2.1.jar            spring-core-2.5.4.jar
> >>> cas-workflow-0.11-SNAPSHOT.jar        concurrent-1.3.4.jar
> >>> joda-time-2.5.jar            spring-dao-2.0.8.jar
> >>> cglib-2.1_3.jar                dom4j-1.6.1.jar
> >>> jsr305-1.3.9.jar            spring-hibernate3-2.0.8.jar
> >>> commons-codec-1.3.jar            ehcache-1.2.3.jar
> >>> jug-2.0.0-asl.jar            spring-jdbc-2.0.8.jar
> >>>
> >>> No PGETaskInstance (from cas-pge-0.x.jar), ProductCrawler (from
> >>> cas-crawler-0.x.jar) and CoreMetKeys (from cas-filemgr-0.x.jar).
> >>>
> >>> The libraries should be defined within the specified pom.xml files
> >>>with scope set to  runtime e.g within workflow/pom.xml you should
> >>>define
> >>>
> >>>    <dependency>
> >>>      <groupId>org.apache.oodt</groupId>
> >>>      <artifactId>cas-pge</artifactId>
> >>>      <version>${project.parent.version}</version>
> >>>      <scope>runtime</scope>
> >>>    </dependency>
> >>>
> >>> ... etc
> >>>
> >>> Can you please submit a patch for both modules?
> >>> Thanks
> >>> Lewis
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Sep 10, 2015 at 1:25 PM, Mallder, Valerie <
> >>>Valerie.Mallder@jhuapl.edu>
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> Have a question for ya.  How in the world do y'all run the workflow
> >>>> manager and resource manager right out of the box?  I mean, aside
> >>>> from having to configure the properties files and policy files, how
> >>>> do you get by with just what's in their lib directories?
> >>>>
> >>>> When I run a simple system consisting of filemgr and workflow
> >>>>manager  and PGE tasks, the workflow manager crashes because it is
> >>>>unable to  load the following classes PGETaskInstance (from
> >>>>cas-pge-0.x.jar),  ProductCrawler (from cas-crawler-0.x.jar) and
> >>>>CoreMetKeys (from  cas-filemgr-0.x.jar). None of these jar files are
> >>>>in the workflow/lib  directory by default. I always end up copying
> >>>>them to the workflow/lib directory.
> >>>>
> >>>> The same thing happens if I add the resource manager to the mix,
> >>>> the resource manager crashes right off the bat because it can't
> >>>> load TaskJobInput (from caas-workflow-0.x.jar), StdPGETaskInstance
> >>>> (from cas-pge-0.x.jar), ProductCrawler (cas-crawler-0.x.jar) and
> >>>> CoreMetKeys (from cas-filemgr-0.x.jar). None of these jars are in
> >>>> the resource/lib directory by default, and I have to copy them over.
> >>>>
> >>>> How can this be?  Am I the only one this has happened to?
> >>>>
> >>>> I opened a JIRA issue for this but then I thought I better ask
> >>>> whether or not there is a valid reason why these jars are not
> >>>> included in workflow/lib and resource/lib directories. And if there
> >>>> is, then does that mean that everyone does the same thing I do and
> >>>> just copy these jar files to workflow/lib and resource/lib?
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Val
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Valerie A. Mallder
> >>>>
> >>>> New Horizons Deputy Mission System Engineer The Johns Hopkins
> >>>> University/Applied Physics Laboratory
> >>>> 11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
> >>>> 240-228-7846 (Office) 410-504-2233 (Blackberry)
> >>>
> >>>
> >>> --
> >>> *Lewis*

Mime
View raw message