oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: OODT Documentation
Date Thu, 11 Jun 2015 14:34:27 GMT
Hi Andrea,

I sent an email - did you get it? My suggestion would be to make sure
that you update your batch stub script per my instructions here:

http://s.apache.org/YDI

Let me know thanks.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++




-----Original Message-----
From: Andrea DeMarco <dr.andrea.demarco@gmail.com>
Date: Thursday, June 11, 2015 at 7:30 AM
To: jpluser <chris.a.mattmann@jpl.nasa.gov>
Cc: "Wyngaard, Jane R (3980-Affiliate)" <Jane.R.Wyngaard@jpl.nasa.gov>,
"Starch, Michael D (398M)" <Michael.D.Starch@jpl.nasa.gov>, Kristian ZARB
ADAMI <kza@astro.ox.ac.uk>, "Ramirez, Paul M (398M)"
<paul.m.ramirez@jpl.nasa.gov>, Alessio Magro <lessju@gmail.com>,
"dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: Re: OODT Documentation

>Hi Chris,
>
>Was wondering if there are any updates on your end, or perhaps
>anything we can try on our installation?
>
>Thanks,
>Andrea
>
>On 9 June 2015 at 16:36, Andrea DeMarco <dr.andrea.demarco@gmail.com>
>wrote:
>> Hi Chris,
>>
>> Some additional detail:
>>
>> When stopping/starting the workflow manager before kicking the PGE, the
>> terminal (post PGE submission) also prints out the following:
>>
>> andrea@issa:~/oodt/workflow/bin$ ./wmgr stop
>> andrea@issa:~/oodt/workflow/bin$ ./wmgr start
>> Workflow Manager started PID file
>> (/home/andrea/oodt/workflow/run/cas.workflow.pid).
>> andrea@issa:~/oodt/workflow/bin$ Jun 09, 2015 4:34:24 PM
>> org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager loadProperties
>> INFO: Loading Workflow Manager Configuration Properties from:
>> [/home/andrea/oodt/workflow/etc/workflow.properties]
>> Jun 09, 2015 4:34:25 PM
>> org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
>> INFO: Workflow Manager started by andrea
>> Jun 09, 2015 4:34:29 PM
>> org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
>> INFO: WorkflowManager: Received event: fileconcatenator-pge
>> Jun 09, 2015 4:34:29 PM
>> org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
>> INFO: WorkflowManager: Workflow FileConcatenatorWorkflow retrieved for
>>event
>> fileconcatenator-pge
>> Jun 09, 2015 4:34:29 PM
>> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>> setRClient
>> INFO: Resource Manager Job Submission enabled to:
>>[http://localhost:9002]
>> Jun 09, 2015 4:34:29 PM
>> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>> checkTaskRequiredMetadata
>> INFO: All required metadata fields present for task: [FileConcatenator]
>> Jun 09, 2015 4:34:29 PM
>> org.apache.oodt.cas.resource.system.XmlRpcResourceManagerClient
>>submitJob
>> FINEST: [{job.queueName=high,
>> job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob,
>> job.name=urn:oodt:FileConcatenator, job.id=, job.status=, job.load=2,
>> job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput},
>> 
>>{task.instance.class=org.apache.oodt.pge.examples.fileconcatenator.FileCo
>>ncatenatorPGETask,
>> 
>>task.config={PGETask_ConfigFilePath=/home/andrea/oodt/pge/file_concatenat
>>or/pge-configs/PGEConfig.xml,
>> 
>>PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatransfer
>>.LocalDataTransferFactory,
>> 
>>PCS_ActionRepoFile=file:/home/andrea/oodt/crawler/policy/crawler-config.x
>>ml,
>> PCS_MetFileExtension=met, PGETask_DumpMetadata=true,
>> PCS_WorkflowManagerUrl=http://localhost:9001,
>> PCS_FileManagerUrl=http://localhost:9000,
>>PGETask_Name=FileConcatenator},
>> task.metadata={TaskId=[urn:oodt:FileConcatenator],
>> WorkflowManagerUrl=[http://issa:9001],
>> JobId=[a2eecfc3-0eb4-11e5-ac1b-fff04e088d71], ProcessingNode=[issa],
>> RunID=[testNumber1],
>> WorkflowInstId=[a2eecfc3-0eb4-11e5-ac1b-fff04e088d71]}}]
>>
>> Not sure what it all means, but it may help :)
>>
>> Cheers,
>> Andrea
>>
>> On 9 June 2015 at 16:03, Andrea DeMarco <dr.andrea.demarco@gmail.com>
>>wrote:
>>>
>>> Hi Chris,
>>>
>>> No problem - I can imagine that checklist to be quite extensive :)
>>>
>>> I'll try and help as best I can:
>>>
>>> 1) The output when starting the batch_stub is:
>>> andrea@issa:~/oodt/resmgr/bin$ ./batch_stub 2001
>>> [1] 2535
>>> andrea@issa:~/oodt/resmgr/bin$ Jun 09, 2015 3:53:04 PM
>>> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub <init>
>>> INFO: XmlRpc Batch Stub started by andrea
>>>
>>> 2) The latest oodt.out log file (after running the below) is attached .
>>>
>>> 3) Kicking of the PGE with the following command (and terminal output):
>>>
>>> andrea@issa:~/oodt/workflow/bin$ ./wmgr-client --url
>>>http://localhost:9001
>>> --operation --sendEvent --eventName fileconcatenator-pge --metaData
>>>--key
>>> RunID testNumber1
>>> Sending event 'fileconcatenator-pge'... SUCCESS
>>>
>>> I have now shut down my VM and restarted everything. When kicking off
>>>the
>>> PGE, this is the terminal output from the same terminal that started
>>>the
>>> batch_stub:
>>>
>>> Jun 09, 2015 3:57:26 PM
>>> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub
>>>genericExecuteJob
>>> INFO: stub attempting to execute class:
>>> [org.apache.oodt.cas.workflow.structs.TaskJob]
>>> Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance
>>> createPgeMetadata
>>> INFO: Converting workflow configuration to static metadata...
>>> Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance
>>> createPgeMetadata
>>> INFO: Loading workflow context metadata...
>>> Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance
>>> createPgeConfig
>>> INFO: Create PgeConfig...
>>> Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance
>>> createPgeConfig
>>> INFO: Using default PgeConfigBuilder:
>>> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
>>> Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance run
>>> SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
>>> value: null
>>> java.io.IOException: Failed to build PgeConfig : Failed to parse value:
>>> null
>>> at
>>> 
>>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeC
>>>onfigBuilder.java:87)
>>> at
>>> 
>>>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.
>>>java:231)
>>> at 
>>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>>> at 
>>>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>>> at
>>> 
>>>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.r
>>>un(XmlRpcBatchStub.java:229)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.Exception: Failed to parse value: null
>>> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>>> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>>> at
>>> 
>>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeC
>>>onfigBuilder.java:77)
>>> ... 5 more
>>> Caused by: java.lang.NullPointerException
>>> at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
>>> at java.util.regex.Matcher.reset(Matcher.java:308)
>>> at java.util.regex.Matcher.<init>(Matcher.java:228)
>>> at java.util.regex.Pattern.matcher(Pattern.java:1088)
>>> at
>>> 
>>>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplace
>>>ment(PathUtils.java:321)
>>> at
>>> 
>>>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUti
>>>ls.java:96)
>>> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:486)
>>> ... 7 more
>>>
>>> org.apache.oodt.cas.resource.structs.exceptions.JobInputException:
>>>Failed
>>> to run task
>>> at 
>>>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104)
>>> at
>>> 
>>>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.r
>>>un(XmlRpcBatchStub.java:229)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by:
>>> 
>>>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExce
>>>ption:
>>> PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
>>>null
>>> at 
>>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
>>> at 
>>>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>>> ... 2 more
>>> Caused by: java.io.IOException: Failed to build PgeConfig : Failed to
>>> parse value: null
>>> at
>>> 
>>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeC
>>>onfigBuilder.java:87)
>>> at
>>> 
>>>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.
>>>java:231)
>>> at 
>>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>>> ... 3 more
>>> Caused by: java.lang.Exception: Failed to parse value: null
>>> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>>> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>>> at
>>> 
>>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeC
>>>onfigBuilder.java:77)
>>> ... 5 more
>>> Caused by: java.lang.NullPointerException
>>> at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
>>> at java.util.regex.Matcher.reset(Matcher.java:308)
>>> at java.util.regex.Matcher.<init>(Matcher.java:228)
>>> at java.util.regex.Pattern.matcher(Pattern.java:1088)
>>> at
>>> 
>>>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplace
>>>ment(PathUtils.java:321)
>>> at
>>> 
>>>org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUti
>>>ls.java:96)
>>> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:486)
>>> ... 7 more
>>>
>>>
>>>
>>> Cheers,
>>> Andrea
>>>
>>>
>>> On 9 June 2015 at 15:47, Mattmann, Chris A (3980)
>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> Thanks Andrea, OK bear with me as we go through our checklist :)
>>>>
>>>> Does anything show in $OODT_HOME/log/oodt.log?
>>>>
>>>> How are you kicking off the PGE? Does the terminal in which you
>>>> started the batch stub print anything?
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Chief Architect
>>>> Instrument Software and Science Data Systems Section (398)
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 168-519, Mailstop: 168-527
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW:  http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Associate Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Andrea DeMarco <dr.andrea.demarco@gmail.com>
>>>> Date: Tuesday, June 9, 2015 at 6:44 AM
>>>> To: jpluser <chris.a.mattmann@jpl.nasa.gov>
>>>> Cc: "Wyngaard, Jane R (3980-Affiliate)"
>>>><Jane.R.Wyngaard@jpl.nasa.gov>,
>>>> "Starch, Michael D (398M)" <Michael.D.Starch@jpl.nasa.gov>, Kristian
>>>>ZARB
>>>> ADAMI <kza@astro.ox.ac.uk>, "Ramirez, Paul M (398M)"
>>>> <paul.m.ramirez@jpl.nasa.gov>, Alessio Magro <lessju@gmail.com>,
>>>> "dev@oodt.apache.org" <dev@oodt.apache.org>
>>>> Subject: Re: OODT Documentation
>>>>
>>>> >Hi Chris,
>>>> >
>>>> >
>>>> >Yup, the jar file is present in both resmgr/lib and workflow/lib. You
>>>> > can
>>>> >have a look at my JAR contents at:
>>>> >
>>>>
>>>> > 
>>>>>https://www.dropbox.com/s/1h0siibxf8sv5ut/fileconcatenator-pge-1.0.jar
>>>>>?dl=
>>>> >0
>>>>
>>>> > 
>>>>><https://www.dropbox.com/s/1h0siibxf8sv5ut/fileconcatenator-pge-1.0.ja
>>>>>r?dl
>>>> >=0>  - perhaps something's off there?
>>>> >
>>>> >
>>>> >Cheers,
>>>> >Andrea
>>>> >
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Andrea DeMarco
>> Institute of Space Sciences and Astronomy
>> University of Malta
>
>
>
>-- 
>Best Regards,
>Andrea DeMarco
>Institute of Space Sciences and Astronomy
>University of Malta

Mime
View raw message