Return-Path: X-Original-To: apmail-oodt-dev-archive@www.apache.org Delivered-To: apmail-oodt-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AC9F71757F for ; Tue, 14 Oct 2014 20:01:06 +0000 (UTC) Received: (qmail 95768 invoked by uid 500); 14 Oct 2014 20:01:06 -0000 Delivered-To: apmail-oodt-dev-archive@oodt.apache.org Received: (qmail 95732 invoked by uid 500); 14 Oct 2014 20:01:06 -0000 Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@oodt.apache.org Delivered-To: mailing list dev@oodt.apache.org Received: (qmail 95721 invoked by uid 99); 14 Oct 2014 20:01:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Oct 2014 20:01:06 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Valerie.Mallder@jhuapl.edu designates 128.244.251.37 as permitted sender) Received: from [128.244.251.37] (HELO piper.jhuapl.edu) (128.244.251.37) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Oct 2014 20:01:01 +0000 Received: from aplex01.dom1.jhuapl.edu (aplex01.dom1.jhuapl.edu [128.244.198.5]) by piper.jhuapl.edu with smtp (TLS: TLSv1/SSLv3,256bits,AES256-SHA) id 1264_4d72_6c4e6a12_7c4d_4cb1_9b09_a5b61afb0e7b; Tue, 14 Oct 2014 16:00:39 -0400 X-CrossPremisesHeadersFilteredBySendConnector: aplex01.dom1.jhuapl.edu Received: from aplex01.dom1.jhuapl.edu (128.244.198.5) by aplex01.dom1.jhuapl.edu (128.244.198.5) with Microsoft SMTP Server (TLS) id 15.0.847.32; Tue, 14 Oct 2014 15:58:56 -0400 Received: from aplex01.dom1.jhuapl.edu ([fe80::f159:e1c5:d8d7:cf40]) by aplex01.dom1.jhuapl.edu ([fe80::f159:e1c5:d8d7:cf40%22]) with mapi id 15.00.0847.030; Tue, 14 Oct 2014 15:58:56 -0400 From: "Mallder, Valerie" To: "dev@oodt.apache.org" Subject: RE: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement Thread-Topic: Failed to build PgeConfig, exception in PathUtils.doDynamicReplacement Thread-Index: Ac/k2GLZsYPja5c8QFqPOeCTR3z27gADbMoAAL4YoYA= Date: Tue, 14 Oct 2014 19:58:55 +0000 Message-ID: <83a188f8de4a4780ae5be622a3ae48eb@aplex01.dom1.jhuapl.edu> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.244.103.238] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OrganizationHeadersPreserved: aplex01.dom1.jhuapl.edu X-Virus-Checked: Checked by ClamAV on apache.org Hi Chris, Thanks for your emails from Friday and Saturday. They were extremely helpfu= l and I have now worked my way through all of my setup issues so far. I wa= s using the new style CASE-PGE keys because Rishi used those in his working= example and at one point I had been trying to mimic that example. I didn't= know anything about the legacy mode flag. But, after changing the keys to = the old style, everything started to come together a little faster. To start my crawler at the beginning of my pipeline, I chose the approach t= hat Rishi had taken where the pgeConfig starts the crawler_launcher script = directly rather than rely on the definition of the output files to trigger = the crawler to run. This was because the met extractor I am using is a per= l script, and your DRAT system only shows how to specify a class for the me= tFileWriterClass attribute. I could not find anything to show me how to spe= cify a perl script as the met file writer instead. Also, having to specify= the file types to crawl for by using a regular expression in the config fi= le seemed like an unnecessary duplication of the definition of the files ty= pes to crawl for (since the regular expression just specifies a mime-type t= hat is already defined in mime-types.xml). I didn't want to specify the mim= e-type in two different places. So, now I've been through the process of setting up my CAS-PGE to ingest th= e raw science and spacecraft housekeeping telemetry files, and run a postIn= gestSuccess action to copy some of those files to a new directory for furth= er processing. The next step in my pipeline may be a little tricky. I need to query the fi= lemgr and pass the results of the query (it could be a long list) as an inp= ut parameter to the first algorithm in the pipeline which, happens to be an= other perl script. Does anyone know how I might do this in a PGE config fi= le? Thanks!! Val Valerie A. Mallder New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied Physics Laboratory > -----Original Message----- > From: Mattmann, Chris A (3980) [mailto:chris.a.mattmann@jpl.nasa.gov] > Sent: Friday, October 10, 2014 7:59 PM > To: dev@oodt.apache.org > Subject: Re: Failed to build PgeConfig, exception in > PathUtils.doDynamicReplacement > > Thanks Val you are close! > > Looking at what you show below, I think the issue is one of the following= : > > 1. You are using the new style CAS-PGE Keys. Try using the old ones (e.g.= , the > ones present here: > http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/example= s/ > WorkflowTask/tasks.xml > > (note the ?_? and not the ?/? used in keys). > These keys are used by CAS-PGE if you see something in your wmgr bin scri= pt (or > your resource manager batch_stub script) stating ?legacyMode=3Dtrue?). > If you > are using RADIX, I believe that?s the case: > > http://s.apache.org/hO > > 2. Regarding better building environments. The m2e plugin for Eclipse is = fantastic > nowadays and can literally checkout a multi-module Maven project from SVN > (when paired with Subversive as a plugin or Subclipse). Another thing to = check out > is this page on the wiki: > > https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Developer+H= el > p > > > Let me know if that fixes it. Sorry for all the trouble :) Trust me it wi= ll be worth it. > CAS-PGE is awesome once working. > > Cheers, > Chris > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > +++++ > Chris Mattmann, Ph.D. > Chief Architect > Instrument Software and Science Data Systems Section (398) NASA Jet > Propulsion Laboratory Pasadena, CA 91109 USA > Office: 168-519, Mailstop: 168-527 > Email: chris.a.mattmann@nasa.gov > WWW: http://sunset.usc.edu/~mattmann/ > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > +++++ > Adjunct Associate Professor, Computer Science Department University of > Southern California, Los Angeles, CA 90089 USA > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > +++++ > > > > > > > -----Original Message----- > From: , Valerie > Reply-To: "dev@oodt.apache.org" > Date: Friday, October 10, 2014 at 3:40 PM > To: "dev@oodt.apache.org" > Subject: Failed to build PgeConfig, exception in PathUtils.doDynamicRepla= cement > > >Hi All, > > > >I'm still working on this! Still trying to get a CA-PGE Task to run. > >I'm almost there. Now it at least trys to build the PGE config file. But= the > >XmlFilePgeConfigBuilder is failing. I looks like it is picking up a > >null string somewhere, but it also looks like it is crashing in a > >strange place. It is unable to read the .xml file that contains my > >... but, the correct path and file are shown in > >the output log messages I've include below which also shows the stack > >trace. I looks like it is failing during a recursive call to > >PathUtils.doDynamicReplacement. I saw some chatter about some prior > >errors in this code in the mailing list archives from 2010 and 2012. So > >I am hoping someone might remember and be able to tell me if I am doing > >something that is causing this error. I included my tasks.xml and > >fei-crawler-pge-config.xml file after the log messages. > > > >At this point, I think need to start running these processes in an > >environment where I can debug this better. The runtime output and > >stack traces simply aren't enough for me to track this down. I am > >newbie to using eclipse and mvn, so, does anyone have some notes on the > >best way to import oodt-0.7 into an exclipe workspace and tell it to > >build it using mvn?? I am hoping I can leverage other peoples > >knowledge of how to do this so that I can do it quickly and not waste > >another week. I imported the oodt-0.7 directory into eclipse so I could > >view the files easier, but I just made eclipse link to the where the > >files are located rather that copy them into a workspace folder. If > >there's a better way, please tell me. And now, how do I tell exclipse to= build > stuff? > > > >Thanks!! > >Valerie > > > > > > > >Using CATALINA_BASE: > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat > >Using CATALINA_HOME: > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat > >Using CATALINA_TMPDIR: > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat/temp > >Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55 > >Workflow Manager started PID file > >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/ > >run > >/cas.workflow.pid). > >Oct 10, 2014 5:53:52 PM > >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > >loadProperties > >INFO: Loading Workflow Manager Configuration Properties from: > >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workflow/ > >etc > >/workflow.properties] > >Oct 10, 2014 5:53:52 PM > >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory > >getResmgrUrl > >INFO: No Resource Manager URL provided or malformed URL: executing jobs > >locally. URL: [null] Oct 10, 2014 5:53:52 PM > >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM > >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent > >INFO: WorkflowManager: Received event: startJediPipeline Oct 10, 2014 > >5:54:12 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > >handleEvent > >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for event > >startJediPipeline Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > >checkTaskRequiredMetadata > >INFO: Task: [feiCrawlerTaskName] has no required metadata fields Oct > >10, 2014 5:54:13 PM > >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > >executeTaskLocally > >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014 5:54:13 > >PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >INFO: Converting workflow configuration to static metadata... > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGETask/WorkflowManagerUrl] > >value =3D [http://localhost:9001] Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D > >[PGETask/Ingest/CrawlerConfigFile] > >value =3D > >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/craw > >ler > >/policy/crawler-config.xml] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGETask/Name] value =3D > >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGETask/Ingest/ActionIds] value > >=3D [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGETask/DumpMetadata] value =3D > >[true] Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D > >[PGETask/Query/ClientTransferServiceFactory] value =3D > >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGE_HOME] value =3D > >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGETask/Query/FileManagerUrl] > >value =3D [http://localhost:9000] Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D > >[PGETask/Ingest/MimeExtractorRepo] > >value =3D > >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/exte > >nsi > >ons/policy/mime-extractor-map.xml] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGETask/ConfigFilePath] value = =3D > >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/exte > >nsi > >ons/config/fei-crawler-pge-config.xml] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding static metadata: key =3D [PGETask/Ingest/FileManagerUrl] > >value =3D [http://localhost:9000] Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >INFO: Loading workflow context metadata... > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding dynamic metadata: key =3D [WorkflowInstId] value =3D > >[f8730997-50c7-11e4-b9aa-57625eee7ebd] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding dynamic metadata: key =3D [JobId] value =3D > >[f8730997-50c7-11e4-b9aa-57625eee7ebd] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding dynamic metadata: key =3D [WorkflowManagerUrl] value =3D > >[http://slothrop.jhuapl.edu:9001] > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding dynamic metadata: key =3D [TaskId] value =3D > >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.pge.PGETaskInstance > >createPgeMetadata > >FINEST: Adding dynamic metadata: key =3D [ProcessingNode] value =3D > >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.pge.PGETaskInstance > >createPgeConfig > >INFO: Create PgeConfig... > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance > >createPgeConfig > >INFO: Using default PgeConfigBuilder: > >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder > >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance run > >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse > >value: null > >java.io.IOException: Failed to build PgeConfig : Failed to parse value: > >null > > at > >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge > >Con > >figBuilder.java:87) > > at > >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance > >.ja > >va:230) > > at > >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > > at > >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ex > >ecu > >teTaskLocally(IterativeWorkflowProcessorThread.java:574) > > at > >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ru > >n(I > >terativeWorkflowProcessorThread.java:321) > > at > >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Sourc= e) > > at java.lang.Thread.run(Thread.java:745) > >Caused by: java.lang.Exception: Failed to parse value: null > > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501= ) > > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480= ) > > at > >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge > >Con > >figBuilder.java:77) > > ... 6 more > >Caused by: java.lang.NullPointerException > > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) > > at java.util.regex.Matcher.reset(Matcher.java:308) > > at java.util.regex.Matcher.(Matcher.java:228) > > at java.util.regex.Pattern.matcher(Pattern.java:1088) > > at > >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplac > >eme > >nt(PathUtils.java:321) > > at > >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUt > >ils > >.java:96) > > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488= ) > > ... 8 more > > > >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExc > >ept > >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse > >value: null > > at > >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150) > > at > >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ex > >ecu > >teTaskLocally(IterativeWorkflowProcessorThread.java:574) > > at > >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.ru > >n(I > >terativeWorkflowProcessorThread.java:321) > > at > >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Sourc= e) > > at java.lang.Thread.run(Thread.java:745) > >Caused by: java.io.IOException: Failed to build PgeConfig : Failed to > >parse value: null > > at > >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge > >Con > >figBuilder.java:87) > > at > >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance > >.ja > >va:230) > > at > >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) > > ... 4 more > >Caused by: java.lang.Exception: Failed to parse value: null > > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501= ) > > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480= ) > > at > >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePge > >Con > >figBuilder.java:77) > > ... 6 more > >Caused by: java.lang.NullPointerException > > at java.util.regex.Matcher.getTextLength(Matcher.java:1234) > > at java.util.regex.Matcher.reset(Matcher.java:308) > > at java.util.regex.Matcher.(Matcher.java:228) > > at java.util.regex.Pattern.matcher(Pattern.java:1088) > > at > >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplac > >eme > >nt(PathUtils.java:321) > > at > >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUt > >ils > >.java:96) > > at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488= ) > > ... 8 more > >Oct 10, 2014 5:54:13 PM > >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > >executeTaskLocally > >WARNING: Exception executing task: [feiCrawlerTaskName] locally: Message= : > >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: > >null > > > > > > > > > >Content of tasks.xml > > > > > > > > >class=3D"org.apache.oodt.cas.pge.StdPGETaskInstance"> > > > > > > > > > > >value=3D"file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-depl= o > >y/e xtensions/config/fei-crawler-pge-config.xml"/> > > > > >value=3D"http://localhost:9001" /> > > >value=3D"http://localhost:9000" /> > > >value=3D"http://localhost:9000"/> > > > > >value=3D"org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFacto= r > >y"/ > >> > > >value=3D"file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-depl= o > >y/c > >rawler/policy/crawler-config.xml"/> > > >value=3D"file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-depl= o > >y/e xtensions/policy/mime-extractor-map.xml"/> > > >value=3D"MoveFileToLevel0Dir"/> > > >value=3D"/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pg= e"/> > > > > > > > > > > > > > > > >Contents of fei-crawler-pge-config.xml > > > > > > > > > > > > >thing ever starts --> > > >dir=3D"/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data= / > >pge > >/jobs" shell=3D"/bin/sh"> > > echo "Current Working Directory is `pwd`" > > >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/pge.l > >og< > >/cmd> > > > > > > > > > > > > > > >/st > >aging"/> > > > > > > > > > > > > > > >envReplace=3D"true"/> > > > > > > > > > > > > > > > >Valerie A. Mallder > > > >New Horizons Deputy Mission System Engineer The Johns Hopkins > >University/Applied Physics Laboratory > >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 > >240-228-7846 (Office) 410-504-2233 (Blackberry) > >