oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Iwunze, Michael C (GSFC-4700)[NOAA-JPSS]" <michael.iwu...@nasa.gov>
Subject Re: Resource Manager client question
Date Wed, 09 May 2012 15:46:48 GMT
Hey Cam and Sheryl,

        Thanks for the information you provided me. Things are making a
little more sense now to me. I know I will have more questions in the near

On 5/7/12 11:26 PM, "Cameron Goodale" <goodale@apache.org> wrote:

> Hey Mike and Sheryl,
> Mike was asking me for some similar advice and I plain ran outta talent on
> this topic.  From what I can tell Mike would like to run his python scripts
> on Resource Manager without the need for setting up Workflow or PGE.
> At the time I hadn't really thought through all the configuration files
> needed, but having stewed on it I thought I should reply.  Now my current
>  SnowDS implementation is to have the Workflow Task reference a CAS-PGE
> (which contains the execution block for my python program i want to run).
>  Then the Workflow is merely configured to farm the jobs out to the
> Resource Manager.
> Here is a list of questions that I have started to wonder about with Mike's
> help, any answers would be appreciated:
> 1.  Can Resource Manager + Batchstubs be used without any additional OODT
> components?
> 2.  Is PGE required to run/wrap non-Java programs so they can run within
> Resource Manager?
> Closing comments to Mike:
>     If you are planning to use OODT for data management, it
> is initially very tempting to only setup and configure the minimal set of
> components because you will feel productive and it feels like progress is
> being made.  Trust me I know since I was in your shoes about 6 months ago
> when trying to get some image processing IDL code to run and I bably needed
> to see progress (notice I didn't use the works "make progress").  Because I
> wanted to use (what I thought was) the "easier" solution I ended up
> hardcoding paths to resources my python code needed in the code instead of
> passing the parameters into the code in the first place.  This worked
> reasonably well as long as everything stayed the same....but then it didn't
> so I had to re-visit my "easier" setup and fix it.
>     Recently I have been working to undo my mistakes and python has been
> very forgiving, but the best part was that  all the strange and mystic
> Workflow setups and PGEConfig.xml files actually started to make a whole
> lot more sense.  I am now able to configure and stand up a complete
> workflow config, then jump into PGEConfig and get the input parameters to
> my python code.  This means if the input files i need to process changes I
> don't need to change my python code, instead I can merely pass in a
> different set of parameters into the workflow and they will persist to my
> wrapped python.
>     In short I know that combing through all the xml config is tough,
> especially when things are not working as quickly as you would like.  I
> understand how defeated and frustrating it can be to have the component
> fail and just feel lost, not knowing what is causing the problem.  I know
> the documentation isn't perfect and sometimes it is missing altogether, but
> the people that are on this list will bend over backwards to help you
> understand (some will even share their config files with line-by-line
> comments included at no extra charge ;)
> Thank you Sheryl for being awesome and helpful (you always are).  Mike keep
> the questions coming and I will be sure to add in my $0.02 when I am able
> to.
> Best Regards,
> Cameron
> On Mon, May 7, 2012 at 5:09 PM, Sheryl John <sheryljj@gmail.com> wrote:
>> Hi Mike,
>> Yup, you can run your python scripts, java programs etc. from CAS-PGE which
>> is used with the Workflow Manager. Check out this cas-pge guide [1] and the
>> other wiki pages related to workflow.
>> You can use Resource Manager to run tasks sent from the Workflow Manager.
>> I've recently started testing this but there are others on the list who can
>> guide you more on the Resource Manager.
>> HTH!
>> Sheryl
>> [1] https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>> On Mon, May 7, 2012 at 3:43 PM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] <
>> michael.iwunze@nasa.gov> wrote:
>>> I have two questions, I am able to run the Resource Manager with no
>>> issues. I have some python scripts and possibly some other programs  I
>>> would like to run using the Resource Manager. From what I know so far I
>>> believe the cas-pge component needs to be used in conjunction with the
>>> Resource Manager and is used as a wrapper program for running my scripts.
>>> Can someone give me more information on how this can be accomplished or
>> are
>>> there any examples to view?
>>>  I would also like to be able to utilize the Job Scheduler, Monitor and
>>> Job queue classes that are part of the Resource Manager. I can't find any
>>> examples of how they are used anywhere. And if examples do exist can
>>> someone point me in the right direction or give me more information on
>> this?
>>> Thanks
>>> Mike
>> --
>> -Sheryl

View raw message