oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mattmann <mattm...@apache.org>
Subject Re: [VOTE] Apache OODT 1.1 RC #1
Date Wed, 19 Jul 2017 15:51:38 GMT
Absolutely, here is a simple use case:

If you look at CAS-PGE files, they typically include setup to “stage files” 
(already archived in File Manager) to the local job working directly. Many times
this is just handled in the bash script (or Python or whatever) that CAS-PGE generates
itself. However it’s such a repetitive task and *part of* the actual workflow for running
a job, we added a simple interface to do so. See here:

https://raw.githubusercontent.com/apache/oodt/master/pge/src/test/resources/pge-config.xml

And notice this block:

  <fileStaging dir="[StagingDir]" force="[ForceStaging]">
    <stageFiles metadataKey="InputFiles" />
  </fileStaging>

What the above does is inspect the value of the InputFiles metadata as provided
to CAS-PGE (even if it’s a list) and for each of those files, it “stages” them (aka
uses 
the associated and declared DataTransferer for this CAS-PGE instance) to the directory
defined by [StagingDir].

Cool huh?

Cheers,
Chris


P.S. It doesn’t work now, but will shortly ;)


On 7/19/17, 8:43 AM, "Mallder, Valerie" <Valerie.Mallder@jhuapl.edu> wrote:

    Yes, adding a note to CHANGES.txt is fine.  The fix to the fileStaging feature of CAS-PGE
sounds good, although I've never understood what that feature is supposed to do :)  When you
have some free time I would love to hear about what you are using it for.
    
    Thanks!
    
    
    -----Original Message-----
    From: Chris Mattmann [mailto:mattmann@apache.org] 
    Sent: Wednesday, July 19, 2017 11:37 AM
    To: dev@oodt.apache.org
    Subject: Re: [VOTE] Apache OODT 1.1 RC #1
    
    Thank you Val!
    
    I may spin another RC not only because of the below (which I think we should just note
in CHANGES.txt (I am happy to add a note on it), but also b/c I found an interesting bug while
testing OODT 1.1 RC #1 with DRAT – the bug relates to using fileStaging – it doesn’t
explicitly set a Product.PRODUCT_STRUCTURE* and as so gets an NPE if you try and use it. I
never encountered this before b/c I never used OODT’s native file staging support in CAS-PGE.
It is a simple fix, and I think will add a lot of value (fileStaging is inoperable without
it in CAS-PGE).
    
    Due to that, I will spin another RC shortly.
    
    Thank you!
    
    Cheers,
    Chris
    
    
    
    
    On 7/19/17, 7:55 AM, "Mallder, Valerie" <Valerie.Mallder@jhuapl.edu> wrote:
    
        +1
        
        I downloaded the source and built it with "mvn package -skipTests" and it built ok.
I did a system test by running some data through my pipeline that uses cas-workflow-1.1, cas-filemgr-1.1
and cas-pge-1.1 and everything worked as I expected.  And it even helped find some things
that I needed to tweak in my test environment :) 
        
        The only problem I should note is that had to delete the workflow repository directory
and allow cas-workflow-1.1 to create a repository from scratch. cas-workflow-1.1 reported
an incompatibility with workflow repository that was created with cas-workflow-0.10 and has
been used with cas-workflow-1.0 for a while now. (Also note that, if the repository directory
exists but it is empty, workflow manager will crash. So, I had to delete the repository directory
itself rather than just delete the contents within it.)
        
         The error I received is below. It looks like the change from lucene-core-2.0.0.jar
to the lucene-core-6.1.0.jar caused this incompatibility between cas-workflow-1.0 and cas-workflow-1.1.
Deleting my pipeline's workflow respository and allowing  cas-workflow-1.1 to create a new
one was not a problem for me, but I can't speak for everyone.
        
        Jul 19, 2017 10:45:02 AM org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository
addWorkflowInstanceToCatalog
        WARNING: Unable to index workflow instance: [d8dad567-6c90-11e7-9e4e-7574e3a12106]:
Message: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/project/oodt/dev/jedi/data/workflow/slothrop/segments"))):
-1 (needs to be between 1071082519 and 1071082519). This version of Lucene only supports indexes
created with release 5.0 and later.
        java.lang.NullPointerException
        
        Val
        
        
        
        -----Original Message-----
        From: Chris Mattmann [mailto:mattmann@apache.org] 
        Sent: Tuesday, July 18, 2017 6:54 PM
        To: dev@oodt.apache.org
        Subject: [VOTE] Apache OODT 1.1 RC #1
        
        Hi Folks,
        
        I have posted a 1st release candidate for the Apache OODT 1.1 release. The source
code is at:
        
        https://dist.apache.org/repos/dist/dev/oodt/ 
        
        For more detailed information, see the included CHANGES.txt file for details on release
contents and latest changes. The release was made using the OODT release process, documented
on the Wiki here:
        
        https://cwiki.apache.org/confluence/display/OODT/Release+Process
        
        The release was made from the OODT 1.1 tag at:
        
        https://github.com/apache/oodt/tree/1.1
        
        A staged Maven repository is available at:
        
        https://repository.apache.org/content/repositories/orgapacheoodt-1012/
        
        Please vote on releasing these packages as Apache OODT 1.1. The vote is open for at
least the next 72 hours.
        
        Only votes from OODT PMC are binding, but folks are welcome to check the release candidate
and voice their approval or disapproval. The vote passes if at least three binding +1 votes
are cast.
        
        [ ] +1 Release the packages as Apache OODT 1.1
        
        [ ] -1 Do not release the packages because...
        
        Thanks!
        
        Chris Mattmann
        
        P.S. Here is my +1.
        
        
        
        
        
        
    
    
    



Mime
View raw message