oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mattmann <mattm...@apache.org>
Subject Re: [VOTE] Apache OODT 1.1 RC #1
Date Fri, 21 Jul 2017 20:11:32 GMT
Hi BW, have you tried RC #2? +1 still?

Cheers,
Chris




On 7/19/17, 5:32 PM, "BW" <webb@apache.org> wrote:

    +1
    
    On Wed, Jul 19, 2017 at 8:51 AM Chris Mattmann <mattmann@apache.org> wrote:
    
    > Absolutely, here is a simple use case:
    >
    > If you look at CAS-PGE files, they typically include setup to “stage files”
    > (already archived in File Manager) to the local job working directly. Many
    > times
    > this is just handled in the bash script (or Python or whatever) that
    > CAS-PGE generates
    > itself. However it’s such a repetitive task and *part of* the actual
    > workflow for running
    > a job, we added a simple interface to do so. See here:
    >
    >
    > https://raw.githubusercontent.com/apache/oodt/master/pge/src/test/resources/pge-config.xml
    >
    > And notice this block:
    >
    >   <fileStaging dir="[StagingDir]" force="[ForceStaging]">
    >     <stageFiles metadataKey="InputFiles" />
    >   </fileStaging>
    >
    > What the above does is inspect the value of the InputFiles metadata as
    > provided
    > to CAS-PGE (even if it’s a list) and for each of those files, it “stages”
    > them (aka uses
    > the associated and declared DataTransferer for this CAS-PGE instance) to
    > the directory
    > defined by [StagingDir].
    >
    > Cool huh?
    >
    > Cheers,
    > Chris
    >
    >
    > P.S. It doesn’t work now, but will shortly ;)
    >
    >
    > On 7/19/17, 8:43 AM, "Mallder, Valerie" <Valerie.Mallder@jhuapl.edu>
    > wrote:
    >
    >     Yes, adding a note to CHANGES.txt is fine.  The fix to the fileStaging
    > feature of CAS-PGE sounds good, although I've never understood what that
    > feature is supposed to do :)  When you have some free time I would love to
    > hear about what you are using it for.
    >
    >     Thanks!
    >
    >
    >     -----Original Message-----
    >     From: Chris Mattmann [mailto:mattmann@apache.org]
    >     Sent: Wednesday, July 19, 2017 11:37 AM
    >     To: dev@oodt.apache.org
    >     Subject: Re: [VOTE] Apache OODT 1.1 RC #1
    >
    >     Thank you Val!
    >
    >     I may spin another RC not only because of the below (which I think we
    > should just note in CHANGES.txt (I am happy to add a note on it), but also
    > b/c I found an interesting bug while testing OODT 1.1 RC #1 with DRAT – the
    > bug relates to using fileStaging – it doesn’t explicitly set a
    > Product.PRODUCT_STRUCTURE* and as so gets an NPE if you try and use it. I
    > never encountered this before b/c I never used OODT’s native file staging
    > support in CAS-PGE. It is a simple fix, and I think will add a lot of value
    > (fileStaging is inoperable without it in CAS-PGE).
    >
    >     Due to that, I will spin another RC shortly.
    >
    >     Thank you!
    >
    >     Cheers,
    >     Chris
    >
    >
    >
    >
    >     On 7/19/17, 7:55 AM, "Mallder, Valerie" <Valerie.Mallder@jhuapl.edu>
    > wrote:
    >
    >         +1
    >
    >         I downloaded the source and built it with "mvn package -skipTests"
    > and it built ok. I did a system test by running some data through my
    > pipeline that uses cas-workflow-1.1, cas-filemgr-1.1 and cas-pge-1.1 and
    > everything worked as I expected.  And it even helped find some things that
    > I needed to tweak in my test environment :)
    >
    >         The only problem I should note is that had to delete the workflow
    > repository directory and allow cas-workflow-1.1 to create a repository from
    > scratch. cas-workflow-1.1 reported an incompatibility with workflow
    > repository that was created with cas-workflow-0.10 and has been used with
    > cas-workflow-1.0 for a while now. (Also note that, if the repository
    > directory exists but it is empty, workflow manager will crash. So, I had to
    > delete the repository directory itself rather than just delete the contents
    > within it.)
    >
    >          The error I received is below. It looks like the change from
    > lucene-core-2.0.0.jar to the lucene-core-6.1.0.jar caused this
    > incompatibility between cas-workflow-1.0 and cas-workflow-1.1. Deleting my
    > pipeline's workflow respository and allowing  cas-workflow-1.1 to create a
    > new one was not a problem for me, but I can't speak for everyone.
    >
    >         Jul 19, 2017 10:45:02 AM
    > org.apache.oodt.cas.workflow.instrepo.LuceneWorkflowInstanceRepository
    > addWorkflowInstanceToCatalog
    >         WARNING: Unable to index workflow instance:
    > [d8dad567-6c90-11e7-9e4e-7574e3a12106]: Message: Format version is not
    > supported (resource
    > BufferedChecksumIndexInput(MMapIndexInput(path="/project/oodt/dev/jedi/data/workflow/slothrop/segments"))):
    > -1 (needs to be between 1071082519 and 1071082519). This version of Lucene
    > only supports indexes created with release 5.0 and later.
    >         java.lang.NullPointerException
    >
    >         Val
    >
    >
    >
    >         -----Original Message-----
    >         From: Chris Mattmann [mailto:mattmann@apache.org]
    >         Sent: Tuesday, July 18, 2017 6:54 PM
    >         To: dev@oodt.apache.org
    >         Subject: [VOTE] Apache OODT 1.1 RC #1
    >
    >         Hi Folks,
    >
    >         I have posted a 1st release candidate for the Apache OODT 1.1
    > release. The source code is at:
    >
    >         https://dist.apache.org/repos/dist/dev/oodt/
    >
    >         For more detailed information, see the included CHANGES.txt file
    > for details on release contents and latest changes. The release was made
    > using the OODT release process, documented on the Wiki here:
    >
    >         https://cwiki.apache.org/confluence/display/OODT/Release+Process
    >
    >         The release was made from the OODT 1.1 tag at:
    >
    >         https://github.com/apache/oodt/tree/1.1
    >
    >         A staged Maven repository is available at:
    >
    >
    > https://repository.apache.org/content/repositories/orgapacheoodt-1012/
    >
    >         Please vote on releasing these packages as Apache OODT 1.1. The
    > vote is open for at least the next 72 hours.
    >
    >         Only votes from OODT PMC are binding, but folks are welcome to
    > check the release candidate and voice their approval or disapproval. The
    > vote passes if at least three binding +1 votes are cast.
    >
    >         [ ] +1 Release the packages as Apache OODT 1.1
    >
    >         [ ] -1 Do not release the packages because...
    >
    >         Thanks!
    >
    >         Chris Mattmann
    >
    >         P.S. Here is my +1.
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    



Mime
View raw message