oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Starch <starc...@umich.edu>
Subject Re: Amazon S3 data transfer for filemanager
Date Thu, 12 Mar 2015 20:52:17 GMT
John,

I should be more verbose.  The java classpath traditionally did not pick up
multiple jars, so it was super labor intensive to setup.  These days you
can use * inside the classpath to pick up multiple jars....but it must be
in " " because otherwise the shell will glob the * if it is outside of
quotes.

If this doesn't work, try the other recommendations in:


http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath

-Michael


On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <starchmd@umich.edu> wrote:

> John,
>
> Change:
>     -classpath "$FILEMGR_HOME"/lib \
> To:
>     -classpath "$FILEMGR_HOME/lib/*" \
>
> -Michael
>
>
> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <jreynolds@vpicu.net>
> wrote:
>
>> Thanks Michael, if i modify the filemgr-client to look like this (at the
>> end)
>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
>>   -classpath "$FILEMGR_HOME"/lib \
>>
>> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties
>> \
>>   -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \
>>
>> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml
>> \
>>
>> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml
>> \
>>   org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“
>>
>> (replacing ext jars with -classpath) then i get
>> Error: Could not find or load main class
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient
>> i assume i’m doing something wrong with classpath but not sure what
>>
>> > On Mar 12, 2015, at 11:31 AM, Michael Starch <starchmd@umich.edu>
>> wrote:
>> >
>> > John,
>> >
>> > Can you open filemgr-client sh script?  It may set the JAVA_EXT_JARS
>> > there.  If so, it is clobbering the default path for "extension" jars,
>> and
>> > your java encryption jars are not being picked up. If it does set
>> > JAVA_EXT_JARS you have two options:
>> >
>> > 1. Move all your encryption jars into FILEMGR_HOME/lib/
>> > 2. update filemgr-client script to us classpath to specify the jars in
>> the
>> > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS
>> >
>> >
>> > -Michael
>> >
>> >
>> > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jreynolds@vpicu.net>
>> wrote:
>> >
>> >> Hi Michael
>> >> yeah it’s openjdk 1.7 (“1.7.0_75")
>> >> i did download the the unlimited encryption jar from oracle and
>> replaced
>> >> the local_policy / us_export_policy jars in javahome/jre/lib/security
>> >> more i read, maybe limited by jce.jar
>> >>
>> >> i dont have anything special set for extension jars
>> >>
>> >>
>> >>
>> >>> On Mar 12, 2015, at 10:35 AM, Michael Starch <starchmd@umich.edu>
>> wrote:
>> >>>
>> >>> John,
>> >>>
>> >>> What version of the JDK are you running, and what is your extension
>> jars
>> >>> environment variable set to.  Do you have the java cryptology jar
>> >> included
>> >>> (Oracle JDK usually has this, I don't know if Open JDK does).
>> >>>
>> >>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot
>> >> find
>> >>> the java crypto jar used to calculate the given hash.
>> >>>
>> >>> -Michael
>> >>>
>> >>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jreynolds@vpicu.net>
>> >> wrote:
>> >>>
>> >>>> Hi Lewis,
>> >>>> using the latest docker buggtb/oodt image, which i assume is .8
>> >>>> here’s the command i’m running to test the upload
>> >>>>
>> >>>> filemgr-client --url http://localhost:9000 --operation
>> --ingestProduct
>> >>>> --productName test --productStructure Flat --productTypeName
>> GenericFile
>> >>>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt
>> >>>>
>> >>>> i verified that i can upload to the path using the s3 tools on the
>> box /
>> >>>> with same credentials i put in the properties file
>> >>>>
>> >>>> here’s the full exception returned:
>> >>>>
>> >>>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
>> >>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>>       at
>> >>>>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >>>>       at
>> >>>>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>>       at java.lang.reflect.Method.invoke(Method.java:606)
>> >>>>       at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>> >>>>       at
>> >>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>> >>>>       at
>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>> >>>>       at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>> >>>>       at java.lang.Thread.run(Thread.java:745)
>> >>>> Caused by:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
>> >>>>       ... 12 more
>> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate
a
>> >>>> request signature: Unable to calculate a request signature: Algorithm
>> >>>> HmacSHA1 not available
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
>> >>>>       at
>> >>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
>> >>>>       at
>> >>>>
>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
>> >>>>       ... 13 more
>> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate
a
>> >>>> request signature: Algorithm HmacSHA1 not available
>> >>>>       at
>> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
>> >>>>       at
>> >>>>
>> >>
>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
>> >>>>       ... 20 more
>> >>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1
>> >> not
>> >>>> available
>> >>>>       at javax.crypto.Mac.getInstance(Mac.java:176)
>> >>>>       at
>> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
>> >>>>       ... 21 more
>> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error
>> >>>> ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> >> ]
>> >>>> :
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>>       at
>> >>>>
>> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>>       at
>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>> >>>>       at
>> >>>>
>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>> >>>>       at
>> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>> >>>> java.lang.Exception: Failed to ingest product
>> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>> >>>> java.lang.Exception:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error
>> >>>> ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> >> ]
>> >>>> :
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
>> >>>>       at
>> >>>>
>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>> >>>>       at
>> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>> >>>>       at
>> >>>>
>> >>
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
>> >>>> ERROR: Failed to ingest product 'test' : Failed to ingest product
>> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] :
>> >>>> java.lang.Exception:
>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Error
>> >>>> ingesting product
>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1
>> >> ]
>> >>>> :
>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
>> >>>> Failed to upload product reference /root/test.txt to S3 at
>> >>>> usr/src/oodt/data/archive/test/test.txt
>> >>>>
>> >>>>
>> >>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <
>> >>>> lewis.mcgibbney@gmail.com> wrote:
>> >>>>>
>> >>>>> Hi John,
>> >>>>> What beau on of OODT are you using?
>> >>>>> The file types should be encapsulated by the factory implementation
>> I
>> >>>> would
>> >>>>> guess. It should be. Case of mike this the default filemgr store
>> then
>> >> off
>> >>>>> you go.
>> >>>>> Can you provide a paste of how your arriving at your exception
>> please?
>> >>>>> Lewia
>> >>>>>
>> >>>>> On Wednesday, March 11, 2015, John Reynolds <jreynolds@vpicu.net>
>> >> wrote:
>> >>>>>
>> >>>>>> Should S3DataTransferer work with the generic file type
from the
>> get
>> >> go
>> >>>> or
>> >>>>>> is there anything i need to change
>> >>>>>> i was using the radix docker but getting 'Caused by:
>> >>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1
not
>> >>>> available’
>> >>>>>> i tried downloading the jce but no dice
>> >>>>>> i might try just building on my local machine and trying
again
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>>
>> >>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>> >>>>>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>> >>>>>>>
>> >>>>>>> Hey John,
>> >>>>>>> You're right, you've nearly answered everything here
;)
>> >>>>>>> OK, so your properties can be located here
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>> >>>>>>> You'll see the usual key, value pairs in there which
you can ad
>> >> before
>> >>>>>>> build the project and packaging filemgr.properties along
with your
>> >>>>>> compiled
>> >>>>>>> code.
>> >>>>>>> Regarding the following,
>> >>>>>>>
>> >>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <
>> jreynolds@vpicu.net
>> >>>>>> <javascript:;>> wrote:
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> in the near future, i will want to pass in a security
token as
>> well
>> >> to
>> >>>>>> the
>> >>>>>>>> S3 client. to do so i would extend or modify the
s3datatransfer
>> >> class
>> >>>>>>>> (which expires after an hour), what’s the best
practice to read
>> in
>> >>>> this
>> >>>>>>>> transient data from the command line (or wherever
else i would
>> >>>> initiate
>> >>>>>> a
>> >>>>>>>> job from) since this wouldnt go in a properties
file
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Do you mean that your security token expires hourly?
As oppose to
>> >> your
>> >>>> s3
>> >>>>>>> client?
>> >>>>>>> Lets try and defined exactly what you are after here.
>> >>>>>>> Thanks
>> >>>>>>> Lewis
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>> --
>> >>>>> *Lewis*
>> >>>>
>> >>>>
>> >>
>> >>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message