oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Reynolds <jreyno...@vpicu.net>
Subject Re: Amazon S3 data transfer for filemanager
Date Thu, 12 Mar 2015 16:06:23 GMT
Hi Lewis,
using the latest docker buggtb/oodt image, which i assume is .8
here’s the command i’m running to test the upload

filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName test
--productStructure Flat --productTypeName GenericFile --metadataFile file:///root/test.txt.met
--refs file:///root/test.txt

i verified that i can upload to the path using the s3 tools on the box / with same credentials
i put in the properties file

here’s the full exception returned:

rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
	at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
	at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
	at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
	at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
	at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
	at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to
upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752)
	... 12 more
Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable
to calculate a request signature: Algorithm HmacSHA1 not available
	at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)
	at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57)
	at com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393)
	at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76)
	... 13 more
Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Algorithm
HmacSHA1 not available
	at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)
	at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)
	... 20 more
Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available
	at javax.crypto.Mac.getInstance(Mac.java:176)
	at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86)
	... 21 more
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
	at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
	at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
	at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
	at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
	at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198)
	at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
	at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
	at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
java.lang.Exception: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41]
: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303)
	at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112)
	at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
	at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
	at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350)
ERROR: Failed to ingest product 'test' : Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41]
: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error
ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException:
Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt


> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com> wrote:
> 
> Hi John,
> What beau on of OODT are you using?
> The file types should be encapsulated by the factory implementation I would
> guess. It should be. Case of mike this the default filemgr store then off
> you go.
> Can you provide a paste of how your arriving at your exception please?
> Lewia
> 
> On Wednesday, March 11, 2015, John Reynolds <jreynolds@vpicu.net> wrote:
> 
>> Should S3DataTransferer work with the generic file type from the get go or
>> is there anything i need to change
>> i was using the radix docker but getting 'Caused by:
>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available’
>> i tried downloading the jce but no dice
>> i might try just building on my local machine and trying again
>> 
>> Thanks
>> 
>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com <javascript:;>> wrote:
>>> 
>>> Hey John,
>>> You're right, you've nearly answered everything here ;)
>>> OK, so your properties can be located here
>>> 
>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124
>>> You'll see the usual key, value pairs in there which you can ad before
>>> build the project and packaging filemgr.properties along with your
>> compiled
>>> code.
>>> Regarding the following,
>>> 
>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <jreynolds@vpicu.net
>> <javascript:;>> wrote:
>>> 
>>>> 
>>>> in the near future, i will want to pass in a security token as well to
>> the
>>>> S3 client. to do so i would extend or modify the s3datatransfer class
>>>> (which expires after an hour), what’s the best practice to read in this
>>>> transient data from the command line (or wherever else i would initiate
>> a
>>>> job from) since this wouldnt go in a properties file
>>> 
>>> 
>>> Do you mean that your security token expires hourly? As oppose to your s3
>>> client?
>>> Lets try and defined exactly what you are after here.
>>> Thanks
>>> Lewis
>> 
>> 
> 
> -- 
> *Lewis*

Mime
View raw message