oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Bennett <tho...@ska.ac.za>
Subject Re: Remote data transfer
Date Fri, 15 Aug 2014 07:54:39 GMT
Hi Etienne,

There are various methods you can use to download the data.

See this page:
https://cwiki.apache.org/confluence/display/OODT/Getting+products+from+a+remote+FileManager

Recently there is some great work that has been done on using a REST API -
this exists on svn trunk. I don't think it has been released yet.

https://cwiki.apache.org/confluence/display/OODT/File+Manager+REST+API

To use these components you will need to deploy tomcat or jetty.

Shout if you need some help.

Cheers,
Tom




On Thu, Aug 14, 2014 at 4:31 PM, Etienne Koen <etiennek@scs-space.com>
wrote:

> Hi Chris and Tom,
>
> As I have mentioned before in my previous email, I have managed to ingest
> a file to a remote location using the filemgr-client. I am also able to
> query the information remotely using for example the query_tool in this way:
>
> $ ./query_tool --url http://192.168.0.10:9000 --lucene -query
> 'CAS.ProductName:blah.txt'
>
> 978ca28e-23b0-11e4-87fb-4f1c29029486
>
> What component would I use for searching and downloading the actual
> product from the remote file manager? Is the filemgr-client or query_tool
> capable of doing this?
>
> Are there any tutorials you would recommend?
>
> Thanks
> Etienne
>
> ________________________________________
> From: Mattmann, Chris A (3980) [chris.a.mattmann@jpl.nasa.gov]
> Sent: Wednesday, August 13, 2014 6:04 PM
> To: Etienne Koen; Thomas Bennett
> Cc: cschollar@ska.ac.za; dev@oodt.apache.org; Mattmann, Chris A (3980)
> Subject: Re: Remote data transfer
>
> Thanks guys.
>
> Etienne, I hope you don't mind but I've copied dev@oodt.apache.org
>
> on this email. That way you can tap into the entire Apache OODT
> community for help.
>
> The URI has authority component is usually an error indicating
> that you have referenced some environment variable in your config
> (e.g., filemgr.properties in the etc directory) but that variable
> isn't defined. E.g., maybe you have a *.policy.dirs property set
> to file://[SOME_UNDEFINED_VARIABLE]/path/dir/ and SOME_UNDEFINED_VARIABLE
> is undefined.
>
> Can you check that to see if that's the root cause of this issue?
>
> Cheers,
> Chris
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Etienne Koen <etiennek@scs-space.com>
> Date: Wednesday, August 13, 2014 1:42 AM
> To: Thomas Bennett <thomas@ska.ac.za>
> Cc: "cschollar@ska.ac.za" <cschollar@ska.ac.za>, Chris Mattmann
> <chris.mattmann@gmail.com>
> Subject: RE: Remote data transfer
>
> >Hi Tom,
> >
> >I get the following error when using the argument:
> >
> >ERROR: Failed to ingest product 'blah.txt' : URI has an authority
> >component
> >
> >Here both the server and client were using port 9000
> >
> >I get this when both the server and client are running on the same port
> >
> >When communicating on different ports I get:
> >
> ><-- some I/O / HTTP exceptions -->
> >...
> >...
> >
> >ERROR: Failed to ingest product 'blah.txt' : Connection refused
> >
> >Server:9000 and Client:431
> >
> >Do you know what any of this mean?
> >
> >Cheers
> >Etienne
> >
> >________________________________________
> >From: Thomas Bennett [thomas@ska.ac.za]
> >Sent: Wednesday, August 13, 2014 10:02 AM
> >To: Etienne Koen
> >Cc: cschollar@ska.ac.za; chris.mattmann@gmail.com
> >Subject: Re: Remote data transfer
> >
> >Hey Etienne,
> >
> >I've been out of the office the last week but I'm back now.
> >
> >./filemgr-client --url http://localhost:9000 --operation --ingestProduct
> >--productName blah.txt --productStructure Flat --productTypeName
> >GenericFile --metadataFile file:///tmp/blah.txt.met --refs
> >file:///tmp/blah.txt
> >
> >How would this line be modified to achieve what I want to do? I see there
> >is also an argument --clientTransfer --dataTransfer but I am not sure
> >what java class to use for this?
> >
> >You will need to specify the filemgr remotely ie: --url
> >http://192.168.0.1 - are you doing this?
> >
> >I've done remote file transfer before I'll see if I can remember how to
> >do it.
> >
> >Can I log into the CHPC with the usual credentials?
> >
> >Cheers,
> >Tom
> >--
> >Thomas Bennett
> >
> >SKA South Africa
> >Science Processing Team
> >
> >Office: +27 21 5067341
> >Mobile: +27 79 5237105
> >
> >________________________________
> >Disclaimer: This E-mail message, including any attachments, is intended
> >only for the person or entity to which it is addressed, and may contain
> >confidential information. Each page attached hereto must also be read in
> >conjunction with this disclaimer.
> >If you are not the intended recipient you are hereby notified that any
> >disclosure, copying, distribution or reliance upon the contents of this
> >e-mail is strictly prohibited. E.&O.E.
> >
> >Disclaimer: This E-mail message, including any attachments, is intended
> >only for the  person or entity to which it is addressed, and may contain
> >confidential  information. Each page attached hereto must also be read in
> >conjunction with this disclaimer.
> >If you are not the intended recipient you are hereby notified that any
> >disclosure, copying, distribution or reliance upon the contents of this
> >e-mail is strictly prohibited.    E.&O.E.
>
>
> Disclaimer: This E-mail message, including any attachments, is intended
> only for the  person or entity to which it is addressed, and may contain
> confidential  information. Each page attached hereto must also be read in
> conjunction with this disclaimer.
> If you are not the intended recipient you are hereby notified that any
> disclosure, copying, distribution or reliance upon the contents of this
> e-mail is strictly prohibited.    E.&O.E.
>
> Disclaimer: This E-mail message, including any attachments, is intended
> only for the  person or entity to which it is addressed, and may contain
> confidential  information. Each page attached hereto must also be read in
> conjunction with this disclaimer.
> If you are not the intended recipient you are hereby notified that any
> disclosure, copying, distribution or reliance upon the contents of this
> e-mail is strictly prohibited.    E.&O.E.
>



-- 
Thomas Bennett

SKA South Africa
Science Processing Team

Office: +27 21 5067341
Mobile: +27 79 5237105

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message