oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Etienne Koen <etien...@scs-space.com>
Subject RE: Remote data transfer
Date Thu, 14 Aug 2014 07:38:13 GMT
Hi Chris,

Thanks for your help! No problem, please feel free to make our conversation available to the
community!

I had success with ingesting a file to a remote server. Here is the scenario I had:

A file manager running 192.168.0.10 and a client, running on 192.168.0.11, wanting to ingest
a file and archive it on the file manager node (192.168.0.10). I used the tutorial at https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
(Thanks Tom!) as an outline to configure or the necessary parameters, e.g. the archive repository.

Here are a list of variables I declared:

export PROD_NAME=blah.txt
export PROD_REF=/root/source/blah.txt
export PROD_MET=/root/source/blah.txt.met

using it with the command:

./filemgr-client --url http://192.168.0.10:9000 --clientTransfer --dataTransfer org.apache.oodt.cas.filemgr.datatransfer.RemoteDataTransferFactory
--operation --ingestProduct --productName $PROD_NAME --productStructure Flat --productTypeName
GenericFile --metadataFile file://$PROD_MET --refs file://$PROD_REF

This gave a notification on the client side (192.168.0.11) that the file was ingested successfully.
The file appeared on the file manager node (192.168.0.10) under the directory I specified
:-)

However, on the file manager node, I got a warning:

WARNING: No Metadata specified for product [blah.txt] for required field [DataVersion]: Attempting
to continue processing metadata

I guess this is due to not having a metadata extractor running?

I am a bit of a newbie when it comes to cluster software, security etc.. So here is what I
have done prior to running the file manager to make sure the environment was configured to
allow communication:

- I flushed the firewall settings and added a rule to allow communication

$ iptables -F
$ iptables -I INPUT -j ACCEPT

I then checked to see if all ports allowed communication:

$ iptables -L

Cheers
Etienne
________________________________________
From: Mattmann, Chris A (3980) [chris.a.mattmann@jpl.nasa.gov]
Sent: Wednesday, August 13, 2014 6:04 PM
To: Etienne Koen; Thomas Bennett
Cc: cschollar@ska.ac.za; dev@oodt.apache.org; Mattmann, Chris A (3980)
Subject: Re: Remote data transfer

Thanks guys.

Etienne, I hope you don't mind but I've copied dev@oodt.apache.org

on this email. That way you can tap into the entire Apache OODT
community for help.

The URI has authority component is usually an error indicating
that you have referenced some environment variable in your config
(e.g., filemgr.properties in the etc directory) but that variable
isn't defined. E.g., maybe you have a *.policy.dirs property set
to file://[SOME_UNDEFINED_VARIABLE]/path/dir/ and SOME_UNDEFINED_VARIABLE
is undefined.

Can you check that to see if that's the root cause of this issue?

Cheers,
Chris

------------------------
Chris Mattmann
chris.mattmann@gmail.com




-----Original Message-----
From: Etienne Koen <etiennek@scs-space.com>
Date: Wednesday, August 13, 2014 1:42 AM
To: Thomas Bennett <thomas@ska.ac.za>
Cc: "cschollar@ska.ac.za" <cschollar@ska.ac.za>, Chris Mattmann
<chris.mattmann@gmail.com>
Subject: RE: Remote data transfer

>Hi Tom,
>
>I get the following error when using the argument:
>
>ERROR: Failed to ingest product 'blah.txt' : URI has an authority
>component
>
>Here both the server and client were using port 9000
>
>I get this when both the server and client are running on the same port
>
>When communicating on different ports I get:
>
><-- some I/O / HTTP exceptions -->
>...
>...
>
>ERROR: Failed to ingest product 'blah.txt' : Connection refused
>
>Server:9000 and Client:431
>
>Do you know what any of this mean?
>
>Cheers
>Etienne
>
>________________________________________
>From: Thomas Bennett [thomas@ska.ac.za]
>Sent: Wednesday, August 13, 2014 10:02 AM
>To: Etienne Koen
>Cc: cschollar@ska.ac.za; chris.mattmann@gmail.com
>Subject: Re: Remote data transfer
>
>Hey Etienne,
>
>I've been out of the office the last week but I'm back now.
>
>./filemgr-client --url http://localhost:9000 --operation --ingestProduct
>--productName blah.txt --productStructure Flat --productTypeName
>GenericFile --metadataFile file:///tmp/blah.txt.met --refs
>file:///tmp/blah.txt
>
>How would this line be modified to achieve what I want to do? I see there
>is also an argument --clientTransfer --dataTransfer but I am not sure
>what java class to use for this?
>
>You will need to specify the filemgr remotely ie: --url
>http://192.168.0.1 - are you doing this?
>
>I've done remote file transfer before I'll see if I can remember how to
>do it.
>
>Can I log into the CHPC with the usual credentials?
>
>Cheers,
>Tom
>--
>Thomas Bennett
>
>SKA South Africa
>Science Processing Team
>
>Office: +27 21 5067341
>Mobile: +27 79 5237105
>
>________________________________
>Disclaimer: This E-mail message, including any attachments, is intended
>only for the person or entity to which it is addressed, and may contain
>confidential information. Each page attached hereto must also be read in
>conjunction with this disclaimer.
>If you are not the intended recipient you are hereby notified that any
>disclosure, copying, distribution or reliance upon the contents of this
>e-mail is strictly prohibited. E.&O.E.
>
>Disclaimer: This E-mail message, including any attachments, is intended
>only for the  person or entity to which it is addressed, and may contain
>confidential  information. Each page attached hereto must also be read in
>conjunction with this disclaimer.
>If you are not the intended recipient you are hereby notified that any
>disclosure, copying, distribution or reliance upon the contents of this
>e-mail is strictly prohibited.    E.&O.E.


Disclaimer: This E-mail message, including any attachments, is intended only for the  person
or entity to which it is addressed, and may contain confidential  information. Each page attached
hereto must also be read in conjunction with this disclaimer.
If you are not the intended recipient you are hereby notified that any disclosure, copying,
distribution or reliance upon the contents of this e-mail is strictly prohibited.    E.&O.E.

Disclaimer: This E-mail message, including any attachments, is intended only for the  person
or entity to which it is addressed, and may contain confidential  information. Each page attached
hereto must also be read in conjunction with this disclaimer.
If you are not the intended recipient you are hereby notified that any disclosure, copying,
distribution or reliance upon the contents of this e-mail is strictly prohibited.    E.&O.E.

Mime
View raw message