oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Remote data transfer
Date Fri, 22 Aug 2014 03:43:08 GMT
Hey Guys,

The key here is that the data transfer factory that the File Manager
(Server and Client), the Crawler, CAS-PGE, etc., use could be leveraged
to try and test different data transfer technologies.

You guys may also be interested in scoping out my PhD dissertation
which covered this topic extensively:

http://sunset.usc.edu/~mattmann/Dissertation.pdf

Also the software framework for evaluating and classifying different
movement technologies is now up on Github:

https://github.com/chrismattmann/disco/

The real cool thing to do would be to make a "DISCOConnector", which uses
the evaluation framework to pick the right connector for the scenario.

BTW, iRODS can do data movement - we've used it before *with* OODT
as one example of a data movement technology. OODT is happy to
work with it in the context of the Data Transfer interface I mentioned
above.

Cheers,
Chris


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Etienne Koen <etiennek@scs-space.com>
Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Date: Wednesday, August 20, 2014 5:35 AM
To: Thomas Bennett <thomas@ska.ac.za>
Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: RE: Remote data transfer

>Hi Tom
>
>One of the other tools which is currently being tested called iRods has
>this capability/option you are referring to built into it. Does OODT have
>something similar or how would it be done?
>
>Here is the definition of the checksum we want to perform given on the
>confluence page:
>
>"A checksum or hash sum is a small-size datum from an arbitrary block of
>digital data for the purpose of detecting errors which may have been
>introduced during its transmission or storage."
>
>Does this help?
>
>Etienne
>________________________________________
>From: Thomas Bennett [thomas@ska.ac.za]
>Sent: Wednesday, August 20, 2014 1:26 PM
>To: Etienne Koen
>Cc: dev@oodt.apache.org
>Subject: Re: Remote data transfer
>
>Hi Etienne,
>
>I am using TCP. My first thoughts were that a checksum using TCP is not
>necessary since the protocol takes care of the data integrity but there
>seems to be a requirement for this in the test plan to have a chescksum
>capability.
>
>Not to confuse issues, but I perform an md5sum on all data files (hdf5
>files) once they have been created and store it as metadata for the file.
>Does the requirement maybe not refer to this?
>
>I would like to test a UDP protocol with a checksum calculation as well
>for some comparison. Please let me know if you has success with
>implementing/using this functionality!
>
>I will also have a look at the UDT transport mechanism as you proposed.
>
>Woot!
>
>Cheers,
>Tom
>
>________________________________
>Disclaimer: This E-mail message, including any attachments, is intended
>only for the person or entity to which it is addressed, and may contain
>confidential information. Each page attached hereto must also be read in
>conjunction with this disclaimer.
>If you are not the intended recipient you are hereby notified that any
>disclosure, copying, distribution or reliance upon the contents of this
>e-mail is strictly prohibited. E.&O.E.
>
>Disclaimer: This E-mail message, including any attachments, is intended
>only for the  person or entity to which it is addressed, and may contain
>confidential  information. Each page attached hereto must also be read in
>conjunction with this disclaimer.
>If you are not the intended recipient you are hereby notified that any
>disclosure, copying, distribution or reliance upon the contents of this
>e-mail is strictly prohibited.    E.&O.E.


Mime
View raw message