airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Feasibility of using OODT file manager in Airavata
Date Mon, 10 Aug 2015 17:02:35 GMT
Sorry about the delay, here’s my replies Dimuthu:

>
>Hi all,
>
>I have been working on integrating a file staging server for Apache
>Airavata as a part of my GSoC project. Main purpose of doing this is to
>manage and store input/ output files for Airavata experiments.
>
>Main use case is as follows.
>
>1. Clients (users) can push input files and metadata to file server (OODT
>file manager) and should receive the url/ identifier of uploaded file to
>provide to Airavata

The best way to do this is to (in addition to the file manager), deploy
the FM prod web service, here [1]. It’s a simple WAR file. You can
configure the WAR by making sure you have environment variables set for
each of the named variables inside of ‘[‘ and ‘]’ in this file:
http://svn.apache.org/repos/asf/oodt/trunk/webapp/fmprod/src/main/webapp/ME
TA-INF/context.xml


For example, FILEMGR_URL by default is http://localhost:9000. Make sure
the env vars are set before starting Tomcat and/or Jetty for your web
app server.

The FM prod provides REST-ful perm links to products by their IDs, etc.
The API is here:

https://cwiki.apache.org/confluence/display/OODT/File+Manager+REST+API


>2. Users submit experiments to Airavata with the url of uploaded input
>file

You can use the File Manager Ingest API [2] and/or Crawler to take
care of this.

>3. Airavata should be able to fetch the input file over SCP or some other
>mechanism. If the file server exists in same machine where Airavata also
>has been hosted, Airavata can directly fetch the file from the repository
>folder.

For this you’ll want to use PushPull [3]. Another OODT component.
Docs are here:

https://cwiki.apache.org/confluence/x/KraoAQ

> 4. Once the experiment (job) is done, Airavata should be able to push
>output files to file server and notify the client.
>5. Client should be able to download the output file.

Got it.

>
>In addition to that, enforcing security in uploading and downloading
>files is required.
>
>I tried OODT file manager by following this [1] tutorial and managed to
>push and download files using command line tools. Before going in to
>integration, I need to know some details about the feasibility of using
>OODT file manager for this scenario.
>
>1. Can OODT file manager simulate the role of file server that I have
>mentioned earlier without using other modules of OODT project?

Yep.

>2. Is there any licensing issue if I use OODT file manager in Airavata?

Nope.

>3. What is the preferred way of using file manager client to talk to the
>file manager programatically? Simply, is there a Java API for file
>manager client rather than cli commands?
>


From your use case scenario above, I would just install the entire OODT
stack. Since you are going to be running jobs, and pushing data and so
forth, I would seriously have a look at OODT’s CAS-PGE job. It does all
that. It stages a file/data/set of files automatically from the file
manager
and other places to a job working directory. Allows you to integrate and
execute any algorithm, cmd line, API or otherwise, then provides an
automatic
mechanism to extract metadata (using Tika) and to Crawl and capture the
results
using the OODT crawler and file manager. RADIX will install the entire OODT
stack and is a good start point for testing this stuff:

https://cwiki.apache.org/confluence/x/zoOOAQ

Please try RADIX out and keep the questions coming. Thanks!

Cheers,
Chris


[1] 
http://search.maven.org/remotecontent?filepath=org/apache/oodt/cas-product/
0.9/cas-product-0.9.war
[2] http://s.apache.org/nYf
[3] http://svn.apache.org/repos/asf/oodt/trunk/pushpull/

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++





-----Original Message-----
From: DImuthu Upeksha <dimuthu.upeksha2@gmail.com>
Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Date: Thursday, August 6, 2015 at 8:28 PM
To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Cc: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: Re: Feasibility of using OODT file manager in Airavata

>Hi Chris,
>
>
>Any update on this?
>
>
>On Tue, Aug 4, 2015 at 12:05 AM, Chris Mattmann
><chris.mattmann@gmail.com> wrote:
>
>I’ll provide a detailed answer on this today. Sorry been on baby
>duties and enjoying every minute of it! :)
>
>—
>Chris Mattmann
>chris.mattmann@gmail.com
>
>
>
>
>
>
>-----Original Message-----
>From: "Pierce, Marlon" <marpierc@iu.edu>
>Reply-To: <dev@oodt.apache.org>
>Date: Monday, August 3, 2015 at 8:14 AM
>To: "dev@airavata.apache.org" <dev@airavata.apache.org>,
>"dev@oodt.apache.org" <dev@oodt.apache.org>
>Subject: Re: Feasibility of using OODT file manager in Airavata
>
>>Hi Dimuthu-
>>
>>In regards to question #2, there should not be any licensing issues.
>>
>>Marlon
>>
>>
>>From: DImuthu Upeksha
>><dimuthu.upeksha2@gmail.com<mailto:dimuthu.upeksha2@gmail.com>>
>>Reply-To: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>"
>><dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
>>Date: Saturday, August 1, 2015 at 12:40 AM
>>To: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>"
>><dev@airavata.apache.org<mailto:dev@airavata.apache.org>>,
>>"dev@oodt.apache.org<mailto:dev@oodt.apache.org>"
>><dev@oodt.apache.org<mailto:dev@oodt.apache.org>>
>>Subject: Feasibility of using OODT file manager in Airavata
>>
>>Hi all,
>>
>>I have been working on integrating a file staging server for Apache
>>Airavata as a part of my GSoC project. Main purpose of doing this is to
>>manage and store input/ output files for Airavata experiments.
>>
>>Main use case is as follows.
>>
>>1. Clients (users) can push input files and metadata to file server (OODT
>>file manager) and should receive the url/ identifier of uploaded file to
>>provide to Airavata
>>2. Users submit experiments to Airavata with the url of uploaded input
>>file
>>3. Airavata should be able to fetch the input file over SCP or some other
>>mechanism. If the file server exists in same machine where Airavata also
>>has been hosted, Airavata can directly fetch the file from the repository
>>folder.
>> 4. Once the experiment (job) is done, Airavata should be able to push
>>output files to file server and notify the client.
>>5. Client should be able to download the output file.
>>
>>In addition to that, enforcing security in uploading and downloading
>>files is required.
>>
>>I tried OODT file manager by following this [1] tutorial and managed to
>>push and download files using command line tools. Before going in to
>>integration, I need to know some details about the feasibility of using
>>OODT file manager for this scenario.
>>
>>1. Can OODT file manager simulate the role of file server that I have
>>mentioned earlier without using other modules of OODT project?
>>2. Is there any licensing issue if I use OODT file manager in Airavata?
>>3. What is the preferred way of using file manager client to talk to the
>>file manager programatically? Simply, is there a Java API for file
>>manager client rather than cli commands?
>>
>>[1]
>>https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>>
>>Thank You
>>Dimuthu
>>--
>>Regards
>>
>>W.Dimuthu Upeksha
>>Undergraduate
>>
>>Department of Computer Science And Engineering
>>
>>University of Moratuwa, Sri Lanka
>
>
>
>
>
>
>
>
>
>
>-- 
>Regards
>W.Dimuthu Upeksha
>Undergraduate
>Department of Computer Science And Engineering
>University of Moratuwa, Sri Lanka
>
>

Mime
View raw message