airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <glah...@gmail.com>
Subject Re: ec2 input types
Date Fri, 13 Jun 2014 19:17:57 GMT
Logically this should be possible to do in a handler, because you can do
the initializing of the environment in one of the handler and store that
data in jobexecution context and consume them in file transfer handlers.

And provider just focus on the submission of the job. In this approach you
have more flexibility of implementing different operaitons. If you
implement everything in provider all the tasks are in one place and not
decoupled. Best way is to implement these thing in each handler and order
them properly. Its totally fine if handlers are not function if they
arrange in the proper order, but please try to give proper error if one
handler get invoked in wrong place hwich will help developer to find issues
with the configuration.

Lahiru


On Fri, Jun 13, 2014 at 3:54 AM, Nipun Udara <y.b.n.udara@gmail.com> wrote:

> Hi lahiru
>
> Thank you for the details. I am currently checking AdvancedSCPInputHandler
> and AdvancedSCPOutputHandler. Since i need to start ec2 host and ssh
> connected to transfer data and also need to access command line tool to
> copy s3 files isnt it ok to do data transfer from provider once we have
> started ec2 host. any idea regarding this.
>
> regards
> Nipun Udara
>
>
> On Thu, Jun 12, 2014 at 7:58 PM, Lahiru Gunathilake <glahiru@gmail.com>
> wrote:
>
>> Hi Nipun,
>>
>> I think you can implement this as an handler and if the input/output
>> types are URI you can invoke these handler logics. If we want to add new
>> schemas and thrift models we can focus on them little later.
>>
>> First you can write an handlers to transfer inputs and outputs and write
>> test cases to test them. Please refer SSHInputHandler,SSHOutputHandler and
>> AdvancedSCPInputHandler and AdvancedSCPOutputHandler. Initial two handlers
>> transfer files to the remote resource from airavata host and advance
>> handlers transfer files from third party machine to remote machine.
>>
>> Please try to find whether the API allow to do third party transfer, in
>> that case its easy to have one set of handlers to transfer from remote
>> machine to S3 (basically two handlers one for input one for output).
>>
>> Regards
>> Lahiru
>>
>>
>> On Thu, Jun 12, 2014 at 12:12 AM, Raminderjeet Singh <ramifnu@indiana.edu
>> > wrote:
>>
>>> Hi Nipun,
>>>
>>> S3 [1] is the storage service provided by Amazon and it provide
>>> persistent storage for input/output data. There are Amazon API's provided
>>> to develop clients[2]. To add S3 input/output type in Airavata, you need to
>>> change gfac schema and also thrift model. These 2 places will be merged to
>>> one in 0.13 release. They you can expect client to set input as S3 type and
>>> you need to add handlers to upload and download the data from S3.
>>>
>>> Thanks
>>> Raminder
>>>
>>> 1. http://aws.amazon.com/s3/
>>> 2.
>>> http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/s3/transfer/TransferManager.html
>>>
>>>
>>> On Wed, Jun 11, 2014 at 11:53 PM, Nipun Udara <y.b.n.udara@gmail.com>
>>> wrote:
>>>
>>>> Hi raminder singh
>>>>
>>>> I cannot find S3type among other input types. How can i add S3type can
>>>> you give a brief idea about doing that.
>>>>
>>>> Regards
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 8:59 PM, Raminder Singh <
>>>> raminderjsingh@gmail.com> wrote:
>>>>
>>>>> URI and S3 types can be used for data movement. We should support SCP,
>>>>> HTTP,FTP data transfers under URI type.
>>>>>
>>>>> Thanks
>>>>> Raminder
>>>>>
>>>>> On Jun 11, 2014, at 11:10 AM, Nipun Udara <y.b.n.udara@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi all
>>>>> >
>>>>> > current ec2 implementation use only String type inputs. what other
>>>>> input types will be used with ec2. i would be grateful if someone can
give
>>>>> a idea.
>>>>> >
>>>>> > regards
>>>>> > Nipun Udara
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nipun Udara
>>>>
>>>> *Undergraduate*
>>>> *Department of Computer Science & Engineering*
>>>> *University of Moratuwa*
>>>> *Sri Lanka*
>>>>
>>>>
>>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
>
>
>
> --
> Nipun Udara
>
> *Undergraduate*
> *Department of Computer Science & Engineering*
> *University of Moratuwa*
> *Sri Lanka*
>
>


-- 
System Analyst Programmer
PTI Lab
Indiana University

Mime
View raw message