airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marlon Pierce <marpi...@iu.edu>
Subject Fwd: Re: Orchestrator Multi-Threaded vs Single Threaded
Date Wed, 22 Jan 2014 20:11:23 GMT
Forwarding to dev.


-------- Original Message --------
Subject: 	Re: Orchestrator Multi-Threaded vs Single Threaded
Date: 	Wed, 22 Jan 2014 14:55:23 -0500
From: 	Marlon Pierce <marpierc@iu.edu>
To: 	architecture@airavata.apache.org



Thanks, Eran.  We only have O(10) jobs at a time at the moment in
realistic scenarios, but we are also interested in more academic scaling
issues. We definitely don't want to invent something new, so we are
looking for good packages and tools that we can try out with Airavata. 

Thanks for the pointer to Apache Storm.  Others?

Marlon

On 1/22/14 2:42 PM, Eran Chinthaka Withana wrote:
> Hi,
>
> The answer to this depends on the amount of workload you will have in the
> queue at a given point of time. If you are talking about hundreds or may be
> thousands of jobs in a given point of time in a queue, may be a solution
> based on proven Storm like architecture would be ideal as it handles all
> sync issues and provides enough parallelism.
>
> But, if this is about handling few jobs at most at a given point of time, I
> would go for an easy thread sync solution. For example, you can easily make
> the job controller (the one who polls the queue and picks new jobs) a
> singleton class but within it, it creates workers to execute or submit
> jobs.
>
> So may be provide a bit more details on the current and future size of the
> problem?
>
> Thanks,
> Eran Chinthaka Withana
>
>
> On Wed, Jan 22, 2014 at 10:53 AM, Sachith Withana <swsachith@gmail.com>wrote:
>
>> As I have mentioned in the Wiki, there are two types of Job Workers. I'll
>> explain the scenario using the newJobWorker.
>>
>> What the NewJobWorker does is it goes to the Orchestrator table and pulls
>> all the jobs which are marked "ACCEPTED" and submits them. But when there
>> are multiple threads running at the same time we need to ensure that the
>> same job doesn't get submitted more than once by different NewJobWorkers.
>> It's the same with the HangedJobWorkers.
>>
>> I started this discussion to get everyone's opinion on how and where we
>> should handle this synchronization issue.
>> We need to do this in a way that, it will improve the performance.
>>
>> I hope I made myself clear.
>>
>>
>> On Wed, Jan 22, 2014 at 1:24 PM, Marlon Pierce <marpierc@iu.edu> wrote:
>>
>>> Hi Sachith, et al--
>>>
>>> Can you say a little more about this?  Did you cover it in your wiki
>>> entry
>>> (https://cwiki.apache.org/confluence/display/AIRAVATA/Orchestrator)?
>>> Will Java's concurrency package not be adequate?
>>>
>>>
>>> Marlon
>>>
>>> On 1/20/14 10:59 AM, Sachith Withana wrote:
>>>> Hi all,
>>>>
>>>> There are race condition issues when using the multi-threaded
>>> Orchestrator.
>>>> We need to come up with a way to synchronize the threads.
>>>>
>>>> Does anyone have any experiences or ideas regarding the issue?
>>>>
>>>>
>>>> On Mon, Jan 20, 2014 at 9:34 AM, Marlon Pierce <marpierc@iu.edu>
>> wrote:
>>>>> This sounds like a good question for the architects@airavata list.
>>>>>
>>>>>
>>>>> Marlon
>>>>>
>>>>> On 1/20/14 2:29 AM, Saminda Wijeratne wrote:
>>>>>> Suresh suggested we use existing libraries. Does anyone have any
>>>>> experience
>>>>>> in using them?
>>>>>>
>>>>>>
>>>>>> On Sun, Jan 19, 2014 at 3:55 PM, Lahiru Gunathilake <
>> glahiru@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Sachitha,
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jan 19, 2014 at 6:41 PM, Sachith Withana <
>> swsachith@gmail.com
>>>>>> wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> There are race condition issues when using the multi-threaded
>>>>>>>> Orchestrator.
>>>>>>>>
>>>>>>> +1, There's another thread we agreed on this. (subject:
>> Orchestration
>>>>>>> Component implementation review).
>>>>>>>
>>>>>>> Lahiru
>>>>>>>
>>>>>>>>  We need to come up with a way to synchronize the threads.
>>>>>>>>
>>>>>>>> Any suggestions?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>> Sachith Withana
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> System Analyst Programmer
>>>>>>> PTI Lab
>>>>>>> Indiana University
>>>>>>>
>>>
>>
>> --
>> Thanks,
>> Sachith Withana
>>




Mime
View raw message