airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Supun Kamburugamuve <supu...@gmail.com>
Subject Re: Airavata Registry Considerations
Date Fri, 22 May 2015 16:12:28 GMT
Hi Supun,

In normal software developments, it is normal to have these kind of
slowness. We cannot foresee all the things when we develop. The solution is
to improve the performance of important operations rather than re-writing
everything from the beginning. For example for this particular select
operations you can directly use SQL rather than going through JPA.

I'm pretty sure you'll encounter more problems, if you implement this in
MongoDB than in the current MySQL. If that happens, do you think abandoning
that technology and going for a new database will be a good solution? Now
you have more experience with MySQL than MongoDB as well.

Rather than going to abandon everything you have because of one problem,
trying to fix it may be better for you in the long run.

Thanks,
Supun..

On Fri, May 22, 2015 at 11:49 AM, Supun Nakandala <supun.nakandala@gmail.com
> wrote:

> Hi Supun,
>
> I haven't done done profiling of registry based operations. Here what I
> mean by slow performance is mainly the slowness of the SELECT operations in
> PHP Reference Gateway. e.g fetching Projects, fetching experiments. Even a
> simple query to fetch the 20 most recent experiments is embarrassingly slow
> in PGA.
>
> Even though I didn't do a proper profiling of operations I did a query log
> analysis for a SELECT experiment query. This was a simple query to fetch 20
> most recent experiments. I found that JPA layer is generating enormous
> amount of queries for this task rather than one single query (due to the
> select N+1 isssue). This issue is same for fetching a single experiment by
> specifying the id.
>
> I think it is ok to say that current registry has become bottleneck for
> most of the PGA specific operations. But I don't have evidence to show how
> it has become a bottleneck for the Orchestrator or GFac specific
> operations. For that as you have mentioned we need to profile the
> operations. But I think the argument is still valid even for GFac and
> Orchestrator based operations.
>
> I have attached the query log for the above mentioned select operation
> here with. If you observe the query log you can see that every associated
> entity is fetched separately using complex join operations.
>
>
>
> On Fri, May 22, 2015 at 8:05 PM, Supun Kamburugamuve <supun06@gmail.com>
> wrote:
>
>> Hi Supun,
>>
>> In your report it says Slow performance. Do you have any data about this
>> slow performance? For a typical request in what percent the registry slow
>> down the processing compared to overall time it takes to execute that
>> request?
>>
>> Do you have a use case where registry is the bottleneck?
>>
>> Thanks,
>> Supun..
>>
>> On Fri, May 22, 2015 at 9:45 AM, Suresh Marru <smarru@apache.org> wrote:
>>
>>> Hi Supun,
>>>
>>> This is very good analysis, you have nicely embraced the problem. Before
>>> we jump into the solution, we may want to do small POC’s to validate your
>>> claims.
>>>
>>> Thank you for getting a headstart, this also cuts into GSoC goals of
>>> Douglas’s project. So lets work on this collaboratively.
>>>
>>> Hi Madhu,
>>>
>>> Can you please provide guidance on this effort on how to academically
>>> approach the data management challenges of Airavata. The students might
>>> appreciate insights on how to profile and benchmark any possible solutions.
>>>
>>> Cheers,
>>> Suresh
>>>
>>> On May 22, 2015, at 9:18 AM, Supun Nakandala <supun.nakandala@gmail.com>
>>> wrote:
>>>
>>> Hi Devs,
>>>
>>> I have compiled a document based on the analysis I did on current
>>> registry architecture/technology and possible modification and
>>> alternatives. You can find the document at
>>> https://docs.google.com/document/d/1XWAQLtdtCf9nTigAz6r5JINHR99bP0oeYaTgeEIVr4w/edit#
>>>
>>> Thanks
>>> Supun
>>>
>>>
>>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Thank you
> Supun Nakandala
> Dept. Computer Science and Engineering
> University of Moratuwa
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Mime
View raw message