airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <glah...@gmail.com>
Subject Re: Implementing a cache registry
Date Sun, 08 Jun 2014 11:40:34 GMT
Hi Chathuri,

I think we can write an article about the registry performance and
limitations. we can consult Cipres and other gateways to get the practical
numbers of their systems and explain how Airavata works with those numbers
after running another performance test with those numbers.

WDYT ?

Regards
Lahiru


On Fri, May 30, 2014 at 3:44 PM, Chathuri Wimalasena <kamalasini@gmail.com>
wrote:

> I did some performance testing with JConsole. Attaching two images with
> caching and without. I ran getAllUserExperiments 500 times with 180
> experiments in the database.  To run 500 iterations with caching is like 5
> mins but without caching it took more than a hour. In both cases memory and
> cpu usages are not much difference as I feel. But it seems we have a memory
> leak somewhere in our code since memory usage is increasing over the time.
> I will dig more into it.
>
>
> On Fri, May 30, 2014 at 9:30 AM, Marlon Pierce <marpierc@iu.edu> wrote:
>
>> +1 very nice improvement.
>> On 5/29/14 2:41 PM, Saminda Wijeratne wrote:
>> > wow.... thats great news... Could you also monitor memory and CPU usage
>> > impact? See also if there's any drawbacks that'll affect us for using
>> > openJPA caching.
>> >
>> >
>> > On Thu, May 29, 2014 at 11:54 AM, Lahiru Gunathilake <glahiru@gmail.com
>> >
>> > wrote:
>> >
>> >> Sounds great Chathuri, I think we can tweak the caching configuration
>> of
>> >> openJPA and find out the best caching configuration and publish it in
>> wiki.
>> >>
>> >> If you can make the caching parameters configurable in
>> >> airavata-server.properties that would be great.
>> >>
>> >> Regards
>> >> Lahiru
>> >>
>> >>
>> >> On Thu, May 29, 2014 at 2:42 PM, Chathuri Wimalasena <
>> kamalasini@gmail.com
>> >>> wrote:
>> >>> I enabled openJPA caching and the results are pretty impressive. I
>> have
>> >>> 180 experiments in default derby database. I query
>> getAllUserExperiments
>> >>> 100 times and average time is around 7000 - 8000 milliseconds without
>> >>> enabling caching. After enabling caching that number reduced to
>> average of
>> >>> 400 - 500 milliseconds.
>> >>>
>> >>>
>> >>> On Tue, May 27, 2014 at 4:21 PM, Chathuri Wimalasena <
>> >>> kamalasini@gmail.com> wrote:
>> >>>
>> >>>> We can try openJPA caching [1] and see how it improves the
>> performance.
>> >>>> This will have minimal changes to the existing code.
>> >>>>
>> >>>> [1]
>> >>>>
>> http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Ftejb_datcacheconfig.html
>> >>>>
>> >>>>
>> >>>> On Tue, May 27, 2014 at 12:38 PM, Supun Kamburugamuva <
>> supun06@gmail.com
>> >>>>> wrote:
>> >>>>> I'm not 100% percent sure on the architecture and the requirements.
>> May
>> >>>>> be you should look in to Redis or Memcache.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Supun..
>> >>>>>
>> >>>>>
>> >>>>> On Tue, May 27, 2014 at 12:35 PM, Lahiru Gunathilake <
>> glahiru@gmail.com
>> >>>>>> wrote:
>> >>>>>> I think we can implement this from the scratch and won't
be a hard
>> >>>>>> thing to do.
>> >>>>>>
>> >>>>>>
>> >>>>>> On Tue, May 27, 2014 at 12:25 PM, Saminda Wijeratne <
>> >>>>>> samindaw@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> +1.
>> >>>>>>>
>> >>>>>>> Do you know any good frameworks which supports this?
IMO the
>> tricky
>> >>>>>>> part is when to know that the data in cache has expired.
>> Jackrabbit had a
>> >>>>>>> feature where it does a callback whenever something
gets updated
>> in some
>> >>>>>>> tree path. If we restrict all registry access through
a single
>> >>>>>>> component/layer instance we would be able to do the
same by a
>> subscription
>> >>>>>>> pattern.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, May 27, 2014 at 9:04 AM, Lahiru Gunathilake
<
>> >>>>>>> glahiru@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Devs,
>> >>>>>>>>
>> >>>>>>>> In our current implementation we have large number
of Experiment
>> >>>>>>>> retrieval and experiment storing happen in between
experiment
>> creation and
>> >>>>>>>> experiment completion. We do not really parse these
data-model
>> objects
>> >>>>>>>> between component and we simply parse the ids of
these
>> experiment so every
>> >>>>>>>> component has to retrieve them everytime. I think
>> programatically this
>> >>>>>>>> approach looks much cleaner than parsing big objects.
But to
>> make this more
>> >>>>>>>> efficient we can use a cachedRegistry implementation
as another
>> >>>>>>>> implementation of registry and make sure we do not
get objects
>> all the way
>> >>>>>>>> from the database.
>> >>>>>>>>
>> >>>>>>>> Each component can init its own cache registry object
and it will
>> >>>>>>>> build a cache on that module and update the cache
if some other
>> component
>> >>>>>>>> had changed the data-model objects. IMHO if we implement
a good
>> caching
>> >>>>>>>> layer on our current data-model airavata registry
will be more
>> efficient.
>> >>>>>>>>
>> >>>>>>>> WDYT ?
>> >>>>>>>>
>> >>>>>>>> Regards
>> >>>>>>>> Lahiru
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> System Analyst Programmer
>> >>>>>>>> PTI Lab
>> >>>>>>>> Indiana University
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> System Analyst Programmer
>> >>>>>> PTI Lab
>> >>>>>> Indiana University
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Supun Kamburugamuva
>> >>>>> Member, Apache Software Foundation; http://www.apache.org
>> >>>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> >>>>> Blog: http://supunk.blogspot.com
>> >>>>>
>> >>>>>
>> >>
>> >> --
>> >> System Analyst Programmer
>> >> PTI Lab
>> >> Indiana University
>> >>
>>
>>
>


-- 
System Analyst Programmer
PTI Lab
Indiana University

Mime
View raw message