airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chathuri Wimalasena <kamalas...@gmail.com>
Subject Re: Implementing a cache registry
Date Fri, 06 Jun 2014 19:55:12 GMT
With the help of OpneJPA community, I was able to solve the memory leak
issue. It is due to the way we close the entity manager. After fixing that,
we no longer see previous memory leak. Attaching the memory graphs after
the fix and before the fix.

Thanks..
Chathuri



On Fri, May 30, 2014 at 3:50 PM, Supun Kamburugamuva <supun06@gmail.com>
wrote:

> You can use Eclipse memory analyzer for finding the leak.I find it much
> easier than JProfiler.
>
> http://www.eclipse.org/mat/
>
> Thanks,
> Supun..
>
>
> On Fri, May 30, 2014 at 3:44 PM, Chathuri Wimalasena <kamalasini@gmail.com
> > wrote:
>
>> I did some performance testing with JConsole. Attaching two images with
>> caching and without. I ran getAllUserExperiments 500 times with 180
>> experiments in the database.  To run 500 iterations with caching is like 5
>> mins but without caching it took more than a hour. In both cases memory and
>> cpu usages are not much difference as I feel. But it seems we have a memory
>> leak somewhere in our code since memory usage is increasing over the time.
>> I will dig more into it.
>>
>>
>> On Fri, May 30, 2014 at 9:30 AM, Marlon Pierce <marpierc@iu.edu> wrote:
>>
>>> +1 very nice improvement.
>>> On 5/29/14 2:41 PM, Saminda Wijeratne wrote:
>>> > wow.... thats great news... Could you also monitor memory and CPU usage
>>> > impact? See also if there's any drawbacks that'll affect us for using
>>> > openJPA caching.
>>> >
>>> >
>>> > On Thu, May 29, 2014 at 11:54 AM, Lahiru Gunathilake <
>>> glahiru@gmail.com>
>>> > wrote:
>>> >
>>> >> Sounds great Chathuri, I think we can tweak the caching configuration
>>> of
>>> >> openJPA and find out the best caching configuration and publish it in
>>> wiki.
>>> >>
>>> >> If you can make the caching parameters configurable in
>>> >> airavata-server.properties that would be great.
>>> >>
>>> >> Regards
>>> >> Lahiru
>>> >>
>>> >>
>>> >> On Thu, May 29, 2014 at 2:42 PM, Chathuri Wimalasena <
>>> kamalasini@gmail.com
>>> >>> wrote:
>>> >>> I enabled openJPA caching and the results are pretty impressive.
I
>>> have
>>> >>> 180 experiments in default derby database. I query
>>> getAllUserExperiments
>>> >>> 100 times and average time is around 7000 - 8000 milliseconds without
>>> >>> enabling caching. After enabling caching that number reduced to
>>> average of
>>> >>> 400 - 500 milliseconds.
>>> >>>
>>> >>>
>>> >>> On Tue, May 27, 2014 at 4:21 PM, Chathuri Wimalasena <
>>> >>> kamalasini@gmail.com> wrote:
>>> >>>
>>> >>>> We can try openJPA caching [1] and see how it improves the
>>> performance.
>>> >>>> This will have minimal changes to the existing code.
>>> >>>>
>>> >>>> [1]
>>> >>>>
>>> http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Ftejb_datcacheconfig.html
>>> >>>>
>>> >>>>
>>> >>>> On Tue, May 27, 2014 at 12:38 PM, Supun Kamburugamuva <
>>> supun06@gmail.com
>>> >>>>> wrote:
>>> >>>>> I'm not 100% percent sure on the architecture and the
>>> requirements. May
>>> >>>>> be you should look in to Redis or Memcache.
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Supun..
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, May 27, 2014 at 12:35 PM, Lahiru Gunathilake <
>>> glahiru@gmail.com
>>> >>>>>> wrote:
>>> >>>>>> I think we can implement this from the scratch and won't
be a hard
>>> >>>>>> thing to do.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Tue, May 27, 2014 at 12:25 PM, Saminda Wijeratne
<
>>> >>>>>> samindaw@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>>> +1.
>>> >>>>>>>
>>> >>>>>>> Do you know any good frameworks which supports this?
IMO the
>>> tricky
>>> >>>>>>> part is when to know that the data in cache has
expired.
>>> Jackrabbit had a
>>> >>>>>>> feature where it does a callback whenever something
gets updated
>>> in some
>>> >>>>>>> tree path. If we restrict all registry access through
a single
>>> >>>>>>> component/layer instance we would be able to do
the same by a
>>> subscription
>>> >>>>>>> pattern.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Tue, May 27, 2014 at 9:04 AM, Lahiru Gunathilake
<
>>> >>>>>>> glahiru@gmail.com> wrote:
>>> >>>>>>>
>>> >>>>>>>> Hi Devs,
>>> >>>>>>>>
>>> >>>>>>>> In our current implementation we have large
number of Experiment
>>> >>>>>>>> retrieval and experiment storing happen in between
experiment
>>> creation and
>>> >>>>>>>> experiment completion. We do not really parse
these data-model
>>> objects
>>> >>>>>>>> between component and we simply parse the ids
of these
>>> experiment so every
>>> >>>>>>>> component has to retrieve them everytime. I
think
>>> programatically this
>>> >>>>>>>> approach looks much cleaner than parsing big
objects. But to
>>> make this more
>>> >>>>>>>> efficient we can use a cachedRegistry implementation
as another
>>> >>>>>>>> implementation of registry and make sure we
do not get objects
>>> all the way
>>> >>>>>>>> from the database.
>>> >>>>>>>>
>>> >>>>>>>> Each component can init its own cache registry
object and it
>>> will
>>> >>>>>>>> build a cache on that module and update the
cache if some other
>>> component
>>> >>>>>>>> had changed the data-model objects. IMHO if
we implement a good
>>> caching
>>> >>>>>>>> layer on our current data-model airavata registry
will be more
>>> efficient.
>>> >>>>>>>>
>>> >>>>>>>> WDYT ?
>>> >>>>>>>>
>>> >>>>>>>> Regards
>>> >>>>>>>> Lahiru
>>> >>>>>>>>
>>> >>>>>>>> --
>>> >>>>>>>> System Analyst Programmer
>>> >>>>>>>> PTI Lab
>>> >>>>>>>> Indiana University
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> System Analyst Programmer
>>> >>>>>> PTI Lab
>>> >>>>>> Indiana University
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Supun Kamburugamuva
>>> >>>>> Member, Apache Software Foundation; http://www.apache.org
>>> >>>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> >>>>> Blog: http://supunk.blogspot.com
>>> >>>>>
>>> >>>>>
>>> >>
>>> >> --
>>> >> System Analyst Programmer
>>> >> PTI Lab
>>> >> Indiana University
>>> >>
>>>
>>>
>>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>

Mime
View raw message