airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Implementing a cache registry
Date Fri, 06 Jun 2014 20:20:35 GMT
Very nice, thanks for the detective work Chathuri.

Suresh

On Jun 6, 2014, at 3:55 PM, Chathuri Wimalasena <kamalasini@gmail.com> wrote:

> With the help of OpneJPA community, I was able to solve the memory leak issue. It is
due to the way we close the entity manager. After fixing that, we no longer see previous memory
leak. Attaching the memory graphs after the fix and before the fix.
> 
> Thanks..
> Chathuri
> 
> 
> 
> On Fri, May 30, 2014 at 3:50 PM, Supun Kamburugamuva <supun06@gmail.com> wrote:
> You can use Eclipse memory analyzer for finding the leak.I find it much easier than JProfiler.
> 
> http://www.eclipse.org/mat/
> 
> Thanks,
> Supun..
> 
> 
> On Fri, May 30, 2014 at 3:44 PM, Chathuri Wimalasena <kamalasini@gmail.com> wrote:
> I did some performance testing with JConsole. Attaching two images with caching and without.
I ran getAllUserExperiments 500 times with 180 experiments in the database.  To run 500 iterations
with caching is like 5 mins but without caching it took more than a hour. In both cases memory
and cpu usages are not much difference as I feel. But it seems we have a memory leak somewhere
in our code since memory usage is increasing over the time. I will dig more into it. 
> 
> 
> On Fri, May 30, 2014 at 9:30 AM, Marlon Pierce <marpierc@iu.edu> wrote:
> +1 very nice improvement.
> On 5/29/14 2:41 PM, Saminda Wijeratne wrote:
> > wow.... thats great news... Could you also monitor memory and CPU usage
> > impact? See also if there's any drawbacks that'll affect us for using
> > openJPA caching.
> >
> >
> > On Thu, May 29, 2014 at 11:54 AM, Lahiru Gunathilake <glahiru@gmail.com>
> > wrote:
> >
> >> Sounds great Chathuri, I think we can tweak the caching configuration of
> >> openJPA and find out the best caching configuration and publish it in wiki.
> >>
> >> If you can make the caching parameters configurable in
> >> airavata-server.properties that would be great.
> >>
> >> Regards
> >> Lahiru
> >>
> >>
> >> On Thu, May 29, 2014 at 2:42 PM, Chathuri Wimalasena <kamalasini@gmail.com
> >>> wrote:
> >>> I enabled openJPA caching and the results are pretty impressive. I have
> >>> 180 experiments in default derby database. I query getAllUserExperiments
> >>> 100 times and average time is around 7000 - 8000 milliseconds without
> >>> enabling caching. After enabling caching that number reduced to average
of
> >>> 400 - 500 milliseconds.
> >>>
> >>>
> >>> On Tue, May 27, 2014 at 4:21 PM, Chathuri Wimalasena <
> >>> kamalasini@gmail.com> wrote:
> >>>
> >>>> We can try openJPA caching [1] and see how it improves the performance.
> >>>> This will have minimal changes to the existing code.
> >>>>
> >>>> [1]
> >>>> http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Ftejb_datcacheconfig.html
> >>>>
> >>>>
> >>>> On Tue, May 27, 2014 at 12:38 PM, Supun Kamburugamuva <supun06@gmail.com
> >>>>> wrote:
> >>>>> I'm not 100% percent sure on the architecture and the requirements.
May
> >>>>> be you should look in to Redis or Memcache.
> >>>>>
> >>>>> Thanks,
> >>>>> Supun..
> >>>>>
> >>>>>
> >>>>> On Tue, May 27, 2014 at 12:35 PM, Lahiru Gunathilake <glahiru@gmail.com
> >>>>>> wrote:
> >>>>>> I think we can implement this from the scratch and won't be
a hard
> >>>>>> thing to do.
> >>>>>>
> >>>>>>
> >>>>>> On Tue, May 27, 2014 at 12:25 PM, Saminda Wijeratne <
> >>>>>> samindaw@gmail.com> wrote:
> >>>>>>
> >>>>>>> +1.
> >>>>>>>
> >>>>>>> Do you know any good frameworks which supports this? IMO
the tricky
> >>>>>>> part is when to know that the data in cache has expired.
Jackrabbit had a
> >>>>>>> feature where it does a callback whenever something gets
updated in some
> >>>>>>> tree path. If we restrict all registry access through a
single
> >>>>>>> component/layer instance we would be able to do the same
by a subscription
> >>>>>>> pattern.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, May 27, 2014 at 9:04 AM, Lahiru Gunathilake <
> >>>>>>> glahiru@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hi Devs,
> >>>>>>>>
> >>>>>>>> In our current implementation we have large number of
Experiment
> >>>>>>>> retrieval and experiment storing happen in between experiment
creation and
> >>>>>>>> experiment completion. We do not really parse these
data-model objects
> >>>>>>>> between component and we simply parse the ids of these
experiment so every
> >>>>>>>> component has to retrieve them everytime. I think programatically
this
> >>>>>>>> approach looks much cleaner than parsing big objects.
But to make this more
> >>>>>>>> efficient we can use a cachedRegistry implementation
as another
> >>>>>>>> implementation of registry and make sure we do not get
objects all the way
> >>>>>>>> from the database.
> >>>>>>>>
> >>>>>>>> Each component can init its own cache registry object
and it will
> >>>>>>>> build a cache on that module and update the cache if
some other component
> >>>>>>>> had changed the data-model objects. IMHO if we implement
a good caching
> >>>>>>>> layer on our current data-model airavata registry will
be more efficient.
> >>>>>>>>
> >>>>>>>> WDYT ?
> >>>>>>>>
> >>>>>>>> Regards
> >>>>>>>> Lahiru
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> System Analyst Programmer
> >>>>>>>> PTI Lab
> >>>>>>>> Indiana University
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> System Analyst Programmer
> >>>>>> PTI Lab
> >>>>>> Indiana University
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Supun Kamburugamuva
> >>>>> Member, Apache Software Foundation; http://www.apache.org
> >>>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>>>> Blog: http://supunk.blogspot.com
> >>>>>
> >>>>>
> >>
> >> --
> >> System Analyst Programmer
> >> PTI Lab
> >> Indiana University
> >>
> 
> 
> 
> 
> 
> -- 
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
> 
> 
> <after_fix.png><before_fix.png>


Mime
View raw message