airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <glah...@gmail.com>
Subject Re: Implementing a cache registry
Date Sun, 08 Jun 2014 11:35:13 GMT
Hi Chathuri,

Great work !!

Regards
Lahiru


On Fri, Jun 6, 2014 at 4:20 PM, Suresh Marru <smarru@apache.org> wrote:

> Very nice, thanks for the detective work Chathuri.
>
> Suresh
>
> On Jun 6, 2014, at 3:55 PM, Chathuri Wimalasena <kamalasini@gmail.com>
> wrote:
>
> > With the help of OpneJPA community, I was able to solve the memory leak
> issue. It is due to the way we close the entity manager. After fixing that,
> we no longer see previous memory leak. Attaching the memory graphs after
> the fix and before the fix.
> >
> > Thanks..
> > Chathuri
> >
> >
> >
> > On Fri, May 30, 2014 at 3:50 PM, Supun Kamburugamuva <supun06@gmail.com>
> wrote:
> > You can use Eclipse memory analyzer for finding the leak.I find it much
> easier than JProfiler.
> >
> > http://www.eclipse.org/mat/
> >
> > Thanks,
> > Supun..
> >
> >
> > On Fri, May 30, 2014 at 3:44 PM, Chathuri Wimalasena <
> kamalasini@gmail.com> wrote:
> > I did some performance testing with JConsole. Attaching two images with
> caching and without. I ran getAllUserExperiments 500 times with 180
> experiments in the database.  To run 500 iterations with caching is like 5
> mins but without caching it took more than a hour. In both cases memory and
> cpu usages are not much difference as I feel. But it seems we have a memory
> leak somewhere in our code since memory usage is increasing over the time.
> I will dig more into it.
> >
> >
> > On Fri, May 30, 2014 at 9:30 AM, Marlon Pierce <marpierc@iu.edu> wrote:
> > +1 very nice improvement.
> > On 5/29/14 2:41 PM, Saminda Wijeratne wrote:
> > > wow.... thats great news... Could you also monitor memory and CPU usage
> > > impact? See also if there's any drawbacks that'll affect us for using
> > > openJPA caching.
> > >
> > >
> > > On Thu, May 29, 2014 at 11:54 AM, Lahiru Gunathilake <
> glahiru@gmail.com>
> > > wrote:
> > >
> > >> Sounds great Chathuri, I think we can tweak the caching configuration
> of
> > >> openJPA and find out the best caching configuration and publish it in
> wiki.
> > >>
> > >> If you can make the caching parameters configurable in
> > >> airavata-server.properties that would be great.
> > >>
> > >> Regards
> > >> Lahiru
> > >>
> > >>
> > >> On Thu, May 29, 2014 at 2:42 PM, Chathuri Wimalasena <
> kamalasini@gmail.com
> > >>> wrote:
> > >>> I enabled openJPA caching and the results are pretty impressive. I
> have
> > >>> 180 experiments in default derby database. I query
> getAllUserExperiments
> > >>> 100 times and average time is around 7000 - 8000 milliseconds without
> > >>> enabling caching. After enabling caching that number reduced to
> average of
> > >>> 400 - 500 milliseconds.
> > >>>
> > >>>
> > >>> On Tue, May 27, 2014 at 4:21 PM, Chathuri Wimalasena <
> > >>> kamalasini@gmail.com> wrote:
> > >>>
> > >>>> We can try openJPA caching [1] and see how it improves the
> performance.
> > >>>> This will have minimal changes to the existing code.
> > >>>>
> > >>>> [1]
> > >>>>
> http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Ftejb_datcacheconfig.html
> > >>>>
> > >>>>
> > >>>> On Tue, May 27, 2014 at 12:38 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >>>>> wrote:
> > >>>>> I'm not 100% percent sure on the architecture and the
> requirements. May
> > >>>>> be you should look in to Redis or Memcache.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Supun..
> > >>>>>
> > >>>>>
> > >>>>> On Tue, May 27, 2014 at 12:35 PM, Lahiru Gunathilake <
> glahiru@gmail.com
> > >>>>>> wrote:
> > >>>>>> I think we can implement this from the scratch and won't
be a hard
> > >>>>>> thing to do.
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, May 27, 2014 at 12:25 PM, Saminda Wijeratne <
> > >>>>>> samindaw@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> +1.
> > >>>>>>>
> > >>>>>>> Do you know any good frameworks which supports this?
IMO the
> tricky
> > >>>>>>> part is when to know that the data in cache has expired.
> Jackrabbit had a
> > >>>>>>> feature where it does a callback whenever something
gets updated
> in some
> > >>>>>>> tree path. If we restrict all registry access through
a single
> > >>>>>>> component/layer instance we would be able to do the
same by a
> subscription
> > >>>>>>> pattern.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Tue, May 27, 2014 at 9:04 AM, Lahiru Gunathilake
<
> > >>>>>>> glahiru@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Devs,
> > >>>>>>>>
> > >>>>>>>> In our current implementation we have large number
of Experiment
> > >>>>>>>> retrieval and experiment storing happen in between
experiment
> creation and
> > >>>>>>>> experiment completion. We do not really parse these
data-model
> objects
> > >>>>>>>> between component and we simply parse the ids of
these
> experiment so every
> > >>>>>>>> component has to retrieve them everytime. I think
> programatically this
> > >>>>>>>> approach looks much cleaner than parsing big objects.
But to
> make this more
> > >>>>>>>> efficient we can use a cachedRegistry implementation
as another
> > >>>>>>>> implementation of registry and make sure we do
not get objects
> all the way
> > >>>>>>>> from the database.
> > >>>>>>>>
> > >>>>>>>> Each component can init its own cache registry
object and it
> will
> > >>>>>>>> build a cache on that module and update the cache
if some other
> component
> > >>>>>>>> had changed the data-model objects. IMHO if we
implement a good
> caching
> > >>>>>>>> layer on our current data-model airavata registry
will be more
> efficient.
> > >>>>>>>>
> > >>>>>>>> WDYT ?
> > >>>>>>>>
> > >>>>>>>> Regards
> > >>>>>>>> Lahiru
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> System Analyst Programmer
> > >>>>>>>> PTI Lab
> > >>>>>>>> Indiana University
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> System Analyst Programmer
> > >>>>>> PTI Lab
> > >>>>>> Indiana University
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Supun Kamburugamuva
> > >>>>> Member, Apache Software Foundation; http://www.apache.org
> > >>>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > >>>>> Blog: http://supunk.blogspot.com
> > >>>>>
> > >>>>>
> > >>
> > >> --
> > >> System Analyst Programmer
> > >> PTI Lab
> > >> Indiana University
> > >>
> >
> >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
> > <after_fix.png><before_fix.png>
>
>


-- 
System Analyst Programmer
PTI Lab
Indiana University

Mime
View raw message