airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marlon Pierce <marpi...@iu.edu>
Subject Re: Implementing a cache registry
Date Mon, 09 Jun 2014 12:59:23 GMT
+1 for solving and +1 for engaging OpenJPA.

Marlon

On 6/8/14 7:35 AM, Lahiru Gunathilake wrote:
> Hi Chathuri,
>
> Great work !!
>
> Regards
> Lahiru
>
>
> On Fri, Jun 6, 2014 at 4:20 PM, Suresh Marru <smarru@apache.org> wrote:
>
>> Very nice, thanks for the detective work Chathuri.
>>
>> Suresh
>>
>> On Jun 6, 2014, at 3:55 PM, Chathuri Wimalasena <kamalasini@gmail.com>
>> wrote:
>>
>>> With the help of OpneJPA community, I was able to solve the memory leak
>> issue. It is due to the way we close the entity manager. After fixing that,
>> we no longer see previous memory leak. Attaching the memory graphs after
>> the fix and before the fix.
>>> Thanks..
>>> Chathuri
>>>
>>>
>>>
>>> On Fri, May 30, 2014 at 3:50 PM, Supun Kamburugamuva <supun06@gmail.com>
>> wrote:
>>> You can use Eclipse memory analyzer for finding the leak.I find it much
>> easier than JProfiler.
>>> http://www.eclipse.org/mat/
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>> On Fri, May 30, 2014 at 3:44 PM, Chathuri Wimalasena <
>> kamalasini@gmail.com> wrote:
>>> I did some performance testing with JConsole. Attaching two images with
>> caching and without. I ran getAllUserExperiments 500 times with 180
>> experiments in the database.  To run 500 iterations with caching is like 5
>> mins but without caching it took more than a hour. In both cases memory and
>> cpu usages are not much difference as I feel. But it seems we have a memory
>> leak somewhere in our code since memory usage is increasing over the time.
>> I will dig more into it.
>>>
>>> On Fri, May 30, 2014 at 9:30 AM, Marlon Pierce <marpierc@iu.edu> wrote:
>>> +1 very nice improvement.
>>> On 5/29/14 2:41 PM, Saminda Wijeratne wrote:
>>>> wow.... thats great news... Could you also monitor memory and CPU usage
>>>> impact? See also if there's any drawbacks that'll affect us for using
>>>> openJPA caching.
>>>>
>>>>
>>>> On Thu, May 29, 2014 at 11:54 AM, Lahiru Gunathilake <
>> glahiru@gmail.com>
>>>> wrote:
>>>>
>>>>> Sounds great Chathuri, I think we can tweak the caching configuration
>> of
>>>>> openJPA and find out the best caching configuration and publish it in
>> wiki.
>>>>> If you can make the caching parameters configurable in
>>>>> airavata-server.properties that would be great.
>>>>>
>>>>> Regards
>>>>> Lahiru
>>>>>
>>>>>
>>>>> On Thu, May 29, 2014 at 2:42 PM, Chathuri Wimalasena <
>> kamalasini@gmail.com
>>>>>> wrote:
>>>>>> I enabled openJPA caching and the results are pretty impressive.
I
>> have
>>>>>> 180 experiments in default derby database. I query
>> getAllUserExperiments
>>>>>> 100 times and average time is around 7000 - 8000 milliseconds without
>>>>>> enabling caching. After enabling caching that number reduced to
>> average of
>>>>>> 400 - 500 milliseconds.
>>>>>>
>>>>>>
>>>>>> On Tue, May 27, 2014 at 4:21 PM, Chathuri Wimalasena <
>>>>>> kamalasini@gmail.com> wrote:
>>>>>>
>>>>>>> We can try openJPA caching [1] and see how it improves the
>> performance.
>>>>>>> This will have minimal changes to the existing code.
>>>>>>>
>>>>>>> [1]
>>>>>>>
>> http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Ftejb_datcacheconfig.html
>>>>>>>
>>>>>>> On Tue, May 27, 2014 at 12:38 PM, Supun Kamburugamuva <
>> supun06@gmail.com
>>>>>>>> wrote:
>>>>>>>> I'm not 100% percent sure on the architecture and the
>> requirements. May
>>>>>>>> be you should look in to Redis or Memcache.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Supun..
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 27, 2014 at 12:35 PM, Lahiru Gunathilake <
>> glahiru@gmail.com
>>>>>>>>> wrote:
>>>>>>>>> I think we can implement this from the scratch and won't
be a hard
>>>>>>>>> thing to do.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 27, 2014 at 12:25 PM, Saminda Wijeratne <
>>>>>>>>> samindaw@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> +1.
>>>>>>>>>>
>>>>>>>>>> Do you know any good frameworks which supports this?
IMO the
>> tricky
>>>>>>>>>> part is when to know that the data in cache has expired.
>> Jackrabbit had a
>>>>>>>>>> feature where it does a callback whenever something
gets updated
>> in some
>>>>>>>>>> tree path. If we restrict all registry access through
a single
>>>>>>>>>> component/layer instance we would be able to do the
same by a
>> subscription
>>>>>>>>>> pattern.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 27, 2014 at 9:04 AM, Lahiru Gunathilake
<
>>>>>>>>>> glahiru@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Devs,
>>>>>>>>>>>
>>>>>>>>>>> In our current implementation we have large number
of Experiment
>>>>>>>>>>> retrieval and experiment storing happen in between
experiment
>> creation and
>>>>>>>>>>> experiment completion. We do not really parse
these data-model
>> objects
>>>>>>>>>>> between component and we simply parse the ids
of these
>> experiment so every
>>>>>>>>>>> component has to retrieve them everytime. I think
>> programatically this
>>>>>>>>>>> approach looks much cleaner than parsing big
objects. But to
>> make this more
>>>>>>>>>>> efficient we can use a cachedRegistry implementation
as another
>>>>>>>>>>> implementation of registry and make sure we do
not get objects
>> all the way
>>>>>>>>>>> from the database.
>>>>>>>>>>>
>>>>>>>>>>> Each component can init its own cache registry
object and it
>> will
>>>>>>>>>>> build a cache on that module and update the cache
if some other
>> component
>>>>>>>>>>> had changed the data-model objects. IMHO if we
implement a good
>> caching
>>>>>>>>>>> layer on our current data-model airavata registry
will be more
>> efficient.
>>>>>>>>>>> WDYT ?
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Lahiru
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> System Analyst Programmer
>>>>>>>>>>> PTI Lab
>>>>>>>>>>> Indiana University
>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> System Analyst Programmer
>>>>>>>>> PTI Lab
>>>>>>>>> Indiana University
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Supun Kamburugamuva
>>>>>>>> Member, Apache Software Foundation; http://www.apache.org
>>>>>>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>>>>>> Blog: http://supunk.blogspot.com
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> System Analyst Programmer
>>>>> PTI Lab
>>>>> Indiana University
>>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>> <after_fix.png><before_fix.png>
>>
>


Mime
View raw message