incubator-esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Kohler <markus.koh...@gmail.com>
Subject Re: Further analysis of the GC issue
Date Wed, 25 Nov 2009 18:10:43 GMT
BTW, you don't need 10 user , 2 or 3 would be enough.
You just have to take care that the session does not expire.
The issue disappears if you wait long enough, I verified that already.


Regards,
Markus

"The best way to predict the future is to invent it" -- Alan Kay


On Wed, Nov 25, 2009 at 6:35 PM, David Pollak <feeder.of.the.bears@gmail.com
> wrote:

> Markus,
>
> I'll have to look into how the code is currently implemented.  The original
> design called for a message to hang out for quite a while (not be reloaded
> from RDBMS for each user) and lazily render different pieces of itself.
>  Put
> another way, for each message, no matter where it's viewed in the system,
> there should be a single, cached instance of the message that contains the
> rendered Textile markup, etc.
>
> Put another way, a message that appears in 300 timelines should be the same
> message instance and should have its complex values calculated once.  Sure,
> if you go back and look at a message that's a week old, it'll have fallen
> out of memory and need to be re-rendered, but in the normal case, a message
> that's viewed by 300 people as part of their timeline or the public
> timeline
> should only be materialized/rendered once.
>
> Can you point me to a place (maybe even a VM Ware instance) where I can
> reproduce your tests?  I'd love to cycle on making 23GB -> 100MB (1/300th
> the size).
>
> Thanks,
>
> David
>
> On Wed, Nov 25, 2009 at 9:14 AM, Markus Kohler <markus.kohler@gmail.com
> >wrote:
>
> > Hi all,
> > the Garbage Collector issue I was talking about is reproducible.
> > I've uploaded an annotated GC graph to
> >
> >
> http://picasaweb.google.com/lh/photo/wB-RRtb0wIVfpxJkTJPNuw?authkey=Gv1sRgCOve7LThpfvXsQE&feat=directlink
> >
> > I think the "LOGON" phase where I logon all the 300 users looks ok (given
> > that probably textile formatting is involved) but the phase where just
> one
> > user sends one message is certainly not looking good.
> >
> > I took the profiler and the result is a bit shocking. For that one
> message,
> > 881.000.000 objects weighting  23,2 Gbyte where allocated (and reclaimed
> > afterwards). My former record was 2Gbyte ;-)
> >
> > Fortunately I have a theory what happens, without looking into the
> > code,yet,
> > so take it with a grain of salt. It seems that the public time line for
> all
> > users is re-rendered, because 99% of the allocations come
> > from org.apache.esme.comet.PublicTimeline.render(). I guess all the
> actors
> > for all the users are sitting there, not knowing that the user has closed
> > the browser, because the user session has not yet expired.
> >
> > I wonder how we get around this with a real "push" model. If the browser
> > would ask for updates this rendering could be done lazily. Or can we
> "ping"
> > the browser and check whether it responds?
> > On the other side. It should also not be necessary the re-render the
> > message
> > again and again because the result will be the same.
> >
> > I will send Richard some attachments. Not sure whether you will need
> them,
> > they look very similar to the ones we already have.
> >
> > BTW, we should definitely check the use
> > of scala.xml.XML$.loadString(java.lang.String)
> > It's creating a new Parser each time, which is a bit costly because it
> > allocates a new Buffer each time and also hits the disk, when searching
> for
> > the name of the Java class.
> >
> > Greetings,
> > Markus
> >
> >
> >
> > "The best way to predict the future is to invent it" -- Alan Kay
> >
>
>
>
> --
> Lift, the simply functional web framework http://liftweb.net
> Beginning Scala http://www.apress.com/book/view/1430219890
> Follow me: http://twitter.com/dpp
> Surf the harmonics
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message