incubator-esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Pollak <feeder.of.the.be...@gmail.com>
Subject Re: Further analysis of the GC issue
Date Wed, 25 Nov 2009 18:51:06 GMT
On Wed, Nov 25, 2009 at 9:58 AM, Markus Kohler <markus.kohler@gmail.com>wrote:

> Hi David,
> see my comments below...
>
>
>
> "The best way to predict the future is to invent it" -- Alan Kay
>
>
> On Wed, Nov 25, 2009 at 6:35 PM, David Pollak <
> feeder.of.the.bears@gmail.com
> > wrote:
>
> > Markus,
> >
> > I'll have to look into how the code is currently implemented.  The
> original
> > design called for a message to hang out for quite a while (not be
> reloaded
> > from RDBMS for each user) and lazily render different pieces of itself.
> >  Put
> > another way, for each message, no matter where it's viewed in the system,
> > there should be a single, cached instance of the message that contains
> the
> > rendered Textile markup, etc.
> >
> > Put another way, a message that appears in 300 timelines should be the
> same
> > message instance and should have its complex values calculated once.
>  Sure,
> > if you go back and look at a message that's a week old, it'll have fallen
> > out of memory and need to be re-rendered, but in the normal case, a
> message
> > that's viewed by 300 people as part of their timeline or the public
> > timeline
> > should only be materialized/rendered once.
> >
> > Yes, that's how it should be done.
>
> Can you point me to a place (maybe even a VM Ware instance) where I can
> > reproduce your tests?  I'd love to cycle on making 23GB -> 100MB (1/300th
> > the size).
> >
> > I see 2 options (I guess you need to debug):
>
> 1. You doit manually.
> I think it would be enough if you have say 10 users. logon a user with the
> browser, clear the browser cache (no logout) and then logon the next user
> etc.
> Then send a message with the last user.
>
> 2. I pack the selenium test into one fat jar file and you run it locally on
> your machine. I could also provide you my esme_db directory with the 300
> users, not sure whether it would work.
>

Having the selenium tests would be great.

I can use YourKit to analyze the memory usage, but I'm not likely to get the
nifty graphs you've been showing us.


>
> Which option do you prefer?
>
> Regards,
> Markus
>
>
> > Thanks,
> >
> > David
> >
> > On Wed, Nov 25, 2009 at 9:14 AM, Markus Kohler <markus.kohler@gmail.com
> > >wrote:
> >
> > > Hi all,
> > > the Garbage Collector issue I was talking about is reproducible.
> > > I've uploaded an annotated GC graph to
> > >
> > >
> >
> http://picasaweb.google.com/lh/photo/wB-RRtb0wIVfpxJkTJPNuw?authkey=Gv1sRgCOve7LThpfvXsQE&feat=directlink
> > >
> > > I think the "LOGON" phase where I logon all the 300 users looks ok
> (given
> > > that probably textile formatting is involved) but the phase where just
> > one
> > > user sends one message is certainly not looking good.
> > >
> > > I took the profiler and the result is a bit shocking. For that one
> > message,
> > > 881.000.000 objects weighting  23,2 Gbyte where allocated (and
> reclaimed
> > > afterwards). My former record was 2Gbyte ;-)
> > >
> > > Fortunately I have a theory what happens, without looking into the
> > > code,yet,
> > > so take it with a grain of salt. It seems that the public time line for
> > all
> > > users is re-rendered, because 99% of the allocations come
> > > from org.apache.esme.comet.PublicTimeline.render(). I guess all the
> > actors
> > > for all the users are sitting there, not knowing that the user has
> closed
> > > the browser, because the user session has not yet expired.
> > >
> > > I wonder how we get around this with a real "push" model. If the
> browser
> > > would ask for updates this rendering could be done lazily. Or can we
> > "ping"
> > > the browser and check whether it responds?
> > > On the other side. It should also not be necessary the re-render the
> > > message
> > > again and again because the result will be the same.
> > >
> > > I will send Richard some attachments. Not sure whether you will need
> > them,
> > > they look very similar to the ones we already have.
> > >
> > > BTW, we should definitely check the use
> > > of scala.xml.XML$.loadString(java.lang.String)
> > > It's creating a new Parser each time, which is a bit costly because it
> > > allocates a new Buffer each time and also hits the disk, when searching
> > for
> > > the name of the Java class.
> > >
> > > Greetings,
> > > Markus
> > >
> > >
> > >
> > > "The best way to predict the future is to invent it" -- Alan Kay
> > >
> >
> >
> >
> > --
> > Lift, the simply functional web framework http://liftweb.net
> > Beginning Scala http://www.apress.com/book/view/1430219890
> > Follow me: http://twitter.com/dpp
> > Surf the harmonics
> >
>



-- 
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message