incubator-esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Hirsch <hirsch.d...@gmail.com>
Subject Re: Further analysis of the GC issue
Date Sat, 28 Nov 2009 09:19:22 GMT
For some reason the wiki page about the performance test on 11-25 was
lost, I'll have to create once again.....

On Fri, Nov 27, 2009 at 5:47 AM, Richard Hirsch <hirsch.dick@gmail.com> wrote:
> Moved this whole thread to the wiki:
> http://cwiki.apache.org/confluence/display/ESME/Performance+test+2009-11-25
>
> D.
>
> On Thu, Nov 26, 2009 at 2:22 PM, Markus Kohler <markus.kohler@gmail.com> wrote:
>> Hi Michael,
>> No problem :-)
>>
>>
>>
>> Regards,
>> Markus
>>
>> "The best way to predict the future is to invent it" -- Alan Kay
>>
>>
>> On Thu, Nov 26, 2009 at 2:12 PM, Bechauf, Michael
>> <michael.bechauf@sap.com>wrote:
>>
>>> Thanks Markus. That certainly sounds much better. I was confused
>>> yesterday already because 23 GByte memory would be a little difficult to
>>> create when not even the operating system can handle such size. I should
>>> have asked right away. Blame it on jetlag.
>>>
>>> -Michael
>>>
>>> -----Original Message-----
>>> From: Markus Kohler [mailto:markus.kohler@gmail.com]
>>> Sent: Thursday, Nov 26, 2009 1:04 AM
>>> To: esme-dev@incubator.apache.org
>>> Subject: Re: Further analysis of the GC issue
>>>
>>> Hi Michael,
>>> Good to see you here!
>>>
>>> "Memory Analyzer"? that's me ;-)
>>>
>>> The 23 Gbyte are not "retained" at one point in time, but they are the
>>> sum
>>> of all temporary allocated objects, most of memory, (or all of it, there
>>> doesn't seem to be an obvious memory leak), are gone within a
>>> millisecond.
>>> I'm confident that this value can be decreased to 90Mbyte and can be
>>> further
>>> improved down to a few MByte (or even less). We already know that the
>>> 90Mbyte are mostly caused be an inefficient textile parser.
>>>
>>> I also used the Memory Analyzer to look at how much memory is retained,
>>> e.g.
>>> still in use/referenced after the user interaction has been finished.
>>> The
>>> report is here
>>> http://cwiki.apache.org/confluence/display/ESME/Performance+test+-+2009-
>>> 11-22
>>> Also there's room for improvement, potentially caused by the same bug
>>> that
>>> turned 90Mbyte into 23Gbyte, I don't see any major issues yet with
>>> regards
>>> to memory usage.
>>>
>>> This is also related to the state less versus state full discussion, ATM
>>> the
>>> amount of state needed for one user is already very low ( a few hundred
>>> kByte), at least compared to what I'm used to with Enterprise
>>> Applications.
>>> It is at least an order of magnitude lower, which can only partially
>>> explained by ESME being less complex than the typical Enterprise app.
>>> So far I don't see any major road block from the design perspective that
>>> would stop us from scaling very well.
>>>
>>> In my experience, it's quite normal that as soon as someone with a
>>> little
>>> bit of experience in performance takes as closer look at a software,
>>> that a
>>> few dramatic improvements can be made. That makes working as a
>>> performance
>>> analysis expert so gratifying. You suggest a few improvements, which
>>> have an
>>> dramatic impact, and then you walk away before it gets too complicated
>>> ;-)
>>> No, that's not my intention here :-)
>>>
>>>
>>> Markus
>>>
>>> "The best way to predict the future is to invent it" -- Alan Kay
>>>
>>>
>>> On Thu, Nov 26, 2009 at 6:04 AM, Bechauf, Michael
>>> <michael.bechauf@sap.com>wrote:
>>>
>>> > David,
>>> >
>>> > well, "dead wrong" is a strong expression; hopefully I'm still
>>> breathing. I
>>> > don't want to judge without having looked at the code myself, but I
>>> have no
>>> > idea how a massive multi-user system could possibly be designed with
>>> state
>>> > where per-user information is kept in memory for a certain time. I
>>> mean, 23
>>> > GB allocated - that's tough for an SAP transaction server that is not
>>> > mutlithreaded and where the memory management is highly optimized
>>> based on
>>> > shared memory that the work processes can attach to, or rolled out to
>>> a file
>>> > if unused for a whilet. It is, however, deadly for a VM that was never
>>> > designed for such memory consumption and where a GC run can halt the
>>> server.
>>> >
>>> > Anyway, I'll study this a bit more, particularely the Scala
>>> architecture. I
>>> > heard many good things about Scala, but in the end it's all translated
>>> to
>>> > things a VM can understand, and I hope Scala does a good enough job
>>> managing
>>> > this load in a transparent way.
>>> >
>>> > -Michael
>>> >
>>> >
>>> > ----- Original Message -----
>>> > From: David Pollak <feeder.of.the.bears@gmail.com>
>>> > To: esme-dev@incubator.apache.org <esme-dev@incubator.apache.org>
>>> > Sent: Wed Nov 25 23:00:20 2009
>>> > Subject: Re: Further analysis of the GC issue
>>> >
>>> > On Wed, Nov 25, 2009 at 7:16 PM, Bechauf, Michael
>>> > <michael.bechauf@sap.com>wrote:
>>> >
>>> > > Wasn't this exactly the kind of stuff that the Eclipse Memory
>>> Analyzer -
>>> > > donated by SAP - was supposed to fix ? A heap of that size for a
>>> still
>>> > > moderate number of 300 users is crazy, so either there is stuff like
>>> > > circular references that hog memory, or the design model is
>>> fundamentally
>>> > > flawed. I don't understand why ESME needs "sessions" ? How can a
>>> > scaleable
>>> > > server be created if each user will allocate memory until some
>>> timeout.
>>> > In a
>>> > > world of stateless browser-based UIs that's not going to work.
>>> > >
>>> >
>>> > You're actually dead wrong about this.  "Stateless" is not... it's
>>> just
>>> > pushing state and cache someplace else (the RDBMS, memcached, etc.).
>>> > "Stateless" will lead to radical performance problems.  "Stateless"
>>> merely
>>> > moves the caching decisions into code you don't control.  I dealt with
>>> this
>>> > issue first-hand while helping a popular micro-blogging site migrate
>>> from a
>>> > "stateless" to a Scala-based backend.  I'm dealing with this issue
>>> > first-hand helping another popular site that's experiencing
>>> exponential
>>> > growth migrate away from "push everything back to the RDBMS and hope
>>> for
>>> > the
>>> > best."
>>> >
>>> > My original design for ESME is stateful.  My original design for ESME
>>> is
>>> > based on lessoned learned in this very space and was oriented to have
>>> > things
>>> > intelligently cached so that the caching is not based on RDBMS
>>> indexes.
>>> >  I'm
>>> > not sure what happened to cause the particular issues, but it seems
>>> like
>>> > folks are loading messages from the RDBMS rather than asking the
>>> message
>>> > cache for them.
>>> >
>>> >
>>> > >
>>> > > Time for me to look at that code ...
>>> > >
>>> > > -Michael
>>> > >
>>> > >
>>> > > ----- Original Message -----
>>> > > From: Markus Kohler <markus.kohler@gmail.com>
>>> > > To: esme-dev@incubator.apache.org <esme-dev@incubator.apache.org>
>>> > > Sent: Wed Nov 25 12:14:58 2009
>>> > > Subject: Further analysis of the GC issue
>>> > >
>>> > > Hi all,
>>> > > the Garbage Collector issue I was talking about is reproducible.
>>> > > I've uploaded an annotated GC graph to
>>> > >
>>> > >
>>> >
>>> http://picasaweb.google.com/lh/photo/wB-RRtb0wIVfpxJkTJPNuw?authkey=Gv1s
>>> RgCOve7LThpfvXsQE&feat=directlink
>>> > >
>>> > > I think the "LOGON" phase where I logon all the 300 users looks ok
>>> (given
>>> > > that probably textile formatting is involved) but the phase where
>>> just
>>> > one
>>> > > user sends one message is certainly not looking good.
>>> > >
>>> > > I took the profiler and the result is a bit shocking. For that one
>>> > message,
>>> > > 881.000.000 objects weighting  23,2 Gbyte where allocated (and
>>> reclaimed
>>> > > afterwards). My former record was 2Gbyte ;-)
>>> > >
>>> > > Fortunately I have a theory what happens, without looking into the
>>> > > code,yet,
>>> > > so take it with a grain of salt. It seems that the public time line
>>> for
>>> > all
>>> > > users is re-rendered, because 99% of the allocations come
>>> > > from org.apache.esme.comet.PublicTimeline.render(). I guess all the
>>> > actors
>>> > > for all the users are sitting there, not knowing that the user has
>>> closed
>>> > > the browser, because the user session has not yet expired.
>>> > >
>>> > > I wonder how we get around this with a real "push" model. If the
>>> browser
>>> > > would ask for updates this rendering could be done lazily. Or can we
>>> > "ping"
>>> > > the browser and check whether it responds?
>>> > > On the other side. It should also not be necessary the re-render the
>>> > > message
>>> > > again and again because the result will be the same.
>>> > >
>>> > > I will send Richard some attachments. Not sure whether you will need
>>> > them,
>>> > > they look very similar to the ones we already have.
>>> > >
>>> > > BTW, we should definitely check the use
>>> > > of scala.xml.XML$.loadString(java.lang.String)
>>> > > It's creating a new Parser each time, which is a bit costly because
>>> it
>>> > > allocates a new Buffer each time and also hits the disk, when
>>> searching
>>> > for
>>> > > the name of the Java class.
>>> > >
>>> > > Greetings,
>>> > > Markus
>>> > >
>>> > >
>>> > >
>>> > > "The best way to predict the future is to invent it" -- Alan Kay
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Lift, the simply functional web framework http://liftweb.net
>>> > Beginning Scala http://www.apress.com/book/view/1430219890
>>> > Follow me: http://twitter.com/dpp
>>> > Surf the harmonics
>>> >
>>>
>>
>

Mime
View raw message