esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Pollak <feeder.of.the.be...@gmail.com>
Subject Re: Memory allocation analysis Episode II (details)
Date Mon, 30 Nov 2009 23:06:17 GMT
On Mon, Nov 30, 2009 at 2:51 PM, Markus Kohler <markus.kohler@gmail.com>wrote:

> Hi all,
> Here come some details about my memory allocation analysis done after
> Vassil
> fixed the most important issue.
>
> -we are at 4,93Mbyte temporary allocated for one message send
> - 2,7Mbyte alone are allocated for char[]. There's seems to be some easy
> optimization possible, because 1,5Mbyte is allocated in
> StringBuilder.expandCapacity.


How many of these allocations escape the eden pool?

Have you run any of this on a 1.6_16 JVM (the one with escape analysis)?  If
so, does it make any difference?

Also, doing a length on a List is O(n), so I'm not sure the value of getting
the length as part of a pre-allocation strategy.

Is the mkString stuff happening as part of the parse combinator?  If so, we
can change up some strategies pretty easily.


> This is caused by scala not preallocating the
> StringBuilder with the correct size in List.mkString. The reason for this
> behaviour is that mkString is not implemented in List but in the more
> abstract Iterator. Iterator doesn't have a length (for a good reason) and
> therefore mkString has no idea how large the StringBuilder has to be at the
> end.
> I believe (without looking at the Scala source code so far) that mkString
> should be reimplemented in List. We can probably go around this issue
> somehow, until some does implement a better List.mkString in scala (have to
> check the scala sources, whether that's possible at all).
> I think as soon as we fixed this, we should maybe stop with the memory
> allocation optimizations, because we are already very good, and we should
> do
> more tests before we put more effort into this topic.
>
> The rest of the allocations are pretty minor. It's interesting the
> ListBuffer is used not sure whether that is the best choice here. I will
> provide an profiler report, for those how want to look into this.
>
> I will send the corresponding profiling reports to Dick.
>
> - I haven't done a heap dump yet. I want to do 2 of them to compute the
> amount of memory for one user.
>
> Regards,
> Markus
>
> "The best way to predict the future is to invent it" -- Alan Kay
>
>
> On Fri, Nov 27, 2009 at 5:28 PM, Markus Kohler <markus.kohler@gmail.com
> >wrote:
>
> > Hi all,
> > In short, We are down from 90Mbyte to *19Mbyte* for sending one message
> > and I believe it should be possible to get under 10Mbyte  without too
> much
> > effort. With 10  Mbyte we would comply to the performance standard of
> some
> > major ERP vendor, but honestly I believe we should be able to do even
> better
> > ;-)
> >
> > Out of the 19Mbyte(maybe even only 11, it's bit hard to get reproducible
> > results) 9,9 are consumed by scala.xml.XML$.loadString(java.lang.String)
> > mainly because every time the routine is called, a new Parser is
> > instantiated. In addition
> javax.xml.parsers.SAXParserFactory.newInstance()
> > is called, which hits  (unless that has changed in recent JDK's) the file
> > system to find out which XML parser is configured.  *I think ideally this
> > should be fixed in the Scala sources. *
> >
> > The issue with the message formatted 300 times when all users are logged
> on
> > still seems to be there, at least the GC log suggests that, but I guess
> > David will attack that soon. I will send around the profiling reports
> later
> > on.
> >
> > Regards,
> > Markus
> >
> >
> > "The best way to predict the future is to invent it" -- Alan Kay
> >
>



-- 
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message