Mailing-List: contact esme-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: esme-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of markus.kohler@gmail.com
 designates 209.85.217.225 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=CZ15ey9yWCjKAz6LlpuHEThlUIj6jR6IZGgLYwugh5PAqZvX0MyR1Ky6i0LzH5qQmI
         1M9wfBNRq6g+IIBvcfwztqBuOffDqw1YMSSKwRz7OY49Wq0uNJKK401vuEBRM4uPCxgq
         2hbAvYw9MpXZIgR44RIo/Iozw9yK551J4RZCQ=
MIME-Version: 1.0
Date: Mon, 30 Nov 2009 23:51:20 +0100
Message-ID: <771905290911301451o7c81ac97pc3ca4d62fe1ca722@mail.gmail.com>
Subject: Memory allocation analysis Episode II (details)
From: Markus Kohler <markus.kohler@gmail.com>
To: esme-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=0016e644c6eae8752b04799e7a62

--0016e644c6eae8752b04799e7a62
Content-Type: text/plain; charset=ISO-8859-1

Hi all,
Here come some details about my memory allocation analysis done after Vassil
fixed the most important issue.

-we are at 4,93Mbyte temporary allocated for one message send
- 2,7Mbyte alone are allocated for char[]. There's seems to be some easy
optimization possible, because 1,5Mbyte is allocated in
StringBuilder.expandCapacity. This is caused by scala not preallocating the
StringBuilder with the correct size in List.mkString. The reason for this
behaviour is that mkString is not implemented in List but in the more
abstract Iterator. Iterator doesn't have a length (for a good reason) and
therefore mkString has no idea how large the StringBuilder has to be at the
end.
I believe (without looking at the Scala source code so far) that mkString
should be reimplemented in List. We can probably go around this issue
somehow, until some does implement a better List.mkString in scala (have to
check the scala sources, whether that's possible at all).
I think as soon as we fixed this, we should maybe stop with the memory
allocation optimizations, because we are already very good, and we should do
more tests before we put more effort into this topic.

The rest of the allocations are pretty minor. It's interesting the
ListBuffer is used not sure whether that is the best choice here. I will
provide an profiler report, for those how want to look into this.

I will send the corresponding profiling reports to Dick.

- I haven't done a heap dump yet. I want to do 2 of them to compute the
amount of memory for one user.

Regards,
Markus

"The best way to predict the future is to invent it" -- Alan Kay


On Fri, Nov 27, 2009 at 5:28 PM, Markus Kohler <markus.kohler@gmail.com>wrote:

> Hi all,
> In short, We are down from 90Mbyte to *19Mbyte* for sending one message
> and I believe it should be possible to get under 10Mbyte  without too much
> effort. With 10  Mbyte we would comply to the performance standard of some
> major ERP vendor, but honestly I believe we should be able to do even better
> ;-)
>
> Out of the 19Mbyte(maybe even only 11, it's bit hard to get reproducible
> results) 9,9 are consumed by scala.xml.XML$.loadString(java.lang.String)
> mainly because every time the routine is called, a new Parser is
> instantiated. In addition javax.xml.parsers.SAXParserFactory.newInstance()
> is called, which hits  (unless that has changed in recent JDK's) the file
> system to find out which XML parser is configured.  *I think ideally this
> should be fixed in the Scala sources. *
>
> The issue with the message formatted 300 times when all users are logged on
> still seems to be there, at least the GC log suggests that, but I guess
> David will attack that soon. I will send around the profiling reports later
> on.
>
> Regards,
> Markus
>
>
> "The best way to predict the future is to invent it" -- Alan Kay
>

--0016e644c6eae8752b04799e7a62--