Return-Path: Delivered-To: apmail-ibatis-user-java-archive@www.apache.org Received: (qmail 2290 invoked from network); 25 Jul 2005 07:01:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 25 Jul 2005 07:01:35 -0000 Received: (qmail 79502 invoked by uid 500); 25 Jul 2005 07:01:33 -0000 Delivered-To: apmail-ibatis-user-java-archive@ibatis.apache.org Received: (qmail 79483 invoked by uid 500); 25 Jul 2005 07:01:33 -0000 Mailing-List: contact user-java-help@ibatis.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user-java@ibatis.apache.org Delivered-To: mailing list user-java@ibatis.apache.org Received: (qmail 79466 invoked by uid 99); 25 Jul 2005 07:01:32 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jul 2005 00:01:32 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=HTML_60_70,HTML_MESSAGE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of misiek_x@poczta.onet.pl designates 213.180.130.71 as permitted sender) Received: from [213.180.130.71] (HELO smtp11.poczta.onet.pl) (213.180.130.71) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jul 2005 00:01:24 -0700 Received: from polygonatum.man.poznan.pl ([150.254.170.114]:57349 "EHLO polygonatum") by ps11.test.onet.pl with ESMTP id ; Mon, 25 Jul 2005 09:01:22 +0200 Message-ID: <007301c590e6$c5bb2020$72aafe96@polygonatum> From: "Michal Malecki" To: References: <8A9C57F4-4F4D-48B4-B555-7B65D5EA6766@sun.com> <16178eb10507212019addf2d1@mail.gmail.com> <007501c58ea6$14bd41f0$72aafe96@polygonatum> <16178eb10507220615b1584a6@mail.gmail.com> <00c201c58ec0$d44369e0$72aafe96@polygonatum> <16178eb1050722062624745600@mail.gmail.com> <00d801c58ec4$581d8130$72aafe96@polygonatum> <16178eb105072207026f6d4fcc@mail.gmail.com> Subject: Re: Estimating amount of memory need for cache Date: Mon, 25 Jul 2005 09:02:07 +0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0070_01C590F7.88EB48D0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1506 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1506 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N This is a multi-part message in MIME format. ------=_NextPart_000_0070_01C590F7.88EB48D0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable There is a problem with accessing fields, that I think can be only = solves with bytecode manipulation. More specifically - using reflection = we can check all public fields (getFields) and all fields in the = specific class (getDeclaredFields and then setAcesible for all = private/protected/package). There is no way to check = private/protected/package fields of the superclasses. Any ideas are = welcomed, first prototype of the utility - tomorrow regards Michal Malecki You can keep a running set of instance references to ensure that you = don't count the same object twice. Cheers, Clinton On 7/22/05, Michal Malecki wrote: I can think about it during weekend (at last) :) first - two = general questions: 1) this sizeof method would return size of object and all objects = that are referenced by it; so invoking it in the loop for array and = adding the result is not good because some objects can be referenced = many times 2)there is no problem with primitives and strings :) but what we do = with all other classes; it it enough to check all attributes by = reflection (probably invoking setAccessible) until we have only = primitives and arrays of primitives? As I read this article in the JavaWorld (I have never time for that = :() this sizeof is very tricky :( So some simplyfying assumptions must = be made More ideas after weekend Michal Malecki ----- Original Message -----=20 From: Clinton Begin=20 To: user-java@ibatis.apache.org=20 Sent: Friday, July 22, 2005 3:26 PM=20 Subject: Re: Estimating amount of memory need for cache Good idea Michal, Sounds like something useful that could be added to Commons Lang. = Would you have an interest in contributing it, or are you going to make = me write it? ;-) Cheers, Clinton On 7/22/05, Michal Malecki wrote:=20 I think that in many cases estimating memory taken maybe quite = easy (when there are no circual references). Then it's ease to write = utility that would analyze object using reflection or to implement = interface that has size() method; If you know structure of the bean and = there are well known types (strings, primitives maybe dates/timestamps) = estimating size is as easy as writing hashCode :) Michal Malecki ----- Original Message -----=20 From: Clinton Begin=20 To: user-java@ibatis.apache.org=20 Sent: Friday, July 22, 2005 3:15 PM=20 Subject: Re: Estimating amount of memory need for cache Thanks for the info Michal, I suppose then you'd have to double expected size of each = string in your model then, to make the estimate safer. Too bad there isn't something along the lines of System.sizeOf = (Object obj). =20 Cheers, Clinton On 7/22/05, Michal Malecki wrote:=20 Hello, I think it's not quite so e.g. all characters in the memory = are 2 bytes and serialized there are in UTF-8 here is article about it: = http://www.javaworld.com/javaworld/javaqa/2003-12/02-qa-1226-sizeof.html = Regards Michal Malecki You could serialize the results it to disk. That would = give you a pessimistic estimation of the size it will be in memory. = That is, the cost of serialization will be higher than the amount of RAM = used, so you can feel relatively confident that it will be smaller in = memory. Cheers, Clinton On 7/21/05, Nathan Maves wrote:=20 What is the best and most accurate way to calculate the = space needed to cache a query. Say I have a query that returns 5 strings on average of = 20 chars each. The query returns 5000 rows. I know this seems high but just humor me and remember I = have access=20 to big servers here at Sun :) Nathan ------=_NextPart_000_0070_01C590F7.88EB48D0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
There is a problem with accessing = fields, that I=20 think can be only solves with bytecode manipulation. More specifically - = using=20 reflection we can check all public fields (getFields) and all fields in = the=20 specific class (getDeclaredFields and then setAcesible for all=20 private/protected/package). There is no way to check = private/protected/package=20 fields of the superclasses. Any ideas are welcomed, first prototype of = the=20 utility - tomorrow
regards
Michal Malecki

You=20 can keep a running set of instance references to ensure that you don't = count=20 the same object twice.

Cheers,
Clinton

On 7/22/05, Michal=20 Malecki <misiek_x@poczta.onet.pl>=20 wrote:
I can think about it during weekend = (at last)=20 :) first  - two general questions:
1) this sizeof method would return = size of=20 object and all objects that are referenced by it; so invoking it in = the loop=20 for array and adding the result is not good because some objects can = be=20 referenced many times
2)there is no problem with = primitives and=20 strings :) but what we do with all other classes; it it enough to = check all=20 attributes by reflection (probably invoking setAccessible) until we = have=20 only primitives and arrays of primitives?
 
As I read this article in the = JavaWorld (I have=20 never time for that :() this sizeof is very tricky :( So some = simplyfying=20 assumptions must be made
 
More ideas after = weekend
Michal Malecki
-----=20 Original Message -----
From:=20 Clinton = Begin=20 To:=20 user-java@ibatis.apache.org
Sent:=20 Friday, July 22, 2005 3:26 PM
Subject:=20 Re: Estimating amount of memory need for cache


Good idea Michal,

Sounds like something = useful=20 that could be added to Commons Lang.  Would you have an = interest in=20 contributing it, or are you going to make me write it? =20 ;-)

Cheers,
Clinton

On 7/22/05, Michal=20 Malecki <misiek_x@poczta.onet.pl> wrote:=20
I think that in many cases = estimating=20 memory taken maybe quite easy (when there are no circual=20 references). Then it's ease to write utility that would analyze = object=20 using reflection or to implement interface that has size() = method; If=20 you know structure of the bean and there are well known types = (strings,=20 primitives maybe dates/timestamps)  estimating size is as = easy as=20 writing hashCode :)
Michal=20 Malecki
-----=20 Original Message -----
From:=20 Clinton Begin=20
To:=20 user-java@ibatis.apache.org Sent:=20 Friday, July 22, 2005 3:15 PM Subject:=20 Re: Estimating amount of memory need for cache

Thanks for the info Michal,

I = suppose=20 then you'd have to double expected size of each string in your = model=20 then, to make the estimate safer.

Too bad there isn't = something=20 along the lines of System.sizeOf (Object obj). =20

Cheers,
Clinton

On 7/22/05, Michal Malecki <misiek_x@poczta.onet.pl> wrote:=20
Hello,
I think it's not quite so = e.g. all=20 characters in the memory are 2 bytes and serialized there = are in=20 UTF-8
here is article about = it:
http://www.javaworld.com/javaworld/javaqa/2003-12/02-qa-1= 226-sizeof.html=20
Regards
Michal = Malecki

You=20 could serialize the results it to disk.  That would = give you=20 a pessimistic estimation of the size it will be in = memory. =20 That is, the cost of serialization will be higher than the = amount=20 of RAM used, so you can feel relatively confident that it = will be=20 smaller in memory.

Cheers,
Clinton

On 7/21/05, Nathan Maves <Nathan.Maves@sun.com > = wrote:=20
What=20 is the best and most accurate way to calculate the space = needed
to cache a query.

Say I have a query = that=20 returns 5 strings on average of 20 chars = each.

The query=20 returns 5000 rows.

I know this seems high but = just humor=20 me and remember I have access
to big servers here at = Sun=20 = :)

Nathan




------=_NextPart_000_0070_01C590F7.88EB48D0--