felix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre De Rop <pierre.de...@gmail.com>
Subject Re: Bundle cache changes
Date Mon, 04 Oct 2010 16:31:59 GMT
This is surprising, because I double checked (I did the same kind of tests
of differents machines), and I always see that loading a populated cache is
a bit longer than an empty cache.
Ok, it's confusing for now, I will do further investigation and will get
back to you later, if I find something.


On Mon, Oct 4, 2010 at 5:05 PM, Richard S. Hall <heavy@ungoverned.org>wrote:

>  On 10/4/10 10:52, Richard S. Hall wrote:
>
>>
>> Thanks for the installer bundle, I did some tests locally on my Mac using
>> 230 bundles from GlassFish, this is what I see (note: the "purge" command
>> flushes disk buffers):
>>
>> Empty cache
>> -----------------
>> [heavyweight main]$ rm -rf felix-cache; purge; date; java -jar
>> bin/felix.jar
>> Mon Oct  4 10:43:09 EDT 2010
>> 2010-10-04 10:43:24,541 - BundleInstaller starting and waiting for
>> FrameworkEvent.STARTED event ...
>> 2010-10-04 10:43:24,822 - Framework started: Installing bundles from
>> ./load dir
>> 2010-10-04 10:43:54,464 - Done.
>>
>> Populated cache
>> -----------------------
>> [heavyweight main]$ purge; date; java -jar bin/felix.jar
>> Mon Oct  4 10:44:49 EDT 2010
>> 2010-10-04 10:45:12,212 - BundleInstaller starting and waiting for
>> FrameworkEvent.STARTED event ...
>> 2010-10-04 10:45:13,340 - Framework started: Installing bundles from
>> ./load dir
>> 2010-10-04 10:45:13,342 - Done.
>>
>> You can see that the empty cache scenario takes about 45 seconds, whereas
>> the populated cache takes about 24 seconds. So, on my machine, it takes a
>> lot less time to re-load cached bundles...could be a function of my disk
>> speed, I suppose, but it is what I would expect.
>>
>> Not sure how to proceed, other than trying to get more samples.
>>
>
> Interestingly, I tried the same test using the 3.0.3 release (the above was
> with trunk) and I got the follow results:
>
> Empty cache (3.0.3)
> -----------------
>
> rm -rf felix-cache; purge; date; java -jar bin/felix.jar
> Mon Oct  4 10:58:23 EDT 2010
> 2010-10-04 10:58:43,340 - BundleInstaller starting and waiting for
> FrameworkEvent.STARTED event ...
> 2010-10-04 10:58:43,557 - Framework started: Installing bundles from ./load
> dir
> 2010-10-04 10:59:03,373 - Done.
>
> Populated cache (3.0.3)
> -----------------------
>
> purge; date; java -jar bin/felix.jar
> Mon Oct  4 10:59:50 EDT 2010
> 2010-10-04 11:00:24,738 - BundleInstaller starting and waiting for
> FrameworkEvent.STARTED event ...
> 2010-10-04 11:00:26,736 - Framework started: Installing bundles from ./load
> dir
> 2010-10-04 11:00:26,738 - Done.
>
> So, the populated cache takes nearly as long as the empty cache (36 seconds
> compared to 40 seconds)...this also makes sense, since the trunk has patches
> specifically designed to improve re-loading cached bundles.
>
> -> richard
>
>
>
>>>  On Fri, Oct 1, 2010 at 11:30 PM,<heavy@ungoverned.org>   wrote:
>>>>>
>>>>>  On Fri, 1 Oct 2010 23:25:35 +0200, Pierre De Rop<
>>>>> pierre.derop@gmail.com>
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>  On Fri, Oct 1, 2010 at 10:57 PM, Richard S. Hall
>>>>>>> <heavy@ungoverned.org>wrote:
>>>>>>>
>>>>>>>   One thing, was your performance comparison based on using the
new
>>>>>>> cache
>>>>>>>  or
>>>>>>>
>>>>>>>> old?
>>>>>>>>
>>>>>>>> I assume you are using a newly created cache for the framework
>>>>>>>> snapshot
>>>>>>>> and
>>>>>>>> [obviously] an old cache for the 3.0.2, correct?
>>>>>>>>
>>>>>>>>  yes:
>>>>>>>>
>>>>>>> old cache with 3.0.2, and new cache with trunk.
>>>>>>> but I also checked that new fwk/trunk is properly working when
being
>>>>>>> started
>>>>>>> with the old cache (from 3.0.2)
>>>>>>>
>>>>>>>  You should also be testing an already created cache (i.e., a
>>>>>>> framework
>>>>>>>
>>>>>>>> restart), not an empty cache.
>>>>>>>>
>>>>>>>>  In the previous test, I compared with already created cache,
not
>>>>>>>> with
>>>>>>>>
>>>>>>> empty
>>>>>>
>>>>>>  caches.
>>>>>>> With empty caches, here are the results:
>>>>>>>
>>>>>>>
>>>>>>>    - old fwk 3.0.2 / fresh empty cache ->   15.8 sec. (this
is
>>>>>>> surprising:
>>>>>>>
>>>>>>>  I
>>>>>>
>>>>>>     would expect to get a longer delay, since the cache is empty
when
>>>>>>> I
>>>>>>>    start
>>>>>>>    the fwk)
>>>>>>>
>>>>>>> 2010-10-01 23:10:17,403 Starting Felix 3.0.2
>>>>>>> 2010-10-01 23:10:23,885 Starting cluster node "test"
>>>>>>> 2010-10-01 23:10:33,208 Cluster node "test" ready
>>>>>>>
>>>>>>>
>>>>>>>    - new fwk / fresh empty cache ->15.2 sec
>>>>>>>
>>>>>>> 2010-10-01 23:15:46,2 Starting Felix 3.1.0.SNAPSHOT
>>>>>>> 2010-10-01 23:15:52,417 Starting cluster node "test"
>>>>>>> 2010-10-01 23:16:01,225 Cluster node "test" ready
>>>>>>>
>>>>>>> hmmm, I don't understand why the fwk startup time is better when
the
>>>>>>>
>>>>>>>  cache
>>>>>>
>>>>>>  is empty (15.2 sec) than when the cache is non-empty (21.2 sec from
>>>>>>> previous
>>>>>>> mail)
>>>>>>> Ok, I will do more tests ... may be I did something wrong ? ...
to be
>>>>>>> continued ...
>>>>>>>
>>>>>>>  Hmm. Yeah, that seems a little odd. In that case, you should
just
>>>>>> delete
>>>>>> your cache each time to get better startup performance! ;-)
>>>>>>
>>>>>> Definitely let me know what you find out. I too would expect framework
>>>>>> restarts to be faster since we don't have to copy the JAR files first.
>>>>>>
>>>>>> ->   richard
>>>>>>
>>>>>>  /pierre
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  Is that what you are doing? Just want to make sure I understand
what
>>>>>>> is
>>>>>>>
>>>>>>>> being measured. :-)
>>>>>>>>
>>>>>>>>
>>>>>>>> ->   richard
>>>>>>>>
>>>>>>>> On 10/1/10 16:43, Pierre De Rop wrote:
>>>>>>>>
>>>>>>>>  Hi Richard,
>>>>>>>>
>>>>>>>>> I tried our app server with the trunk (old cache with
the new
>>>>>>>>>
>>>>>>>>>  fwk/trunk)
>>>>>>>>
>>>>>>> ->
>>>>>>>
>>>>>>>> everything works ok
>>>>>>>>> (I install 195 bundles, and start 41 bundles)
>>>>>>>>>
>>>>>>>>> With new cache/fwk trunk ->    it's also ok.
>>>>>>>>>
>>>>>>>>> I also compared the performance of the startup time between
felix
>>>>>>>>>
>>>>>>>>>  2.0.2
>>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>>> felix trunk.
>>>>>>>>> To do this, I managed to ensure that my linux IO buffers
cache are
>>>>>>>>>
>>>>>>>>>  empty
>>>>>>>>
>>>>>>> before doing each test, by doing this:
>>>>>>>
>>>>>>>> sync
>>>>>>>>> echo 1>    /proc/sys/vm/drop_caches
>>>>>>>>> echo 2>    /proc/sys/vm/drop_caches
>>>>>>>>> echo 3>    /proc/sys/vm/drop_caches
>>>>>>>>>
>>>>>>>>> This ensures that my linux has released all buffer cache
disk, like
>>>>>>>>> it
>>>>>>>>> is
>>>>>>>>> the case when I boot my host (cold start).
>>>>>>>>> Now, for felix 3.0.2, I load my server in around *21,1
*seconds
>>>>>>>>> (it's
>>>>>>>>> slow
>>>>>>>>> because all disk buffers are empty):
>>>>>>>>>
>>>>>>>>>   2010-10-01 22:15:07,880 Starting Felix 3.0.2
>>>>>>>>>   2010-10-01 22:15:18,121 Starting cluster node "test"
>>>>>>>>>   2010-10-01 22:15:28,966 Cluster node "test" ready
>>>>>>>>>
>>>>>>>>> and with felix/trunk, indeed, the server starts a little
bit
>>>>>>>>> quicker,
>>>>>>>>> but
>>>>>>>>> not so much: around *in 20,3* seconds (I also flushed
disk buffers
>>>>>>>>> before
>>>>>>>>> starting the test):
>>>>>>>>>
>>>>>>>>>   2010-10-01 22:17:47,505 Starting Felix 3.1.0.SNAPSHOT
>>>>>>>>>   2010-10-01 22:17:56,815 Starting cluster node "test"
>>>>>>>>>   2010-10-01 22:18:07,798 Cluster node "test" ready
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /pierre
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 1, 2010 at 5:39 PM, Richard S. Hall<
>>>>>>>>> heavy@ungoverned.org
>>>>>>>>>
>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>>    p.s. I deploy snapshots of the everything, including
the
>>>>>>>>> convenience
>>>>>>>>>
>>>>>>>>>  framework distribution or you can build from trunk.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/1/10 11:37, Richard S. Hall wrote:
>>>>>>>>>>
>>>>>>>>>>   Hey everyone,
>>>>>>>>>>
>>>>>>>>>>  I've made some changes to the bundle cache layout
and handling to
>>>>>>>>>>> improve
>>>>>>>>>>> performance and clean up code. I've tried to
maintain backward
>>>>>>>>>>> compatibility
>>>>>>>>>>> with old caches, but I'd appreciate is someone
could try it out
>>>>>>>>>>> on
>>>>>>>>>>> their
>>>>>>>>>>> caches to see if it works.
>>>>>>>>>>>
>>>>>>>>>>> ** Back up your caches just in case! **
>>>>>>>>>>>
>>>>>>>>>>> Note that the new cache format will not work
with older
>>>>>>>>>>> frameworks.
>>>>>>>>>>>
>>>>>>>>>>>  So
>>>>>>>>>>
>>>>>>>>> you
>>>>>>>
>>>>>>>> cannot use a cache created with this new framework with previous
>>>>>>>>>>> frameworks;
>>>>>>>>>>> however, previously created caches should mostly
continue to work
>>>>>>>>>>> between
>>>>>>>>>>> the two although there will be a loss of fidelity
since changes
>>>>>>>>>>> to
>>>>>>>>>>> state
>>>>>>>>>>> are
>>>>>>>>>>> only saved to the new way or the old way depending
on if you are
>>>>>>>>>>> running
>>>>>>>>>>> on
>>>>>>>>>>> the new framework or the old one.
>>>>>>>>>>>
>>>>>>>>>>> For the curious, I've combined five bundle cache
files (id,
>>>>>>>>>>>
>>>>>>>>>>>  location,
>>>>>>>>>>
>>>>>>>>> state, start level, last modified, and refresh count)
into a single
>>>>>>>
>>>>>>>> file
>>>>>>>>>>> and
>>>>>>>>>>> try to avoid excessive file accesses. This appears
to improve
>>>>>>>>>>>
>>>>>>>>>>>  startup
>>>>>>>>>>
>>>>>>>>> time
>>>>>>>
>>>>>>>> when you have a cache with lots of bundles, but your mileage
will
>>>>>>>>>>>
>>>>>>>>>>>  vary
>>>>>>>>>>
>>>>>>>>> based
>>>>>>>
>>>>>>>> on platform and other factors. Although, you won't necessarily
see
>>>>>>>>>>>
>>>>>>>>>>>  any
>>>>>>>>>>
>>>>>>>>> improvements if you are using an old cache, since it
will revert to
>>>>>>>
>>>>>>>> the
>>>>>>>>>>> old
>>>>>>>>>>> way.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> ->    richard
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message