felix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard S. Hall" <he...@ungoverned.org>
Subject Re: Bundle cache changes
Date Mon, 04 Oct 2010 20:26:52 GMT
  On 10/4/10 12:31, Pierre De Rop wrote:
> This is surprising, because I double checked (I did the same kind of tests
> of differents machines), and I always see that loading a populated cache is
> a bit longer than an empty cache.
> Ok, it's confusing for now, I will do further investigation and will get
> back to you later, if I find something.

Let's keep investigating this.

In the meantime, I decided to rollback the change for the next release 
so we can have more time to investigate it because right now I need to 
get a bug fix out. So, we won't introduce any bundle cache structure 
changes in 3.0.4, but maybe in 3.0.5.

-> richard

>
> On Mon, Oct 4, 2010 at 5:05 PM, Richard S. Hall<heavy@ungoverned.org>wrote:
>
>>   On 10/4/10 10:52, Richard S. Hall wrote:
>>
>>> Thanks for the installer bundle, I did some tests locally on my Mac using
>>> 230 bundles from GlassFish, this is what I see (note: the "purge" command
>>> flushes disk buffers):
>>>
>>> Empty cache
>>> -----------------
>>> [heavyweight main]$ rm -rf felix-cache; purge; date; java -jar
>>> bin/felix.jar
>>> Mon Oct  4 10:43:09 EDT 2010
>>> 2010-10-04 10:43:24,541 - BundleInstaller starting and waiting for
>>> FrameworkEvent.STARTED event ...
>>> 2010-10-04 10:43:24,822 - Framework started: Installing bundles from
>>> ./load dir
>>> 2010-10-04 10:43:54,464 - Done.
>>>
>>> Populated cache
>>> -----------------------
>>> [heavyweight main]$ purge; date; java -jar bin/felix.jar
>>> Mon Oct  4 10:44:49 EDT 2010
>>> 2010-10-04 10:45:12,212 - BundleInstaller starting and waiting for
>>> FrameworkEvent.STARTED event ...
>>> 2010-10-04 10:45:13,340 - Framework started: Installing bundles from
>>> ./load dir
>>> 2010-10-04 10:45:13,342 - Done.
>>>
>>> You can see that the empty cache scenario takes about 45 seconds, whereas
>>> the populated cache takes about 24 seconds. So, on my machine, it takes a
>>> lot less time to re-load cached bundles...could be a function of my disk
>>> speed, I suppose, but it is what I would expect.
>>>
>>> Not sure how to proceed, other than trying to get more samples.
>>>
>> Interestingly, I tried the same test using the 3.0.3 release (the above was
>> with trunk) and I got the follow results:
>>
>> Empty cache (3.0.3)
>> -----------------
>>
>> rm -rf felix-cache; purge; date; java -jar bin/felix.jar
>> Mon Oct  4 10:58:23 EDT 2010
>> 2010-10-04 10:58:43,340 - BundleInstaller starting and waiting for
>> FrameworkEvent.STARTED event ...
>> 2010-10-04 10:58:43,557 - Framework started: Installing bundles from ./load
>> dir
>> 2010-10-04 10:59:03,373 - Done.
>>
>> Populated cache (3.0.3)
>> -----------------------
>>
>> purge; date; java -jar bin/felix.jar
>> Mon Oct  4 10:59:50 EDT 2010
>> 2010-10-04 11:00:24,738 - BundleInstaller starting and waiting for
>> FrameworkEvent.STARTED event ...
>> 2010-10-04 11:00:26,736 - Framework started: Installing bundles from ./load
>> dir
>> 2010-10-04 11:00:26,738 - Done.
>>
>> So, the populated cache takes nearly as long as the empty cache (36 seconds
>> compared to 40 seconds)...this also makes sense, since the trunk has patches
>> specifically designed to improve re-loading cached bundles.
>>
>> ->  richard
>>
>>
>>
>>>>   On Fri, Oct 1, 2010 at 11:30 PM,<heavy@ungoverned.org>    wrote:
>>>>>>   On Fri, 1 Oct 2010 23:25:35 +0200, Pierre De Rop<
>>>>>> pierre.derop@gmail.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>   On Fri, Oct 1, 2010 at 10:57 PM, Richard S. Hall
>>>>>>>> <heavy@ungoverned.org>wrote:
>>>>>>>>
>>>>>>>>    One thing, was your performance comparison based on using
the new
>>>>>>>> cache
>>>>>>>>   or
>>>>>>>>
>>>>>>>>> old?
>>>>>>>>>
>>>>>>>>> I assume you are using a newly created cache for the
framework
>>>>>>>>> snapshot
>>>>>>>>> and
>>>>>>>>> [obviously] an old cache for the 3.0.2, correct?
>>>>>>>>>
>>>>>>>>>   yes:
>>>>>>>>>
>>>>>>>> old cache with 3.0.2, and new cache with trunk.
>>>>>>>> but I also checked that new fwk/trunk is properly working
when being
>>>>>>>> started
>>>>>>>> with the old cache (from 3.0.2)
>>>>>>>>
>>>>>>>>   You should also be testing an already created cache (i.e.,
a
>>>>>>>> framework
>>>>>>>>
>>>>>>>>> restart), not an empty cache.
>>>>>>>>>
>>>>>>>>>   In the previous test, I compared with already created
cache, not
>>>>>>>>> with
>>>>>>>>>
>>>>>>>> empty
>>>>>>>   caches.
>>>>>>>> With empty caches, here are the results:
>>>>>>>>
>>>>>>>>
>>>>>>>>     - old fwk 3.0.2 / fresh empty cache ->    15.8 sec.
(this is
>>>>>>>> surprising:
>>>>>>>>
>>>>>>>>   I
>>>>>>>      would expect to get a longer delay, since the cache is empty
when
>>>>>>>> I
>>>>>>>>     start
>>>>>>>>     the fwk)
>>>>>>>>
>>>>>>>> 2010-10-01 23:10:17,403 Starting Felix 3.0.2
>>>>>>>> 2010-10-01 23:10:23,885 Starting cluster node "test"
>>>>>>>> 2010-10-01 23:10:33,208 Cluster node "test" ready
>>>>>>>>
>>>>>>>>
>>>>>>>>     - new fwk / fresh empty cache ->15.2 sec
>>>>>>>>
>>>>>>>> 2010-10-01 23:15:46,2 Starting Felix 3.1.0.SNAPSHOT
>>>>>>>> 2010-10-01 23:15:52,417 Starting cluster node "test"
>>>>>>>> 2010-10-01 23:16:01,225 Cluster node "test" ready
>>>>>>>>
>>>>>>>> hmmm, I don't understand why the fwk startup time is better
when the
>>>>>>>>
>>>>>>>>   cache
>>>>>>>   is empty (15.2 sec) than when the cache is non-empty (21.2
sec from
>>>>>>>> previous
>>>>>>>> mail)
>>>>>>>> Ok, I will do more tests ... may be I did something wrong
? ... to be
>>>>>>>> continued ...
>>>>>>>>
>>>>>>>>   Hmm. Yeah, that seems a little odd. In that case, you should
just
>>>>>>> delete
>>>>>>> your cache each time to get better startup performance! ;-)
>>>>>>>
>>>>>>> Definitely let me know what you find out. I too would expect
framework
>>>>>>> restarts to be faster since we don't have to copy the JAR files
first.
>>>>>>>
>>>>>>> ->    richard
>>>>>>>
>>>>>>>   /pierre
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>   Is that what you are doing? Just want to make sure I understand
what
>>>>>>>> is
>>>>>>>>
>>>>>>>>> being measured. :-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ->    richard
>>>>>>>>>
>>>>>>>>> On 10/1/10 16:43, Pierre De Rop wrote:
>>>>>>>>>
>>>>>>>>>   Hi Richard,
>>>>>>>>>
>>>>>>>>>> I tried our app server with the trunk (old cache
with the new
>>>>>>>>>>
>>>>>>>>>>   fwk/trunk)
>>>>>>>> ->
>>>>>>>>
>>>>>>>>> everything works ok
>>>>>>>>>> (I install 195 bundles, and start 41 bundles)
>>>>>>>>>>
>>>>>>>>>> With new cache/fwk trunk ->     it's also ok.
>>>>>>>>>>
>>>>>>>>>> I also compared the performance of the startup time
between felix
>>>>>>>>>>
>>>>>>>>>>   2.0.2
>>>>>>>> and
>>>>>>>>
>>>>>>>>> felix trunk.
>>>>>>>>>> To do this, I managed to ensure that my linux IO
buffers cache are
>>>>>>>>>>
>>>>>>>>>>   empty
>>>>>>>> before doing each test, by doing this:
>>>>>>>>
>>>>>>>>> sync
>>>>>>>>>> echo 1>     /proc/sys/vm/drop_caches
>>>>>>>>>> echo 2>     /proc/sys/vm/drop_caches
>>>>>>>>>> echo 3>     /proc/sys/vm/drop_caches
>>>>>>>>>>
>>>>>>>>>> This ensures that my linux has released all buffer
cache disk, like
>>>>>>>>>> it
>>>>>>>>>> is
>>>>>>>>>> the case when I boot my host (cold start).
>>>>>>>>>> Now, for felix 3.0.2, I load my server in around
*21,1 *seconds
>>>>>>>>>> (it's
>>>>>>>>>> slow
>>>>>>>>>> because all disk buffers are empty):
>>>>>>>>>>
>>>>>>>>>>    2010-10-01 22:15:07,880 Starting Felix 3.0.2
>>>>>>>>>>    2010-10-01 22:15:18,121 Starting cluster node
"test"
>>>>>>>>>>    2010-10-01 22:15:28,966 Cluster node "test" ready
>>>>>>>>>>
>>>>>>>>>> and with felix/trunk, indeed, the server starts a
little bit
>>>>>>>>>> quicker,
>>>>>>>>>> but
>>>>>>>>>> not so much: around *in 20,3* seconds (I also flushed
disk buffers
>>>>>>>>>> before
>>>>>>>>>> starting the test):
>>>>>>>>>>
>>>>>>>>>>    2010-10-01 22:17:47,505 Starting Felix 3.1.0.SNAPSHOT
>>>>>>>>>>    2010-10-01 22:17:56,815 Starting cluster node
"test"
>>>>>>>>>>    2010-10-01 22:18:07,798 Cluster node "test" ready
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> /pierre
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 1, 2010 at 5:39 PM, Richard S. Hall<
>>>>>>>>>> heavy@ungoverned.org
>>>>>>>>>>
>>>>>>>>>>   wrote:
>>>>>>>>>>>     p.s. I deploy snapshots of the everything,
including the
>>>>>>>>>> convenience
>>>>>>>>>>
>>>>>>>>>>   framework distribution or you can build from trunk.
>>>>>>>>>>>
>>>>>>>>>>> On 10/1/10 11:37, Richard S. Hall wrote:
>>>>>>>>>>>
>>>>>>>>>>>    Hey everyone,
>>>>>>>>>>>
>>>>>>>>>>>   I've made some changes to the bundle cache
layout and handling to
>>>>>>>>>>>> improve
>>>>>>>>>>>> performance and clean up code. I've tried
to maintain backward
>>>>>>>>>>>> compatibility
>>>>>>>>>>>> with old caches, but I'd appreciate is someone
could try it out
>>>>>>>>>>>> on
>>>>>>>>>>>> their
>>>>>>>>>>>> caches to see if it works.
>>>>>>>>>>>>
>>>>>>>>>>>> ** Back up your caches just in case! **
>>>>>>>>>>>>
>>>>>>>>>>>> Note that the new cache format will not work
with older
>>>>>>>>>>>> frameworks.
>>>>>>>>>>>>
>>>>>>>>>>>>   So
>>>>>>>>>> you
>>>>>>>>> cannot use a cache created with this new framework with
previous
>>>>>>>>>>>> frameworks;
>>>>>>>>>>>> however, previously created caches should
mostly continue to work
>>>>>>>>>>>> between
>>>>>>>>>>>> the two although there will be a loss of
fidelity since changes
>>>>>>>>>>>> to
>>>>>>>>>>>> state
>>>>>>>>>>>> are
>>>>>>>>>>>> only saved to the new way or the old way
depending on if you are
>>>>>>>>>>>> running
>>>>>>>>>>>> on
>>>>>>>>>>>> the new framework or the old one.
>>>>>>>>>>>>
>>>>>>>>>>>> For the curious, I've combined five bundle
cache files (id,
>>>>>>>>>>>>
>>>>>>>>>>>>   location,
>>>>>>>>>> state, start level, last modified, and refresh count)
into a single
>>>>>>>>> file
>>>>>>>>>>>> and
>>>>>>>>>>>> try to avoid excessive file accesses. This
appears to improve
>>>>>>>>>>>>
>>>>>>>>>>>>   startup
>>>>>>>>>> time
>>>>>>>>> when you have a cache with lots of bundles, but your
mileage will
>>>>>>>>>>>>   vary
>>>>>>>>>> based
>>>>>>>>> on platform and other factors. Although, you won't necessarily
see
>>>>>>>>>>>>   any
>>>>>>>>>> improvements if you are using an old cache, since
it will revert to
>>>>>>>>> the
>>>>>>>>>>>> old
>>>>>>>>>>>> way.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> ->     richard
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>

Mime
View raw message