Return-Path: Delivered-To: apmail-felix-dev-archive@www.apache.org Received: (qmail 97841 invoked from network); 6 Oct 2010 05:51:52 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Oct 2010 05:51:52 -0000 Received: (qmail 31421 invoked by uid 500); 6 Oct 2010 05:51:52 -0000 Delivered-To: apmail-felix-dev-archive@felix.apache.org Received: (qmail 31076 invoked by uid 500); 6 Oct 2010 05:51:50 -0000 Mailing-List: contact dev-help@felix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@felix.apache.org Delivered-To: mailing list dev@felix.apache.org Received: (qmail 31066 invoked by uid 99); 6 Oct 2010 05:51:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 05:51:49 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of fmeschbe@gmail.com designates 209.85.161.49 as permitted sender) Received: from [209.85.161.49] (HELO mail-fx0-f49.google.com) (209.85.161.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 05:51:42 +0000 Received: by fxm15 with SMTP id 15so4946781fxm.22 for ; Tue, 05 Oct 2010 22:51:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=ThL3a0r1fAt2tR+THaiqHO+YAe+gMUYkCGI0Mgbu/c8=; b=fP3uYVjZ1YGRgYXReMxprje/hDO2e/j5hEGBD+YOlkU9pbcXrJaatH3B7e/CK8SK0o LLfWmiN+Qe8MkKh4IeoLEt8GcgvrOaVJ33dxHHgrBZoY3/U0E4dSSi7xHs+15by1Iscz GgAoSU/qFJ7egoj9mpWsrnAAWFZ1kq0CDGOYI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=wpGch9hE8XGrPgrUZeYFx9uJheoqpuw7piwKeulJ26U2RutwKZ2pSkTuURUp1lsJWt Cbtw8t7jW3j2j8l4TutJgTK88rVKGRPfzsoG6Yguth+lirssVxFCF0wLbZ1qf5QA9ih0 sAWPXnFlVLX2ylS1U6I1ZZfyhYDHrQIyIgDxU= Received: by 10.223.125.148 with SMTP id y20mr11890075far.94.1286344281349; Tue, 05 Oct 2010 22:51:21 -0700 (PDT) Received: from [192.168.1.21] (cable-static-182-112.eblcom.ch [87.102.182.112]) by mx.google.com with ESMTPS id m17sm1637087fag.0.2010.10.05.22.51.18 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 05 Oct 2010 22:51:19 -0700 (PDT) Message-ID: <4CAC0E55.6020409@gmail.com> Date: Wed, 06 Oct 2010 07:51:17 +0200 From: Felix Meschberger User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8 MIME-Version: 1.0 To: dev@felix.apache.org Subject: Re: Bundle cache changes References: <4CA60020.6000802@ungoverned.org> <4CA600CB.9060101@ungoverned.org> <4CA64B1C.5000004@ungoverned.org> <53f1cd2eaf68d4afee9eeb78c68e44de@ungoverned.us> <4CA76EEA.9000102@ungoverned.org> <4CA9EA35.90003@ungoverned.org> <4CA9ED34.8080802@ungoverned.org> <4CAA388C.1040600@ungoverned.org> <4CAACCEE.5070800@gmail.com> <4CAB245D.5010402@ungoverned.org> In-Reply-To: <4CAB245D.5010402@ungoverned.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Hi On 05.10.2010 15:13, Richard S. Hall wrote: > On 10/5/10 2:59, Felix Meschberger wrote: >> Hi, >> >> On 04.10.2010 22:26, Richard S. Hall wrote: >>> On 10/4/10 12:31, Pierre De Rop wrote: >>>> This is surprising, because I double checked (I did the same kind of >>>> tests >>>> of differents machines), and I always see that loading a populated >>>> cache is >>>> a bit longer than an empty cache. >>>> Ok, it's confusing for now, I will do further investigation and will >>>> get >>>> back to you later, if I find something. >>> Let's keep investigating this. >>> >>> In the meantime, I decided to rollback the change for the next release >>> so we can have more time to investigate it because right now I need to >>> get a bug fix out. So, we won't introduce any bundle cache structure >>> changes in 3.0.4, but maybe in 3.0.5. >> This sounds like a minor-release-number-worthy change ;-) > > I'm not sure that's necessary. It is still backward compatible, but it > isn't forward compatible. This means: once you ran the framework with the new bundle cache and installed and/or updated bundles you cannot run the existing cache with an older version of the framework any longer. Right ? In this case, IMHO a minor version number increase might provide a hint at this situation. Regards Felix > > -> richard > >> Regards >> Felix >> >>> -> richard >>> >>>> On Mon, Oct 4, 2010 at 5:05 PM, Richard S. >>>> Hallwrote: >>>> >>>>> On 10/4/10 10:52, Richard S. Hall wrote: >>>>> >>>>>> Thanks for the installer bundle, I did some tests locally on my Mac >>>>>> using >>>>>> 230 bundles from GlassFish, this is what I see (note: the "purge" >>>>>> command >>>>>> flushes disk buffers): >>>>>> >>>>>> Empty cache >>>>>> ----------------- >>>>>> [heavyweight main]$ rm -rf felix-cache; purge; date; java -jar >>>>>> bin/felix.jar >>>>>> Mon Oct 4 10:43:09 EDT 2010 >>>>>> 2010-10-04 10:43:24,541 - BundleInstaller starting and waiting for >>>>>> FrameworkEvent.STARTED event ... >>>>>> 2010-10-04 10:43:24,822 - Framework started: Installing bundles from >>>>>> ./load dir >>>>>> 2010-10-04 10:43:54,464 - Done. >>>>>> >>>>>> Populated cache >>>>>> ----------------------- >>>>>> [heavyweight main]$ purge; date; java -jar bin/felix.jar >>>>>> Mon Oct 4 10:44:49 EDT 2010 >>>>>> 2010-10-04 10:45:12,212 - BundleInstaller starting and waiting for >>>>>> FrameworkEvent.STARTED event ... >>>>>> 2010-10-04 10:45:13,340 - Framework started: Installing bundles from >>>>>> ./load dir >>>>>> 2010-10-04 10:45:13,342 - Done. >>>>>> >>>>>> You can see that the empty cache scenario takes about 45 seconds, >>>>>> whereas >>>>>> the populated cache takes about 24 seconds. So, on my machine, it >>>>>> takes a >>>>>> lot less time to re-load cached bundles...could be a function of my >>>>>> disk >>>>>> speed, I suppose, but it is what I would expect. >>>>>> >>>>>> Not sure how to proceed, other than trying to get more samples. >>>>>> >>>>> Interestingly, I tried the same test using the 3.0.3 release (the >>>>> above was >>>>> with trunk) and I got the follow results: >>>>> >>>>> Empty cache (3.0.3) >>>>> ----------------- >>>>> >>>>> rm -rf felix-cache; purge; date; java -jar bin/felix.jar >>>>> Mon Oct 4 10:58:23 EDT 2010 >>>>> 2010-10-04 10:58:43,340 - BundleInstaller starting and waiting for >>>>> FrameworkEvent.STARTED event ... >>>>> 2010-10-04 10:58:43,557 - Framework started: Installing bundles from >>>>> ./load >>>>> dir >>>>> 2010-10-04 10:59:03,373 - Done. >>>>> >>>>> Populated cache (3.0.3) >>>>> ----------------------- >>>>> >>>>> purge; date; java -jar bin/felix.jar >>>>> Mon Oct 4 10:59:50 EDT 2010 >>>>> 2010-10-04 11:00:24,738 - BundleInstaller starting and waiting for >>>>> FrameworkEvent.STARTED event ... >>>>> 2010-10-04 11:00:26,736 - Framework started: Installing bundles from >>>>> ./load >>>>> dir >>>>> 2010-10-04 11:00:26,738 - Done. >>>>> >>>>> So, the populated cache takes nearly as long as the empty cache (36 >>>>> seconds >>>>> compared to 40 seconds)...this also makes sense, since the trunk has >>>>> patches >>>>> specifically designed to improve re-loading cached bundles. >>>>> >>>>> -> richard >>>>> >>>>> >>>>> >>>>>>> On Fri, Oct 1, 2010 at 11:30 PM, wrote: >>>>>>>>> On Fri, 1 Oct 2010 23:25:35 +0200, Pierre De Rop< >>>>>>>>> pierre.derop@gmail.com> >>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> On Fri, Oct 1, 2010 at 10:57 PM, Richard S. Hall >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> One thing, was your performance comparison based on using >>>>>>>>>>> the new >>>>>>>>>>> cache >>>>>>>>>>> or >>>>>>>>>>> >>>>>>>>>>>> old? >>>>>>>>>>>> >>>>>>>>>>>> I assume you are using a newly created cache for the framework >>>>>>>>>>>> snapshot >>>>>>>>>>>> and >>>>>>>>>>>> [obviously] an old cache for the 3.0.2, correct? >>>>>>>>>>>> >>>>>>>>>>>> yes: >>>>>>>>>>>> >>>>>>>>>>> old cache with 3.0.2, and new cache with trunk. >>>>>>>>>>> but I also checked that new fwk/trunk is properly working when >>>>>>>>>>> being >>>>>>>>>>> started >>>>>>>>>>> with the old cache (from 3.0.2) >>>>>>>>>>> >>>>>>>>>>> You should also be testing an already created cache (i.e., a >>>>>>>>>>> framework >>>>>>>>>>> >>>>>>>>>>>> restart), not an empty cache. >>>>>>>>>>>> >>>>>>>>>>>> In the previous test, I compared with already created cache, >>>>>>>>>>>> not >>>>>>>>>>>> with >>>>>>>>>>>> >>>>>>>>>>> empty >>>>>>>>>> caches. >>>>>>>>>>> With empty caches, here are the results: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - old fwk 3.0.2 / fresh empty cache -> 15.8 sec. >>>>>>>>>>> (this is >>>>>>>>>>> surprising: >>>>>>>>>>> >>>>>>>>>>> I >>>>>>>>>> would expect to get a longer delay, since the cache is >>>>>>>>>> empty when >>>>>>>>>>> I >>>>>>>>>>> start >>>>>>>>>>> the fwk) >>>>>>>>>>> >>>>>>>>>>> 2010-10-01 23:10:17,403 Starting Felix 3.0.2 >>>>>>>>>>> 2010-10-01 23:10:23,885 Starting cluster node "test" >>>>>>>>>>> 2010-10-01 23:10:33,208 Cluster node "test" ready >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> - new fwk / fresh empty cache ->15.2 sec >>>>>>>>>>> >>>>>>>>>>> 2010-10-01 23:15:46,2 Starting Felix 3.1.0.SNAPSHOT >>>>>>>>>>> 2010-10-01 23:15:52,417 Starting cluster node "test" >>>>>>>>>>> 2010-10-01 23:16:01,225 Cluster node "test" ready >>>>>>>>>>> >>>>>>>>>>> hmmm, I don't understand why the fwk startup time is better >>>>>>>>>>> when the >>>>>>>>>>> >>>>>>>>>>> cache >>>>>>>>>> is empty (15.2 sec) than when the cache is non-empty (21.2 sec >>>>>>>>>> from >>>>>>>>>>> previous >>>>>>>>>>> mail) >>>>>>>>>>> Ok, I will do more tests ... may be I did something wrong ? ... >>>>>>>>>>> to be >>>>>>>>>>> continued ... >>>>>>>>>>> >>>>>>>>>>> Hmm. Yeah, that seems a little odd. In that case, you should >>>>>>>>>>> just >>>>>>>>>> delete >>>>>>>>>> your cache each time to get better startup performance! ;-) >>>>>>>>>> >>>>>>>>>> Definitely let me know what you find out. I too would expect >>>>>>>>>> framework >>>>>>>>>> restarts to be faster since we don't have to copy the JAR files >>>>>>>>>> first. >>>>>>>>>> >>>>>>>>>> -> richard >>>>>>>>>> >>>>>>>>>> /pierre >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Is that what you are doing? Just want to make sure I >>>>>>>>>>> understand what >>>>>>>>>>> is >>>>>>>>>>> >>>>>>>>>>>> being measured. :-) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -> richard >>>>>>>>>>>> >>>>>>>>>>>> On 10/1/10 16:43, Pierre De Rop wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Richard, >>>>>>>>>>>> >>>>>>>>>>>>> I tried our app server with the trunk (old cache with the new >>>>>>>>>>>>> >>>>>>>>>>>>> fwk/trunk) >>>>>>>>>>> -> >>>>>>>>>>> >>>>>>>>>>>> everything works ok >>>>>>>>>>>>> (I install 195 bundles, and start 41 bundles) >>>>>>>>>>>>> >>>>>>>>>>>>> With new cache/fwk trunk -> it's also ok. >>>>>>>>>>>>> >>>>>>>>>>>>> I also compared the performance of the startup time between >>>>>>>>>>>>> felix >>>>>>>>>>>>> >>>>>>>>>>>>> 2.0.2 >>>>>>>>>>> and >>>>>>>>>>> >>>>>>>>>>>> felix trunk. >>>>>>>>>>>>> To do this, I managed to ensure that my linux IO buffers >>>>>>>>>>>>> cache are >>>>>>>>>>>>> >>>>>>>>>>>>> empty >>>>>>>>>>> before doing each test, by doing this: >>>>>>>>>>> >>>>>>>>>>>> sync >>>>>>>>>>>>> echo 1> /proc/sys/vm/drop_caches >>>>>>>>>>>>> echo 2> /proc/sys/vm/drop_caches >>>>>>>>>>>>> echo 3> /proc/sys/vm/drop_caches >>>>>>>>>>>>> >>>>>>>>>>>>> This ensures that my linux has released all buffer cache >>>>>>>>>>>>> disk, like >>>>>>>>>>>>> it >>>>>>>>>>>>> is >>>>>>>>>>>>> the case when I boot my host (cold start). >>>>>>>>>>>>> Now, for felix 3.0.2, I load my server in around *21,1 >>>>>>>>>>>>> *seconds >>>>>>>>>>>>> (it's >>>>>>>>>>>>> slow >>>>>>>>>>>>> because all disk buffers are empty): >>>>>>>>>>>>> >>>>>>>>>>>>> 2010-10-01 22:15:07,880 Starting Felix 3.0.2 >>>>>>>>>>>>> 2010-10-01 22:15:18,121 Starting cluster node "test" >>>>>>>>>>>>> 2010-10-01 22:15:28,966 Cluster node "test" ready >>>>>>>>>>>>> >>>>>>>>>>>>> and with felix/trunk, indeed, the server starts a little bit >>>>>>>>>>>>> quicker, >>>>>>>>>>>>> but >>>>>>>>>>>>> not so much: around *in 20,3* seconds (I also flushed disk >>>>>>>>>>>>> buffers >>>>>>>>>>>>> before >>>>>>>>>>>>> starting the test): >>>>>>>>>>>>> >>>>>>>>>>>>> 2010-10-01 22:17:47,505 Starting Felix 3.1.0.SNAPSHOT >>>>>>>>>>>>> 2010-10-01 22:17:56,815 Starting cluster node "test" >>>>>>>>>>>>> 2010-10-01 22:18:07,798 Cluster node "test" ready >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> /pierre >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Oct 1, 2010 at 5:39 PM, Richard S. Hall< >>>>>>>>>>>>> heavy@ungoverned.org >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> p.s. I deploy snapshots of the everything, including the >>>>>>>>>>>>> convenience >>>>>>>>>>>>> >>>>>>>>>>>>> framework distribution or you can build from trunk. >>>>>>>>>>>>>> On 10/1/10 11:37, Richard S. Hall wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hey everyone, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've made some changes to the bundle cache layout and >>>>>>>>>>>>>> handling to >>>>>>>>>>>>>>> improve >>>>>>>>>>>>>>> performance and clean up code. I've tried to maintain >>>>>>>>>>>>>>> backward >>>>>>>>>>>>>>> compatibility >>>>>>>>>>>>>>> with old caches, but I'd appreciate is someone could try it >>>>>>>>>>>>>>> out >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>> caches to see if it works. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ** Back up your caches just in case! ** >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Note that the new cache format will not work with older >>>>>>>>>>>>>>> frameworks. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So >>>>>>>>>>>>> you >>>>>>>>>>>> cannot use a cache created with this new framework with >>>>>>>>>>>> previous >>>>>>>>>>>>>>> frameworks; >>>>>>>>>>>>>>> however, previously created caches should mostly continue >>>>>>>>>>>>>>> to work >>>>>>>>>>>>>>> between >>>>>>>>>>>>>>> the two although there will be a loss of fidelity since >>>>>>>>>>>>>>> changes >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> state >>>>>>>>>>>>>>> are >>>>>>>>>>>>>>> only saved to the new way or the old way depending on if >>>>>>>>>>>>>>> you are >>>>>>>>>>>>>>> running >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>> the new framework or the old one. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For the curious, I've combined five bundle cache files (id, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> location, >>>>>>>>>>>>> state, start level, last modified, and refresh count) into a >>>>>>>>>>>>> single >>>>>>>>>>>> file >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> try to avoid excessive file accesses. This appears to >>>>>>>>>>>>>>> improve >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> startup >>>>>>>>>>>>> time >>>>>>>>>>>> when you have a cache with lots of bundles, but your mileage >>>>>>>>>>>> will >>>>>>>>>>>>>>> vary >>>>>>>>>>>>> based >>>>>>>>>>>> on platform and other factors. Although, you won't necessarily >>>>>>>>>>>> see >>>>>>>>>>>>>>> any >>>>>>>>>>>>> improvements if you are using an old cache, since it will >>>>>>>>>>>>> revert to >>>>>>>>>>>> the >>>>>>>>>>>>>>> old >>>>>>>>>>>>>>> way. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -> richard >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >