From dev-return-23756-apmail-apr-dev-archive=apr.apache.org@apr.apache.org Sat Feb 19 02:35:26 2011 Return-Path: Delivered-To: apmail-apr-dev-archive@www.apache.org Received: (qmail 37301 invoked from network); 19 Feb 2011 02:35:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 19 Feb 2011 02:35:25 -0000 Received: (qmail 58198 invoked by uid 500); 19 Feb 2011 02:35:25 -0000 Delivered-To: apmail-apr-dev-archive@apr.apache.org Received: (qmail 57813 invoked by uid 500); 19 Feb 2011 02:35:23 -0000 Mailing-List: contact dev-help@apr.apache.org; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Id: Delivered-To: mailing list dev@apr.apache.org Received: (qmail 57805 invoked by uid 99); 19 Feb 2011 02:35:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Feb 2011 02:35:22 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gstein@gmail.com designates 209.85.216.50 as permitted sender) Received: from [209.85.216.50] (HELO mail-qw0-f50.google.com) (209.85.216.50) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Feb 2011 02:35:17 +0000 Received: by qwd6 with SMTP id 6so3982506qwd.37 for ; Fri, 18 Feb 2011 18:34:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=sEIObtQXs6Mz/G840t+NSPRBpFPMXDR02ysh517H3RY=; b=bIJSGyP28hyzPLNX5JJhjlImbFtOHgsESOa14jHh5gc7NmxatwHYPWztaFGx25WjRg bohNiq/uESpye+zc8FtLvr2QTxfAuAtv6WVEtiDm2nGeVM6Z34tmiNjYLASijxGPW1Gc w2MSSJCqRCVGJ6rdRM8Nx/sNa9zjWO5Tjrvv0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=tH/a0JGzQa0nsxSnb02G7vorV117nUURVxr3m/qJa4OAK9G8FUz/hiYBInLvjNzpIo tppZ1FlsFR4Sj+nVSHCCo+uzRYE1+9aq170OFjLbc8GHAYVmiR2Civ2Lr22BbaM4y1wQ yzX3OceGa6gGtufqs7aSp/XxnkhTYIlC2525E= MIME-Version: 1.0 Received: by 10.229.188.1 with SMTP id cy1mr1144269qcb.110.1298082895898; Fri, 18 Feb 2011 18:34:55 -0800 (PST) Received: by 10.229.92.21 with HTTP; Fri, 18 Feb 2011 18:34:55 -0800 (PST) In-Reply-To: <201102182256.00175.sf@sfritsch.de> References: <20110203202002.8FE1A23889E3@eris.apache.org> <201102062334.45009.sf@sfritsch.de> <201102182256.00175.sf@sfritsch.de> Date: Fri, 18 Feb 2011 21:34:55 -0500 Message-ID: Subject: Re: reducing memory fragmentation From: Greg Stein To: Stefan Fritsch Cc: dev@apr.apache.org Content-Type: text/plain; charset=ISO-8859-1 On Fri, Feb 18, 2011 at 16:55, Stefan Fritsch wrote: > On Thursday 17 February 2011, Jim Jagielski wrote: >> Please also look at Greg's pocore memory allocation stuff... >> his hash stuff is also quite nice. Would be useful to be >> able to use that as well.... > > That looks like a much bigger change than what I have just commited. > But I agree that before trying to optimize apr's allocator, one should > try pocore's. > > Have you thought about how to do this? Probably every apr pool would > be a wrapper for a pocore pool. But what about the other users of > apr_allocator, like bucket allocators? There is a branch in apr for wrapping pocore's pools and hash tables[1]. Obviously, the indirection slows it down, but it does demonstrate how it would work. (and it does: I've run the entire svn test suite using this wrapping) Just one comment: use pocore's child pools rather than its "post" notion. I need to get rid of the posts and simply speed up child pools. The basic problem with the apr allocator is its granularity. The reuse of memory is also a problem. pocore will store arbitrary-sized segments and unused fragments and whatnot into a red-black tree to quickly grab it and reuse it. My experiments show that mmap/munmap are NOT speed-advantageous on MacOS. But if you're looking at long-term memory usage and avoiding fragmentation... I don't have a good way to test that. That said, pocore should not be subject to fragmentation like apr. Its coalescing feature (designed w/ the APIs present, but not yet coded) will avoid much fragmentation. Cheers, -g [1] http://svn.apache.org/repos/asf/apr/apr/branches/gstein-pocore/ ... see memory/unix/apr_pools.c and tables/apr_hash.c