Return-Path: Delivered-To: apmail-hbase-user-archive@www.apache.org Received: (qmail 39829 invoked from network); 12 Jan 2011 06:47:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 12 Jan 2011 06:47:34 -0000 Received: (qmail 21039 invoked by uid 500); 12 Jan 2011 06:47:33 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 20897 invoked by uid 500); 12 Jan 2011 06:47:30 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 20889 invoked by uid 99); 12 Jan 2011 06:47:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 06:47:29 +0000 X-ASF-Spam-Status: No, hits=3.7 required=10.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of octo47@gmail.com designates 209.85.161.41 as permitted sender) Received: from [209.85.161.41] (HELO mail-fx0-f41.google.com) (209.85.161.41) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 06:47:22 +0000 Received: by fxm12 with SMTP id 12so259189fxm.14 for ; Tue, 11 Jan 2011 22:47:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=LoAai2NtC0h34C0sn4YJ39GKb1aTO6bUsYteq0UD7c8=; b=Zbtf4MlL24rc2X6h2RpC3Xp+2pythbn1Xoja+0uR0TFfaQiZqNrtWxOS6L8Ch9+OdQ 4jeSEyKX/7lk0TThHTIOZnqKYyxz6MmBExe6hBIy2TkKmwnSR4SFiRxs2EyC2+fx1eE4 8r7I+X2ONQ3xYRnQAWrtR5NCeMf7MaJE3GSMw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=uk92DaDCQdIs/CfzRYzejQ+Fs3oUk4wIMPV5WdcTnyUN3QykV7kIpAtJaBsDNfXOaH yEdq+O489sTYAfc3ZTi5xWgXiKWYZz+hs+bm9RwualDj+0ACF4rvdDtAB8YrSEJkvRJ0 +EfPKWQs2aD2wki5clLlozxZhP5aP6MbOmgYY= Received: by 10.223.87.67 with SMTP id v3mr549551fal.130.1294814821589; Tue, 11 Jan 2011 22:47:01 -0800 (PST) MIME-Version: 1.0 Received: by 10.223.117.67 with HTTP; Tue, 11 Jan 2011 22:46:41 -0800 (PST) In-Reply-To: References: <61DF20EF-BF0A-4160-B06F-A4E8B9745B6D@xebia.com> <4D2C88AB.4000706@mozilla.com> From: Andrey Stepachev Date: Wed, 12 Jan 2011 09:46:41 +0300 Message-ID: Subject: Re: Java Commited Virtual Memory significally larged then Heap Memory To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=20cf304345487da4230499a091d9 X-Virus-Checked: Checked by ClamAV on apache.org --20cf304345487da4230499a091d9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable My bad. All things work. Thanks for Todd Lipcon :) 2011/1/11 Andrey Stepachev > I tried to set MALLOC_ARENA_MAX=3D2. But still the same issue like in LZO > problem thread. All those 65M blocks here. And JVM continues to eat memor= y > on heavy write load. And yes, I use "improved" kernel > Linux 2.6.34.7-0.5. > > 2011/1/11 Xavier Stevens > > Are you using a newer linux kernel with the new and "improved" memory >> allocator? >> >> If so try setting this in hadoop-env.sh: >> >> export MALLOC_ARENA_MAX=3D >> >> Maybe start by setting it to 4. You can thank Todd Lipcon if this works >> for you. >> >> Cheers, >> >> >> -Xavier >> >> On 1/11/11 7:24 AM, Andrey Stepachev wrote: >> > No. I don't use LZO. I tried even remove any native support (i.e. all >> .so >> > from class path) >> > and use java gzip. But nothing. >> > >> > >> > 2011/1/11 Friso van Vollenhoven >> > >> >> Are you using LZO by any chance? If so, which version? >> >> >> >> Friso >> >> >> >> >> >> On 11 jan 2011, at 15:57, Andrey Stepachev wrote: >> >> >> >>> After starting the hbase in jro=D1=81kit found the same memory leaka= ge. >> >>> >> >>> After the launch >> >>> >> >>> Every 2,0 s: date & & ps - sort =3D- rss-eopid, rss, vsz, pcpu | hea= d >> >>> Tue Jan 11 16:49:31 2011 >> >>> >> >>> 11 16:49:31 MSK 2011 >> >>> PID RSS VSZ% CPU >> >>> 7863 2547760 5576744 78.7 >> >>> >> >>> >> >>> >> >>> JR dumps: >> >>> >> >>> Total mapped 5576740KB (reserved =3D 2676404KB) - Java heap 2048000K= B >> >>> (reserved =3D 1472176KB) - GC tables 68512KB - Thread stacks 37236KB= (# >> >>> threads =3D 111) - Compiled code 1048576KB (used =3D 2599KB) - Inter= nal >> >>> 1224KB - OS 549688KB - Other 1800976KB - Classblocks 1280KB (malloce= d >> >>> =3D 1110KB # 3285) - Java class data 20224KB (malloced =3D 20002KB #= 15134 >> >>> in 3285 classes) - Native memory tracking 1024KB (malloced =3D 325KB= +10 >> >>> KB # 20) >> >>> >> >>> >> >>> >> >>> After running the mr which make high write load (~1hour) >> >>> >> >>> Every 2,0 s: date & & ps - sort =3D- rss-eopid, rss, vsz, pcpu | hea= d >> >>> Tue Jan 11 17:08:56 2011 >> >>> >> >>> 11 17:08:56 MSK 2011 >> >>> PID RSS VSZ% CPU >> >>> 7863 4072396 5459572 100 >> >>> >> >>> >> >>> >> >>> JR said not important below specify why) >> >>> >> >>> http://paste.ubuntu.com/552820/ >> >>> >> >>> >> >>> >> >>> 7863: >> >>> Total mapped 5742628KB +165888KB (reserved=3D114400= 0KB >> >>> -1532404KB) >> >>> - Java heap 2048000KB (reserved=3D0KB >> >> -1472176KB) >> >>> - GC tables 68512KB >> >>> - Thread stacks 38028KB +792KB (#threads=3D114 +3= ) >> >>> - Compiled code 1048576KB (used=3D3376KB +77= 6KB) >> >>> - Internal 1480KB +256KB >> >>> - OS 517944KB -31744KB >> >>> - Other 1996792KB +195816KB >> >>> - Classblocks 1280KB (malloced=3D1156KB >> >>> +45KB #3421 +136) >> >>> - Java class data 20992KB +768KB (malloced=3D20843K= B >> >>> +840KB #15774 +640 in 3421 classes) >> >>> - Native memory tracking 1024KB (malloced=3D325KB >> +10KB >> >> #20) >> >>> >> >>> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++ >> >>> OS *java r x 0x0000000000400000.( >> >> 76KB) >> >>> OS *java rw 0x0000000000612000 ( >> >> 4KB) >> >>> OS *[heap] rw 0x0000000000613000.( >> >> 478712KB) >> >>> INT Poll r 0x000000007fffe000 ( >> >> 4KB) >> >>> INT Membar rw 0x000000007ffff000.( >> >> 4KB) >> >>> MSP Classblocks (1/2) rw 0x0000000082ec0000 ( >> >> 768KB) >> >>> MSP Classblocks (2/2) rw 0x0000000082f80000 ( >> >> 512KB) >> >>> HEAP Java heap rw >> >> 0x0000000083000000.(2048000KB) >> >>> rw 0x00007f2574000000 ( >> >> 65500KB) >> >>> 0x00007f2577ff7000.( >> >> 36KB) >> >>> rw 0x00007f2584000000 ( >> >> 65492KB) >> >>> 0x00007f2587ff5000.( >> >> 44KB) >> >>> rw 0x00007f258c000000 ( >> >> 65500KB) >> >>> 0x00007f258fff7000 ( >> >> 36KB) >> >>> rw 0x00007f2590000000 ( >> >> 65500KB) >> >>> 0x00007f2593ff7000 ( >> >> 36KB) >> >>> rw 0x00007f2594000000 ( >> >> 65500KB) >> >>> 0x00007f2597ff7000 ( >> >> 36KB) >> >>> rw 0x00007f2598000000 ( >> >> 131036KB) >> >>> 0x00007f259fff7000 ( >> >> 36KB) >> >>> rw 0x00007f25a0000000 ( >> >> 65528KB) >> >>> 0x00007f25a3ffe000 ( >> >> 8KB) >> >>> rw 0x00007f25a4000000 ( >> >> 65496KB) >> >>> 0x00007f25a7ff6000 ( >> >> 40KB) >> >>> rw 0x00007f25a8000000 ( >> >> 65496KB) >> >>> 0x00007f25abff6000 ( >> >> 40KB) >> >>> rw 0x00007f25ac000000 ( >> >> 65504KB) >> >>> >> >>> >> >>> So, the difference was in the pieces of memory like this: >> >>> >> >>> rw 0x00007f2590000000 (65500KB) >> >>> 0x00007f2593ff7000 (36KB) >> >>> >> >>> >> >>> Looks like HLog allocates memory (looks like HLog, becase it is very >> >> similar >> >>> size) >> >>> >> >>> If we count this blocks we get amount of lost memory: >> >>> >> >>> 65M * 32 + 132M =3D 2212M >> >>> >> >>> So, it looks like HLog allcates to many memory, and question is: how >> to >> >>> restrict it? >> >>> >> >>> 2010/12/30 Andrey Stepachev >> >>> >> >>>> Hi All. >> >>>> >> >>>> After heavy load into hbase (single node, nondistributed test syste= m) >> I >> >> got >> >>>> 4Gb process size of my HBase java process. >> >>>> On 6GB machine there was no room for anything else (disk cache and = so >> >> on). >> >>>> Does anybody knows, what is going on, and how you solve this. What >> heap >> >>>> memory is set on you hosts >> >>>> and how much of RSS hbase process actually use. >> >>>> >> >>>> I don't see such things before, all tomcat and other java apps don'= t >> >> eats >> >>>> significally more memory then -Xmx. >> >>>> >> >>>> Connection name: pid: 23476 org.apache.hadoop.hbase.master.HMaste= r >> >>>> start Virtual Machine: Java HotSpot(TM) 64-Bit Server VM versio= n >> >>>> 17.1-b03 Vendor: Sun Microsystems Inc. Name: 23476@mars >> >> Uptime: 12 >> >>>> hours 4 minutes Process CPU time: 5 hours 45 minutes JIT >> compiler: >> >> HotSpot >> >>>> 64-Bit Server Compiler Total compile time: 19,223 seconds >> >>>> ------------------------------ >> >>>> Current heap size: 703 903 kbytes Maximum heap size: 2 0= 30 >> >> 976kbytes Committed memory: >> >>>> 2 030 976 kbytes Pending finalization: 0 objects Garbage >> >>>> collector: Name =3D 'ParNew', Collections =3D 9 990, Total time s= pent =3D >> 5 >> >>>> minutes Garbage collector: Name =3D 'ConcurrentMarkSweep', >> Collections >> >> =3D >> >>>> 20, Total time spent =3D 35,754 seconds >> >>>> ------------------------------ >> >>>> Operating System: Linux 2.6.34.7-0.5-xen Architecture: amd= 64 >> >> Number of processors: >> >>>> 8 Committed virtual memory: 4 403 512 kbytes Total physical >> >>>> memory: 6 815 744 kbytes Free physical memory: 82 720 kbyt= es >> >> Total swap space: >> >>>> 8 393 924 kbytes Free swap space: 8 050 880 kbytes >> >>>> >> >>>> >> >>>> >> >>>> >> >> >> > > --20cf304345487da4230499a091d9--