Return-Path: X-Original-To: apmail-tomcat-users-archive@www.apache.org Delivered-To: apmail-tomcat-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C176DC15C for ; Thu, 7 Jun 2012 18:51:56 +0000 (UTC) Received: (qmail 38679 invoked by uid 500); 7 Jun 2012 18:51:53 -0000 Delivered-To: apmail-tomcat-users-archive@tomcat.apache.org Received: (qmail 38615 invoked by uid 500); 7 Jun 2012 18:51:53 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 38604 invoked by uid 99); 7 Jun 2012 18:51:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jun 2012 18:51:53 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cerebrotecnologico@gmail.com designates 209.85.214.45 as permitted sender) Received: from [209.85.214.45] (HELO mail-bk0-f45.google.com) (209.85.214.45) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jun 2012 18:51:48 +0000 Received: by bkwj10 with SMTP id j10so1173605bkw.18 for ; Thu, 07 Jun 2012 11:51:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=yYX17riEnvifaFLYuJuxPw/UzyBlPNc9uz6YgYxA+yM=; b=dwkbA0+sZzi267HDVjqmLK5EqaOQIhtOz84HqIifHmpA1soD3lqey/+E+sAZmZmMhy JSEkwuulNQq0w7spwVuNDTN+ccRFXLFpQlB5nVYD10gZGCkAjv07Cl0kf8YemDbAR8Az EiToRbo2FWd7SHsloVczG4z/fGGN6sK/yOuOWGiMCjWmVCCOqXBznWhQTIvBtP+HRmNO Pl/vYXHKg0gdnFfxcz/ZXlnCPCPC+oynEa6Ro9gTckwLoBVpCoyTjq92OIM500EDSAaa ii+g0M2A3UGjha2Clran8IgOoxly4jpLjRUb74vBqKOnAhqk815bEwWablXR0blq1KEV P0wg== MIME-Version: 1.0 Received: by 10.204.153.193 with SMTP id l1mr645431bkw.120.1339095086525; Thu, 07 Jun 2012 11:51:26 -0700 (PDT) Received: by 10.223.157.1 with HTTP; Thu, 7 Jun 2012 11:51:26 -0700 (PDT) In-Reply-To: References: Date: Thu, 7 Jun 2012 11:51:26 -0700 Message-ID: Subject: Re: Java process killed by oom-killer in Ubuntu From: Jorge Medina To: users@tomcat.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org It seems my system is running with an relatively old version of the Tomcat Native Library and old versions of APR and OpenSSL. Tomcat Native 1.1.19 APR 1.3.8 OpenSSL 0.9.8k These are the latest versions available: Tomcat Native 1.1.23 APR 1.4.6 OpenSSL 1.0.1c I will try disabling the Tomcat Native Library to see if this solves the problem. Has anyone experience problems of Tomcat 7 running with these versions? -Jorge On Wed, Jun 6, 2012 at 2:33 PM, Jorge Medina wrote: > Hello, > =A0 I have an application that runs under Tomcat 7.0.23 that > periodically crashes. The java process running tomcat keeps growing in > memory until the Linux oom-killer kills the process. I do not get an > OutOfMemoryError because the memory leak is not in the Java heap. In > fact, it seems the heap is only using 4GB of the max 6GB specified in > the -Xmx parameter. Nevertheless the total memory held by the java > process keeps growing up to 16GB when the OS kills the process. > =A0 =A0I haven't been able to find the conditions to reproduce this > problem, so I am not able to replicate it. Nevertheless it keeps > occurring; sometimes at midnight with no user activity sometimes in > the middle of a busy day. > > =A0 The web application uses Spring/Postgres/Mongo. > > =A0 I know this is not a Tomcat related problem, but some of you may > have experience a similar problem and may have some suggestions on how > to troubleshoot it. > =A0 I already have read many of the links that come after searching the > web for "java invoked oom-killer" but I still don't have any clue on > what causes the problem and how to solve it. > =A0 It looks like a memory leak in native code, not java code; so my > usual java toolset is not useful. > > =A0 Tomcat runs behind nginx in a EC2 instance. The application uses > Sun (now Oracle) JDK 1.6. > > =A0 Any suggestions on what should I look at? > > > -Jorge > > > Jun =A04 16:02:49 ip-10-83-35-78 kernel: [1468800.179218] 3795110 pages n= on-shared > Jun =A05 06:50:07 ip-10-83-35-78 rsyslogd: [origin software=3D"rsyslogd" > swVersion=3D"4.2.0" x-pid=3D"599" x-info=3D"http://www.rsyslog.com"] > rsyslogd was HUPed, type 'lightweight'. > Jun =A05 22:06:40 ip-10-83-35-78 kernel: [1576977.209487] java invoked > oom-killer: gfp_mask=3D0x201da, order=3D0, oom_adj=3D0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209492] java cpuset=3D/ > mems_allowed=3D0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209496] Pid: 15618, > comm: java Not tainted 2.6.32-317-ec2 #36-Ubuntu > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209498] Call Trace: > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209507] > [] ? cpuset_print_task_mems_allowed+0x8c/0xc0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209513] > [] oom_kill_process+0xe3/0x210 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209516] > [] __out_of_memory+0x50/0xb0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209519] > [] out_of_memory+0x5f/0xc0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209523] > [] __alloc_pages_slowpath+0x561/0x580 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209526] > [] __alloc_pages_nodemask+0x171/0x180 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209530] > [] __do_page_cache_readahead+0xd7/0x220 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209534] > [] ra_submit+0x1c/0x20 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209536] > [] filemap_fault+0x3fe/0x450 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209541] > [] __do_fault+0x50/0x680 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209547] > [] ? __dequeue_entity+0x2b/0x50 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209550] > [] handle_mm_fault+0x260/0x4f0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209555] > [] do_page_fault+0x147/0x390 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209558] > [] page_fault+0x28/0x30 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209560] Mem-Info: > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209561] DMA per-cpu: > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209563] CPU =A0 =A00: h= i: > =A00, btch: =A0 1 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209565] CPU =A0 =A01: h= i: > =A00, btch: =A0 1 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209567] CPU =A0 =A02: h= i: > =A00, btch: =A0 1 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209569] CPU =A0 =A03: h= i: > =A00, btch: =A0 1 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209570] DMA32 per-cpu: > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209572] CPU =A0 =A00: h= i: > 155, btch: =A038 usd: =A044 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209574] CPU =A0 =A01: h= i: > 155, btch: =A038 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209575] CPU =A0 =A02: h= i: > 155, btch: =A038 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209577] CPU =A0 =A03: h= i: > 155, btch: =A038 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209578] Normal per-cpu: > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209580] CPU =A0 =A00: h= i: > 155, btch: =A038 usd: =A032 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209582] CPU =A0 =A01: h= i: > 155, btch: =A038 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209584] CPU =A0 =A02: h= i: > 155, btch: =A038 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209585] CPU =A0 =A03: h= i: > 155, btch: =A038 usd: =A0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209590] > active_anon:3513144 inactive_anon:266669 isolated_anon:0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209591] > active_file:101 inactive_file:15 isolated_file:0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209592] > unevictable:16 dirty:2 writeback:0 unstable:0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209593] =A0free:19129 > slab_reclaimable:959 slab_unreclaimable:2729 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209594] =A0mapped:0 > shmem:52 pagetables:0 bounce:0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209600] DMA > free:16384kB min:16kB low:20kB high:24kB active_anon:0kB > inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB > dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB > slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB > bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209604] > lowmem_reserve[]: 0 4024 15134 15134 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209611] DMA32 > free:48712kB min:4184kB low:5228kB high:6276kB active_anon:3632260kB > inactive_anon:24572kB active_file:224kB inactive_file:28kB > unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:4120800kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB > shmem:4kB slab_reclaimable:472kB slab_unreclaimable:240kB > kernel_stack:80kB pagetables:0kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:107 all_unreclaimable? no > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209616] > lowmem_reserve[]: 0 0 11109 11109 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209623] Normal > free:11420kB min:11548kB low:14432kB high:17320kB > active_anon:10420316kB inactive_anon:1042104kB active_file:180kB > inactive_file:32kB unevictable:64kB isolated(anon):0kB > isolated(file):0kB present:11376528kB mlocked:64kB dirty:8kB > writeback:0kB mapped:0kB shmem:204kB slab_reclaimable:3364kB > slab_unreclaimable:10676kB kernel_stack:1960kB pagetables:0kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:99 > all_unreclaimable? no > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209628] > lowmem_reserve[]: 0 0 0 0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209631] DMA: 0*4kB > 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB > 4*4096kB =3D 16384kB > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209639] DMA32: 336*4kB > 194*8kB 141*16kB 97*32kB 73*64kB 57*128kB 28*256kB 18*512kB 6*1024kB > 1*2048kB 1*4096kB =3D 48896kB > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209646] Normal: > 953*4kB 23*8kB 11*16kB 14*32kB 12*64kB 11*128kB 8*256kB 2*512kB > 2*1024kB 0*2048kB 0*4096kB =3D 11916kB > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209653] 225 total > pagecache pages > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209654] 0 pages in swap= cache > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209656] Swap cache > stats: add 0, delete 0, find 0/0 > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209657] Free swap =A0= =3D 0kB > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.209658] Total swap =3D = 0kB > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.252355] 3934208 pages R= AM > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.252359] 118009 pages re= served > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.252361] 749 pages share= d > Jun =A05 22:06:41 ip-10-83-35-78 kernel: [1576977.252362] 3795607 pages n= on-shared > Jun =A06 06:48:07 ip-10-83-35-78 rsyslogd: [origin software=3D"rsyslogd" > swVersion=3D"4.2.0" x-pid=3D"599" x-info=3D"http://www.rsyslog.com"] > rsyslogd was HUPed, type 'lightweight'. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org