tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorge Medina <cerebrotecnolog...@gmail.com>
Subject Re: Java process killed by oom-killer in Ubuntu
Date Thu, 07 Jun 2012 18:54:36 GMT
Thanks Martin, but the memory error is not in the Java heap. Basically
my java process keeps growing far beyond the maximum java heap size
allowed.

-Jorge

On Wed, Jun 6, 2012 at 5:57 PM, Martin Gainty <mgainty@hotmail.com> wrote:
> neceista empezar tomcat con JMX por ejemplo:
> $CATALINA_HOME/bin/java -Dcom.sun.management.jmxremote -jar bootstrap.jar
>
> entonces..
>
> necesita monitorear heap y stack para proceso de TC durante la vida del
> proceso de tomcat con Jconsole
> por ejemplo
> $JAVA_HOME/bin/jconsole localhost:8080
> http://docs.oracle.com/javase/1.5.0/docs/guide/management/jconsole.html
>
> Saludos Cordiales desde EEUU
> Martin Gainty
> ______________________________________________
> Porfavor..no altere o disrupta esta communicacion..Gracias
>
>
>> Date: Wed, 6 Jun 2012 14:33:22 -0700
>> Subject: Java process killed by oom-killer in Ubuntu
>> From: cerebrotecnologico@gmail.com
>> To: users@tomcat.apache.org
>
>>
>> Hello,
>> I have an application that runs under Tomcat 7.0.23 that
>> periodically crashes. The java process running tomcat keeps growing in
>> memory until the Linux oom-killer kills the process. I do not get an
>> OutOfMemoryError because the memory leak is not in the Java heap. In
>> fact, it seems the heap is only using 4GB of the max 6GB specified in
>> the -Xmx parameter. Nevertheless the total memory held by the java
>> process keeps growing up to 16GB when the OS kills the process.
>> I haven't been able to find the conditions to reproduce this
>> problem, so I am not able to replicate it. Nevertheless it keeps
>> occurring; sometimes at midnight with no user activity sometimes in
>> the middle of a busy day.
>>
>> The web application uses Spring/Postgres/Mongo.
>>
>> I know this is not a Tomcat related problem, but some of you may
>> have experience a similar problem and may have some suggestions on how
>> to troubleshoot it.
>> I already have read many of the links that come after searching the
>> web for "java invoked oom-killer" but I still don't have any clue on
>> what causes the problem and how to solve it.
>> It looks like a memory leak in native code, not java code; so my
>> usual java toolset is not useful.
>>
>> Tomcat runs behind nginx in a EC2 instance. The application uses
>> Sun (now Oracle) JDK 1.6.
>>
>> Any suggestions on what should I look at?
>>
>>
>> -Jorge
>>
>>
>> Jun 4 16:02:49 ip-10-83-35-78 kernel: [1468800.179218] 3795110 pages
>> non-shared
>> Jun 5 06:50:07 ip-10-83-35-78 rsyslogd: [origin software="rsyslogd"
>> swVersion="4.2.0" x-pid="599" x-info="http://www.rsyslog.com"]
>> rsyslogd was HUPed, type 'lightweight'.
>> Jun 5 22:06:40 ip-10-83-35-78 kernel: [1576977.209487] java invoked
>> oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209492] java cpuset=/
>> mems_allowed=0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209496] Pid: 15618,
>> comm: java Not tainted 2.6.32-317-ec2 #36-Ubuntu
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209498] Call Trace:
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209507]
>> [<ffffffff8107cbbc>] ? cpuset_print_task_mems_allowed+0x8c/0xc0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209513]
>> [<ffffffff810b1723>] oom_kill_process+0xe3/0x210
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209516]
>> [<ffffffff810b18a0>] __out_of_memory+0x50/0xb0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209519]
>> [<ffffffff810b195f>] out_of_memory+0x5f/0xc0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209523]
>> [<ffffffff810b4641>] __alloc_pages_slowpath+0x561/0x580
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209526]
>> [<ffffffff810b47d1>] __alloc_pages_nodemask+0x171/0x180
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209530]
>> [<ffffffff810b76f7>] __do_page_cache_readahead+0xd7/0x220
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209534]
>> [<ffffffff810b785c>] ra_submit+0x1c/0x20
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209536]
>> [<ffffffff810b01fe>] filemap_fault+0x3fe/0x450
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209541]
>> [<ffffffff810cbef0>] __do_fault+0x50/0x680
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209547]
>> [<ffffffff8102afdb>] ? __dequeue_entity+0x2b/0x50
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209550]
>> [<ffffffff810cde30>] handle_mm_fault+0x260/0x4f0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209555]
>> [<ffffffff814b3ab7>] do_page_fault+0x147/0x390
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209558]
>> [<ffffffff814b18e8>] page_fault+0x28/0x30
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209560] Mem-Info:
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209561] DMA per-cpu:
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209563] CPU 0: hi:
>> 0, btch: 1 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209565] CPU 1: hi:
>> 0, btch: 1 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209567] CPU 2: hi:
>> 0, btch: 1 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209569] CPU 3: hi:
>> 0, btch: 1 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209570] DMA32 per-cpu:
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209572] CPU 0: hi:
>> 155, btch: 38 usd: 44
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209574] CPU 1: hi:
>> 155, btch: 38 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209575] CPU 2: hi:
>> 155, btch: 38 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209577] CPU 3: hi:
>> 155, btch: 38 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209578] Normal per-cpu:
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209580] CPU 0: hi:
>> 155, btch: 38 usd: 32
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209582] CPU 1: hi:
>> 155, btch: 38 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209584] CPU 2: hi:
>> 155, btch: 38 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209585] CPU 3: hi:
>> 155, btch: 38 usd: 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209590]
>> active_anon:3513144 inactive_anon:266669 isolated_anon:0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209591]
>> active_file:101 inactive_file:15 isolated_file:0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209592]
>> unevictable:16 dirty:2 writeback:0 unstable:0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209593] free:19129
>> slab_reclaimable:959 slab_unreclaimable:2729
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209594] mapped:0
>> shmem:52 pagetables:0 bounce:0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209600] DMA
>> free:16384kB min:16kB low:20kB high:24kB active_anon:0kB
>> inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB
>> isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB
>> dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>> slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
>> bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209604]
>> lowmem_reserve[]: 0 4024 15134 15134
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209611] DMA32
>> free:48712kB min:4184kB low:5228kB high:6276kB active_anon:3632260kB
>> inactive_anon:24572kB active_file:224kB inactive_file:28kB
>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>> present:4120800kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB
>> shmem:4kB slab_reclaimable:472kB slab_unreclaimable:240kB
>> kernel_stack:80kB pagetables:0kB unstable:0kB bounce:0kB
>> writeback_tmp:0kB pages_scanned:107 all_unreclaimable? no
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209616]
>> lowmem_reserve[]: 0 0 11109 11109
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209623] Normal
>> free:11420kB min:11548kB low:14432kB high:17320kB
>> active_anon:10420316kB inactive_anon:1042104kB active_file:180kB
>> inactive_file:32kB unevictable:64kB isolated(anon):0kB
>> isolated(file):0kB present:11376528kB mlocked:64kB dirty:8kB
>> writeback:0kB mapped:0kB shmem:204kB slab_reclaimable:3364kB
>> slab_unreclaimable:10676kB kernel_stack:1960kB pagetables:0kB
>> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:99
>> all_unreclaimable? no
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209628]
>> lowmem_reserve[]: 0 0 0 0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209631] DMA: 0*4kB
>> 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB
>> 4*4096kB = 16384kB
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209639] DMA32: 336*4kB
>> 194*8kB 141*16kB 97*32kB 73*64kB 57*128kB 28*256kB 18*512kB 6*1024kB
>> 1*2048kB 1*4096kB = 48896kB
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209646] Normal:
>> 953*4kB 23*8kB 11*16kB 14*32kB 12*64kB 11*128kB 8*256kB 2*512kB
>> 2*1024kB 0*2048kB 0*4096kB = 11916kB
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209653] 225 total
>> pagecache pages
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209654] 0 pages in swap
>> cache
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209656] Swap cache
>> stats: add 0, delete 0, find 0/0
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209657] Free swap = 0kB
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.209658] Total swap = 0kB
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252355] 3934208 pages RAM
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252359] 118009 pages
>> reserved
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252361] 749 pages shared
>> Jun 5 22:06:41 ip-10-83-35-78 kernel: [1576977.252362] 3795607 pages
>> non-shared
>> Jun 6 06:48:07 ip-10-83-35-78 rsyslogd: [origin software="rsyslogd"
>> swVersion="4.2.0" x-pid="599" x-info="http://www.rsyslog.com"]
>> rsyslogd was HUPed, type 'lightweight'.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message