cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulo Motta <pauloricard...@gmail.com>
Subject Re: Cassandra causing OOM Killer to strike on new cluster running 3.4
Date Wed, 20 Apr 2016 11:43:27 GMT
2.2.6 should be released in the next couple of weeks, but I also attached a
patch file to the issue if you want to patch 2.2.5 manually.

2016-04-20 0:19 GMT-03:00 Satoshi Hikida <satoshi@imagine-orb.com>:

> Hi,
>
> I'm looking forward to a  patch (file) for this bug(CASSANDRA-11344) to
> apply C* version 2.2.5. Is there available patch for that version? I
> watched link(https://issues.apache.org/jira/browse/CASSANDRA-11344) but
> couldn't find patch file or something like that.  Or is there any
> workaround to avoid this bug with version 2.2.5?
>
> Regards,
> Satoshi
>
> On Tue, Mar 15, 2016 at 4:47 AM, Adam Plumb <aplumb@fiksu.com> wrote:
>
>> OK so good news, I'm running with the patched jar file in my cluster and
>> haven't seen any issues.  The bloom filter off-heap memory usage is between
>> 1.5GB and 2GB per node, which is much more in-line with what I'm expecting!
>>  (thumbsup)
>>
>> On Mon, Mar 14, 2016 at 9:42 AM, Adam Plumb <aplumb@fiksu.com> wrote:
>>
>>> Thanks for the link!  Luckily the cluster I'm running is not yet in
>>> production and running with dummy data so I will throw that jar on the
>>> nodes and I'll let you know how things shake out.
>>>
>>> On Sun, Mar 13, 2016 at 11:02 PM, Paulo Motta <pauloricardomg@gmail.com>
>>> wrote:
>>>
>>>> You could be hitting CASSANDRA-11344 (
>>>> https://issues.apache.org/jira/browse/CASSANDRA-11344).  If that's the
>>>> case, you may try to replace your cassandra jar on an affected node with
a
>>>> version with this fix in place and force bloom filter regeneration to see
>>>> if if it fixes your problem. You can build with "ant jar" from this branch:
>>>> https://github.com/pauloricardomg/cassandra/tree/3.4-11344
>>>>
>>>> You can force bloom filter regeneration by either removing your
>>>> *Filter.db files (make sure to backup them before for safety) or changing
>>>> the bloom_filter_fp_chance before restarting affected nodes with the fixed
>>>> jar.
>>>>
>>>> 2016-03-13 19:51 GMT-03:00 Adam Plumb <aplumb@fiksu.com>:
>>>>
>>>>> So it's looking like the bloom filter off heap memory usage is ramping
>>>>> up and up until the OOM killer kills the java process.  I relaunched
on
>>>>> instances with 60GB of memory and the same thing is happening.  A node
will
>>>>> start using more and more RAM until the process is killed, then another
>>>>> node will start using more and more until it is also killed.
>>>>>
>>>>> Is this the expected behavior?  It doesn't seem ideal to me.  Is there
>>>>> anything obvious that I'm doing wrong?
>>>>>
>>>>> On Fri, Mar 11, 2016 at 11:31 AM, Adam Plumb <aplumb@fiksu.com>
wrote:
>>>>>
>>>>>> Here is the creation syntax for the entire schema.  The xyz table
has
>>>>>> about 2.1 billion keys and the def table has about 230 million keys.
 Max
>>>>>> row size is about 3KB, mean row size is 700B.
>>>>>>
>>>>>> CREATE KEYSPACE abc WITH replication = {'class':
>>>>>>> 'NetworkTopologyStrategy', 'us-east': 3};
>>>>>>> CREATE TABLE xyz (
>>>>>>>   id text,
>>>>>>>   secondary_id int,
>>>>>>>   data text,
>>>>>>>   PRIMARY KEY(id)
>>>>>>> )
>>>>>>>   WITH
>>>>>>>   compaction = { 'class': 'LeveledCompactionStrategy' }
>>>>>>>   and compression = {'class': 'LZ4Compressor'};
>>>>>>> CREATE INDEX secondary_id_index ON abc.xyz (secondary_id);
>>>>>>> CREATE TABLE def (
>>>>>>>   id text,
>>>>>>>   secondary_id int,
>>>>>>>   data text,
>>>>>>>   PRIMARY KEY(id)
>>>>>>> )
>>>>>>>   WITH
>>>>>>>   compaction = { 'class': 'LeveledCompactionStrategy' }
>>>>>>>   and compression = {'class': 'LZ4Compressor'};
>>>>>>> CREATE INDEX secondary_id_index_def ON abc.def (secondary_id);
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 11, 2016 at 11:24 AM, Jack Krupansky <
>>>>>> jack.krupansky@gmail.com> wrote:
>>>>>>
>>>>>>> What is your schema and data like - in particular, how wide are
your
>>>>>>> partitions (number of rows and typical row size)?
>>>>>>>
>>>>>>> Maybe you just need (a lot) more heap for rows during the repair
>>>>>>> process.
>>>>>>>
>>>>>>> -- Jack Krupansky
>>>>>>>
>>>>>>> On Fri, Mar 11, 2016 at 11:19 AM, Adam Plumb <aplumb@fiksu.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> These are brand new boxes only running Cassandra.  Yeah the
kernel
>>>>>>>> is what is killing the JVM, and this does appear to be a
memory leak in
>>>>>>>> Cassandra.  And Cassandra is the only thing running, aside
from the basic
>>>>>>>> services needed for Amazon Linux to run.
>>>>>>>>
>>>>>>>> On Fri, Mar 11, 2016 at 11:17 AM, Sebastian Estevez <
>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>
>>>>>>>>> Sacrifice child in dmesg is your OS killing the process
with the
>>>>>>>>> most ram. That means you're actually running out of memory
at the Linux
>>>>>>>>> level outside of the JVM.
>>>>>>>>>
>>>>>>>>> Are you running anything other than Cassandra on this
box?
>>>>>>>>>
>>>>>>>>> If so, does it have a memory leak?
>>>>>>>>>
>>>>>>>>> all the best,
>>>>>>>>>
>>>>>>>>> Sebastián
>>>>>>>>> On Mar 11, 2016 11:14 AM, "Adam Plumb" <aplumb@fiksu.com>
wrote:
>>>>>>>>>
>>>>>>>>>> I've got a new cluster of 18 nodes running Cassandra
3.4 that I
>>>>>>>>>> just launched and loaded data into yesterday (roughly
2TB of total storage)
>>>>>>>>>> and am seeing runaway memory usage.  These nodes
are EC2 c3.4xlarges with
>>>>>>>>>> 30GB RAM and the heap size is set to 8G with a new
heap size of 1.6G.
>>>>>>>>>>
>>>>>>>>>> Last night I finished loading up the data, then ran
an
>>>>>>>>>> incremental repair on one of the nodes just to ensure
that everything was
>>>>>>>>>> working (nodetool repair).  Over night all 18 nodes
ran out of memory and
>>>>>>>>>> were killed by the OOM killer.  I restarted them
this morning and they all
>>>>>>>>>> came up fine, but just started churning through memory
and got killed
>>>>>>>>>> again.  I restarted them again and they're doing
the same thing.  I'm not
>>>>>>>>>> getting any errors in the system log, since the process
is getting killed
>>>>>>>>>> abruptly (which makes me think this is a native memory
issue, not heap)
>>>>>>>>>>
>>>>>>>>>> Obviously this behavior isn't the best.  I'm willing
to provide
>>>>>>>>>> any data people need to help debug this, these nodes
are still up and
>>>>>>>>>> running.  I'm also in IRC if anyone wants to jump
on there.
>>>>>>>>>>
>>>>>>>>>> Here is the output of ps aux:
>>>>>>>>>>
>>>>>>>>>> 497       64351  108 89.5 187156072 27642988 ?  SLl
 15:13  62:15
>>>>>>>>>>> java -ea -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
>>>>>>>>>>> -XX:ThreadPriorityPolicy=42 -Xms7536M -Xmx7536M
-Xmn1600M
>>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003
>>>>>>>>>>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>>>>>>>>>>> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
>>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
>>>>>>>>>>> -XX:+UseTLAB -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45
>>>>>>>>>>> -XX:-ParallelRefProcEnabled -XX:-AlwaysPreTouch
-XX:+UseBiasedLocking
>>>>>>>>>>> -XX:+UseTLAB -XX:+ResizeTLAB -Djava.net.preferIPv4Stack=true
>>>>>>>>>>> -Dcom.sun.management.jmxremote.port=7199
>>>>>>>>>>> -Dcom.sun.management.jmxremote.rmi.port=7199
>>>>>>>>>>> -Dcom.sun.management.jmxremote.ssl=false
>>>>>>>>>>> -Dcom.sun.management.jmxremote.authenticate=false
>>>>>>>>>>> -XX:+CMSClassUnloadingEnabled -Dlogback.configurationFile=logback.xml
-D
>>>>>>>>>>> *cas*sandra.logdir=/usr/local/*cas*sandra/logs
-D*cas*
>>>>>>>>>>> sandra.storagedir=/usr/local/*cas*sandra/data
-D*cas*
>>>>>>>>>>> sandra-pidfile=/var/run/*cas*sandra/*cas*sandra.pid
-cp
>>>>>>>>>>> /usr/local/*cas*sandra/conf:/usr/local/*cas*
>>>>>>>>>>> sandra/build/classes/main:/usr/local/*cas*
>>>>>>>>>>> sandra/build/classes/thrift:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/airline-0.6.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/antlr-runtime-3.5.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/apache-*cas*sandra-3.4.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/apache-*cas*sandra-clientutil-3.4.jar:/usr/local/
>>>>>>>>>>> *cas*sandra/lib/apache-*cas*sandra-thrift-3.4.jar:/usr/local/
>>>>>>>>>>> *cas*sandra/lib/asm-5.0.4.jar:/usr/local/*cas*sandra/lib/*cas*
>>>>>>>>>>> sandra-driver-core-3.0.0-shaded.jar:/usr/local/*ca*
>>>>>>>>>>> sandra/lib/commons-cli-1.1.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/commons-codec-1.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/commons-lang3-3.1.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/commons-math3-3.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/compress-lzf-0.8.4.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/concurrent-trees-2.4.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/disruptor-3.0.1.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/ecj-4.4.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/guava-18.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/high-scale-lib-1.0.6.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/hppc-0.5.4.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/jackson-core-asl-1.9.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/jamm-0.3.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/javax.inject.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/jbcrypt-0.3m.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/jflex-1.6.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/jna-4.0.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/joda-time-2.4.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/json-simple-1.1.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/libthrift-0.9.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/logback-classic-1.1.3.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/logback-core-1.1.3.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/lz4-1.3.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/metrics-core-3.1.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/metrics-logback-3.1.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/netty-all-4.0.23.Final.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/ohc-core-0.4.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/ohc-core-j8-0.4.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/primitive-1.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/reporter-config3-3.0.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/reporter-config-base-3.0.0.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/sigar-1.6.4.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/slf4j-api-1.7.7.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/snakeyaml-1.11.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/snappy-java-1.1.1.7.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/ST4-4.0.8.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/stream-2.5.2.jar:/usr/local/*cas*
>>>>>>>>>>> sandra/lib/thrift-server-0.3.7.jar:/usr/local/*cas*sandra/lib/jsr223/*/*.jar
>>>>>>>>>>> org.apache.*cas*sandra.service.CassandraDaemon
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  Here is some dmesg output:
>>>>>>>>>>
>>>>>>>>>> [40003.010117] java invoked oom-killer: gfp_mask=0x280da,
>>>>>>>>>> order=0, oom_score_adj=0
>>>>>>>>>> [40003.013042] java cpuset=/ mems_allowed=0
>>>>>>>>>> [40003.014789] CPU: 3 PID: 37757 Comm: java Tainted:
G
>>>>>>>>>> E   4.1.7-15.23.amzn1.x86_64 #1
>>>>>>>>>> [40003.017852] Hardware name: Xen HVM domU, BIOS
4.2.amazon
>>>>>>>>>> 12/07/2015
>>>>>>>>>> [40003.020066]  0000000000000000 ffff8800ebaaba18
>>>>>>>>>> ffffffff814da12c 0000000000000000
>>>>>>>>>> [40003.022870]  ffff880763594c80 ffff8800ebaabac8
>>>>>>>>>> ffffffff814d7939 ffff8800ebaaba78
>>>>>>>>>> [40003.025674]  ffffffff811bf8f7 ffff880770679c00
>>>>>>>>>> ffff88077001c190 0000000000000080
>>>>>>>>>> [40003.028660] Call Trace:
>>>>>>>>>> [40003.029613]  [<ffffffff814da12c>] dump_stack+0x45/0x57
>>>>>>>>>> [40003.031486]  [<ffffffff814d7939>] dump_header+0x7f/0x1fe
>>>>>>>>>> [40003.033390]  [<ffffffff811bf8f7>] ? mem_cgroup_iter+0x137/0x3d0
>>>>>>>>>> [40003.035475]  [<ffffffff8107f496>] ? __queue_work+0x136/0x320
>>>>>>>>>> [40003.037594]  [<ffffffff8115d11c>] oom_kill_process+0x1cc/0x3b0
>>>>>>>>>> [40003.039825]  [<ffffffff8115d67e>] __out_of_memory+0x31e/0x530
>>>>>>>>>> [40003.041938]  [<ffffffff8115da2b>] out_of_memory+0x5b/0x80
>>>>>>>>>> [40003.043857]  [<ffffffff81162a79>]
>>>>>>>>>> __alloc_pages_nodemask+0x8a9/0x8d0
>>>>>>>>>> [40003.046105]  [<ffffffff811a48fa>]
>>>>>>>>>> alloc_page_interleave+0x3a/0x90
>>>>>>>>>> [40003.048419]  [<ffffffff811a79c3>] alloc_pages_vma+0x143/0x200
>>>>>>>>>> [40003.050582]  [<ffffffff81188035>] handle_mm_fault+0x1355/0x1770
>>>>>>>>>> [40003.052674]  [<ffffffff8118e4c5>] ? do_mmap_pgoff+0x2f5/0x3c0
>>>>>>>>>> [40003.054737]  [<ffffffff8105dafc>] __do_page_fault+0x17c/0x420
>>>>>>>>>> [40003.056858]  [<ffffffff8118c976>] ? SyS_mmap_pgoff+0x116/0x270
>>>>>>>>>> [40003.059082]  [<ffffffff8105ddc2>] do_page_fault+0x22/0x30
>>>>>>>>>> [40003.061084]  [<ffffffff814e2ad8>] page_fault+0x28/0x30
>>>>>>>>>> [40003.062938] Mem-Info:
>>>>>>>>>> [40003.063762] active_anon:5437903 inactive_anon:1025
>>>>>>>>>> isolated_anon:0
>>>>>>>>>>  active_file:51 inactive_file:8 isolated_file:0
>>>>>>>>>>  unevictable:2088582 dirty:0 writeback:0 unstable:0
>>>>>>>>>>  slab_reclaimable:82028 slab_unreclaimable:12209
>>>>>>>>>>  mapped:31065 shmem:20 pagetables:37089 bounce:0
>>>>>>>>>>  free:35830 free_pcp:3141 free_cma:0
>>>>>>>>>> [40003.075549] Node 0 DMA free:15872kB min:8kB low:8kB
high:12kB
>>>>>>>>>> active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB
>>>>>>>>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15988kB
>>>>>>>>>> managed:15904kB mlocked:0kB dirty:0kB writeback:0kB
mapped:0kB shmem:0kB
>>>>>>>>>> slab_reclaimable:32kB slab_unreclaimable:0kB kernel_stack:0kB
>>>>>>>>>> pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB
>>>>>>>>>> free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable?
yes
>>>>>>>>>> [40003.090267] lowmem_reserve[]: 0 3746 30128 30128
>>>>>>>>>> [40003.092182] Node 0 DMA32 free:108236kB min:2756kB
low:3444kB
>>>>>>>>>> high:4132kB active_anon:2400616kB inactive_anon:4060kB
active_file:0kB
>>>>>>>>>> inactive_file:0kB unevictable:1049732kB isolated(anon):0kB
>>>>>>>>>> isolated(file):0kB present:3915776kB managed:3840296kB
mlocked:1049732kB
>>>>>>>>>> dirty:4kB writeback:0kB mapped:16564kB shmem:12kB
slab_reclaimable:243852kB
>>>>>>>>>> slab_unreclaimable:8832kB kernel_stack:1152kB pagetables:16532kB
>>>>>>>>>> unstable:0kB bounce:0kB free_pcp:5716kB local_pcp:220kB
free_cma:0kB
>>>>>>>>>> writeback_tmp:0kB pages_scanned:5408 all_unreclaimable?
yes
>>>>>>>>>> [40003.108802] lowmem_reserve[]: 0 0 26382 26382
>>>>>>>>>> [40003.110578] Node 0 Normal free:19212kB min:19412kB
low:24264kB
>>>>>>>>>> high:29116kB active_anon:19350996kB inactive_anon:40kB
active_file:212kB
>>>>>>>>>> inactive_file:80kB unevictable:7304596kB isolated(anon):0kB
>>>>>>>>>> isolated(file):0kB present:27525120kB managed:27015196kB
mlocked:7304596kB
>>>>>>>>>> dirty:0kB writeback:0kB mapped:107696kB shmem:68kB
slab_reclaimable:84228kB
>>>>>>>>>> slab_unreclaimable:40004kB kernel_stack:10000kB pagetables:131824kB
>>>>>>>>>> unstable:0kB bounce:0kB free_pcp:6848kB local_pcp:692kB
free_cma:0kB
>>>>>>>>>> writeback_tmp:0kB pages_scanned:38332 all_unreclaimable?
yes
>>>>>>>>>> [40003.128300] lowmem_reserve[]: 0 0 0 0
>>>>>>>>>> [40003.129844] Node 0 DMA: 0*4kB 0*8kB 0*16kB 2*32kB
(UE) 3*64kB
>>>>>>>>>> (UE) 2*128kB (UE) 2*256kB (UE) 1*512kB (E) 2*1024kB
(UE) 2*2048kB (ER)
>>>>>>>>>> 2*4096kB (M) = 15872kB
>>>>>>>>>> [40003.135917] Node 0 DMA32: 193*4kB (UEM) 254*8kB
(UEM) 714*16kB
>>>>>>>>>> (UE) 1344*32kB (UEMR) 249*64kB (UEMR) 120*128kB (UER)
53*256kB (ER)
>>>>>>>>>> 10*512kB (ER) 1*1024kB (E) 0*2048kB 0*4096kB = 108244kB
>>>>>>>>>> [40003.142956] Node 0 Normal: 3956*4kB (UE) 0*8kB
1*16kB (R)
>>>>>>>>>> 8*32kB (R) 3*64kB (R) 2*128kB (R) 3*256kB (R) 0*512kB
0*1024kB 1*2048kB (R)
>>>>>>>>>> 0*4096kB = 19360kB
>>>>>>>>>> [40003.148749] Node 0 hugepages_total=0 hugepages_free=0
>>>>>>>>>> hugepages_surp=0 hugepages_size=2048kB
>>>>>>>>>> [40003.151777] 31304 total pagecache pages
>>>>>>>>>> [40003.153288] 0 pages in swap cache
>>>>>>>>>> [40003.154528] Swap cache stats: add 0, delete 0,
find 0/0
>>>>>>>>>> [40003.156377] Free swap  = 0kB
>>>>>>>>>> [40003.157423] Total swap = 0kB
>>>>>>>>>> [40003.158465] 7864221 pages RAM
>>>>>>>>>> [40003.159522] 0 pages HighMem/MovableOnly
>>>>>>>>>> [40003.160984] 146372 pages reserved
>>>>>>>>>> [40003.162244] [ pid ]   uid  tgid total_vm     
rss nr_ptes
>>>>>>>>>> nr_pmds swapents oom_score_adj name
>>>>>>>>>> [40003.165398] [ 2560]     0  2560     2804     
181      11
>>>>>>>>>>   3        0         -1000 udevd
>>>>>>>>>> [40003.168638] [ 3976]     0  3976     2334     
123       9
>>>>>>>>>>   3        0             0 dhclient
>>>>>>>>>> [40003.171895] [ 4017]     0  4017    11626     
 89      23
>>>>>>>>>>   4        0         -1000 auditd
>>>>>>>>>> [40003.175080] [ 4035]     0  4035    61861     
 99      23
>>>>>>>>>>   3        0             0 rsyslogd
>>>>>>>>>> [40003.178198] [ 4046]     0  4046     3462     
 98      10
>>>>>>>>>>   3        0             0 irqbalance
>>>>>>>>>> [40003.181559] [ 4052]     0  4052     1096     
 22       7
>>>>>>>>>>   3        0             0 rngd
>>>>>>>>>> [40003.184683] [ 4067]    32  4067     8815     
 99      22
>>>>>>>>>>   3        0             0 rpcbind
>>>>>>>>>> [40003.187772] [ 4084]    29  4084     9957     
201      24
>>>>>>>>>>   3        0             0 rpc.statd
>>>>>>>>>> [40003.191099] [ 4115]    81  4115     5442     
 60      15
>>>>>>>>>>   3        0             0 dbus-daemon
>>>>>>>>>> [40003.194438] [ 4333]     0  4333    19452     
522      40
>>>>>>>>>>   3        0         -1000 sshd
>>>>>>>>>> [40003.197432] [ 4361]    38  4361     7321     
562      19
>>>>>>>>>>   3        0             0 ntpd
>>>>>>>>>> [40003.200609] [ 4376]     0  4376    22238     
720      46
>>>>>>>>>>   3        0             0 sendmail
>>>>>>>>>> [40003.203868] [ 4384]    51  4384    20103     
674      41
>>>>>>>>>>   3        0             0 sendmail
>>>>>>>>>> [40003.206963] [ 4515]     0  4515     4267     
 38      13
>>>>>>>>>>   3        0             0 atd
>>>>>>>>>> [40003.210100] [ 6730]     0  6730    29888     
547      13
>>>>>>>>>>   3        0             0 crond
>>>>>>>>>> [40003.213267] [13533]   497 13533 47235415  7455314
  36074
>>>>>>>>>> 167        0             0 java
>>>>>>>>>> [40003.216364] [13674]   498 13674    49154     3168
     51
>>>>>>>>>>   3        0             0 supervisord
>>>>>>>>>> [40003.219721] [13680]   498 13680    51046     5350
     69
>>>>>>>>>>   3        0             0 python
>>>>>>>>>> [40003.222908] [13682]   498 13682    36172     5602
     75
>>>>>>>>>>   3        0             0 python
>>>>>>>>>> [40003.225952] [13683]   498 13683    32633     5319
     68
>>>>>>>>>>   3        0             0 python
>>>>>>>>>> [40003.229108] [13684]   498 13684    29577     5003
     63
>>>>>>>>>>   3        0             0 python
>>>>>>>>>> [40003.232263] [13719]   498 13719  1035920    41287
    234
>>>>>>>>>>   8        0             0 java
>>>>>>>>>> [40003.235287] [13753]   498 13753    34605     5645
     70
>>>>>>>>>>   3        0             0 python
>>>>>>>>>> [40003.238322] [14143]     0 14143     1615     
420       9
>>>>>>>>>>   3        0             0 agetty
>>>>>>>>>> [40003.241582] [14145]     0 14145     1078     
377       8
>>>>>>>>>>   3        0             0 mingetty
>>>>>>>>>> [40003.244752] [14147]     0 14147     1078     
354       8
>>>>>>>>>>   3        0             0 mingetty
>>>>>>>>>> [40003.247833] [14149]     0 14149     1078     
373       8
>>>>>>>>>>   3        0             0 mingetty
>>>>>>>>>> [40003.251193] [14151]     0 14151     1078     
367       7
>>>>>>>>>>   3        0             0 mingetty
>>>>>>>>>> [40003.254342] [14153]     0 14153     1078     
348       8
>>>>>>>>>>   3        0             0 mingetty
>>>>>>>>>> [40003.257443] [14154]     0 14154     2803     
182      10
>>>>>>>>>>   3        0         -1000 udevd
>>>>>>>>>> [40003.260688] [14155]     0 14155     2803     
182      10
>>>>>>>>>>   3        0         -1000 udevd
>>>>>>>>>> [40003.263782] [14157]     0 14157     1078     
369       8
>>>>>>>>>>   3        0             0 mingetty
>>>>>>>>>> [40003.266895] Out of memory: Kill process 13533
(java) score 970
>>>>>>>>>> or sacrifice child
>>>>>>>>>> [40003.269702] Killed process 13533 (java) total-vm:188941660kB,
>>>>>>>>>> anon-rss:29710828kB, file-rss:110428kB
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message