couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Herman Chan" <herman...@gmail.com>
Subject OOM and setting overcommit_memory on ubuntu
Date Thu, 24 Oct 2013 17:21:48 GMT
Hi guys,

Just got a OOM error this morning, the relevant syslog is attach below 
[1].

Was reading up on linux memory management and wondering if  setting 
overcommit_memory = 1 would help nor not?

Any suggestions?

Thanks,
Herman



[1]
Oct 24 12:17:29 ip-10-165-26-8 kernel: [18709846.900722] beam.smp 
invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900729] beam.smp 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900734] Pid: 8570, 
comm: beam.smp Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900737] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900749]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900753]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900757]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900762]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900767]  
[<ffffffff81157ee3>] alloc_pages_vma+0xb3/0x190
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900773]  
[<ffffffff8113880c>] do_anonymous_page.isra.38+0x7c/0x2f0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900777]  
[<ffffffff8113c291>] handle_pte_fault+0x1e1/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900783]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900787]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900791]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900796]  
[<ffffffff8100a56d>] ? xen_force_evtchn_callback+0xd/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900800]  
[<ffffffff8100ad42>] ? check_events+0x12/0x20
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900808]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900813]  
[<ffffffff81013878>] ? __switch_to+0x288/0x360
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900819]  
[<ffffffff810553ed>] ? set_next_entity+0xad/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900823]  
[<ffffffff81055593>] ? finish_task_switch+0x53/0xf0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900828]  
[<ffffffff8165505c>] ? __schedule+0x3cc/0x6f0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900832]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900835] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900837] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900840] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900843] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900845] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900848] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900850] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900853] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900855] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900857] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900860] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900861] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900864] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900866] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900869] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900871] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900876] 
active_anon:3704594 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900878]  active_file:42 
inactive_file:25 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900879]  unevictable:0 
dirty:0 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900880]  free:16939 
slab_reclaimable:46925 slab_unreclaimable:13210
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900881]  mapped:5 
shmem:44 pagetables:8838 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900884] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900895] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900900] Node 0 DMA32 
free:48400kB min:4176kB low:5220kB high:6264kB active_anon:3931268kB 
inactive_anon:0kB active_file:68kB inactive_file:16kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:47040kB 
slab_unreclaimable:5420kB kernel_stack:48kB pagetables:8544kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:18908 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900911] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900916] Node 0 Normal 
free:11476kB min:11548kB low:14432kB high:17320kB active_anon:10887108kB 
inactive_anon:144kB active_file:100kB inactive_file:84kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:8kB shmem:176kB slab_reclaimable:140660kB 
slab_unreclaimable:47420kB kernel_stack:808kB pagetables:26808kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:294 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900927] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900932] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900944] Node 0 DMA32: 
7848*4kB 1632*8kB 18*16kB 7*32kB 8*64kB 8*128kB 4*256kB 0*512kB 1*1024kB 
0*2048kB 0*4096kB = 48544kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900955] Node 0 Normal: 
1948*4kB 1*8kB 7*16kB 7*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 11464kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900967] 111 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900969] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900971] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900973] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.900974] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943923] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943928] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943929] 316 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943931] 3817610 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943933] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943950] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943967] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943975] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943981] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.943991] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944002] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944018] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944026] [  709]   101   
709    63526      982   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944032] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944039] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944045] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944053] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944060] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944065] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944071] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944081] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944089] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944095] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944110] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944122] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944135] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944144] [ 8556]     0  
8556  4053583  3694047   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944153] [ 8572]     0  
8572     1040       20   2       0             0 heart
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944159] [ 8575]     0  
8575     1100       26   3       0             0 sh
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944170] [ 8578]     0  
8578    28165     1377   1       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944175] [ 8579]     0  
8579     1845       21   3       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944184] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944194] [ 8582]     0  
8582    28160     1330   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944201] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944207] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944213] [11606]     0 
11606    27790      988   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944228] [11608]     0 
11608    27621      793   3       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944235] Out of memory: 
Kill process 8556 (beam.smp) score 935 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.944264] Killed process 
8575 (sh) total-vm:4400kB, anon-rss:104kB, file-rss:0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948375] beam.smp 
invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948381] beam.smp 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948385] Pid: 8569, 
comm: beam.smp Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948388] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948400]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948404]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948408]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948413]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948421]  
[<ffffffff81216c90>] ? noalloc_get_block_write+0x30/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948428]  
[<ffffffff811565b6>] alloc_pages_current+0xb6/0x120
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948434]  
[<ffffffff811164f7>] __page_cache_alloc+0xb7/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948438]  
[<ffffffff811184c2>] filemap_fault+0x212/0x3c0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948444]  
[<ffffffff81138af2>] __do_fault+0x72/0x550
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948450]  
[<ffffffff81013878>] ? __switch_to+0x288/0x360
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948454]  
[<ffffffff8113c1aa>] handle_pte_fault+0xfa/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948459]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948463]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948468]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948473]  
[<ffffffff8100a56d>] ? xen_force_evtchn_callback+0xd/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948477]  
[<ffffffff8100ad42>] ? check_events+0x12/0x20
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948484]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948488]  
[<ffffffff816574de>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948495]  
[<ffffffff811ba99c>] ? ep_poll+0x1cc/0x380
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948503]  
[<ffffffff8105f300>] ? try_to_wake_up+0x200/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948507]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948510] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948512] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948515] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948518] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948520] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948523] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948524] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948528] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948530] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948532] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948534] CPU    3: hi:  
186, btch:  31 usd:   7
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948536] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948539] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948541] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948544] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948546] CPU    3: hi:  
186, btch:  31 usd:  66
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948551] 
active_anon:3704626 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948552]  active_file:56 
inactive_file:7 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948554]  unevictable:0 
dirty:0 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948555]  free:16972 
slab_reclaimable:46925 slab_unreclaimable:13210
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948556]  mapped:1 
shmem:44 pagetables:8838 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948559] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948570] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948575] Node 0 DMA32 
free:48544kB min:4176kB low:5220kB high:6264kB active_anon:3931396kB 
inactive_anon:0kB active_file:76kB inactive_file:0kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:47040kB 
slab_unreclaimable:5420kB kernel_stack:48kB pagetables:8544kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:18984 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948590] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948595] Node 0 Normal 
free:11464kB min:11548kB low:14432kB high:17320kB active_anon:10887108kB 
inactive_anon:144kB active_file:148kB inactive_file:28kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:0kB shmem:176kB slab_reclaimable:140660kB 
slab_unreclaimable:47420kB kernel_stack:808kB pagetables:26808kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:434 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948606] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948611] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948623] Node 0 DMA32: 
7848*4kB 1632*8kB 18*16kB 7*32kB 8*64kB 8*128kB 4*256kB 0*512kB 1*1024kB 
0*2048kB 0*4096kB = 48544kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948634] Node 0 Normal: 
1917*4kB 1*8kB 7*16kB 7*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 11340kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948646] 107 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948648] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948650] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948652] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.948654] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992077] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992081] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992082] 316 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992084] 3817568 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992087] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992103] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992108] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992112] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992116] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992120] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992124] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992129] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992133] [  709]   101   
709    63526      982   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992137] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992140] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992144] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992148] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992152] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992155] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992159] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992163] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992167] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992171] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992175] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992179] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992183] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992188] [ 8556]     0  
8556  4053583  3694047   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992193] [ 8572]     0  
8572     1040       20   2       0             0 heart
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992197] [ 8578]     0  
8578    28165     1377   1       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992201] [ 8579]     0  
8579     1845       21   3       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992205] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992210] [ 8582]     0  
8582    28160     1330   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992214] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992219] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992223] [11606]     0 
11606    27790      988   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992227] [11608]     0 
11608    27621      793   3       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992231] Out of memory: 
Kill process 8556 (beam.smp) score 935 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709846.992258] Killed process 
8582 (couchjs) total-vm:112640kB, anon-rss:5316kB, file-rss:4kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937054] rsyslogd 
invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937061] rsyslogd 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937066] Pid: 724, comm: 
rsyslogd Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937069] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937081]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937085]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937089]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937094]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937102]  
[<ffffffff81216c90>] ? noalloc_get_block_write+0x30/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937109]  
[<ffffffff811565b6>] alloc_pages_current+0xb6/0x120
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937115]  
[<ffffffff811164f7>] __page_cache_alloc+0xb7/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937119]  
[<ffffffff811184c2>] filemap_fault+0x212/0x3c0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937125]  
[<ffffffff81138af2>] __do_fault+0x72/0x550
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937129]  
[<ffffffff8113c1aa>] handle_pte_fault+0xfa/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937135]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937139]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937144]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937151]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937155]  
[<ffffffff816572ce>] ? _raw_spin_lock+0xe/0x20
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937161]  
[<ffffffff81176bdd>] ? vfs_read+0x10d/0x180
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937165]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937168] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937170] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937173] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937176] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937178] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937180] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937182] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937185] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937188] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937190] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937192] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937194] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937197] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937199] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937201] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937204] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937209] 
active_anon:3704699 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937210]  active_file:39 
inactive_file:33 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937211]  unevictable:0 
dirty:0 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937212]  free:16925 
slab_reclaimable:46921 slab_unreclaimable:13207
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937213]  mapped:19 
shmem:44 pagetables:8781 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937216] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937227] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937232] Node 0 DMA32 
free:48512kB min:4176kB low:5220kB high:6264kB active_anon:3931440kB 
inactive_anon:0kB active_file:80kB inactive_file:64kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:47024kB 
slab_unreclaimable:5412kB kernel_stack:48kB pagetables:8544kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937244] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937249] Node 0 Normal 
free:11308kB min:11548kB low:14432kB high:17320kB active_anon:10887356kB 
inactive_anon:144kB active_file:76kB inactive_file:68kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:60kB shmem:176kB 
slab_reclaimable:140660kB slab_unreclaimable:47416kB kernel_stack:792kB 
pagetables:26580kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:223 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937260] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937265] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937276] Node 0 DMA32: 
7884*4kB 1623*8kB 13*16kB 6*32kB 7*64kB 7*128kB 5*256kB 0*512kB 1*1024kB 
0*2048kB 0*4096kB = 48568kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937288] Node 0 Normal: 
1947*4kB 2*8kB 7*16kB 8*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 11500kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937299] 172 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937301] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937304] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937306] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.937307] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980335] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980340] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980341] 319 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980343] 3817595 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980346] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980362] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980366] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980371] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980375] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980379] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980383] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980387] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980391] [  709]   101   
709    63526      982   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980395] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980399] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980402] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980406] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980410] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980413] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980417] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980421] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980424] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980428] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980433] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980436] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980441] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980446] [ 8556]     0  
8556  4053583  3695508   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980450] [ 8572]     0  
8572     1040       20   2       0             0 heart
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980454] [ 8578]     0  
8578    28165     1377   1       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980458] [ 8579]     0  
8579     1845       21   3       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980462] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980467] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980472] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980476] [11606]     0 
11606    27790      988   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980479] [11608]     0 
11608    27621      793   3       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980483] Out of memory: 
Kill process 8556 (beam.smp) score 935 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709848.980509] Killed process 
11606 (couchjs) total-vm:111160kB, anon-rss:3948kB, file-rss:4kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900770] beam.smp 
invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900777] beam.smp 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900782] Pid: 8570, 
comm: beam.smp Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900785] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900797]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900801]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900804]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900810]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900815]  
[<ffffffff81157ee3>] alloc_pages_vma+0xb3/0x190
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900821]  
[<ffffffff8113880c>] do_anonymous_page.isra.38+0x7c/0x2f0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900825]  
[<ffffffff8113c291>] handle_pte_fault+0x1e1/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900831]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900835]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900839]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900844]  
[<ffffffff8100a56d>] ? xen_force_evtchn_callback+0xd/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900849]  
[<ffffffff8100ad42>] ? check_events+0x12/0x20
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900856]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900862]  
[<ffffffff81013878>] ? __switch_to+0x288/0x360
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900868]  
[<ffffffff810553ed>] ? set_next_entity+0xad/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900871]  
[<ffffffff81055593>] ? finish_task_switch+0x53/0xf0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900876]  
[<ffffffff8165505c>] ? __schedule+0x3cc/0x6f0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900880]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900883] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900886] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900889] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900892] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900894] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900896] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900898] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900901] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900904] CPU    1: hi:  
186, btch:  31 usd:  48
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900906] CPU    2: hi:  
186, btch:  31 usd:  42
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900908] CPU    3: hi:  
186, btch:  31 usd:  30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900910] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900913] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900915] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900917] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900919] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900925] 
active_anon:3704909 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900926]  active_file:0 
inactive_file:23 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900927]  unevictable:0 
dirty:0 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900928]  free:16951 
slab_reclaimable:46921 slab_unreclaimable:13207
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900929]  mapped:0 
shmem:44 pagetables:8726 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900932] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900943] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900948] Node 0 DMA32 
free:48372kB min:4176kB low:5220kB high:6264kB active_anon:3932476kB 
inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:47024kB 
slab_unreclaimable:5412kB kernel_stack:48kB pagetables:8324kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:270 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900960] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900964] Node 0 Normal 
free:11552kB min:11548kB low:14432kB high:17320kB active_anon:10887160kB 
inactive_anon:144kB active_file:8kB inactive_file:108kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:28kB shmem:176kB 
slab_reclaimable:140660kB slab_unreclaimable:47416kB kernel_stack:792kB 
pagetables:26580kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:8553 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900976] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900981] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.900993] Node 0 DMA32: 
7762*4kB 1599*8kB 14*16kB 9*32kB 8*64kB 7*128kB 5*256kB 0*512kB 1*1024kB 
0*2048kB 0*4096kB = 48064kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.901004] Node 0 Normal: 
2000*4kB 3*8kB 7*16kB 8*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 11720kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.901016] 130 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.901018] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.901020] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.901022] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.901023] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944183] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944187] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944189] 339 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944191] 3817495 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944194] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944209] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944214] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944218] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944222] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944226] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944230] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944235] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944238] [  709]   101   
709    63526      982   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944242] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944246] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944250] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944254] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944257] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944261] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944265] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944268] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944272] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944276] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944280] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944284] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944289] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944293] [ 8556]     0  
8556  4053583  3696750   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944297] [ 8572]     0  
8572     1040       20   0       0             0 heart
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944302] [ 8578]     0  
8578    28165     1377   1       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944305] [ 8579]     0  
8579     1845       21   3       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944309] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944313] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944318] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944322] [11608]     0 
11608    27621      793   3       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944326] Out of memory: 
Kill process 8556 (beam.smp) score 936 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709849.944353] Killed process 
11608 (couchjs) total-vm:110484kB, anon-rss:3172kB, file-rss:0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960104] beam.smp 
invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960111] beam.smp 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960116] Pid: 8568, 
comm: beam.smp Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960119] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960130]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960135]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960138]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960143]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960151]  
[<ffffffff811565b6>] alloc_pages_current+0xb6/0x120
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960157]  
[<ffffffff811164f7>] __page_cache_alloc+0xb7/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960162]  
[<ffffffff811184c2>] filemap_fault+0x212/0x3c0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960167]  
[<ffffffff81138af2>] __do_fault+0x72/0x550
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960172]  
[<ffffffff8113c1aa>] handle_pte_fault+0xfa/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960178]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960182]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960186]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960194]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960200]  
[<ffffffff810553ed>] ? set_next_entity+0xad/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960206]  
[<ffffffff8109fcec>] ? do_futex+0x7c/0x1b0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960210]  
[<ffffffff8100a850>] ? xen_clocksource_read+0x20/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960214]  
[<ffffffff8109ff62>] ? sys_futex+0x142/0x1a0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960221]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960224] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960226] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960229] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960232] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960234] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960236] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960238] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960241] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960244] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960246] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960248] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960250] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960253] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960255] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960257] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960260] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960265] 
active_anon:3705233 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960266]  active_file:3 
inactive_file:63 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960267]  unevictable:0 
dirty:6 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960268]  free:16993 
slab_reclaimable:46921 slab_unreclaimable:13207
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960270]  mapped:9 
shmem:44 pagetables:8681 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960272] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960283] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960289] Node 0 DMA32 
free:48540kB min:4176kB low:5220kB high:6264kB active_anon:3933660kB 
inactive_anon:0kB active_file:4kB inactive_file:60kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:24kB writeback:0kB mapped:36kB shmem:0kB slab_reclaimable:47024kB 
slab_unreclaimable:5412kB kernel_stack:32kB pagetables:8140kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2376 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960300] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960305] Node 0 Normal 
free:11552kB min:11548kB low:14432kB high:17320kB active_anon:10887272kB 
inactive_anon:144kB active_file:8kB inactive_file:192kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:0kB shmem:176kB slab_reclaimable:140660kB 
slab_unreclaimable:47416kB kernel_stack:784kB pagetables:26584kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:619 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960317] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960321] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960333] Node 0 DMA32: 
7733*4kB 1625*8kB 18*16kB 22*32kB 9*64kB 8*128kB 5*256kB 0*512kB 
1*1024kB 0*2048kB 0*4096kB = 48828kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960345] Node 0 Normal: 
1964*4kB 3*8kB 7*16kB 8*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 11576kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960356] 110 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960358] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960360] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960362] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709850.960364] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003300] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003304] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003306] 317 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003307] 3817515 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003309] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003324] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003330] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003334] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003338] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003342] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003347] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003351] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003355] [  709]   101   
709    63526      986   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003359] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003363] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003367] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003370] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003374] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003378] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003381] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003385] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003389] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003393] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003397] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003401] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003405] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003410] [ 8556]     0  
8556  4053583  3697869   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003414] [ 8572]     0  
8572     1040       20   0       0             0 heart
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003419] [ 8578]     0  
8578    28165     1377   1       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003422] [ 8579]     0  
8579     1845       21   3       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003426] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003431] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003435] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003440] Out of memory: 
Kill process 8556 (beam.smp) score 936 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.003466] Killed process 
8572 (heart) total-vm:4160kB, anon-rss:80kB, file-rss:0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004172] beam.smp 
invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004176] beam.smp 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004181] Pid: 8568, 
comm: beam.smp Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004184] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004195]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004199]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004203]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004208]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004215]  
[<ffffffff811565b6>] alloc_pages_current+0xb6/0x120
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004221]  
[<ffffffff811164f7>] __page_cache_alloc+0xb7/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004226]  
[<ffffffff811184c2>] filemap_fault+0x212/0x3c0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004231]  
[<ffffffff81138af2>] __do_fault+0x72/0x550
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004235]  
[<ffffffff8113c1aa>] handle_pte_fault+0xfa/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004241]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004245]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004252]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004259]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004265]  
[<ffffffff810553ed>] ? set_next_entity+0xad/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004271]  
[<ffffffff8109fcec>] ? do_futex+0x7c/0x1b0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004276]  
[<ffffffff8100a850>] ? xen_clocksource_read+0x20/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004280]  
[<ffffffff8109ff62>] ? sys_futex+0x142/0x1a0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004284]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004287] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004289] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004293] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004295] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004298] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004300] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004302] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004305] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004307] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004310] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004312] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004314] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004316] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004319] CPU    1: hi:  
186, btch:  31 usd:  69
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004321] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004324] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004329] 
active_anon:3705233 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004330]  active_file:3 
inactive_file:63 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004331]  unevictable:0 
dirty:6 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004332]  free:16993 
slab_reclaimable:46921 slab_unreclaimable:13207
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004334]  mapped:9 
shmem:44 pagetables:8681 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004336] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004348] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004353] Node 0 DMA32 
free:48540kB min:4176kB low:5220kB high:6264kB active_anon:3933660kB 
inactive_anon:0kB active_file:4kB inactive_file:60kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:24kB writeback:0kB mapped:36kB shmem:0kB slab_reclaimable:47024kB 
slab_unreclaimable:5412kB kernel_stack:32kB pagetables:8140kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2544 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004365] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004369] Node 0 Normal 
free:11552kB min:11548kB low:14432kB high:17320kB active_anon:10887272kB 
inactive_anon:144kB active_file:8kB inactive_file:192kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:0kB shmem:176kB slab_reclaimable:140660kB 
slab_unreclaimable:47416kB kernel_stack:784kB pagetables:26584kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:619 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004381] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004386] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004398] Node 0 DMA32: 
7733*4kB 1625*8kB 18*16kB 22*32kB 9*64kB 8*128kB 5*256kB 0*512kB 
1*1024kB 0*2048kB 0*4096kB = 48828kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004410] Node 0 Normal: 
1933*4kB 3*8kB 7*16kB 8*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 11452kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004421] 110 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004424] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004426] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004428] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.004429] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047348] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047352] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047353] 317 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047355] 3817477 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047358] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047373] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047378] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047382] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047386] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047390] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047394] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047398] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047402] [  709]   101   
709    63526      986   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047406] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047410] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047414] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047418] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047422] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047425] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047429] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047433] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047436] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047441] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047445] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047449] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047453] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047458] [ 8556]     0  
8556  4053583  3697869   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047463] [ 8578]     0  
8578    28165     1377   1       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047467] [ 8579]     0  
8579     1845       21   3       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047471] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047476] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047480] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047485] Out of memory: 
Kill process 8556 (beam.smp) score 936 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709851.047508] Killed process 
8578 (couchjs) total-vm:112660kB, anon-rss:5504kB, file-rss:4kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905251] rs:main Q:Reg 
invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905258] rs:main Q:Reg 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905263] Pid: 718, comm: 
rs:main Q:Reg Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905266] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905278]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905282]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905286]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905291]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905300]  
[<ffffffff81216c90>] ? noalloc_get_block_write+0x30/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905307]  
[<ffffffff811565b6>] alloc_pages_current+0xb6/0x120
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905313]  
[<ffffffff811164f7>] __page_cache_alloc+0xb7/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905317]  
[<ffffffff811184c2>] filemap_fault+0x212/0x3c0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905323]  
[<ffffffff81138af2>] __do_fault+0x72/0x550
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905327]  
[<ffffffff8113c1aa>] handle_pte_fault+0xfa/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905333]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905337]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905341]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905345]  
[<ffffffff8100a56d>] ? xen_force_evtchn_callback+0xd/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905350]  
[<ffffffff8100ad42>] ? check_events+0x12/0x20
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905357]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905363]  
[<ffffffff8103dd15>] ? pvclock_clocksource_read+0x55/0xf0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905369]  
[<ffffffff8109fcec>] ? do_futex+0x7c/0x1b0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905374]  
[<ffffffff8100a939>] ? xen_clocksource_get_cycles+0x9/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905379]  
[<ffffffff8109ff62>] ? sys_futex+0x142/0x1a0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905383]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905386] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905388] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905392] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905394] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905397] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905399] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905401] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905404] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905407] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905409] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905411] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905413] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905416] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905418] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905420] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905423] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905428] 
active_anon:3705433 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905429]  active_file:62 
inactive_file:24 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905430]  unevictable:0 
dirty:0 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905431]  free:16810 
slab_reclaimable:46909 slab_unreclaimable:13235
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905433]  mapped:2 
shmem:44 pagetables:8622 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905435] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905447] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905452] Node 0 DMA32 
free:48500kB min:4176kB low:5220kB high:6264kB active_anon:3934052kB 
inactive_anon:0kB active_file:88kB inactive_file:84kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:47008kB 
slab_unreclaimable:5404kB kernel_stack:16kB pagetables:8140kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905463] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905468] Node 0 Normal 
free:10860kB min:11548kB low:14432kB high:17320kB active_anon:10887680kB 
inactive_anon:144kB active_file:160kB inactive_file:12kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:0kB shmem:176kB slab_reclaimable:140628kB 
slab_unreclaimable:47536kB kernel_stack:760kB pagetables:26348kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:258 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905480] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905484] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905496] Node 0 DMA32: 
7722*4kB 1612*8kB 21*16kB 19*32kB 9*64kB 8*128kB 5*256kB 0*512kB 
1*1024kB 0*2048kB 0*4096kB = 48632kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905508] Node 0 Normal: 
1796*4kB 23*8kB 0*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 10888kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905519] 77 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905522] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905524] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905526] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.905528] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948637] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948642] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948643] 320 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948645] 3817773 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948647] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948664] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948670] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948675] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948680] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948685] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948691] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948697] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948702] [  709]   101   
709    63526      986   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948708] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948712] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948716] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948719] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948723] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948727] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948730] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948734] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948738] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948742] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948746] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948751] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948755] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948760] [ 8556]     0  
8556  4053583  3699505   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948764] [ 8579]     0  
8579     1845       21   3       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948769] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948773] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948777] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948781] Out of memory: 
Kill process 8556 (beam.smp) score 936 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.948807] Killed process 
8579 (inet_gethost) total-vm:7380kB, anon-rss:84kB, file-rss:0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952808] rs:main Q:Reg 
invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952814] rs:main Q:Reg 
cpuset=/ mems_allowed=0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952819] Pid: 718, comm: 
rs:main Q:Reg Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952822] Call Trace:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952834]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952838]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952841]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952846]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952854]  
[<ffffffff81216c90>] ? noalloc_get_block_write+0x30/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952861]  
[<ffffffff811565b6>] alloc_pages_current+0xb6/0x120
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952867]  
[<ffffffff811164f7>] __page_cache_alloc+0xb7/0xd0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952872]  
[<ffffffff811184c2>] filemap_fault+0x212/0x3c0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952877]  
[<ffffffff81138af2>] __do_fault+0x72/0x550
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952881]  
[<ffffffff8113c1aa>] handle_pte_fault+0xfa/0x200
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952887]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952892]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952896]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952900]  
[<ffffffff8100a56d>] ? xen_force_evtchn_callback+0xd/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952905]  
[<ffffffff8100ad42>] ? check_events+0x12/0x20
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952912]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952918]  
[<ffffffff8103dd15>] ? pvclock_clocksource_read+0x55/0xf0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952924]  
[<ffffffff8109fcec>] ? do_futex+0x7c/0x1b0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952929]  
[<ffffffff8100a939>] ? xen_clocksource_get_cycles+0x9/0x10
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952932]  
[<ffffffff8109ff62>] ? sys_futex+0x142/0x1a0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952937]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952940] Mem-Info:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952942] Node 0 DMA 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952945] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952948] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952950] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952953] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952955] Node 0 DMA32 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952958] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952960] CPU    1: hi:  
186, btch:  31 usd:  29
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952963] CPU    2: hi:  
186, btch:  31 usd:  30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952965] CPU    3: hi:  
186, btch:  31 usd:  30
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952967] Node 0 Normal 
per-cpu:
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952970] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952972] CPU    1: hi:  
186, btch:  31 usd:  56
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952974] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952977] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952982] 
active_anon:3705434 inactive_anon:36 isolated_anon:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952983]  active_file:13 
inactive_file:97 isolated_file:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952984]  unevictable:0 
dirty:0 writeback:0 unstable:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952985]  free:16788 
slab_reclaimable:46909 slab_unreclaimable:13251
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952987]  mapped:1 
shmem:44 pagetables:8622 bounce:0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.952989] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953001] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953006] Node 0 DMA32 
free:48384kB min:4176kB low:5220kB high:6264kB active_anon:3934056kB 
inactive_anon:0kB active_file:72kB inactive_file:196kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:47008kB 
slab_unreclaimable:5404kB kernel_stack:16kB pagetables:8140kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1037 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953018] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953022] Node 0 Normal 
free:10888kB min:11548kB low:14432kB high:17320kB active_anon:10887680kB 
inactive_anon:144kB active_file:0kB inactive_file:192kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:0kB shmem:176kB slab_reclaimable:140628kB 
slab_unreclaimable:47600kB kernel_stack:760kB pagetables:26348kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:813 
all_unreclaimable? yes
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953034] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953039] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953051] Node 0 DMA32: 
7715*4kB 1603*8kB 16*16kB 13*32kB 9*64kB 8*128kB 5*256kB 0*512kB 
1*1024kB 0*2048kB 0*4096kB = 48260kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953063] Node 0 Normal: 
1765*4kB 23*8kB 0*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 10764kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953075] 105 total 
pagecache pages
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953077] 0 pages in swap 
cache
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953079] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953081] Free swap  = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.953083] Total swap = 
0kB
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996017] 3934192 pages 
RAM
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996021] 99281 pages 
reserved
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996023] 303 pages 
shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996024] 3817735 pages 
non-shared
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996028] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996047] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996052] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996056] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996060] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996064] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996068] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996073] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996077] [  709]   101   
709    63526      986   2       0             0 rsyslogd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996081] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996085] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996089] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996093] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996096] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996100] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996104] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996107] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996111] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996115] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996119] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996124] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996128] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996133] [ 8556]     0  
8556  4053583  3699505   1       0             0 beam.smp
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996137] [ 8580]     0  
8580     1845       28   0       0             0 inet_gethost
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996141] [ 8583]     0  
8583    28411     1600   0       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996145] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996149] Out of memory: 
Kill process 8556 (beam.smp) score 936 or sacrifice child
Oct 24 12:17:36 ip-10-165-26-8 kernel: [18709856.996171] Killed process 
8583 (couchjs) total-vm:113644kB, anon-rss:6396kB, file-rss:4kB
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966663] inet_gethost 
invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, 
oom_score_adj=0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966670] inet_gethost 
cpuset=/ mems_allowed=0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966674] Pid: 8580, 
comm: inet_gethost Not tainted 3.2.0-40-virtual #64-Ubuntu
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966677] Call Trace:
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966689]  
[<ffffffff811195d1>] dump_header+0x91/0xe0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966693]  
[<ffffffff81119955>] oom_kill_process+0x85/0xb0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966697]  
[<ffffffff81119cfa>] out_of_memory+0xfa/0x220
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966702]  
[<ffffffff8111f6e3>] __alloc_pages_nodemask+0x8c3/0x8e0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966710]  
[<ffffffff81216c90>] ? noalloc_get_block_write+0x30/0x30
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966717]  
[<ffffffff811565b6>] alloc_pages_current+0xb6/0x120
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966723]  
[<ffffffff811164f7>] __page_cache_alloc+0xb7/0xd0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966728]  
[<ffffffff811184c2>] filemap_fault+0x212/0x3c0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966733]  
[<ffffffff81138af2>] __do_fault+0x72/0x550
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966737]  
[<ffffffff8113a2d2>] ? do_wp_page+0x382/0x740
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966741]  
[<ffffffff8113c1aa>] handle_pte_fault+0xfa/0x200
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966747]  
[<ffffffff8100638e>] ? xen_pmd_val+0xe/0x10
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966751]  
[<ffffffff81005309>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966755]  
[<ffffffff8113d419>] handle_mm_fault+0x269/0x370
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966762]  
[<ffffffff8165adcb>] do_page_fault+0x14b/0x520
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966769]  
[<ffffffff8129b433>] ? security_file_permission+0x93/0xb0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966774]  
[<ffffffff811766a1>] ? rw_verify_area+0x61/0xf0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966778]  
[<ffffffff81176b80>] ? vfs_read+0xb0/0x180
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966782]  
[<ffffffff81657a35>] page_fault+0x25/0x30
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966785] Mem-Info:
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966787] Node 0 DMA 
per-cpu:
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966790] CPU    0: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966793] CPU    1: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966795] CPU    2: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966798] CPU    3: hi:   
  0, btch:   1 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966800] Node 0 DMA32 
per-cpu:
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966803] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966805] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966807] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966810] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966811] Node 0 Normal 
per-cpu:
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966814] CPU    0: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966817] CPU    1: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966819] CPU    2: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966822] CPU    3: hi:  
186, btch:  31 usd:   0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966827] 
active_anon:3705378 inactive_anon:36 isolated_anon:0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966828]  active_file:49 
inactive_file:20 isolated_file:0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966829]  unevictable:0 
dirty:0 writeback:0 unstable:0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966831]  free:16963 
slab_reclaimable:46904 slab_unreclaimable:13246
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966832]  mapped:39 
shmem:44 pagetables:8562 bounce:0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966835] Node 0 DMA 
free:7880kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB 
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:7624kB mlocked:0kB dirty:0kB writeback:0kB 
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB 
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966846] 
lowmem_reserve[]: 0 4016 15112 15112
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966851] Node 0 DMA32 
free:48500kB min:4176kB low:5220kB high:6264kB active_anon:3933992kB 
inactive_anon:0kB active_file:28kB inactive_file:24kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:4112640kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:136kB shmem:0kB slab_reclaimable:47008kB 
slab_unreclaimable:5404kB kernel_stack:16kB pagetables:8152kB 
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3814 
all_unreclaimable? yes
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966863] 
lowmem_reserve[]: 0 0 11095 11095
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966868] Node 0 Normal 
free:11472kB min:11548kB low:14432kB high:17320kB active_anon:10887520kB 
inactive_anon:144kB active_file:168kB inactive_file:56kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:11362176kB mlocked:0kB 
dirty:0kB writeback:0kB mapped:20kB shmem:176kB 
slab_reclaimable:140608kB slab_unreclaimable:47580kB kernel_stack:736kB 
pagetables:26096kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:2464 all_unreclaimable? yes
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966880] 
lowmem_reserve[]: 0 0 0 0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966884] Node 0 DMA: 
2*4kB 2*8kB 1*16kB 3*32kB 3*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 
2*2048kB 0*4096kB = 7880kB
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966897] Node 0 DMA32: 
7754*4kB 1622*8kB 23*16kB 12*32kB 8*64kB 7*128kB 6*256kB 0*512kB 
1*1024kB 0*2048kB 0*4096kB = 48712kB
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966908] Node 0 Normal: 
1957*4kB 4*8kB 7*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 
1*2048kB 0*4096kB = 11492kB
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966920] 153 total 
pagecache pages
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966922] 0 pages in swap 
cache
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966924] Swap cache 
stats: add 0, delete 0, find 0/0
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966926] Free swap  = 
0kB
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709857.966928] Total swap = 
0kB
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009722] 3934192 pages 
RAM
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009732] 99281 pages 
reserved
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009743] 318 pages 
shared
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009748] 3817572 pages 
non-shared
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009753] [ pid ]   uid  
tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009770] [  308]     0   
308     4308       48   1       0             0 upstart-udev-br
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009778] [  310]     0   
310     5384      138   0     -17         -1000 udevd
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009787] [  363]     0   
363     5366      104   0     -17         -1000 udevd
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009795] [  364]     0   
364     5366      104   3     -17         -1000 udevd
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009801] [  454]     0   
454     3797       47   0       0             0 upstart-socket-
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009808] [  503]     0   
503     1816      126   0       0             0 dhclient3
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009815] [  645]     0   
645    12489      153   2     -17         -1000 sshd
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009820] [  709]   101   
709    63526     1027   2       0             0 rsyslogd
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009827] [  716]   102   
716     5979       79   0       0             0 dbus-daemon
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009833] [  770]     0   
770     3626       42   0       0             0 getty
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009838] [  776]     0   
776     3626       39   1       0             0 getty
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009843] [  781]     0   
781     3626       42   0       0             0 getty
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009848] [  782]     0   
782     3626       40   3       0             0 getty
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009856] [  785]     0   
785     3626       42   3       0             0 getty
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009861] [  789]     0   
789     1082       35   3       0             0 acpid
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009867] [  790]     0   
790     4778       54   2       0             0 cron
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009875] [  791]     0   
791     4227       41   0       0             0 atd
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009881] [  873]   103   
873    46899      306   1       0             0 whoopsie
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009886] [  892]     0   
892     3626       39   2       0             0 getty
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009894] [ 8494]     0  
8494     1100       38   0       0             0 couchdb
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009902] [ 8555]     0  
8555     1100       39   1       0             0 couchdb
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009916] [ 8556]     0  
8556  4053527  3700994   1       0             0 beam.smp
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009922] [ 8580]     0  
8580     1845       28   1       0             0 inet_gethost
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009932] [11605]     0 
11605    28309     1504   2       0             0 couchjs
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009938] Out of memory: 
Kill process 8556 (beam.smp) score 937 or sacrifice child
Oct 24 12:17:37 ip-10-165-26-8 kernel: [18709858.009968] Killed process 
11605 (couchjs) total-vm:113236kB, anon-rss:6012kB, file-rss:4kB

Mime
View raw message