cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Bhardwaj <rahul.bhard...@indiamart.com>
Subject Re: error='Cannot allocate memory' (errno=12)
Date Mon, 11 May 2015 12:24:39 GMT
Hi Robert,

I saw somewhere you answering the same prob but no solution found. plz
check again.


regards:
Rahul Bhardwaj




On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:

> Bu it is giving same error with java 7 and open jdk
>
> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal <anishek@gmail.com>
> wrote:
>
>> Well i havent used 2.1.x cassandra neither java 8 but any reason for not
>> using oracle JDK as i thought thats what is recommended. i saw a thread
>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
>> 2.1.x versions.
>>
>>
>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> PFA of error log​
>>>  hs_err_pid9656.log
>>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>>> ​
>>>
>>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
>>> rahul.bhardwaj@indiamart.com> wrote:
>>>
>>>> free RAM:
>>>>
>>>>
>>>> free -m
>>>>              total       used       free     shared    buffers
>>>> cached
>>>> Mem:         64398      23753      40644          0        108
>>>> 8324
>>>> -/+ buffers/cache:      15319      49078
>>>> Swap:         2925         15       2909
>>>>
>>>>
>>>>  ulimit -a
>>>> core file size          (blocks, -c) 0
>>>> data seg size           (kbytes, -d) unlimited
>>>> scheduling priority             (-e) 0
>>>> file size               (blocks, -f) unlimited
>>>> pending signals                 (-i) 515041
>>>> max locked memory       (kbytes, -l) 64
>>>> max memory size         (kbytes, -m) unlimited
>>>> open files                      (-n) 1024
>>>> pipe size            (512 bytes, -p) 8
>>>> POSIX message queues     (bytes, -q) 819200
>>>> real-time priority              (-r) 0
>>>> stack size              (kbytes, -s) 10240
>>>> cpu time               (seconds, -t) unlimited
>>>> max user processes              (-u) 515041
>>>> virtual memory          (kbytes, -v) unlimited
>>>> file locks                      (-x) unlimited
>>>>
>>>>
>>>> Also attaching complete error file
>>>>
>>>>
>>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <anishek@gmail.com>
>>>> wrote:
>>>>
>>>>> the memory cassandra is trying to allocate is pretty small. you sure
>>>>> there is no hardware failure on the machine. what is the free ram on
the
>>>>> box ?
>>>>>
>>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> We have cluster of 3 nodes with  64GB RAM each. My cluster was
>>>>>> running in healthy state. Suddenly one machine's cassandra daemon
stops
>>>>>> working and shut down.
>>>>>>
>>>>>> On restarting it after 2 minutes it again stops and is getting stop
>>>>>> after returning below error in cassandra.log
>>>>>>
>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>>>>> allocate memory' (errno=12)
>>>>>> #
>>>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>>> continue.
>>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes
>>>>>> for committing reserved memory.
>>>>>> # An error report file with more information is saved as:
>>>>>> # /tmp/hs_err_pid23215.log
>>>>>> INFO  09:50:41 Loading settings from
>>>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>>>> INFO  09:50:41 Node
>>>>>> configuration:[authenticator=AllowAllAuthenticator;
>>>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>>>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>>>>> commit_failure_policy=stop;
>>>>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>>>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>>>>> concurrent_writes=32; counter_cache_save_period=7200;
>>>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>>>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>>>>> dynamic_snitch_reset_interval_in_ms=600000;
>>>>>> dynamic_snitch_update_interval_in_ms=100;
>>>>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>>>>> index_summary_capacity_in_mb=null;
>>>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>>>>> internode_compression=all; key_cache_save_period=14400;
>>>>>> key_cache_size_in_mb=null; listen_address=null;
>>>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>>>>> read_request_timeout_in_ms=999999999;
>>>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>>>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>>>>> rpc_port=9160; rpc_server_type=sync;
>>>>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>>>>> parameters=[{seeds=206.191.151.199}]}];
>>>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>>>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>>>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>>>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>>>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>>>>> write_request_timeout_in_ms=999999999]
>>>>>> ERROR 09:50:41 Exception encountered during startup
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>         at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>>>>>         at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>> ~[na:1.7.0_60]
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>> ~[na:1.7.0_60]
>>>>>>         at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>> ~[na:1.7.0_60]
>>>>>>         at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>> ~[na:1.7.0_60]
>>>>>>         at
>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>         at
>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>         at
>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>         at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>         at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>         at
>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>         at
>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>         at
>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>         at java.lang.Thread.start0(Native Method)
>>>>>>         at java.lang.Thread.start(Thread.java:714)
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>>         at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>>         at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>>         at
>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>>         at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>>         at
>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>>         at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>>         at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>>         at
>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>>         at
>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>>         at
>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>> Exception encountered during startup: unable to create new native
>>>>>> thread
>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>>>>> allocate memory' (errno=12
>>>>>>
>>>>>> This is our production cluster and rest 2 nodes are still in healthy
>>>>>> state.
>>>>>> We have also changed heap size from 10G to 20 G but problem still
>>>>>> persists. Please help in finding the reason behind this.
>>>>>>
>>>>>>
>>>>>> Regards:
>>>>>> Rahul Bhardwaj
>>>>>>
>>>>>>
>>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest
updates
>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>>> Mobile Channel:
>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>> <http://m.indiamart.com/>
>>>>>>
>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>> this and more: <https://plus.google.com/+indiamart>
>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>> Mobile Channel:
>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>> <http://m.indiamart.com/>
>>>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Check out some smart tips to generate more business - *IndiaMART
>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>
>>
>>
>

-- 

Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this 
and more: <https://plus.google.com/+indiamart> 
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile 
Channel: 
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>

<https://play.google.com/store/apps/details?id=com.indiamart.m> 
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Check out some smart tips to generate more business - *IndiaMART Learning 
Centre* <http://corporate.indiamart.com/quick-learn/>

Mime
View raw message