cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "J. Ryan Earl" <...@jryanearl.us>
Subject Re: error='Cannot allocate memory' (errno=12)
Date Wed, 13 May 2015 01:08:12 GMT
I see your ulimit -a above, missed that.  You should increase nofile
ulimit.  If you used JNA, you'd need to increase memlock too, but you
probably aren't using JNA.  1024 nofile is default and far too small, try
making that like 64K.  Thread handles can count against file descriptors
limits, similar to how pipes, sockets, etc can count against file
descriptor limits.

On Tue, May 12, 2015 at 7:55 PM, J. Ryan Earl <oss@jryanearl.us> wrote:

> What's your ulimit -a output?  Did you adjust nproc and nofile ulimits
> up?  Do you have JNA installed?  What about memlock ulimit and in
> sysctl.conf => kernel.shmmax?
>
> What's in cassandra.log?
>
> On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> Hi Robert,
>>
>> I saw somewhere you answering the same prob but no solution found. plz
>> check again.
>>
>>
>> regards:
>> Rahul Bhardwaj
>>
>>
>>
>>
>> On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> Bu it is giving same error with java 7 and open jdk
>>>
>>> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal <anishek@gmail.com>
>>> wrote:
>>>
>>>> Well i havent used 2.1.x cassandra neither java 8 but any reason for
>>>> not using oracle JDK as i thought thats what is recommended. i saw a thread
>>>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
>>>> 2.1.x versions.
>>>>
>>>>
>>>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>
>>>>> PFA of error log​
>>>>>  hs_err_pid9656.log
>>>>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>>>>> ​
>>>>>
>>>>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>>> free RAM:
>>>>>>
>>>>>>
>>>>>> free -m
>>>>>>              total       used       free     shared    buffers
>>>>>> cached
>>>>>> Mem:         64398      23753      40644          0        108
>>>>>> 8324
>>>>>> -/+ buffers/cache:      15319      49078
>>>>>> Swap:         2925         15       2909
>>>>>>
>>>>>>
>>>>>>  ulimit -a
>>>>>> core file size          (blocks, -c) 0
>>>>>> data seg size           (kbytes, -d) unlimited
>>>>>> scheduling priority             (-e) 0
>>>>>> file size               (blocks, -f) unlimited
>>>>>> pending signals                 (-i) 515041
>>>>>> max locked memory       (kbytes, -l) 64
>>>>>> max memory size         (kbytes, -m) unlimited
>>>>>> open files                      (-n) 1024
>>>>>> pipe size            (512 bytes, -p) 8
>>>>>> POSIX message queues     (bytes, -q) 819200
>>>>>> real-time priority              (-r) 0
>>>>>> stack size              (kbytes, -s) 10240
>>>>>> cpu time               (seconds, -t) unlimited
>>>>>> max user processes              (-u) 515041
>>>>>> virtual memory          (kbytes, -v) unlimited
>>>>>> file locks                      (-x) unlimited
>>>>>>
>>>>>>
>>>>>> Also attaching complete error file
>>>>>>
>>>>>>
>>>>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <anishek@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> the memory cassandra is trying to allocate is pretty small. you
sure
>>>>>>> there is no hardware failure on the machine. what is the free
ram on the
>>>>>>> box ?
>>>>>>>
>>>>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> We have cluster of 3 nodes with  64GB RAM each. My cluster
was
>>>>>>>> running in healthy state. Suddenly one machine's cassandra
daemon stops
>>>>>>>> working and shut down.
>>>>>>>>
>>>>>>>> On restarting it after 2 minutes it again stops and is getting
stop
>>>>>>>> after returning below error in cassandra.log
>>>>>>>>
>>>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>>>>>>> allocate memory' (errno=12)
>>>>>>>> #
>>>>>>>> # There is insufficient memory for the Java Runtime Environment
to
>>>>>>>> continue.
>>>>>>>> # Native memory allocation (malloc) failed to allocate 12288
bytes
>>>>>>>> for committing reserved memory.
>>>>>>>> # An error report file with more information is saved as:
>>>>>>>> # /tmp/hs_err_pid23215.log
>>>>>>>> INFO  09:50:41 Loading settings from
>>>>>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>>>>>> INFO  09:50:41 Node
>>>>>>>> configuration:[authenticator=AllowAllAuthenticator;
>>>>>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>>>>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>>>>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>>>>>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>>>>>>> commit_failure_policy=stop;
>>>>>>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>>>>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>>>>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>>>>>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>>>>>>> concurrent_writes=32; counter_cache_save_period=7200;
>>>>>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>>>>>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>>>>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>>>>>>> dynamic_snitch_reset_interval_in_ms=600000;
>>>>>>>> dynamic_snitch_update_interval_in_ms=100;
>>>>>>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>>>>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>>>>>>> index_summary_capacity_in_mb=null;
>>>>>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>>>>>>> internode_compression=all; key_cache_save_period=14400;
>>>>>>>> key_cache_size_in_mb=null; listen_address=null;
>>>>>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>>>>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>>>>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>>>>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>>>>>>> read_request_timeout_in_ms=999999999;
>>>>>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>>>>>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>>>>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>>>>>>> rpc_port=9160; rpc_server_type=sync;
>>>>>>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>>>>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>>>>>>> parameters=[{seeds=206.191.151.199}]}];
>>>>>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>>>>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>>>>>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>>>>>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>>>>>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>>>>>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>>>>>>> write_request_timeout_in_ms=999999999]
>>>>>>>> ERROR 09:50:41 Exception encountered during startup
>>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>>         at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>>>>>>>         at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>>         at java.lang.Thread.start0(Native Method)
>>>>>>>>         at java.lang.Thread.start(Thread.java:714)
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>>>>         at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>>>>         at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>>>> Exception encountered during startup: unable to create new
native
>>>>>>>> thread
>>>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>>>>>>> allocate memory' (errno=12
>>>>>>>>
>>>>>>>> This is our production cluster and rest 2 nodes are still
in
>>>>>>>> healthy state.
>>>>>>>> We have also changed heap size from 10G to 20 G but problem
still
>>>>>>>> persists. Please help in finding the reason behind this.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards:
>>>>>>>> Rahul Bhardwaj
>>>>>>>>
>>>>>>>>
>>>>>>>> Follow IndiaMART.com <http://www.indiamart.com> for
latest updates
>>>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>>>> <https://www.facebook.com/IndiaMART>
>>>>>>>> <https://twitter.com/IndiaMART> Mobile Channel:
>>>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>>>> <http://m.indiamart.com/>
>>>>>>>>
>>>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates
on
>>>>> this and more: <https://plus.google.com/+indiamart>
>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>> Mobile Channel:
>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>> <http://m.indiamart.com/>
>>>>>
>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>>
>>>>
>>>>
>>>
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Check out some smart tips to generate more business - *IndiaMART
>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>
>
>

Mime
View raw message