Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A7F0E18622 for ; Mon, 11 May 2015 10:34:24 +0000 (UTC) Received: (qmail 8029 invoked by uid 500); 11 May 2015 10:34:22 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 7985 invoked by uid 500); 11 May 2015 10:34:22 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 7975 invoked by uid 99); 11 May 2015 10:34:22 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 May 2015 10:34:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C9490182223 for ; Mon, 11 May 2015 10:34:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.911 X-Spam-Level: ** X-Spam-Status: No, score=2.911 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, T_REMOTE_IMAGE=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=indiamart.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id IN-4iyJuw_Z3 for ; Mon, 11 May 2015 10:34:10 +0000 (UTC) Received: from mail-ig0-f182.google.com (mail-ig0-f182.google.com [209.85.213.182]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id ACD9124A48 for ; Mon, 11 May 2015 10:34:09 +0000 (UTC) Received: by igbpi8 with SMTP id pi8so69349199igb.0 for ; Mon, 11 May 2015 03:34:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=indiamart.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=HPWNoVnT3/i9p0KL/PeT3WMuKtEBLorFXCDKkRfvMlE=; b=g2c0jeoRgG589QEYpgRbjcX6I8deEc2EzAjnQveTjxMY2aVGkcemcQU3DGa32QdiAa ucSbULwZPcI17JvcfryCOWPUGAK2NlOlv6AKwpgO+6fDNPkSB4I540YzFxUEZFPfgBZd TlQQBekKmIPbjU4YQyw0+rpYvDiL0K7bOx6Sc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=HPWNoVnT3/i9p0KL/PeT3WMuKtEBLorFXCDKkRfvMlE=; b=OdcLxHHE7LacrSzyhik3168a+NeTnpp0PaATFTXSlJIzU8YhYWJ1OzAMfvCykILB7q RnODBqpYn1WMDKijQTTLtlhFkHqKiZ5xg28SfthFMZki3o/DagIKPHddszifY5BdMsxQ VaACR2S9TJ+bbtUHxtJQP1r8MhnHvszljU3DFO98TKXXVo0jJURGdvX0ihBIQ1Q51irz L+E9bnWkr+et8f5/hCB8DIrWXfdTNuMc53qZ9E+HGIB90HJG1Iknc4YW7AgCIpq7eeIz IeZMjqTmoykXvjaBogc7Xxbb3ecAWKovlO2LBa27vl5t96A0YF/LZ1QLp/sSsRsmlTe6 +aug== X-Gm-Message-State: ALoCoQkgd5031GSYez0ttByRmImMTrpCdr1wlNY9fc2Lv8feDn87q/2uTbbreDeQN9I0d6G2JaF6vHaJ6zXEtL4v9YYYCp25VFw2WkD634uRSLnFkBi1Rh5Z5LRrkES0uRgCll/7vRop MIME-Version: 1.0 X-Received: by 10.43.171.70 with SMTP id nt6mr9742738icc.73.1431340448982; Mon, 11 May 2015 03:34:08 -0700 (PDT) Received: by 10.79.76.196 with HTTP; Mon, 11 May 2015 03:34:08 -0700 (PDT) In-Reply-To: References: Date: Mon, 11 May 2015 16:04:08 +0530 Message-ID: Subject: Re: error='Cannot allocate memory' (errno=12) From: Rahul Bhardwaj To: user Content-Type: multipart/alternative; boundary=001a11c2e33e03f1310515cbeb19 --001a11c2e33e03f1310515cbeb19 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable PFA of error log=E2=80=8B hs_err_pid9656.log =E2=80=8B On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj < rahul.bhardwaj@indiamart.com> wrote: > free RAM: > > > free -m > total used free shared buffers cached > Mem: 64398 23753 40644 0 108 8324 > -/+ buffers/cache: 15319 49078 > Swap: 2925 15 2909 > > > ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 515041 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 515041 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > Also attaching complete error file > > > On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal > wrote: > >> the memory cassandra is trying to allocate is pretty small. you sure >> there is no hardware failure on the machine. what is the free ram on the >> box ? >> >> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj < >> rahul.bhardwaj@indiamart.com> wrote: >> >>> Hi All, >>> >>> We have cluster of 3 nodes with 64GB RAM each. My cluster was running >>> in healthy state. Suddenly one machine's cassandra daemon stops working= and >>> shut down. >>> >>> On restarting it after 2 minutes it again stops and is getting stop >>> after returning below error in cassandra.log >>> >>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: >>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error=3D'Cannot >>> allocate memory' (errno=3D12) >>> # >>> # There is insufficient memory for the Java Runtime Environment to >>> continue. >>> # Native memory allocation (malloc) failed to allocate 12288 bytes for >>> committing reserved memory. >>> # An error report file with more information is saved as: >>> # /tmp/hs_err_pid23215.log >>> INFO 09:50:41 Loading settings from >>> file:/etc/cassandra/default.conf/cassandra.yaml >>> INFO 09:50:41 Node configuration:[authenticator=3DAllowAllAuthenticato= r; >>> authorizer=3DAllowAllAuthorizer; auto_snapshot=3Dtrue; >>> batch_size_warn_threshold_in_kb=3D5; batchlog_replay_throttle_in_kb=3D1= 024; >>> cas_contention_timeout_in_ms=3D1000; client_encryption_options=3D; >>> cluster_name=3DTest Cluster; column_index_size_in_kb=3D64; >>> commit_failure_policy=3Dstop; >>> commitlog_directory=3D/var/lib/cassandra/commitlog; >>> commitlog_segment_size_in_mb=3D64; commitlog_sync=3Dperiodic; >>> commitlog_sync_period_in_ms=3D10000; compaction_throughput_mb_per_sec= =3D16; >>> concurrent_compactors=3D4; concurrent_counter_writes=3D32; concurrent_r= eads=3D32; >>> concurrent_writes=3D32; counter_cache_save_period=3D7200; >>> counter_cache_size_in_mb=3Dnull; counter_write_request_timeout_in_ms=3D= 5000; >>> cross_node_timeout=3Dfalse; data_file_directories=3D[/var/lib/cassandra= /data]; >>> disk_failure_policy=3Dstop; dynamic_snitch_badness_threshold=3D0.1; >>> dynamic_snitch_reset_interval_in_ms=3D600000; >>> dynamic_snitch_update_interval_in_ms=3D100; >>> endpoint_snitch=3DGossipingPropertyFileSnitch; hinted_handoff_enabled= =3Dtrue; >>> hinted_handoff_throttle_in_kb=3D1024; incremental_backups=3Dfalse; >>> index_summary_capacity_in_mb=3Dnull; >>> index_summary_resize_interval_in_minutes=3D60; inter_dc_tcp_nodelay=3Df= alse; >>> internode_compression=3Dall; key_cache_save_period=3D14400; >>> key_cache_size_in_mb=3Dnull; listen_address=3Dnull; >>> max_hint_window_in_ms=3D10800000; max_hints_delivery_threads=3D2; >>> memtable_allocation_type=3Dheap_buffers; native_transport_port=3D9042; >>> num_tokens=3D256; partitioner=3Dorg.apache.cassandra.dht.Murmur3Partiti= oner; >>> permissions_validity_in_ms=3D2000; range_request_timeout_in_ms=3D100000= 0; >>> read_request_timeout_in_ms=3D999999999; >>> request_scheduler=3Dorg.apache.cassandra.scheduler.NoScheduler; >>> request_timeout_in_ms=3D999999999; row_cache_save_period=3D0; >>> row_cache_size_in_mb=3D0; rpc_address=3Dnull; rpc_keepalive=3Dtrue; >>> rpc_port=3D9160; rpc_server_type=3Dsync; >>> saved_caches_directory=3D/var/lib/cassandra/saved_caches; >>> seed_provider=3D[{class_name=3Dorg.apache.cassandra.locator.SimpleSeedP= rovider, >>> parameters=3D[{seeds=3D206.191.151.199}]}]; >>> server_encryption_options=3D; snapshot_before_compaction=3Dfa= lse; >>> ssl_storage_port=3D7001; sstable_preemptive_open_interval_in_mb=3D50; >>> start_native_transport=3Dtrue; start_rpc=3Dtrue; storage_port=3D7000; >>> thrift_framed_transport_size_in_mb=3D15; tombstone_failure_threshold=3D= 100000; >>> tombstone_warn_threshold=3D1000; trickle_fsync=3Dfalse; >>> trickle_fsync_interval_in_kb=3D10240; truncate_request_timeout_in_ms=3D= 60000; >>> write_request_timeout_in_ms=3D999999999] >>> ERROR 09:50:41 Exception encountered during startup >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60] >>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.ja= va:949) >>> ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecut= or.java:1590) >>> ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Schedul= edThreadPoolExecutor.java:333) >>> ~[na:1.7.0_60] >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay= (ScheduledThreadPoolExecutor.java:594) >>> ~[na:1.7.0_60] >>> at >>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.s= cheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageServic= e.java:721) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.j= ava:643) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.j= ava:535) >>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java= :324) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.j= ava:448) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> at >>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:= 537) >>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>> java.lang.OutOfMemoryError: unable to create new native thread >>> at java.lang.Thread.start0(Native Method) >>> at java.lang.Thread.start(Thread.java:714) >>> at >>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.ja= va:949) >>> at >>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecut= or.java:1590) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Schedul= edThreadPoolExecutor.java:333) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay= (ScheduledThreadPoolExecutor.java:594) >>> at >>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.s= cheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61) >>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188) >>> at >>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageServic= e.java:721) >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.j= ava:643) >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.j= ava:535) >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java= :324) >>> at >>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.j= ava:448) >>> at >>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:= 537) >>> Exception encountered during startup: unable to create new native threa= d >>> Java HotSpot(TM) 64-Bit Server VM warning: INFO: >>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error=3D'Cannot >>> allocate memory' (errno=3D12 >>> >>> This is our production cluster and rest 2 nodes are still in healthy >>> state. >>> We have also changed heap size from 10G to 20 G but problem still >>> persists. Please help in finding the reason behind this. >>> >>> >>> Regards: >>> Rahul Bhardwaj >>> >>> >>> Follow IndiaMART.com for latest updates on >>> this and more: >>> >>> Mobile Channel: >>> >>> >>> >>> >>> >>> Check out some smart tips to generate more business - *IndiaMART >>> Learning Centre* >> >> >> > --=20 Follow IndiaMART.com for latest updates on this= =20 and more: =20 Mobile= =20 Channel:=20 =20 =20 Check out some smart tips to generate more business - *IndiaMART Learning= =20 Centre* --001a11c2e33e03f1310515cbeb19 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
PFA of error log=E2=80=8B
=C2=A0hs_err_pid9656.log
=E2= =80=8B

On Mo= n, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <rahul.bhardwaj@indi= amart.com> wrote:
free RAM:


free -m
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0total =C2=A0 =C2=A0 =C2=A0= used =C2=A0 =C2=A0 =C2=A0 free =C2=A0 =C2=A0 shared =C2=A0 =C2=A0buffers = =C2=A0 =C2=A0 cached
Mem: =C2=A0 =C2=A0 =C2=A0 =C2=A0 64398 =C2= =A0 =C2=A0 =C2=A023753 =C2=A0 =C2=A0 =C2=A040644 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0108 =C2=A0 =C2=A0 =C2=A0 8324
<= div>-/+ buffers/cache: =C2=A0 =C2=A0 =C2=A015319 =C2=A0 =C2=A0 =C2=A049078<= /div>
Swap: =C2=A0 =C2=A0 =C2=A0 =C2=A0 2925 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 15 =C2=A0 =C2=A0 =C2=A0 2909


<= div>
=C2=A0ulimit -a
core file size =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0(blocks, -c) 0
data seg size =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 (kbytes, -d) unlimited
scheduling priority =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (-e) 0
file size =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (blocks, -f) unlimited
pending= signals =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (-i) 51504= 1
max locked memory =C2=A0 =C2=A0 =C2=A0 (kbytes, -l) 64
max memory size =C2=A0 =C2=A0 =C2=A0 =C2=A0 (kbytes, -m) unlimited
<= div>open files =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0(-n) 1024
pipe size =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0(512 bytes, -p) 8
POSIX message queues =C2=A0 =C2=A0= (bytes, -q) 819200
real-time priority =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0(-r) 0
stack size =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0(kbytes, -s) 10240
cpu time =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (seconds, -t) unlimited
ma= x user processes =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(-u) 51504= 1
virtual memory =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(kbytes, -v) u= nlimited
file locks =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(-x) unlimited


Also attaching complete error file


<= div class=3D"gmail_quote">On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal = <anishek@gmail.com> wrote:
the memory cassandra is trying to allocate is pretty smal= l. you sure there is no hardware failure on the machine. what is the free r= am on the box ?

On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <= rahul.bhardwaj@indiamart.com> wrote:
Hi All,

We have cluster of 3 nodes with =C2=A064GB RAM each. My cluster was runnin= g in healthy state. Suddenly one machine's cassandra daemon stops worki= ng and shut down.

On restarting it after 2 minutes= it again stops and is getting stop after returning below error in cassandr= a.log

Java HotSpot(TM) 64-Bit Server VM warni= ng: INFO: os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error=3D&= #39;Cannot allocate memory' (errno=3D12)
#
# There = is insufficient memory for the Java Runtime Environment to continue.
<= div># Native memory allocation (malloc) failed to allocate 12288 bytes for = committing reserved memory.
# An error report file with more info= rmation is saved as:
# /tmp/hs_err_pid23215.log
INFO = =C2=A009:50:41 Loading settings from file:/etc/cassandra/default.conf/cassa= ndra.yaml
INFO =C2=A009:50:41 Node configuration:[authenticator= =3DAllowAllAuthenticator; authorizer=3DAllowAllAuthorizer; auto_snapshot=3D= true; batch_size_warn_threshold_in_kb=3D5; batchlog_replay_throttle_in_kb= =3D1024; cas_contention_timeout_in_ms=3D1000; client_encryption_options=3D&= lt;REDACTED>; cluster_name=3DTest Cluster; column_index_size_in_kb=3D64;= commit_failure_policy=3Dstop; commitlog_directory=3D/var/lib/cassandra/com= mitlog; commitlog_segment_size_in_mb=3D64; commitlog_sync=3Dperiodic; commi= tlog_sync_period_in_ms=3D10000; compaction_throughput_mb_per_sec=3D16; conc= urrent_compactors=3D4; concurrent_counter_writes=3D32; concurrent_reads=3D3= 2; concurrent_writes=3D32; counter_cache_save_period=3D7200; counter_cache_= size_in_mb=3Dnull; counter_write_request_timeout_in_ms=3D5000; cross_node_t= imeout=3Dfalse; data_file_directories=3D[/var/lib/cassandra/data]; disk_fai= lure_policy=3Dstop; dynamic_snitch_badness_threshold=3D0.1; dynamic_snitch_= reset_interval_in_ms=3D600000; dynamic_snitch_update_interval_in_ms=3D100; = endpoint_snitch=3DGossipingPropertyFileSnitch; hinted_handoff_enabled=3Dtru= e; hinted_handoff_throttle_in_kb=3D1024; incremental_backups=3Dfalse; index= _summary_capacity_in_mb=3Dnull; index_summary_resize_interval_in_minutes=3D= 60; inter_dc_tcp_nodelay=3Dfalse; internode_compression=3Dall; key_cache_sa= ve_period=3D14400; key_cache_size_in_mb=3Dnull; listen_address=3Dnull; max_= hint_window_in_ms=3D10800000; max_hints_delivery_threads=3D2; memtable_allo= cation_type=3Dheap_buffers; native_transport_port=3D9042; num_tokens=3D256;= partitioner=3Dorg.apache.cassandra.dht.Murmur3Partitioner; permissions_val= idity_in_ms=3D2000; range_request_timeout_in_ms=3D1000000; read_request_tim= eout_in_ms=3D999999999; request_scheduler=3Dorg.apache.cassandra.scheduler.= NoScheduler; request_timeout_in_ms=3D999999999; row_cache_save_period=3D0; = row_cache_size_in_mb=3D0; rpc_address=3Dnull; rpc_keepalive=3Dtrue; rpc_por= t=3D9160; rpc_server_type=3Dsync; saved_caches_directory=3D/var/lib/cassand= ra/saved_caches; seed_provider=3D[{class_name=3Dorg.apache.cassandra.locato= r.SimpleSeedProvider, parameters=3D[{seeds=3D206.191.151.199}]}]; server_en= cryption_options=3D<REDACTED>; snapshot_before_compaction=3Dfalse; ss= l_storage_port=3D7001; sstable_preemptive_open_interval_in_mb=3D50; start_n= ative_transport=3Dtrue; start_rpc=3Dtrue; storage_port=3D7000; thrift_frame= d_transport_size_in_mb=3D15; tombstone_failure_threshold=3D100000; tombston= e_warn_threshold=3D1000; trickle_fsync=3Dfalse; trickle_fsync_interval_in_k= b=3D10240; truncate_request_timeout_in_ms=3D60000; write_request_timeout_in= _ms=3D999999999]
ERROR 09:50:41 Exception encountered during star= tup
java.lang.OutOfMemoryError: unable to create new native threa= d
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.start0(Native M= ethod) ~[na:1.7.0_60]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Th= read.start(Thread.java:714) ~[na:1.7.0_60]
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecu= tor.java:949) ~[na:1.7.0_60]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.= util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1= 590) ~[na:1.7.0_60]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.conc= urrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecut= or.java:333) ~[na:1.7.0_60]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.u= til.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(Scheduled= ThreadPoolExecutor.java:594) ~[na:1.7.0_60]
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExec= utor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
=C2=A0 =C2=A0 =C2=A0= =C2=A0 at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188) ~[ap= ache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 at org.apache.cassandra.service.StorageService.prepareToJoin(StorageSer= vice.java:721) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.StorageService.ini= tServer(StorageService.java:643) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHO= T]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.St= orageService.initServer(StorageService.java:535) ~[apache-cassandra-2.1.2.j= ar:2.1.2-SNAPSHOT]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cass= andra.service.CassandraDaemon.setup(CassandraDaemon.java:324) [apache-cassa= ndra-2.1.2.jar:2.1.2-SNAPSHOT]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org= .apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448= ) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDa= emon.java:537) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
java.l= ang.OutOfMemoryError: unable to create new native thread
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.start0(Native Method)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.start(Thread.java:714)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.ThreadPoolExecuto= r.addWorker(ThreadPoolExecutor.java:949)
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExe= cutor.java:1590)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurr= ent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.= java:333)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.Sch= eduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor= .java:594)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.co= ncurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(Debug= gableScheduledThreadPoolExecutor.java:61)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.StorageSer= vice.prepareToJoin(StorageService.java:721)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at org.apache.cassandra.service.StorageService.initServer(StorageSer= vice.java:643)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandr= a.service.StorageService.initServer(StorageService.java:535)
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.CassandraDaemon.se= tup(CassandraDaemon.java:324)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.= apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.service.Cass= andraDaemon.main(CassandraDaemon.java:537)
Exception encountered = during startup: unable to create new native thread
Java HotSpot(T= M) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fd459520000, 65= 536, 1) failed; error=3D'Cannot allocate memory' (errno=3D12
<= /div>

This is our production cluster and rest 2 nodes ar= e still in healthy state.
We have also changed heap size from 10G= to 20 G but problem still persists. Please help in finding the reason behi= nd this.


Regards:
Rahul B= hardwaj


Follow=C2=A0= IndiaMART.com=C2=A0for latest updates on this= and more: Mobile Channel:
=
Check out some smart tips to generate more business<= /span> - IndiaMART Learning Centre





Follow=C2=A0IndiaMART.co= m=C2=A0for latest updates on this and more: Mobile Channel: <= img src=3D"http://www.indiamart.com/newsletters/mailer/images/mapp.gif">

Check out some smart tips to generate more business - IndiaMART Learning Centre --001a11c2e33e03f1310515cbeb19--