From user-return-63016-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Jan 16 08:45:59 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 326EA180645 for ; Wed, 16 Jan 2019 08:45:58 +0100 (CET) Received: (qmail 61955 invoked by uid 500); 16 Jan 2019 07:45:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 61943 invoked by uid 99); 16 Jan 2019 07:45:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jan 2019 07:45:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A9126180D17 for ; Wed, 16 Jan 2019 07:45:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.798 X-Spam-Level: * X-Spam-Status: No, score=1.798 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Pxtq9JTp7AB0 for ; Wed, 16 Jan 2019 07:45:53 +0000 (UTC) Received: from mail-vs1-f46.google.com (mail-vs1-f46.google.com [209.85.217.46]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 469F85F67C for ; Wed, 16 Jan 2019 07:45:52 +0000 (UTC) Received: by mail-vs1-f46.google.com with SMTP id x1so3312626vsc.10 for ; Tue, 15 Jan 2019 23:45:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ktOLm/ZrzwBIwmMfcUl3+UwuM2ne7tb0NmJPAz9b0yI=; b=nwBUF8fLHD7iQ0oNieks4JT/78/l3Hxa2rE8skZBttOduqh/h581ThHSiBPXQ0ZDk7 1O3xJlHvG9R3Un8mGg5+6VgIXo/lGgfb4FUeMFKwJQ4DA8dyMuwijb4YHsoThuBphdde aRz36Ncysi3cpqFQbxLaet1Qj+bEUhqVMgH9fglnTi3Z1681ldvENI6VYtxHOwwmB4mW DyCqJMN3eN7/0oPXmx5YJlJx7kgbyqfpkM43Btmv9y0lhz6pzMNg7DfekSr3lOsNIp6P wMFcQmcT2aYQfkeW/p0lUaAcmXN4CrmccBsGZxFI6Ucfu/71jshfeAQuxCLpMHU0Mgi9 0QUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ktOLm/ZrzwBIwmMfcUl3+UwuM2ne7tb0NmJPAz9b0yI=; b=gNeboY70tZ3VJ/Y4jY5l9+XE4m6Rm8W+Y9EWTBXaEZcSF59+9F3zxlobciFfzxwR0V von9ApZbnTMkFkS3v2CI6HfAqqUC8ubCjInKA9KMgDxQO+CNS8cGy1R3KQGDj8G71CkK R2rp/esCIBf6UzUw2pIp1plSYt8P9lffWKZQZrsFXa3ZuI0adELXlikjFT8Rb4Phyxmy Azzxb9AEPdovu9dsvqQ8pJY1uT3VrwWpkWKjW4cOKCh84i4z5D1645VOyJ1XhS5CsefS 3d3kE0DJI2cHI4itUGlLDfMNNtO3E+23c+LO0mHpinYU2giLWiN/xkkZcHPoN08uhj78 cJbA== X-Gm-Message-State: AJcUukec5mLC1UeE/d78ASn48AtwNr8pkKpV7yCCV42ONrB5XbrpqylL 6eXSf4Qa3gTn0hBxp/5/bj1mrNMb9QqAmdh+oTaYIQ== X-Google-Smtp-Source: ALg8bN5eiXonOAroKiQdAkDxAIW2LwYSSLd8Jw66AbAHwMl8qBdUYO7500PHOD1PEYHaip+VbTgTZ4EKbfCprHNZQYs= X-Received: by 2002:a67:4909:: with SMTP id w9mr3259444vsa.189.1547624751228; Tue, 15 Jan 2019 23:45:51 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Roy Burstein Date: Wed, 16 Jan 2019 09:45:40 +0200 Message-ID: Subject: Re: Cassandra 3.11.3 map errors To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="0000000000008aebc6057f8e7777" --0000000000008aebc6057f8e7777 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Bob , We did not use the reaper(yet) just trying to run tests against the new C* v3 cluster . The nodes keeps crashing all the time and these are the error we are getting . Any other ideas ? Thanks! Roy On Wed, Jan 16, 2019 at 8:51 AM Bobbie Haynes wrote= : > Hi Roy, > I don't think the Memory Leak issue is related to MAP errors > .I was also using Reaper in our cluster.I have seen Memory Leak issue (ER= ROR > [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED= ) > when some of SStables got corrupted because of disk space issue we had wh= en > compactions were running. > I guess you have to report this Memory Leak issue to Reaper tool JIRA. > > Thanks, > Bob > > On Mon, Jan 14, 2019 at 8:44 AM Roy Burstein > wrote: > >> Hi , >> >> We are testing C* 3.11.3 and we have mapping issue and possibly leaked >> memory. >> It might be related to our configuration,any ideas would be helpful . >> >> >> >> Cassandra version: 3.11.3 >> OS: CentOS Linux release 7.4.1708 (Core) >> Kernel: 3.10.0-957.1.3.el7.x86_64 >> JDK: jdk1.8.0_131 >> Heap: same errors with 16GB / 32GB / 64GB. >> >> *We are seeing this errors in production:* >> >> *java.io.IOException: Map failed:* >> >> ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763 CassandraDaemon.= java:228 - Exception in thread Thread[CompactionExecutor:5017,1,main] >> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed >> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.ja= va:157) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.util.MmappedRegions$State.add(Mmapped= Regions.java:310) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.util.MmappedRegions$State.access$400(= MmappedRegions.java:246) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.util.MmappedRegions.updateState(Mmapp= edRegions.java:181) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.util.MmappedRegions.(MmappedReg= ions.java:73) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.util.MmappedRegions.(MmappedReg= ions.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegion= s.java:104) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.util.FileHandle$Builder.complete(File= Handle.java:362) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.ope= nEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEa= rly(SSTableRewriter.java:179) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTabl= eRewriter.java:134) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.db.compaction.writers.DefaultCompactionW= riter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.3= .jar:3.11.3] >> at org.apache.cassandra.db.compaction.writers.CompactionAwareWri= ter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.3.jar:3.= 11.3] >> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow= (CompactionTask.java:201) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnabl= e.java:28) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.db.compaction.CompactionTask.executeInte= rnal(CompactionTask.java:85) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.db.compaction.AbstractCompactionTask.exe= cute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3] >> at org.apache.cassandra.db.compaction.CompactionManager$Backgrou= ndCompactionCandidate.run(CompactionManager.java:274) ~[apache-cassandra-3.= 11.3.jar:3.11.3] >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors= .java:511) ~[na:1.8.0_131] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na= :1.8.0_131] >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolE= xecutor.java:1142) ~[na:1.8.0_131] >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPool= Executor.java:617) [na:1.8.0_131] >> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$thr= eadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.= jar:3.11.3] >> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] >> Caused by: java.io.IOException: Map failed >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na= :1.8.0_131] >> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.ja= va:153) ~[apache-cassandra-3.11.3.jar:3.11.3] >> ... 23 common frames omitted >> Caused by: java.lang.OutOfMemoryError: Map failed >> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131= ] >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na= :1.8.0_131] >> ... 24 common frames omitted >> >> *LEAK DETECTED:* >> >> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK D= ETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6a4ef= 142) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741= :Memory@[6b91a27c5290..6b91a27de290) was not released before the reference = was garbage collected >> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK D= ETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6c458= f8a) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1179238225:/v= ar/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e9= 0037011e9a45847402874bbd7/mc-1209-big-Index.db was not released before the = reference was garbage collected >> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK D= ETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5b908= 23b) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@783549664:= /var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8= e90037011e9a45847402874bbd7/mc-1209-big-Data.db was not released before the= reference was garbage collected >> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK D= ETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6ecdf= 763) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$= Tidy@1710583516:[Memory@[0..3e24), Memory@[0..45e88)] was not released befo= re the reference was garbage collected >> >> >> *Limits of Cassandra process:* >> >> [root@cass063 ~ ]# cat /proc/`ps -ef | grep CassandraDaemon | grep -v g= rep | awk '\{print $2}'`/limits >> Limit Soft Limit Hard Limit Uni= ts >> Max cpu time unlimited unlimited sec= onds >> Max file size unlimited unlimited byt= es >> Max data size unlimited unlimited byt= es >> Max stack size 8388608 unlimited byt= es >> Max core file size 0 unlimited byt= es >> Max resident set unlimited unlimited byt= es >> Max processes 32768 32768 pro= cesses >> Max open files 100000 100000 fil= es >> Max locked memory unlimited unlimited byt= es >> Max address space unlimited unlimited byt= es >> Max file locks unlimited unlimited loc= ks >> Max pending signals 766985 766985 sig= nals >> Max msgqueue size 819200 819200 byt= es >> Max nice priority 0 0 >> Max realtime priority 0 0 >> Max realtime timeout unlimited unlimited us >> >> >> >> *max_map_count parameter on OS:* >> >> [root@cass063 ~]# sysctl vm.max_map_count >> vm.max_map_count =3D 1073741824 >> >> >> >> >> *cassandra.yaml:* >> >> cluster_name: 'Cass Cluster' >> num_tokens: 256 >> hinted_handoff_enabled: false >> max_hint_window_in_ms: 10800000 >> hinted_handoff_throttle_in_kb: 1024 >> max_hints_delivery_threads: 2 >> hints_directory: /var/lib/cassandra/hints >> hints_flush_period_in_ms: 10000 >> max_hints_file_size_in_mb: 128 >> batchlog_replay_throttle_in_kb: 1024 >> authenticator: AllowAllAuthenticator >> authorizer: AllowAllAuthorizer >> role_manager: CassandraRoleManager >> roles_validity_in_ms: 2000 >> permissions_validity_in_ms: 2000 >> credentials_validity_in_ms: 2000 >> partitioner: org.apache.cassandra.dht.Murmur3Partitioner >> data_file_directories: >> - /var/lib/cassandra/data/disk1 >> commitlog_directory: /var/lib/cassandra/data/disk1/commitlog >> cdc_enabled: false >> disk_failure_policy: stop >> commit_failure_policy: stop >> prepared_statements_cache_size_mb: >> thrift_prepared_statements_cache_size_mb: >> key_cache_size_in_mb: 0 >> key_cache_save_period: 3600 >> row_cache_size_in_mb: 0 >> row_cache_save_period: 0 >> counter_cache_size_in_mb: >> counter_cache_save_period: 7200 >> saved_caches_directory: /var/lib/cassandra/data/disk1/saved_caches >> commitlog_sync: periodic >> commitlog_sync_period_in_ms: 10000 >> commitlog_segment_size_in_mb: 32 >> seed_provider: >> - class_name: org.apache.cassandra.locator.SimpleSeedProvider >> parameters: >> - seeds: "10.110.30.1,10.110.30.2,10.110.30.3" >> concurrent_reads: 48 >> concurrent_writes: 96 >> concurrent_counter_writes: 32 >> concurrent_materialized_view_writes: 32 >> file_cache_size_in_mb: 10240 >> memtable_offheap_space_in_mb: 10240 >> memtable_cleanup_threshold: 0.1 >> memtable_allocation_type: offheap_buffers >> commitlog_total_space_in_mb: 8192 >> memtable_flush_writers: 8 >> index_summary_capacity_in_mb: >> index_summary_resize_interval_in_minutes: 60 >> trickle_fsync: true >> trickle_fsync_interval_in_kb: 10240 >> storage_port: 7000 >> ssl_storage_port: 7001 >> listen_address: 10.106.62.34 >> start_native_transport: true >> native_transport_port: 9042 >> start_rpc: false >> rpc_address: 0.0.0.0 >> rpc_port: 9160 >> broadcast_rpc_address: 10.106.62.34 >> rpc_keepalive: true >> rpc_server_type: hsha >> rpc_max_threads: 128 >> thrift_framed_transport_size_in_mb: 15 >> incremental_backups: false >> snapshot_before_compaction: false >> auto_snapshot: true >> column_index_size_in_kb: 64 >> column_index_cache_size_in_kb: 2 >> concurrent_compactors: 32 >> compaction_throughput_mb_per_sec: 500 >> sstable_preemptive_open_interval_in_mb: 50 >> stream_throughput_outbound_megabits_per_sec: 0 >> read_request_timeout_in_ms: 10000 >> range_request_timeout_in_ms: 10000 >> write_request_timeout_in_ms: 60000 >> counter_write_request_timeout_in_ms: 10000 >> cas_contention_timeout_in_ms: 1000 >> truncate_request_timeout_in_ms: 60000 >> request_timeout_in_ms: 10000 >> slow_query_log_timeout_in_ms: 500 >> cross_node_timeout: false >> phi_convict_threshold: 12 >> endpoint_snitch: GossipingPropertyFileSnitch >> dynamic_snitch_update_interval_in_ms: 100 >> dynamic_snitch_reset_interval_in_ms: 600000 >> dynamic_snitch_badness_threshold: 0.5 >> request_scheduler: org.apache.cassandra.scheduler.NoScheduler >> server_encryption_options: >> internode_encryption: none >> keystore: conf/.keystore >> keystore_password: cassandra >> truststore: conf/.truststore >> truststore_password: cassandra >> client_encryption_options: >> enabled: false >> optional: false >> keystore: conf/.keystore >> keystore_password: cassandra >> internode_compression: dc >> inter_dc_tcp_nodelay: false >> tracetype_query_ttl: 86400 >> tracetype_repair_ttl: 604800 >> enable_user_defined_functions: false >> enable_scripted_user_defined_functions: false >> enable_materialized_views: true >> windows_timer_interval: 1 >> transparent_data_encryption_options: >> enabled: false >> chunk_length_kb: 64 >> cipher: AES/CBC/PKCS5Padding >> key_alias: testing:1 >> key_provider: >> - class_name: org.apache.cassandra.security.JKSKeyProvider >> parameters: >> - keystore: conf/.keystore >> keystore_password: cassandra >> store_type: JCEKS >> key_password: cassandra >> tombstone_warn_threshold: 1000 >> tombstone_failure_threshold: 100000 >> batch_size_warn_threshold_in_kb: 5 >> batch_size_fail_threshold_in_kb: 50 >> unlogged_batch_across_partitions_warn_threshold: 10 >> compaction_large_partition_warning_threshold_mb: 10 >> gc_warn_threshold_in_ms: 1000 >> back_pressure_enabled: false >> back_pressure_strategy: >> - class_name: org.apache.cassandra.net.RateBasedBackPressure >> parameters: >> - high_ratio: 0.90 >> factor: 5 >> flow: FAST >> >> >> >> *A lot of maps, 200K maps of cassandra process,*: >> >> [root@cass063 ~]# wc -l /proc/`ps -ef | grep CassandraDaemon | grep -v g= rep | awk '{print $2}'`/maps >> 239587 /proc/202664/maps >> >> Thanks, >> Roy >> > --0000000000008aebc6057f8e7777 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Bob ,=C2=A0
We did not use the reaper(yet) just try= ing to run tests against the new C* v3 cluster .
The nodes keeps = crashing all the time and these are the error we=C2=A0are getting .
Any other ideas ?
Thanks!
Roy

On Wed, Jan 16, 2= 019 at 8:51 AM Bobbie Haynes <h= aynes30349@gmail.com> wrote:
Hi Roy,=C2=A0
=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 I don't think the Memory Leak issue is related to= MAP errors .I was also using Reaper in our cluster.I have seen Memory Leak= issue (ERROR [Reference-Reaper:1] 201= 9-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED) when some of SStables g= ot corrupted because of disk space issue we had when compactions were runni= ng.
I guess you have= to report this Memory Leak issue to Reaper tool JIRA.

Thanks,
Bob

On Mon, Jan 14, 2019 at 8:4= 4 AM Roy Burstein <burstein.roy@gmail.com> wrote:

Hi ,=C2=A0

We are testing C* 3.11.3 and we hav= e mapping issue and possibly leaked memory.
It might be related to our c= onfiguration,any ideas would be helpful .=C2=A0

=C2=A0

Cassandr= a version:=C2=A03.11.3
OS:=C2=A0CentOS Linux release 7.4.1708 (Core)
Kernel:=C2=A03.10.0-957.1.3.el7.x86_64
JDK:=C2=A0jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.

We are seeing this errors in production:

java.io.IOException: Map failed:

ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763 Cassandra=
Daemon.java:228 - Exception in thread Thread[Compaction=
Executor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:=
157) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedReg=
ions.java:310) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(Mma=
ppedRegions.java:246) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedR=
egions.java:181) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(Mmapped=
Regions.java:73) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(Mmapped=
Regions.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.j=
ava:104) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHan=
dle.java:362) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEa=
rly(BigTableWriter.java:290) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly=
(SSTableRewriter.java:179) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRe=
writer.java:134) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWrit=
er.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.3.ja=
r:3.11.3]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter=
.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.3.jar:3.11.=
3]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(Co=
mpactionTask.java:201) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j=
ava:28) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInterna=
l(CompactionTask.java:85) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execut=
e(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundC=
ompactionCandidate.run(CompactionManager.java:274) ~[apache-cassandra-3.11.=
3.jar:3.11.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja=
va:511) ~[na:1.8.0_131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.=
8.0_131]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec=
utor.java:1142) ~[na:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe=
cutor.java:617) [na:1.8.0_131]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$thread=
LocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar=
:3.11.3]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.=
8.0_131]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:=
153) ~[apache-cassandra-3.11.3.jar:3.11.3]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.=
8.0_131]
        ... 24 common frames omitted

LEAK DETECTED:

ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 -=
 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$Stat=
e@6a4ef142) to class org.apache.cassandra.io.util.Safe=
Memory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290) was not re=
leased before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETE=
CTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a=
) to class org.apache.cassandra.io.util.FileHandle$Cle=
anup@1179238225:/var/lib/cassandra/data/disk1/sessions=
_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-bi=
g-Index.db was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETE=
CTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5b90823b=
) to class org.apache.cassandra.io.util.MmappedRegions=
$Tidier@783549664:/var/lib/cassandra/data/disk1/sessio=
ns_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-=
big-Data.db was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETE=
CTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6ecdf763=
) to class org.apache.cassandra.utils.concurrent.Wrapp=
edSharedCloseable$Tidy@1710583516:[Memory@[0..3e24), Memory@[0..45e88)] was=
 not released before the reference was garbage collected

=C2=A0
Limits of Cassandra process:

 [root@cass063 ~ ]# cat /proc/`ps -ef | grep CassandraDaemon | gr=
ep -v grep | awk '\{print $2}'`/limits
 Limit=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0Soft Limit=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Hard Limit=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Units
 Max cpu time=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 seconds
 Max file size=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unlimited=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 bytes
 Max data size=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unlimited=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 bytes
 Max stack size=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 8388608=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 bytes
 Max core file size=C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 bytes
 Max resident set=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 bytes
 Max processes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A032768=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 32768=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 processes
 Max open files=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 100000=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0100000=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0files
 Max locked memory=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unlimited=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 bytes
 Max address space=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unlimited=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 bytes
 Max file locks=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 locks
 Max pending signals=C2=A0 =C2=A0 =C2=A0 =C2=A0766985=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0766985=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0signals
 Max msgqueue size=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0819200=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0819200=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0bytes
 Max nice priority=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0
 Max realtime priority=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0
 Max realtime timeout=C2=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 unlimited=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 us

=C2=A0

max_map_count parameter on OS:

 [root@cass063 ~]# sysctl vm.max_map_count
 vm.max_map_count =3D 1073741824
 =C2=A0

=C2=A0

cassandra.yaml:

 cluster_name: 'Cass Cluster'
 num_tokens: 256
 hinted_handoff_enabled: false
 max_hint_window_in_ms: 10800000
 hinted_handoff_throttle_in_kb: 1024
 max_hints_delivery_threads: 2
 hints_directory: /var/lib/cassandra/hints
 hints_flush_period_in_ms: 10000
 max_hints_file_size_in_mb: 128
 batchlog_replay_throttle_in_kb: 1024
 authenticator: AllowAllAuthenticator
 authorizer: AllowAllAuthorizer
 role_manager: CassandraRoleManager
 roles_validity_in_ms: 2000
 permissions_validity_in_ms: 2000
 credentials_validity_in_ms: 2000
 partitioner: org.apache.cassandra.dht.Murmur3Partitioner
 data_file_directories:
 =C2=A0 =C2=A0 - /var/lib/cassandra/data/disk1
 commitlog_directory: /var/lib/cassandra/data/disk1/co=
mmitlog
 cdc_enabled: false
 disk_failure_policy: stop
 commit_failure_policy: stop
 prepared_statements_cache_size_mb:
 thrift_prepared_statements_cache_size_mb:
 key_cache_size_in_mb: 0
 key_cache_save_period: 3600
 row_cache_size_in_mb: 0
 row_cache_save_period: 0
 counter_cache_size_in_mb:
 counter_cache_save_period: 7200
 saved_caches_directory: /var/lib/cassandra/data/disk1=
/saved_caches
 commitlog_sync: periodic
 commitlog_sync_period_in_ms: 10000
 commitlog_segment_size_in_mb: 32
 seed_provider:
 =C2=A0 =C2=A0 - class_name: org.apache.cassandra.locator.SimpleSeedProvide=
r
 =C2=A0 =C2=A0 =C2=A0 parameters:
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - seeds: "10.110.30.1,=
10.110.30.2,10.110.30.3"
 concurrent_reads: 48
 concurrent_writes: 96
 concurrent_counter_writes: 32
 concurrent_materialized_view_writes: 32
 file_cache_size_in_mb: 10240
 memtable_offheap_space_in_mb: 10240
 memtable_cleanup_threshold: 0.1
 memtable_allocation_type: offheap_buffers
 commitlog_total_space_in_mb: 8192
 memtable_flush_writers: 8
 index_summary_capacity_in_mb:
 index_summary_resize_interval_in_minutes: 60
 trickle_fsync: true
 trickle_fsync_interval_in_kb: 10240
 storage_port: 7000
 ssl_storage_port: 7001
 listen_address: 10.106.62.34
 start_native_transport: true
 native_transport_port: 9042
 start_rpc: false
 rpc_address: 0.0.0.0
 rpc_port: 9160
 broadcast_rpc_address: 10.106.62.34
 rpc_keepalive: true
 rpc_server_type: hsha
 rpc_max_threads: 128
 thrift_framed_transport_size_in_mb: 15
 incremental_backups: false
 snapshot_before_compaction: false
 auto_snapshot: true
 column_index_size_in_kb: 64
 column_index_cache_size_in_kb: 2
 concurrent_compactors: 32
 compaction_throughput_mb_per_sec: 500
 sstable_preemptive_open_interval_in_mb: 50
 stream_throughput_outbound_megabits_per_sec: 0
 read_request_timeout_in_ms: 10000
 range_request_timeout_in_ms: 10000
 write_request_timeout_in_ms: 60000
 counter_write_request_timeout_in_ms: 10000
 cas_contention_timeout_in_ms: 1000
 truncate_request_timeout_in_ms: 60000
 request_timeout_in_ms: 10000
 slow_query_log_timeout_in_ms: 500
 cross_node_timeout: false
 phi_convict_threshold: 12
 endpoint_snitch: GossipingPropertyFileSnitch
 dynamic_snitch_update_interval_in_ms: 100
 dynamic_snitch_reset_interval_in_ms: 600000
 dynamic_snitch_badness_threshold: 0.5
 request_scheduler: org.apache.cassandra.scheduler.NoScheduler
 server_encryption_options:
 =C2=A0 =C2=A0 internode_encryption: none
 =C2=A0 =C2=A0 keystore: conf/.keystore
 =C2=A0 =C2=A0 keystore_password: cassandra
 =C2=A0 =C2=A0 truststore: conf/.truststore
 =C2=A0 =C2=A0 truststore_password: cassandra
 client_encryption_options:
 =C2=A0 =C2=A0 enabled: false
 =C2=A0 =C2=A0 optional: false
 =C2=A0 =C2=A0 keystore: conf/.keystore
 =C2=A0 =C2=A0 keystore_password: cassandra
 internode_compression: dc
 inter_dc_tcp_nodelay: false
 tracetype_query_ttl: 86400
 tracetype_repair_ttl: 604800
 enable_user_defined_functions: false
 enable_scripted_user_defined_functions: false
 enable_materialized_views: true
 windows_timer_interval: 1
 transparent_data_encryption_options:
 =C2=A0 =C2=A0 enabled: false
 =C2=A0 =C2=A0 chunk_length_kb: 64
 =C2=A0 =C2=A0 cipher: AES/CBC/PKCS5Padding
 =C2=A0 =C2=A0 key_alias: testing:1
 =C2=A0 =C2=A0 key_provider:
 =C2=A0 =C2=A0 =C2=A0 - class_name: org.apache.cassandra.security.JKSKeyPro=
vider
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 parameters:
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - keystore: conf/.keystore
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 keystore_password: cassandra
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 store_type: JCEKS
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 key_password: cassandra
 tombstone_warn_threshold: 1000
 tombstone_failure_threshold: 100000
 batch_size_warn_threshold_in_kb: 5
 batch_size_fail_threshold_in_kb: 50
 unlogged_batch_across_partitions_warn_threshold: 10
 compaction_large_partition_warning_threshold_mb: 10
 gc_warn_threshold_in_ms: 1000
 back_pressure_enabled: false
 back_pressure_strategy:
 =C2=A0 =C2=A0 - class_name: org.apache.cassandra.net.RateBasedBackPressure
 =C2=A0 =C2=A0 =C2=A0 parameters:
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - high_ratio: 0.90
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 factor: 5
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 flow: FAST

=C2=A0

A lot of maps, 200K maps of cassandra process,:

[root@cass063 ~]# wc -l /proc/`ps -ef | grep CassandraDaemon | gr=
ep -v grep | awk '{print $2}'`/maps
239587 /proc/202664/maps

=C2=A0Thanks,
Roy

--0000000000008aebc6057f8e7777--