cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roy Burstein <burstein....@gmail.com>
Subject Re: Cassandra 3.11.3 map errors
Date Wed, 16 Jan 2019 07:45:40 GMT
Hi Bob ,
We did not use the reaper(yet) just trying to run tests against the new C*
v3 cluster .
The nodes keeps crashing all the time and these are the error we are
getting .
Any other ideas ?
Thanks!
Roy

On Wed, Jan 16, 2019 at 8:51 AM Bobbie Haynes <haynes30349@gmail.com> wrote:

> Hi Roy,
>             I don't think the Memory Leak issue is related to MAP errors
> .I was also using Reaper in our cluster.I have seen Memory Leak issue (ERROR
> [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED)
> when some of SStables got corrupted because of disk space issue we had when
> compactions were running.
> I guess you have to report this Memory Leak issue to Reaper tool JIRA.
>
> Thanks,
> Bob
>
> On Mon, Jan 14, 2019 at 8:44 AM Roy Burstein <burstein.roy@gmail.com>
> wrote:
>
>> Hi ,
>>
>> We are testing C* 3.11.3 and we have mapping issue and possibly leaked
>> memory.
>> It might be related to our configuration,any ideas would be helpful .
>>
>>
>>
>> Cassandra version: 3.11.3
>> OS: CentOS Linux release 7.4.1708 (Core)
>> Kernel: 3.10.0-957.1.3.el7.x86_64
>> JDK: jdk1.8.0_131
>> Heap: same errors with 16GB / 32GB / 64GB.
>>
>> *We are seeing this errors in production:*
>>
>> *java.io.IOException: Map failed:*
>>
>> ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763 CassandraDaemon.java:228
- Exception in thread Thread[CompactionExecutor:5017,1,main]
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
>>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
>>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
>>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
>>         at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
>> Caused by: java.io.IOException: Map failed
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_131]
>>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>         ... 23 common frames omitted
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>         at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_131]
>>         ... 24 common frames omitted
>>
>> *LEAK DETECTED:*
>>
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED:
a reference (org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED:
a reference (org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1179238225:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Index.db
was not released before the reference was garbage collected
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED:
a reference (org.apache.cassandra.utils.concurrent.Ref$State@5b90823b) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@783549664:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Data.db
was not released before the reference was garbage collected
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED:
a reference (org.apache.cassandra.utils.concurrent.Ref$State@6ecdf763) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1710583516:[Memory@[0..3e24),
Memory@[0..45e88)] was not released before the reference was garbage collected
>>
>>
>> *Limits of Cassandra process:*
>>
>>  [root@cass063 ~ ]# cat /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk
'\{print $2}'`/limits
>>  Limit                     Soft Limit           Hard Limit           Units
>>  Max cpu time              unlimited            unlimited            seconds
>>  Max file size             unlimited            unlimited            bytes
>>  Max data size             unlimited            unlimited            bytes
>>  Max stack size            8388608              unlimited            bytes
>>  Max core file size        0                    unlimited            bytes
>>  Max resident set          unlimited            unlimited            bytes
>>  Max processes             32768                32768                processes
>>  Max open files            100000               100000               files
>>  Max locked memory         unlimited            unlimited            bytes
>>  Max address space         unlimited            unlimited            bytes
>>  Max file locks            unlimited            unlimited            locks
>>  Max pending signals       766985               766985               signals
>>  Max msgqueue size         819200               819200               bytes
>>  Max nice priority         0                    0
>>  Max realtime priority     0                    0
>>  Max realtime timeout      unlimited            unlimited            us
>>
>>
>>
>> *max_map_count parameter on OS:*
>>
>>  [root@cass063 ~]# sysctl vm.max_map_count
>>  vm.max_map_count = 1073741824
>>
>>
>>
>>
>> *cassandra.yaml:*
>>
>>  cluster_name: 'Cass Cluster'
>>  num_tokens: 256
>>  hinted_handoff_enabled: false
>>  max_hint_window_in_ms: 10800000
>>  hinted_handoff_throttle_in_kb: 1024
>>  max_hints_delivery_threads: 2
>>  hints_directory: /var/lib/cassandra/hints
>>  hints_flush_period_in_ms: 10000
>>  max_hints_file_size_in_mb: 128
>>  batchlog_replay_throttle_in_kb: 1024
>>  authenticator: AllowAllAuthenticator
>>  authorizer: AllowAllAuthorizer
>>  role_manager: CassandraRoleManager
>>  roles_validity_in_ms: 2000
>>  permissions_validity_in_ms: 2000
>>  credentials_validity_in_ms: 2000
>>  partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>  data_file_directories:
>>      - /var/lib/cassandra/data/disk1
>>  commitlog_directory: /var/lib/cassandra/data/disk1/commitlog
>>  cdc_enabled: false
>>  disk_failure_policy: stop
>>  commit_failure_policy: stop
>>  prepared_statements_cache_size_mb:
>>  thrift_prepared_statements_cache_size_mb:
>>  key_cache_size_in_mb: 0
>>  key_cache_save_period: 3600
>>  row_cache_size_in_mb: 0
>>  row_cache_save_period: 0
>>  counter_cache_size_in_mb:
>>  counter_cache_save_period: 7200
>>  saved_caches_directory: /var/lib/cassandra/data/disk1/saved_caches
>>  commitlog_sync: periodic
>>  commitlog_sync_period_in_ms: 10000
>>  commitlog_segment_size_in_mb: 32
>>  seed_provider:
>>      - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>>        parameters:
>>            - seeds: "10.110.30.1,10.110.30.2,10.110.30.3"
>>  concurrent_reads: 48
>>  concurrent_writes: 96
>>  concurrent_counter_writes: 32
>>  concurrent_materialized_view_writes: 32
>>  file_cache_size_in_mb: 10240
>>  memtable_offheap_space_in_mb: 10240
>>  memtable_cleanup_threshold: 0.1
>>  memtable_allocation_type: offheap_buffers
>>  commitlog_total_space_in_mb: 8192
>>  memtable_flush_writers: 8
>>  index_summary_capacity_in_mb:
>>  index_summary_resize_interval_in_minutes: 60
>>  trickle_fsync: true
>>  trickle_fsync_interval_in_kb: 10240
>>  storage_port: 7000
>>  ssl_storage_port: 7001
>>  listen_address: 10.106.62.34
>>  start_native_transport: true
>>  native_transport_port: 9042
>>  start_rpc: false
>>  rpc_address: 0.0.0.0
>>  rpc_port: 9160
>>  broadcast_rpc_address: 10.106.62.34
>>  rpc_keepalive: true
>>  rpc_server_type: hsha
>>  rpc_max_threads: 128
>>  thrift_framed_transport_size_in_mb: 15
>>  incremental_backups: false
>>  snapshot_before_compaction: false
>>  auto_snapshot: true
>>  column_index_size_in_kb: 64
>>  column_index_cache_size_in_kb: 2
>>  concurrent_compactors: 32
>>  compaction_throughput_mb_per_sec: 500
>>  sstable_preemptive_open_interval_in_mb: 50
>>  stream_throughput_outbound_megabits_per_sec: 0
>>  read_request_timeout_in_ms: 10000
>>  range_request_timeout_in_ms: 10000
>>  write_request_timeout_in_ms: 60000
>>  counter_write_request_timeout_in_ms: 10000
>>  cas_contention_timeout_in_ms: 1000
>>  truncate_request_timeout_in_ms: 60000
>>  request_timeout_in_ms: 10000
>>  slow_query_log_timeout_in_ms: 500
>>  cross_node_timeout: false
>>  phi_convict_threshold: 12
>>  endpoint_snitch: GossipingPropertyFileSnitch
>>  dynamic_snitch_update_interval_in_ms: 100
>>  dynamic_snitch_reset_interval_in_ms: 600000
>>  dynamic_snitch_badness_threshold: 0.5
>>  request_scheduler: org.apache.cassandra.scheduler.NoScheduler
>>  server_encryption_options:
>>      internode_encryption: none
>>      keystore: conf/.keystore
>>      keystore_password: cassandra
>>      truststore: conf/.truststore
>>      truststore_password: cassandra
>>  client_encryption_options:
>>      enabled: false
>>      optional: false
>>      keystore: conf/.keystore
>>      keystore_password: cassandra
>>  internode_compression: dc
>>  inter_dc_tcp_nodelay: false
>>  tracetype_query_ttl: 86400
>>  tracetype_repair_ttl: 604800
>>  enable_user_defined_functions: false
>>  enable_scripted_user_defined_functions: false
>>  enable_materialized_views: true
>>  windows_timer_interval: 1
>>  transparent_data_encryption_options:
>>      enabled: false
>>      chunk_length_kb: 64
>>      cipher: AES/CBC/PKCS5Padding
>>      key_alias: testing:1
>>      key_provider:
>>        - class_name: org.apache.cassandra.security.JKSKeyProvider
>>          parameters:
>>            - keystore: conf/.keystore
>>              keystore_password: cassandra
>>              store_type: JCEKS
>>              key_password: cassandra
>>  tombstone_warn_threshold: 1000
>>  tombstone_failure_threshold: 100000
>>  batch_size_warn_threshold_in_kb: 5
>>  batch_size_fail_threshold_in_kb: 50
>>  unlogged_batch_across_partitions_warn_threshold: 10
>>  compaction_large_partition_warning_threshold_mb: 10
>>  gc_warn_threshold_in_ms: 1000
>>  back_pressure_enabled: false
>>  back_pressure_strategy:
>>      - class_name: org.apache.cassandra.net.RateBasedBackPressure
>>        parameters:
>>          - high_ratio: 0.90
>>            factor: 5
>>            flow: FAST
>>
>>
>>
>> *A lot of maps, 200K maps of cassandra process,*:
>>
>> [root@cass063 ~]# wc -l /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk
'{print $2}'`/maps
>> 239587 /proc/202664/maps
>>
>>  Thanks,
>> Roy
>>
>

Mime
View raw message