cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Xie (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-14653) The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
Date Fri, 17 Aug 2018 10:14:00 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Peter Xie updated CASSANDRA-14653:
----------------------------------
    Description: 
We use cassandra as backend storage for Janusgraph. when we loading huge data (~2 billion
vertex, ~10 billion edges), we met some problems.

 

At first, we use STCS as compaction strategy , but met below exception.  we checked the
value of  "max memory lock" is unlimited and "file map count" is 1 million, these values
should enough for loading data. last we found this problem is caused by the virtual memory
are all cosumed by cassandra.  So not additional virtual memory can be used by compaction
task , and below exception is thrown out.   
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952 JVMStabilityInspector.javv
 a:74 - OutOfMemory error letting the JVM handle the error:
 java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the virtual memory
problem. But we found another problem : Many sstables which has been compacted are still retained on
disk,  these old sstables consume so many disk space, it's causing no enough disk for saving
real data. we found that so many files like "mc_txn_compaction_xxx.log" are created under
the data directory. 

After some times' investigaton, we found that this problem is caused by "NonPeriodicTasks"
thread pools.  this pools is always using only one thread for processing clean task and compaction.
this thread pool is instanced with class DebuggableScheduledThreadPoolExecutor,

and DebuggableScheduledThreadPoolExecutor is inherit from class  ScheduledThreadPoolExecutor.

By reading the code of class DebuggableScheduledThreadPoolExecutor,  found DebuggableScheduledThreadPoolExecutor
is using an unbound task queue, and core pool size is 1.   Why here use the unbound queue
for queuing submitted tasks?  If we using unbound queue, the thread pool wouldn't  increasing
thread even there so many task are blocked in queue, because unbound queue never would be
full.  I think here should use bound queue, so when task is heavily, more threads would created
for processing them. 
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String threadPoolName,
int priority)
 Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName, priority));
setRejectedExecutionHandler(rejectedExecutionHandler); }
  

public ScheduledThreadPoolExecutor(int corePoolSize,
 ThreadFactory threadFactory)
 Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new DelayedWorkQueue(),
threadFactory); }
{quote}
 Below is the case about clean task after compaction.  there nearly 3 hours delay for removing
file "mc-56525". 
{quote} 

TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664 LifecycleTransaction.java:363 - Staging
for obsolescence BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
 ..........
 TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 - removing /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big
from list of files tracked for test_2.edgestore
 ............
 TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, before
barrier
 TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
 TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, completed
{quote}
 

 

 

 

 

 

 

  was:
We use cassandra as backend storage for Janusgraph. when we loading huge data (~2 billion
vertex, ~10 billion edges), we met some problems.

 

At first, we use STCS as compaction strategy , but met below exception.  we checked the
value of  "max memory lock" is unlimited and "file map count" is 1 million, these values
should enough for loading data. last we found this problem is caused by the virtual memory
are all cosumed by cassandra.  So not additional virtual memory can be used by compaction
task , and below exception is thrown out.   
{quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952 JVMStabilityInspector.javv
 a:74 - OutOfMemory error letting the JVM handle the error:
 java.lang.OutOfMemoryError: Map failed
{quote}
So, we change compaction strategy to LCS, this change seems can resolve the virtual memory
problem. But we found another problem : Many sstables which has been compacted are still retained on
disk,  these old sstable consume so many disk space, it's causing no enough disk for saving
real data. we found that so many files like "mc_txn_compaction_xxx.log" are created under
the data directory. 

After some times' investigaton, we found that this problem is caused by "NonPeriodicTasks"
thread pools.  this pools is always using only one thread for processing clean task and compaction.
this thread pool is instanced with class DebuggableScheduledThreadPoolExecutor,

and DebuggableScheduledThreadPoolExecutor is inherit from class  ScheduledThreadPoolExecutor.

By reading the code of class DebuggableScheduledThreadPoolExecutor,  found DebuggableScheduledThreadPoolExecutor
is using an unbound task queue, and core pool size is 1.   Why here use the unbound queue
for queuing submitted tasks?  If we using unbound queue, the thread pool wouldn't  increasing
thread even there so many task are blocked in queue, because unbound queue never would be
full.  I think here should use bound queue, so when task is heavily, more threads would created
for processing them. 
{quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String threadPoolName,
int priority)
 Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName, priority));
setRejectedExecutionHandler(rejectedExecutionHandler); }
  

public ScheduledThreadPoolExecutor(int corePoolSize,
 ThreadFactory threadFactory)
 Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new DelayedWorkQueue(),
threadFactory); }
{quote}
 Below is the case about clean task after compaction.  there nearly 3 hours delay for removing
file "mc-56525". 
{quote} 

TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664 LifecycleTransaction.java:363 - Staging
for obsolescence BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
 ..........
 TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 - removing /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big
from list of files tracked for test_2.edgestore
 ............
 TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, before
barrier
 TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
 TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, completed
{quote}
 

 

 

 

 

 

 


> The performance of "NonPeriodicTasks" pools defined in class ScheduledExecutors is low
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-14653
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14653
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Compaction
>         Environment: Cassandra nodes :
> 3 nodes, 330G physical memory per node , and four data directory (ssd)  per node.
>            Reporter: Peter Xie
>            Priority: Major
>
> We use cassandra as backend storage for Janusgraph. when we loading huge data (~2 billion
vertex, ~10 billion edges), we met some problems.
>  
> At first, we use STCS as compaction strategy , but met below exception.  we checked
the value of  "max memory lock" is unlimited and "file map count" is 1 million, these values
should enough for loading data. last we found this problem is caused by the virtual memory
are all cosumed by cassandra.  So not additional virtual memory can be used by compaction
task , and below exception is thrown out.   
> {quote}ERROR [CompactionExecutor:267] 2018-08-09 02:28:40,952 JVMStabilityInspector.javv
>  a:74 - OutOfMemory error letting the JVM handle the error:
>  java.lang.OutOfMemoryError: Map failed
> {quote}
> So, we change compaction strategy to LCS, this change seems can resolve the virtual memory
problem. But we found another problem : Many sstables which has been compacted are still retained on
disk,  these old sstables consume so many disk space, it's causing no enough disk for saving
real data. we found that so many files like "mc_txn_compaction_xxx.log" are created under
the data directory. 
> After some times' investigaton, we found that this problem is caused by "NonPeriodicTasks"
thread pools.  this pools is always using only one thread for processing clean task and compaction.
this thread pool is instanced with class DebuggableScheduledThreadPoolExecutor,
> and DebuggableScheduledThreadPoolExecutor is inherit from class  ScheduledThreadPoolExecutor.
> By reading the code of class DebuggableScheduledThreadPoolExecutor,  found DebuggableScheduledThreadPoolExecutor
is using an unbound task queue, and core pool size is 1.   Why here use the unbound queue
for queuing submitted tasks?  If we using unbound queue, the thread pool wouldn't  increasing
thread even there so many task are blocked in queue, because unbound queue never would be
full.  I think here should use bound queue, so when task is heavily, more threads would created
for processing them. 
> {quote}public DebuggableScheduledThreadPoolExecutor(int corePoolSize, String threadPoolName,
int priority)
>  Unknown macro: \{ super(corePoolSize, new NamedThreadFactory(threadPoolName, priority));
setRejectedExecutionHandler(rejectedExecutionHandler); }
>   
> public ScheduledThreadPoolExecutor(int corePoolSize,
>  ThreadFactory threadFactory)
>  Unknown macro: \{ super(corePoolSize, Integer.MAX_VALUE, 0, NANOSECONDS, new DelayedWorkQueue(),
threadFactory); }
> {quote}
>  Below is the case about clean task after compaction.  there nearly 3 hours delay
for removing file "mc-56525". 
> {quote} 
> TRACE [CompactionExecutor:81] 2018-08-16 21:22:29,664 LifecycleTransaction.java:363 -
Staging for obsolescence BigTableReader(path='/sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big-Data.db')
>  ..........
>  TRACE [CompactionExecutor:81] 2018-08-16 21:22:41,162 Tracker.java:165 - removing /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big
from list of files tracked for test_2.edgestore
>  ............
>  TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,179 SSTableReader.java:2175 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, before
barrier
>  TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,180 SSTableReader.java:2181 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, after
barrier
>  TRACE [NonPeriodicTasks:1] 2018-08-17 00:28:47,182 SSTableReader.java:2196 - Async instance
tidier for /sdb/data/test_2/edgestore-365b0b70a05911e8806001ebe60a5ce7/mc-56525-big, completed
> {quote}
>  
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message