hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikas Vishwakarma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-16499) slow replication for small HBase clusters
Date Thu, 25 Aug 2016 04:32:20 GMT
Vikas Vishwakarma created HBASE-16499:
-----------------------------------------

             Summary: slow replication for small HBase clusters
                 Key: HBASE-16499
                 URL: https://issues.apache.org/jira/browse/HBASE-16499
             Project: HBase
          Issue Type: Bug
            Reporter: Vikas Vishwakarma
            Assignee: Vikas Vishwakarma


For small clusters 10-20 nodes we recently observed that replication is progressing very slowly
when we do bulk writes and there is lot of lag accumulation on AgeOfLastShipped / SizeOfLogQueue.
From the logs we observed that the number of threads used for shipping wal edits in parallel
comes from the following equation in HBaseInterClusterReplicationEndpoint

int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
      replicationSinkMgr.getSinks().size());
... 
      for (int i=0; i<n; i++) {
        entryLists.add(new ArrayList<HLog.Entry>(entries.size()/n+1)); <-- batch
size
      }
...
        for (int i=0; i<entryLists.size(); i++) {
         .....
            // RuntimeExceptions encountered here bubble up and are handled in ReplicationSource
            pool.submit(createReplicator(entryLists.get(i), i));  <-- concurrency 
            futures++;
          }
        }

maxThreads is fixed & configurable and since we are taking min of the three values n gets
decided based replicationSinkMgr.getSinks().size() when we have enough edits to replicate

replicationSinkMgr.getSinks().size() is decided based on 
int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
where ratio is this.ratio = conf.getFloat("replication.source.ratio", DEFAULT_REPLICATION_SOURCE_RATIO);

Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small clusters of size 10-20
RegionServers  the value we get for numSinks and hence n is very small like 1 or 2. This substantially
reduces the pool concurrency used for shipping wal edits in parallel effectively slowing down
replication for small clusters and causing lot of lag accumulation in AgeOfLastShipped. Sometimes
it takes tens of hours to clear off the entire replication queue even after the client has
finished writing on the source side. 

We are running tests by varying replication.source.ratio and have seen multi-fold improvement
in total replication time (will update the results here). I wanted to propose here that we
should increase the default value for replication.source.ratio also so that we have sufficient
concurrency even for small clusters. We figured it out after lot of iterations and debugging
so probably slightly higher default will save the trouble. 






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message