hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3504) HLog performance improvement
Date Fri, 04 Feb 2011 19:44:31 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990696#comment-12990696
] 

Jonathan Gray commented on HBASE-3504:
--------------------------------------

We're measuring 20-40ms increment operations with all time being spent in WAL.  When jstacking,
all threads are waiting on updateLock.  At some point (and this seems to be a common behavior
under heavy concurrency) the lock becomes such a point of contention that performance all
of a sudden drops dramatically.  We'll see 0-2ms WAL on lower loaded servers and then 20-40ms
on higher loaded servers... seemingly there is a tipping point.  This happened with other
locks as well that we have removed or moved to read/write locks.

Basically, the cluster is crawling because a couple RS have insanely long WAL times with every
thread piled on the updateLock.  It is the only current bottleneck we see (unless we turn
off the WAL).  The actual sync latency is still low, it's just the updateLock contention.

The critical point of seqids and HLogs is that the range of seqids in each HLog must never
overlap (that's where the write lock comes in) and we must be able to determine the highest
seqid in an HLog.  Also, that seqids within a row are in order.  Seqids across rows do not
need to be in order; within a region, it's only important to ensure we properly track the
oldest seqid in the MemStore and the latest seqid in each HLog.

> HLog performance improvement
> ----------------------------
>
>                 Key: HBASE-3504
>                 URL: https://issues.apache.org/jira/browse/HBASE-3504
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> The HLog.updateLock protects the rolling of logs with concurrent writes to the HDFS log
file. This is a scalability bottleneck for a workload that comprises mostly of counter-increments.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message