hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1148) Convert FSDataset to ReadWriteLock
Date Tue, 05 Apr 2011 18:05:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016054#comment-13016054
] 

Todd Lipcon commented on HDFS-1148:
-----------------------------------

bq. Todd, I want to understand what methods have lot contention

I actually don't remember anymore - this was a while back that I saw this, and only once I
added HDFS-941. Since it was a read workload, it makes sense that it would be getBlockInputStream,
metaFileExists, getVisibleLength, and getMetaDataInputStream.

bq. I am not sure what you mean mostly uncontended, because I understand the problem description
as there are lot of contentions

Sorry, what I meant here is that, once it's converted to read-write lock, there is very little
contention for the exclusive (write) lock. It's very rare to write small blocks, whereas small
frequent reads come often from applications like HBase. So, we mostly see lots of readers
and only the occasional writer.

> Convert FSDataset to ReadWriteLock
> ----------------------------------
>
>                 Key: HDFS-1148
>                 URL: https://issues.apache.org/jira/browse/HDFS-1148
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hdfs-1148-old.txt, patch-HDFS-1148-rel0.20.2.txt
>
>
> In benchmarking HDFS-941 I noticed that for the random read workload, the FSDataset lock
is highly contended. After converting it to a ReentrantReadWriteLock, I saw a ~25% improvement
on both latency and ops/second.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message