hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang Xie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
Date Wed, 23 Jul 2014 07:00:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14071440#comment-14071440
] 

Liang Xie commented on HDFS-6735:
---------------------------------

bq. I'd think you'd want a comment at least in LocatedBlocks#underConstruction warning an
upper layer is dependent on it being final in case LocatedBlocks changes and starts to allow
blocks complete under a stream.
done.

bq.  locatedBlocks.insertRange(targetBlockIdx, newBlocks.getLocatedBlocks()); ... be inside
a synchronization too? Could two threads be updating block locations at same time?
yes, it's possible. but we could not put a "synchronization" that, it's different from "synchronized
(this) { + pos = offset; + blockEnd = blk.getStartOffset() + blk.getBlockSize() - 1; + currentLocatedBlock
= blk; + }",  because in pread scenario, the "updatePosition" is false, so will never go into
the "synchronized (this) { + pos = offset; + blockEnd = blk.getStartOffset() + blk.getBlockSize()
- 1; + currentLocatedBlock = blk; + }".   And if we put a "synchronization" there, so if pread
reach here, it's still blocked by other monitor holder, e.g. read()  :) but we can have a
"synchronized" or rwLock in Locatedblocks class. let me try

> A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-6735
>                 URL: https://issues.apache.org/jira/browse/HDFS-6735
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-6735-v2.txt, HDFS-6735.txt
>
>
> In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread
path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor
patch against the first encourtered lock, around getFileLength, in deed, after reading code
and testing, it shows still other locks we could improve.
> In this jira, i'll make a patch against other locks, and a simple test case to show the
issue and the improved result.
> This is important for HBase application, since in current HFile read path, we issue all
read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution
is another story i had a plan to do, but probably will take more time than i expected)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message