hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5776) Support 'hedged' reads in DFSClient
Date Wed, 19 Feb 2014 19:44:26 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

stack updated HDFS-5776:

    Attachment: HDFS-5776v21.txt

This patch has a few small differences that come of some time spent testing:

1. Adds DEBUG level logging of the one-time setup of the hedged reads pool.
2. Gives the hedged read pool threads a 'hedged' prefix.
3. Changes the 'cancel' behavior so it does NOT cancel ongoing reads.

3. is the biggest change.  What I've found is that hdfs reads do not take kindly to being
interrupted.  The exception types that bubble up are of a few versions -- InterruptedIOException,
ClosedByInterruptException, and IOEs whose cause is a IE -- but I also encountered complaints
coming up out of protobuf decoding messages likely because the read was cancelled partway
through.  Then there was a bunch of logging noise -- WARN-level logging -- because of the
interrupt exceptions and the fact that on interrupt, the node we were reading against would
get added to the dead list.

I had a patch that was more involved dealing w/ the interrupt exceptions and redoing the WARNs
but it was getting very involved and I was coming to rely on an untrod path, that of interrupted
reads.... so I let it go for now for now.

This patch lets outstanding reads finish.

Let me chat w/ [~xieliang007] to possibly get production numbers on benefit of patch as is.

> Support 'hedged' reads in DFSClient
> -----------------------------------
>                 Key: HDFS-5776
>                 URL: https://issues.apache.org/jira/browse/HDFS-5776
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-5776-v10.txt, HDFS-5776-v11.txt, HDFS-5776-v12.txt, HDFS-5776-v12.txt,
HDFS-5776-v13.wip.txt, HDFS-5776-v14.txt, HDFS-5776-v15.txt, HDFS-5776-v17.txt, HDFS-5776-v17.txt,
HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, HDFS-5776-v5.txt, HDFS-5776-v6.txt,
HDFS-5776-v7.txt, HDFS-5776-v8.txt, HDFS-5776-v9.txt, HDFS-5776.txt, HDFS-5776v18.txt, HDFS-5776v21.txt
> This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size"
to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics,
we could export the interested metric valus into client system(e.g. HBase's regionserver metric).
> The core logic is in pread code path, we decide to goto the original fetchBlockByteRange
or the new introduced fetchBlockByteRangeSpeculative per the above config items.

This message was sent by Atlassian JIRA

View raw message