hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Percy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5131) Need a DEFAULT-like pipeline recovery policy that works for writers that flush
Date Mon, 26 Aug 2013 09:34:52 GMT
Mike Percy created HDFS-5131:

             Summary: Need a DEFAULT-like pipeline recovery policy that works for writers
that flush
                 Key: HDFS-5131
                 URL: https://issues.apache.org/jira/browse/HDFS-5131
             Project: Hadoop HDFS
          Issue Type: Improvement
    Affects Versions: 2.0.6-alpha
            Reporter: Mike Percy

The Hadoop 2 pipeline-recovery mechanism currently has four policies: DISABLE (never do recovery),
NEVER (never do recovery unless client asks for it), ALWAYS (block until we have recovered
the write pipeline to minimum replication levels), and DEFAULT (try to do ALWAYS, but use
a heuristic to "give up" and allow writers to continue if not enough datanodes are available
to recover the pipeline).

The big problem with default is that it specifically falls back to ALWAYS behavior if a client
calls hflush(). On its face, it seems like a reasonable thing to do, but in practice this
means that clients like Flume (as well as, I assume, HBase) just block when the cluster is
low on datanodes.

In order to work around this issue, the easiest thing to do today is set the policy to NEVER
when using Flume to write to the cluster. But obviously that's not ideal.

I believe what clients like Flume need is an additional policy which essentially uses the
heuristic logic used by DEFAULT even in cases where long-lived writers call hflush().

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message