hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HDFS-5131) Need a DEFAULT-like pipeline recovery policy that works for writers that flush
Date Wed, 26 Mar 2014 01:36:15 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tsz Wo Nicholas Sze reassigned HDFS-5131:
-----------------------------------------

    Assignee: Tsz Wo Nicholas Sze

> Need a DEFAULT-like pipeline recovery policy that works for writers that flush
> ------------------------------------------------------------------------------
>
>                 Key: HDFS-5131
>                 URL: https://issues.apache.org/jira/browse/HDFS-5131
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.0.6-alpha
>            Reporter: Mike Percy
>            Assignee: Tsz Wo Nicholas Sze
>
> The Hadoop 2 pipeline-recovery mechanism currently has four policies: DISABLE (never
do recovery), NEVER (never do recovery unless client asks for it), ALWAYS (block until we
have recovered the write pipeline to minimum replication levels), and DEFAULT (try to do ALWAYS,
but use a heuristic to "give up" and allow writers to continue if not enough datanodes are
available to recover the pipeline).
> The big problem with default is that it specifically falls back to ALWAYS behavior if
a client calls hflush(). On its face, it seems like a reasonable thing to do, but in practice
this means that clients like Flume (as well as, I assume, HBase) just block when the cluster
is low on datanodes.
> In order to work around this issue, the easiest thing to do today is set the policy to
NEVER when using Flume to write to the cluster. But obviously that's not ideal.
> I believe what clients like Flume need is an additional policy which essentially uses
the heuristic logic used by DEFAULT even in cases where long-lived writers call hflush().



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message