hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9945) Datanode command for evicting writers
Date Tue, 15 Mar 2016 14:34:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195386#comment-15195386
] 

Kihwal Lee commented on HDFS-9945:
----------------------------------

HDFS-2043 TestHFlush
HDFS-9780 TestRollingFileSystemSinkWithSecureHdfs
HDFS-9950 TestDecommissioningStatus
HDFS-10169 TestEditLog 
HDFS-9767 TestFileAppend
HDFS-6532 TestCrcCorruption
I will work on some of these.

The two checkstyle warnings are about the existing method length being over 150 lines.

> Datanode command for evicting writers
> -------------------------------------
>
>                 Key: HDFS-9945
>                 URL: https://issues.apache.org/jira/browse/HDFS-9945
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-9945.patch, HDFS-9945.v2.patch
>
>
> It will be useful if there is a command to evict writers from a datanode. When a set
of datanodes are being decommissioned, they can get blocked by slow writers at the end.  It
was rare in the old days since mapred jobs didn't last too long, but with many different types
of apps running on today's YARN cluster, we are often see very long tail in datanode decommissioning.
> I propose a new dfsadmin command, {{evictWriters}}, to be added. I initially thought
about having namenode automatically telling datanodes on decommissioning, but realized that
having a command is more flexible. E.g. users can choose not to do this at all, choose when
to evict writers, or whether to try multiple times for whatever reasons.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message