hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9807) Add an optional StorageID to writes
Date Wed, 07 Jun 2017 16:56:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041207#comment-16041207
] 

Andrew Wang commented on HDFS-9807:
-----------------------------------

I believe this (or HDFS-6708) broke backwards compatibility with older clients. I'm running
a 2.6.0-ish client against a 3.0.0-alpha4-ish DN, and am seeing this in the DN log:

{noformat}
2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block token
verification failed: op=WRITE_BLOCK, remoteAddress=/172.28.208.200:53900, message=Block token
with StorageIDs [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with StorageIDs
[]
{noformat}

The client then blacklists each DN when it gets this error, until it runs out of DNs. Dunno
if this retry behavior is fixed in newer clients; InvalidBlockTokenExceptions are kind of
fatal.

What is the expected behavior here?

> Add an optional StorageID to writes
> -----------------------------------
>
>                 Key: HDFS-9807
>                 URL: https://issues.apache.org/jira/browse/HDFS-9807
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Chris Douglas
>            Assignee: Ewan Higgs
>             Fix For: 3.0.0-alpha4
>
>         Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch, HDFS-9807.003.patch, HDFS-9807.004.patch,
HDFS-9807.005.patch, HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch, HDFS-9807.009.patch,
HDFS-9807.010.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the replica is written
the DN {{VolumeChoosingPolicy}} is unaware of any preference or constraints from other policies
affecting placement. This limits heterogeneity to the declared storage types, which are treated
as fungible within the target DN. It should be possible to influence or constrain the DN policy
to select a particular storage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message