hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Krogen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14272) [SBN read] HDFS command line tools does not guarantee consistency
Date Wed, 13 Feb 2019 18:04:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767418#comment-16767418

Erik Krogen commented on HDFS-14272:

I don't see us handling this logic anywhere. I think we need to enhance the checks in {{ObserverReadProxyProvider.ObserverReadInvocationHandler#invoke()}}
so that the very first command always goes to active, thus fetching the active stateID. cc
[~vagarychen] [~csun] [~shv]

> [SBN read] HDFS command line tools does not guarantee consistency
> -----------------------------------------------------------------
>                 Key: HDFS-14272
>                 URL: https://issues.apache.org/jira/browse/HDFS-14272
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: tools
>         Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby + SSL + Kerberos
+ RPC encryption
>            Reporter: Wei-Chiu Chuang
>            Assignee: Erik Krogen
>            Priority: Major
> It is typical for integration tests to create some files and then check their existence.
For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with Consistent Standby
Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the first one, is
not aware of the state id returned from the first bash command. So ObserverNode wouldn't wait
for the the edits to get propagated, and thus fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and this becomes
very annoying. (I am still trying to figure out why this Observer has such a long RPC latency.
But that's another story.)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message