hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13281) Namenode#createFile should be /.reserved/raw/ aware.
Date Thu, 29 Mar 2018 17:27:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419441#comment-16419441

Xiao Chen commented on HDFS-13281:

{quote}in which case there will be no step #3
Didn't get you. can you please elaborate.


I was saying: This patch works for an hdfs client, writing to /.reserved/raw. In this case,
the client writes raw data, so no encrypt in step #3, just raw bytes written:

Let's step back from webhdfs, because here we're changing the NN code. Say you're using hdfs
client, and you write to /.reserved/raw. Before this patch you'll always get a feinfo and
encrypt (which probably isn't correct. But from NN's view there's no 'write-to-raw', it just
resolves to a regular path). After this patch, you get no feinfo, and just write raw (presumably
encrypted) bytes to DN. You'll have to setxattr on that file to NN, otherwise we'll end up
with a file in HDFS that's raw and doesn't have feinfo, which is essentially corrupt data,

Along this line, for webhdfs/hdfs client, after the file is created and been written (1 block
closed, or file closed), and until setxattr, the file will be readable but undecryptable.

I think at the minimum we should setxattr immediately after the file is created. One atomic
way is perhaps pass in xattr at file creation time. What do you think [~shahrs87] [~daryn]?

> Namenode#createFile should be /.reserved/raw/ aware.
> ----------------------------------------------------
>                 Key: HDFS-13281
>                 URL: https://issues.apache.org/jira/browse/HDFS-13281
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: encryption
>    Affects Versions: 2.8.3
>            Reporter: Rushabh S Shah
>            Assignee: Rushabh S Shah
>            Priority: Critical
>         Attachments: HDFS-13281.001.patch, HDFS-13281.002.patch
> If I want to write to /.reserved/raw/<dir> and if that directory happens to be
in EZ, then namenode *should not* create edek and just copy the raw bytes from the source.
>  Namenode#startFileInt should be /.reserved/raw/ aware.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message