hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-13744) OIV tool should better handle control characters present in file or directory names
Date Tue, 04 Sep 2018 21:23:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Mackrory updated HDFS-13744:
---------------------------------
    Attachment: HDFS-13744.03.patch

> OIV tool should better handle control characters present in file or directory names
> -----------------------------------------------------------------------------------
>
>                 Key: HDFS-13744
>                 URL: https://issues.apache.org/jira/browse/HDFS-13744
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs, tools
>    Affects Versions: 2.6.5, 2.9.1, 2.8.4, 2.7.6, 3.0.3
>            Reporter: Zsolt Venczel
>            Assignee: Zsolt Venczel
>            Priority: Critical
>         Attachments: HDFS-13744.01.patch, HDFS-13744.02.patch, HDFS-13744.03.patch
>
>
> In certain cases when control characters or white space is present in file or directory
names OIV tool processors can export data in a misleading format.
> In the below examples we have EXAMPLE_NAME as a file and a directory name where the directory
has a line feed character at the end (the actual production case has multiple line feeds and
multiple spaces)
>  * Delimited processor case:
>  ** misleading example:
> {code:java}
> /user/data/EXAMPLE_NAME
> ,0,2017-04-24 04:34,1969-12-31 16:00,0,0,0,-1,-1,drwxrwxr-x+,user,group
> /user/data/EXAMPLE_NAME,2016-08-26 03:00,2017-05-16 10:05,134217728,1,520,0,0,-rw-rwxr--+,user,group
> {code}
>  * 
>  ** expected example as suggested by [https://tools.ietf.org/html/rfc4180#section-2]:
> {code:java}
> "/user/data/EXAMPLE_NAME%x0A",0,2017-04-24 04:34,1969-12-31 16:00,0,0,0,-1,-1,drwxrwxr-x+,user,group
> "/user/data/EXAMPLE_NAME",2016-08-26 03:00,2017-05-16 10:05,134217728,1,520,0,0,-rw-rwxr--+,user,group
> {code}
>  * XML processor case:
>  ** misleading example:
> {code:java}
> <inode><id>479867791</id><type>DIRECTORY</type><name>EXAMPLE_NAME
> </name><mtime>1493033668294</mtime><permission>user:group:0775</permission></inode>
> <inode><id>113632535</id><type>FILE</type><name>EXAMPLE_NAME</name><replication>3</replication><mtime>1472205657504</mtime><atime>1494954320141</atime><preferredBlockSize>134217728</preferredBlockSize><permission>user:group:0674</permission></inode>
> {code}
>  * 
>  ** expected example as specified in [https://www.w3.org/TR/REC-xml/#sec-line-ends]:
> {code:java}
> <inode><id>479867791</id><type>DIRECTORY</type><name>EXAMPLE_NAME#xA</name><mtime>1493033668294</mtime><permission>user:group:0775</permission></inode>
> <inode><id>113632535</id><type>FILE</type><name>EXAMPLE_NAME</name><replication>3</replication><mtime>1472205657504</mtime><atime>1494954320141</atime><preferredBlockSize>134217728</preferredBlockSize><permission>user:group:0674</permission></inode>
> {code}
>  * JSON:
>  The OIV Web Processor behaves correctly and produces the following:
> {code:java}
> {
>   "FileStatuses": {
>     "FileStatus": [
>       {
>         "fileId": 113632535,
>         "accessTime": 1494954320141,
>         "replication": 3,
>         "owner": "user",
>         "length": 520,
>         "permission": "674",
>         "blockSize": 134217728,
>         "modificationTime": 1472205657504,
>         "type": "FILE",
>         "group": "group",
>         "childrenNum": 0,
>         "pathSuffix": "EXAMPLE_NAME"
>       },
>       {
>         "fileId": 479867791,
>         "accessTime": 0,
>         "replication": 0,
>         "owner": "user",
>         "length": 0,
>         "permission": "775",
>         "blockSize": 0,
>         "modificationTime": 1493033668294,
>         "type": "DIRECTORY",
>         "group": "group",
>         "childrenNum": 0,
>         "pathSuffix": "EXAMPLE_NAME\n"
>       }
>     ]
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message