hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mavin Martin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13023) Distcp with -update feature on first time raw data not working
Date Thu, 14 Apr 2016 00:34:25 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mavin Martin updated HADOOP-13023:
----------------------------------
    Description: 
When attempting to do a distcp with the -update feature toggled on encrypted data, the distcp
shows as successful.  Reading the encrypted file on the target_path does not work since the
keyName does not exist.  

Please see my example to reproduce the issue.

{code}
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
/tmp/gms/ted                                DEF0000000000013
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs dfs -ls -R /tmp
drwxr-xr-x   - WD5-SVT.gmspr0022 WD5-SVT.gmspr0022          0 2016-04-14 00:22 /tmp/gms
drwxr-xr-x   - WD5-SVT.gmspr0022 WD5-SVT.gmspr0022          0 2016-04-14 00:00 /tmp/gms/ted
-rw-r--r--   3 WD5-SVT.gmspr0022 WD5-SVT.gmspr0022         33 2016-04-14 00:00 /tmp/gms/ted/test.txt
[root@769wl02.b13.az2.eng.pdx.wd bin]# hadoop distcp -update /.reserved/raw/tmp/gms/ted /.reserved/raw/tmp/gms-with-update/ted
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
/tmp/gms/ted                                DEF0000000000013
[root@769wl02.b13.az2.eng.pdx.wd bin]# hadoop distcp /.reserved/raw/tmp/gms/ted /.reserved/raw/tmp/gms-no-update/ted
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
/tmp/gms/ted                                DEF0000000000013
/tmp/gms-no-update/ted                      DEF0000000000013
{code}

The crypto zone for gms-with-update should have been created since this is a new destination.
 You can verify this by looking at gms-no-update.

  was:
When attempting to do a distcp with the -update feature toggled on encrypted data, the distcp
shows as successful.  Reading the encrypted file on the target_path does not work since the
keyName does not exist.  

Please see my example to reproduce the issue.

{code}
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
/tmp/gms/ted                                DEF0000000000013
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs dfs -ls -R /tmp
drwxr-xr-x   - WD5-SVT.gmspr0022 WD5-SVT.gmspr0022          0 2016-04-14 00:22 /tmp/gms
drwxr-xr-x   - WD5-SVT.gmspr0022 WD5-SVT.gmspr0022          0 2016-04-14 00:00 /tmp/gms/ted
-rw-r--r--   3 WD5-SVT.gmspr0022 WD5-SVT.gmspr0022         33 2016-04-14 00:00 /tmp/gms/ted/test.txt
[root@769wl02.b13.az2.eng.pdx.wd bin]# hadoop distcp -update /.reserved/raw/tmp/gms/ted /.reserved/raw/tmp/gms2/ted
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
/tmp/gms/ted                                DEF0000000000013
[root@769wl02.b13.az2.eng.pdx.wd bin]# hadoop distcp /.reserved/raw/tmp/gms/ted /.reserved/raw/tmp/gms-no-update/ted
[root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
/tmp/gms/ted                                DEF0000000000013
/tmp/gms-no-update/ted                      DEF0000000000013
{code}

The crypto zone for gms2 should have been created since this is a new destination.  You can
verify this by looking at gms-no-update.


> Distcp with -update feature on first time raw data not working
> --------------------------------------------------------------
>
>                 Key: HADOOP-13023
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13023
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Mavin Martin
>
> When attempting to do a distcp with the -update feature toggled on encrypted data, the
distcp shows as successful.  Reading the encrypted file on the target_path does not work since
the keyName does not exist.  
> Please see my example to reproduce the issue.
> {code}
> [root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
> /tmp/gms/ted                                DEF0000000000013
> [root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs dfs -ls -R /tmp
> drwxr-xr-x   - WD5-SVT.gmspr0022 WD5-SVT.gmspr0022          0 2016-04-14 00:22 /tmp/gms
> drwxr-xr-x   - WD5-SVT.gmspr0022 WD5-SVT.gmspr0022          0 2016-04-14 00:00 /tmp/gms/ted
> -rw-r--r--   3 WD5-SVT.gmspr0022 WD5-SVT.gmspr0022         33 2016-04-14 00:00 /tmp/gms/ted/test.txt
> [root@769wl02.b13.az2.eng.pdx.wd bin]# hadoop distcp -update /.reserved/raw/tmp/gms/ted
/.reserved/raw/tmp/gms-with-update/ted
> [root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
> /tmp/gms/ted                                DEF0000000000013
> [root@769wl02.b13.az2.eng.pdx.wd bin]# hadoop distcp /.reserved/raw/tmp/gms/ted /.reserved/raw/tmp/gms-no-update/ted
> [root@769wl02.b13.az2.eng.pdx.wd bin]# hdfs crypto -listZones
> /tmp/gms/ted                                DEF0000000000013
> /tmp/gms-no-update/ted                      DEF0000000000013
> {code}
> The crypto zone for gms-with-update should have been created since this is a new destination.
 You can verify this by looking at gms-no-update.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message