hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daisuke Kobayashi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9455) In distcp, Invalid Argument Error thrown in case of filesystem operation failure
Date Tue, 05 Jan 2016 10:17:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082774#comment-15082774
] 

Daisuke Kobayashi commented on HDFS-9455:
-----------------------------------------

I see, [~archanat]. When using normal webhdfs protocol over SSL/TLS enabled cluster, it should
catch {{SocketException}} and print a stack trace. Here's a result of my quick test, for instance:

{noformat}
[daisuke@test1 ~]$ hadoop distcp /user/daisuke/hosts webhdfs://target:50470/
16/01/05 19:11:34 WARN security.UserGroupInformation: PriviledgedActionException as:daisuke@HADOOP
(auth:KERBEROS) cause:java.net.SocketException: Unexpected end of file from server
16/01/05 19:11:34 WARN security.UserGroupInformation: PriviledgedActionException as:daisuke@HADOOP
(auth:KERBEROS) cause:java.net.SocketException: Unexpected end of file from server
16/01/05 19:11:34 ERROR tools.DistCp: Log.error:
java.net.SocketException: Unexpected end of file from server
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:336)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:614)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1307)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:239)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:429)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:450)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:697)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:609)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:464)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:493)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:489)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:844)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:859)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1409)
	at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:206)
	at org.apache.hadoop.tools.DistCp.run(DistCp.java:113)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.tools.DistCp.main(DistCp.java:435)
LOG.error: Unexpected end of file from server
{noformat}

Does this make sense?

> In distcp, Invalid Argument Error thrown in case of filesystem operation failure
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-9455
>                 URL: https://issues.apache.org/jira/browse/HDFS-9455
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: distcp, security
>            Reporter: Archana T
>            Assignee: Archana T
>            Priority: Minor
>
> When Filesystem Operation failure happens during discp, 
> Wrong exception : Invalid Argument thrown along with distcp command Usage.
> {color:red} 
> hadoop distcp webhdfs://IP:25003/test/testfile webhdfs://IP:25003/myp
> Invalid arguments: Unexpected end of file from server
> usage: distcp OPTIONS [source_path...] <target_path>
>               OPTIONS
>  -append                       Reuse existing data in target files and
>                                append new data to them if possible
>  -async                        Should distcp execution be blocking
>  -atomic                       Commit all changes or none
>  -bandwidth <arg>              Specify bandwidth per map in MB
>  -delete                       Delete from target, files missing in source
>  -diff <arg>                   Use snapshot diff report to identify the
>                                difference between source and target
>  -f <arg>                      List of files that need to be copied
>  -filelimit <arg>              (Deprecated!) Limit number of files copied
>                                to <= n
>  -i                            Ignore failures during copy
> .....
> {color} 
> Instead Proper Exception has to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message