From common-issues-return-159240-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Oct 10 08:46:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 355B3180672 for ; Wed, 10 Oct 2018 08:46:04 +0200 (CEST) Received: (qmail 45610 invoked by uid 500); 10 Oct 2018 06:46:03 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 45599 invoked by uid 99); 10 Oct 2018 06:46:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Oct 2018 06:46:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A997E1A227B for ; Wed, 10 Oct 2018 06:46:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id l67Hwsfgk5-5 for ; Wed, 10 Oct 2018 06:46:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 52C9B5F43B for ; Wed, 10 Oct 2018 06:46:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id BAF5EE25B0 for ; Wed, 10 Oct 2018 06:46:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 5190624821 for ; Wed, 10 Oct 2018 06:46:00 +0000 (UTC) Date: Wed, 10 Oct 2018 06:46:00 +0000 (UTC) From: "LinJi (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HADOOP-15838) Copy files from SFTP to HDFS using DistCp failed with error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-15838?page=3Dcom.atlass= ian.jira.plugin.system.issuetabpanels:all-tabpanel ] LinJi updated HADOOP-15838: --------------------------- Attachment: (was: =E5=BE=AE=E4=BF=A1=E6=88=AA=E5=9B=BE_201810102243= 16.png) > Copy files from SFTP to HDFS using DistCp failed with error > ----------------------------------------------------------- > > Key: HADOOP-15838 > URL: https://issues.apache.org/jira/browse/HADOOP-15838 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp > Affects Versions: 2.5.0, 2.7.2 > Environment: Hadoop 2.5.0 + kerberos > Reporter: LinJi > Priority: Major > Labels: distcp, hadoop-tools > Fix For: 2.7.5 > > Original Estimate: 96h > Remaining Estimate: 96h > > 1. When I run command: > {code:java} > hadoop distcp sftp://mysftp:1qaz_@WSX@192.168.1.44:/upload/hosts /tmp/JOY= {code} > =C2=A0 > I got error like: > =C2=A0 > {noformat} > 2018-10-10 22:31:37,799 INFO util.KerberosUtil: Using principal pattern: = HTTP/_HOST > 2018-10-10 22:31:39,055 INFO tools.DistCp: Input Options: DistCpOptions{a= tomicCommit=3Dfalse, syncFolder=3Dfalse, deleteMissing=3Dfalse, ignoreFailu= res=3Dfalse, maxMaps=3D20, sslConfigurationFile=3D'null', copyStrategy=3D'u= niformsize', sourceFileListing=3Dnull, sourcePaths=3D[sftp://mysftp:1qaz_@W= SX@192.168.1.44:/upload/hosts], targetPath=3D/tmp/JOY, targetPathExists=3Df= alse} > 2018-10-10 22:31:39,365 ERROR tools.DistCp: Exception encountered > java.io.IOException: Invalid host specified > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.sftp.S= FTPFileSystem.initialize(SFTPFileSystem.java:67) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileSy= stem.createFileSystem(FileSystem.java:2591) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileSy= stem.access$200(FileSystem.java:89) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileSy= stem$Cache.getInternal(FileSystem.java:2643) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileSy= stem$Cache.get(FileSystem.java:2625) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileSy= stem.get(FileSystem.java:368) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.Path.g= etFileSystem(Path.java:296) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Glo= bbedCopyListing.doBuildListing(GlobbedCopyListing.java:76) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Cop= yListing.buildListing(CopyListing.java:84) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.createInputFileListing(DistCp.java:353) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.execute(DistCp.java:160) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.run(DistCp.java:121) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.Tool= Runner.run(ToolRunner.java:70) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.main(DistCp.java:401) > {noformat} > =C2=A0 > 2. When I run command: > {code:java} > hadoop distcp sftp://mysftp:1qaz_%40WSX@192.168.1.44:/upload/hosts /tmp/J= OY{code} > I got error like: > {noformat} > 2018-10-10 22:31:59,909 INFO util.KerberosUtil: Using principal pattern: = HTTP/_HOST > 2018-10-10 22:32:01,286 INFO tools.DistCp: Input Options: DistCpOptions{a= tomicCommit=3Dfalse, syncFolder=3Dfalse, deleteMissing=3Dfalse, ignoreFailu= res=3Dfalse, maxMaps=3D20, sslConfigurationFile=3D'null', copyStrategy=3D'u= niformsize', sourceFileListing=3Dnull, sourcePaths=3D[sftp://mysftp:1qaz_%4= 0WSX@192.168.1.44:/upload/hosts], targetPath=3D/tmp/JOY, targetPathExists= =3Dfalse} > 2018-10-10 22:32:02,190 ERROR tools.DistCp: Exception encountered > java.io.IOException: SSH_MSG_DISCONNECT: 2 Too many authentication failur= es for mysftp > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.sftp.S= FTPFileSystem.connect(SFTPFileSystem.java:143) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0at org.apache.hadoop.fs.sftp.S= FTPFileSystem.getFileStatus(SFTPFileSystem.java:371) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.Globbe= r.getFileStatus(Globber.java:57) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.Globbe= r.glob(Globber.java:252) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileSy= stem.globStatus(FileSystem.java:1623) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Glo= bbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Cop= yListing.buildListing(CopyListing.java:84) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.createInputFileListing(DistCp.java:353) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.execute(DistCp.java:160) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.run(DistCp.java:121) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.Tool= Runner.run(ToolRunner.java:70) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.tools.Dis= tCp.main(DistCp.java:401){noformat} > The SFTP username is=C2=A0mysftp > password is 1qaz_@WSX > =C2=A0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org