Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E4313173C0 for ; Wed, 10 Jun 2015 12:39:00 +0000 (UTC) Received: (qmail 88829 invoked by uid 500); 10 Jun 2015 12:39:00 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 88779 invoked by uid 500); 10 Jun 2015 12:39:00 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 88767 invoked by uid 99); 10 Jun 2015 12:39:00 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Jun 2015 12:39:00 +0000 Date: Wed, 10 Jun 2015 12:39:00 +0000 (UTC) From: "HuanWang (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (YARN-3791) FSDownload MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-3791?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:all-tabpanel ] HuanWang updated YARN-3791: --------------------------- Description:=20 Inadvertently,we set two source ftp path: {code} { { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FILE, nul= l },pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,= DOWNLOADING} ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null }= ,pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOW= NLOADING} the first one is a wrong path,only one source was set this;but Follow the l= og,i saw Starting from the first path source download,All next jobs sources= were downloaded from ftp://10.27.178.207 by default. the log is : 2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.Res= ourceLocalizationService (ResourceLocalizationService.java:addResource(544)= ) - Downloading public rsrc:{ ftp://10.27.89.13:21/home/cbt/common/2/sql.ja= r, 1433225415000, FILE, null } 2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.Res= ourceLocalizationService (ResourceLocalizationService.java:addResource(544)= ) - Downloading public rsrc:{ ftp://10.27.178.207:21/home/cbt/1213/jxf.sql,= 1433225510000, FILE, null } 2015-06-09 11:14:37,883 INFO [Public Localizer] localizer.ResourceLocaliza= tionService (ResourceLocalizationService.java:run(672)) - Failed to downloa= d rsrc { { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FIL= E, null },pending,[(container_20150608111420_41540_1213_1503_)],42376408671= 18938,DOWNLOADING} java.io.IOException: Login failed on server - 10.27.178.207, port - 21 at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.jav= a:133) at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSyst= em.java:390) at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.ja= va:172) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:279) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:52) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-09 11:14:37,885 INFO [Public Localizer] localizer.LocalizedResourc= e (LocalizedResource.java:handle(203)) - Resource ftp://10.27.178.207:21/ho= me/cbt/1213/jxf.sql transitioned from DOWNLOADING to FAILED 2015-06-09 11:14:37,886 INFO [Public Localizer] localizer.ResourceLocaliza= tionService (ResourceLocalizationService.java:run(672)) - Failed to downloa= d rsrc { { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, F= ILE, null },pending,[(container_20150608111420_41540_1213_1503_)],423764086= 6988089,DOWNLOADING} java.io.IOException: Login failed on server - 10.27.178.207, port - 21 at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.jav= a:133) at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSyst= em.java:390) at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.ja= va:172) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:279) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:52) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-09 11:14:37,886 INFO [AsyncDispatcher event handler] container.Con= tainer (ContainerImpl.java:handle(853)) - Container container_2015060811142= 0_41540_1213_1503_ transitioned from LOCALIZING to LOCALIZATION_FAILED 2015-06-09 11:14:37,887 INFO [Public Localizer] localizer.LocalizedResourc= e (LocalizedResource.java:handle(203)) - Resource ftp://10.27.89.13:21/home= /cbt/common/2/sql.jar transitioned from DOWNLOADING to FAILED 2015-06-09 11:14:37,887 INFO [AsyncDispatcher event handler] localizer.Loc= alResourcesTrackerImpl (LocalResourcesTrackerImpl.java:handle(133)) - Conta= iner container_20150608111420_41540_1213_1503_ sent RELEASE event on a reso= urce request { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 143322541500= 0, FILE, null } not present {code} I debug the code of yarn.I found the piont is org.apache.hadoop.fs.FileSyst= em#cache=20 the code source is here: {code} private FileSystem getInternal(URI uri, Configuration conf, Key key) th= rows IOException{ FileSystem fs; synchronized (this) { fs =3D map.get(key); } if (fs !=3D null) { return fs; } fs =3D createFileSystem(uri, conf); synchronized (this) { // refetch the lock again FileSystem oldfs =3D map.get(key); if (oldfs !=3D null) { // a file system is created while lock is re= leasing fs.close(); // close the new file system return oldfs; // return the old file system } =20 // now insert the new file system into the map if (map.isEmpty() && !ShutdownHookManager.get().isShutdownInProgress()) { ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUTDO= WN_HOOK_PRIORITY); } fs.key =3D key; map.put(key, fs); if (conf.getBoolean("fs.automatic.close", true)) { toAutoClose.add(key); } return fs; } } =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D FTPFileSystem.java @Override public void initialize(URI uri, Configuration conf) throws IOException { = // get super.initialize(uri, conf); // get host information from uri (overrides info in conf) String host =3D uri.getHost(); host =3D (host =3D=3D null) ? conf.get("fs.ftp.host", null) : host; if (host =3D=3D null) { throw new IOException("Invalid host specified"); } conf.set("fs.ftp.host", host); // get port information from uri, (overrides info in conf) int port =3D uri.getPort(); port =3D (port =3D=3D -1) ? FTP.DEFAULT_PORT : port; conf.setInt("fs.ftp.host.port", port); // get user/password information from URI (overrides info in conf) String userAndPassword =3D uri.getUserInfo(); if (userAndPassword =3D=3D null) { userAndPassword =3D (conf.get("fs.ftp.user." + host, null) + ":" + co= nf .get("fs.ftp.password." + host, null)); if (userAndPassword =3D=3D null) { throw new IOException("Invalid user/passsword specified"); } } String[] userPasswdInfo =3D userAndPassword.split(":"); conf.set("fs.ftp.user." + host, userPasswdInfo[0]); if (userPasswdInfo.length > 1) { conf.set("fs.ftp.password." + host, userPasswdInfo[1]); } else { conf.set("fs.ftp.password." + host, null); } setConf(conf); this.uri =3D uri; } {code} firstly.we have a source in ftp://10.27.89.13:21 then cache will store thi= s key ,and fs.ftp.host=3D10.27.89.13 ,fs.ftp.user.10.27.89.13=3DXX and=20 fs.ftp.password.10.27.89.13 will be set in conf. secondly,the source in f= tp:/10.27.178.207 come, this is not exit in cache. cache will store this ke= y ,and ,and fs.ftp.host=3D10.27.178.207 ,fs.ftp.user.10.27.178.207=3DXX and= =20 fs.ftp.password.10.27.178.207 will be set in conf. the key point is tha= t : {code} /** * Connect to the FTP server using configuration parameters * *=20 * @return An FTPClient instance * @throws IOException */ private FTPClient connect() throws IOException { FTPClient client =3D null; Configuration conf =3D getConf(); String host =3D conf.get("fs.ftp.host"); int port =3D conf.getInt("fs.ftp.host.port", FTP.DEFAULT_PORT); String user =3D conf.get("fs.ftp.user." + host); String password =3D conf.get("fs.ftp.password." + host); client =3D new FTPClient(); client.connect(host, port); int reply =3D client.getReplyCode(); if (!FTPReply.isPositiveCompletion(reply)) { throw new IOException("Server - " + host + " refused connection on port - " + port); } else if (client.login(user, password)) { client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE); client.setFileType(FTP.BINARY_FILE_TYPE); client.setBufferSize(DEFAULT_BUFFER_SIZE); } else { throw new IOException("Login failed on server - " + host + ", port - = " + port); } return client; } {code} FTPFileSystem use conf to get host .port .username and password. after the first two steps. the fs.ftp.host in conf is set to 10.27.178.207. thirdly,a source in ftp://10.27.89.13:21 come.cache find there is exit. so ftpClient use cache to connect. But the fs.ftp.host in conf is 10.27.178= .207!!!! it's Confusing=EF=BC=81=EF=BC=81=EF=BC=81 was: Inadvertently,we set two source ftp path: { { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FILE, nul= l },pending,[(container_20150608111420_41540_1213_1503_)],4237640867118938,= DOWNLOADING} ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null }= ,pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,DOW= NLOADING} the first one is a wrong path,only one source was set this;but Follow the l= og,i saw Starting from the first path source download,All next jobs sources= were downloaded from ftp://10.27.178.207 by default. the log is : 2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.Res= ourceLocalizationService (ResourceLocalizationService.java:addResource(544)= ) - Downloading public rsrc:{ ftp://10.27.89.13:21/home/cbt/common/2/sql.ja= r, 1433225415000, FILE, null } 2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.Res= ourceLocalizationService (ResourceLocalizationService.java:addResource(544)= ) - Downloading public rsrc:{ ftp://10.27.178.207:21/home/cbt/1213/jxf.sql,= 1433225510000, FILE, null } 2015-06-09 11:14:37,883 INFO [Public Localizer] localizer.ResourceLocaliza= tionService (ResourceLocalizationService.java:run(672)) - Failed to downloa= d rsrc { { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FIL= E, null },pending,[(container_20150608111420_41540_1213_1503_)],42376408671= 18938,DOWNLOADING} java.io.IOException: Login failed on server - 10.27.178.207, port - 21 at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.jav= a:133) at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSyst= em.java:390) at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.ja= va:172) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:279) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:52) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-09 11:14:37,885 INFO [Public Localizer] localizer.LocalizedResourc= e (LocalizedResource.java:handle(203)) - Resource ftp://10.27.178.207:21/ho= me/cbt/1213/jxf.sql transitioned from DOWNLOADING to FAILED 2015-06-09 11:14:37,886 INFO [Public Localizer] localizer.ResourceLocaliza= tionService (ResourceLocalizationService.java:run(672)) - Failed to downloa= d rsrc { { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, F= ILE, null },pending,[(container_20150608111420_41540_1213_1503_)],423764086= 6988089,DOWNLOADING} java.io.IOException: Login failed on server - 10.27.178.207, port - 21 at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.jav= a:133) at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSyst= em.java:390) at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.ja= va:172) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:279) at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.ja= va:52) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-09 11:14:37,886 INFO [AsyncDispatcher event handler] container.Con= tainer (ContainerImpl.java:handle(853)) - Container container_2015060811142= 0_41540_1213_1503_ transitioned from LOCALIZING to LOCALIZATION_FAILED 2015-06-09 11:14:37,887 INFO [Public Localizer] localizer.LocalizedResourc= e (LocalizedResource.java:handle(203)) - Resource ftp://10.27.89.13:21/home= /cbt/common/2/sql.jar transitioned from DOWNLOADING to FAILED 2015-06-09 11:14:37,887 INFO [AsyncDispatcher event handler] localizer.Loc= alResourcesTrackerImpl (LocalResourcesTrackerImpl.java:handle(133)) - Conta= iner container_20150608111420_41540_1213_1503_ sent RELEASE event on a reso= urce request { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 143322541500= 0, FILE, null } not present I debug the code of yarn.I found the piont is org.apache.hadoop.fs.FileSyst= em#cache=20 the code source is here: private FileSystem getInternal(URI uri, Configuration conf, Key key) th= rows IOException{ FileSystem fs; synchronized (this) { fs =3D map.get(key); } if (fs !=3D null) { return fs; } fs =3D createFileSystem(uri, conf); synchronized (this) { // refetch the lock again FileSystem oldfs =3D map.get(key); if (oldfs !=3D null) { // a file system is created while lock is re= leasing fs.close(); // close the new file system return oldfs; // return the old file system } =20 // now insert the new file system into the map if (map.isEmpty() && !ShutdownHookManager.get().isShutdownInProgress()) { ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUTDO= WN_HOOK_PRIORITY); } fs.key =3D key; map.put(key, fs); if (conf.getBoolean("fs.automatic.close", true)) { toAutoClose.add(key); } return fs; } } =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D FTPFileSystem.java @Override public void initialize(URI uri, Configuration conf) throws IOException { = // get super.initialize(uri, conf); // get host information from uri (overrides info in conf) String host =3D uri.getHost(); host =3D (host =3D=3D null) ? conf.get("fs.ftp.host", null) : host; if (host =3D=3D null) { throw new IOException("Invalid host specified"); } conf.set("fs.ftp.host", host); // get port information from uri, (overrides info in conf) int port =3D uri.getPort(); port =3D (port =3D=3D -1) ? FTP.DEFAULT_PORT : port; conf.setInt("fs.ftp.host.port", port); // get user/password information from URI (overrides info in conf) String userAndPassword =3D uri.getUserInfo(); if (userAndPassword =3D=3D null) { userAndPassword =3D (conf.get("fs.ftp.user." + host, null) + ":" + co= nf .get("fs.ftp.password." + host, null)); if (userAndPassword =3D=3D null) { throw new IOException("Invalid user/passsword specified"); } } String[] userPasswdInfo =3D userAndPassword.split(":"); conf.set("fs.ftp.user." + host, userPasswdInfo[0]); if (userPasswdInfo.length > 1) { conf.set("fs.ftp.password." + host, userPasswdInfo[1]); } else { conf.set("fs.ftp.password." + host, null); } setConf(conf); this.uri =3D uri; } firstly.we have a source in ftp://10.27.89.13:21 then cache will store thi= s key ,and fs.ftp.host=3D10.27.89.13 ,fs.ftp.user.10.27.89.13=3DXX and=20 fs.ftp.password.10.27.89.13 will be set in conf. secondly,the source in f= tp:/10.27.178.207 come, this is not exit in cache. cache will store this ke= y ,and ,and fs.ftp.host=3D10.27.178.207 ,fs.ftp.user.10.27.178.207=3DXX and= =20 fs.ftp.password.10.27.178.207 will be set in conf. the key point is tha= t : /** * Connect to the FTP server using configuration parameters * *=20 * @return An FTPClient instance * @throws IOException */ private FTPClient connect() throws IOException { FTPClient client =3D null; Configuration conf =3D getConf(); String host =3D conf.get("fs.ftp.host"); int port =3D conf.getInt("fs.ftp.host.port", FTP.DEFAULT_PORT); String user =3D conf.get("fs.ftp.user." + host); String password =3D conf.get("fs.ftp.password." + host); client =3D new FTPClient(); client.connect(host, port); int reply =3D client.getReplyCode(); if (!FTPReply.isPositiveCompletion(reply)) { throw new IOException("Server - " + host + " refused connection on port - " + port); } else if (client.login(user, password)) { client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE); client.setFileType(FTP.BINARY_FILE_TYPE); client.setBufferSize(DEFAULT_BUFFER_SIZE); } else { throw new IOException("Login failed on server - " + host + ", port - = " + port); } return client; } FTPFileSystem use conf to get host .port .username and password. after the first two steps. the fs.ftp.host in conf is set to 10.27.178.207. thirdly,a source in ftp://10.27.89.13:21 come.cache find there is exit. so ftpClient use cache to connect. But the fs.ftp.host in conf is 10.27.178= .207!!!! it's Confusing=EF=BC=81=EF=BC=81=EF=BC=81 > FSDownload > ---------- > > Key: YARN-3791 > URL: https://issues.apache.org/jira/browse/YARN-3791 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 2.4.0 > Environment: Linux 2.6.32-279.el6.x86_64=20 > Reporter: HuanWang > > Inadvertently,we set two source ftp path: > {code} > { { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, FILE, n= ull },pending,[(container_20150608111420_41540_1213_1503_)],423764086711893= 8,DOWNLOADING} > ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000, FILE, null= },pending,[(container_20150608111420_41540_1213_1503_)],4237640866988089,D= OWNLOADING} > the first one is a wrong path,only one source was set this;but Follow the= log,i saw Starting from the first path source download,All next jobs sourc= es were downloaded from ftp://10.27.178.207 by default. > the log is : > > 2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.R= esourceLocalizationService (ResourceLocalizationService.java:addResource(54= 4)) - Downloading public rsrc:{ ftp://10.27.89.13:21/home/cbt/common/2/sql.= jar, 1433225415000, FILE, null } > 2015-06-09 11:14:34,653 INFO [AsyncDispatcher event handler] localizer.R= esourceLocalizationService (ResourceLocalizationService.java:addResource(54= 4)) - Downloading public rsrc:{ ftp://10.27.178.207:21/home/cbt/1213/jxf.sq= l, 1433225510000, FILE, null } > 2015-06-09 11:14:37,883 INFO [Public Localizer] localizer.ResourceLocali= zationService (ResourceLocalizationService.java:run(672)) - Failed to downl= oad rsrc { { ftp://10.27.178.207:21/home/cbt/1213/jxf.sql, 1433225510000, F= ILE, null },pending,[(container_20150608111420_41540_1213_1503_)],423764086= 7118938,DOWNLOADING} > java.io.IOException: Login failed on server - 10.27.178.207, port - 21 > at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.j= ava:133) > at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSy= stem.java:390) > at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.= java:172) > at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.= java:279) > at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.= java:52) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2015-06-09 11:14:37,885 INFO [Public Localizer] localizer.LocalizedResou= rce (LocalizedResource.java:handle(203)) - Resource ftp://10.27.178.207:21/= home/cbt/1213/jxf.sql transitioned from DOWNLOADING to FAILED > 2015-06-09 11:14:37,886 INFO [Public Localizer] localizer.ResourceLocali= zationService (ResourceLocalizationService.java:run(672)) - Failed to downl= oad rsrc { { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415000,= FILE, null },pending,[(container_20150608111420_41540_1213_1503_)],4237640= 866988089,DOWNLOADING} > java.io.IOException: Login failed on server - 10.27.178.207, port - 21 > at org.apache.hadoop.fs.ftp.FTPFileSystem.connect(FTPFileSystem.j= ava:133) > at org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSy= stem.java:390) > at com.suning.cybertron.superion.util.FSDownload.copy(FSDownload.= java:172) > at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.= java:279) > at com.suning.cybertron.superion.util.FSDownload.call(FSDownload.= java:52) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2015-06-09 11:14:37,886 INFO [AsyncDispatcher event handler] container.C= ontainer (ContainerImpl.java:handle(853)) - Container container_20150608111= 420_41540_1213_1503_ transitioned from LOCALIZING to LOCALIZATION_FAILED > 2015-06-09 11:14:37,887 INFO [Public Localizer] localizer.LocalizedResou= rce (LocalizedResource.java:handle(203)) - Resource ftp://10.27.89.13:21/ho= me/cbt/common/2/sql.jar transitioned from DOWNLOADING to FAILED > 2015-06-09 11:14:37,887 INFO [AsyncDispatcher event handler] localizer.L= ocalResourcesTrackerImpl (LocalResourcesTrackerImpl.java:handle(133)) - Con= tainer container_20150608111420_41540_1213_1503_ sent RELEASE event on a re= source request { ftp://10.27.89.13:21/home/cbt/common/2/sql.jar, 1433225415= 000, FILE, null } not present > {code} > I debug the code of yarn.I found the piont is org.apache.hadoop.fs.FileSy= stem#cache=20 > the code source is here: > {code} > private FileSystem getInternal(URI uri, Configuration conf, Key key) = throws IOException{ > FileSystem fs; > synchronized (this) { > fs =3D map.get(key); > } > if (fs !=3D null) { > return fs; > } > fs =3D createFileSystem(uri, conf); > synchronized (this) { // refetch the lock again > FileSystem oldfs =3D map.get(key); > if (oldfs !=3D null) { // a file system is created while lock is = releasing > fs.close(); // close the new file system > return oldfs; // return the old file system > } > =20 > // now insert the new file system into the map > if (map.isEmpty() > && !ShutdownHookManager.get().isShutdownInProgress()) { > ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUT= DOWN_HOOK_PRIORITY); > } > fs.key =3D key; > map.put(key, fs); > if (conf.getBoolean("fs.automatic.close", true)) { > toAutoClose.add(key); > } > return fs; > } > } > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > FTPFileSystem.java > @Override > public void initialize(URI uri, Configuration conf) throws IOException = { // get > super.initialize(uri, conf); > // get host information from uri (overrides info in conf) > String host =3D uri.getHost(); > host =3D (host =3D=3D null) ? conf.get("fs.ftp.host", null) : host; > if (host =3D=3D null) { > throw new IOException("Invalid host specified"); > } > conf.set("fs.ftp.host", host); > // get port information from uri, (overrides info in conf) > int port =3D uri.getPort(); > port =3D (port =3D=3D -1) ? FTP.DEFAULT_PORT : port; > conf.setInt("fs.ftp.host.port", port); > // get user/password information from URI (overrides info in conf) > String userAndPassword =3D uri.getUserInfo(); > if (userAndPassword =3D=3D null) { > userAndPassword =3D (conf.get("fs.ftp.user." + host, null) + ":" + = conf > .get("fs.ftp.password." + host, null)); > if (userAndPassword =3D=3D null) { > throw new IOException("Invalid user/passsword specified"); > } > } > String[] userPasswdInfo =3D userAndPassword.split(":"); > conf.set("fs.ftp.user." + host, userPasswdInfo[0]); > if (userPasswdInfo.length > 1) { > conf.set("fs.ftp.password." + host, userPasswdInfo[1]); > } else { > conf.set("fs.ftp.password." + host, null); > } > setConf(conf); > this.uri =3D uri; > } > {code} > firstly.we have a source in ftp://10.27.89.13:21 then cache will store t= his key ,and fs.ftp.host=3D10.27.89.13 ,fs.ftp.user.10.27.89.13=3DXX and=20 > fs.ftp.password.10.27.89.13 will be set in conf. secondly,the source in= ftp:/10.27.178.207 come, this is not exit in cache. cache will store this = key ,and ,and fs.ftp.host=3D10.27.178.207 ,fs.ftp.user.10.27.178.207=3DXX a= nd=20 > fs.ftp.password.10.27.178.207 will be set in conf. the key point is t= hat : > {code} > /** > * Connect to the FTP server using configuration parameters * > *=20 > * @return An FTPClient instance > * @throws IOException > */ > private FTPClient connect() throws IOException { > FTPClient client =3D null; > Configuration conf =3D getConf(); > String host =3D conf.get("fs.ftp.host"); > int port =3D conf.getInt("fs.ftp.host.port", FTP.DEFAULT_PORT); > String user =3D conf.get("fs.ftp.user." + host); > String password =3D conf.get("fs.ftp.password." + host); > client =3D new FTPClient(); > client.connect(host, port); > int reply =3D client.getReplyCode(); > if (!FTPReply.isPositiveCompletion(reply)) { > throw new IOException("Server - " + host > + " refused connection on port - " + port); > } else if (client.login(user, password)) { > client.setFileTransferMode(FTP.BLOCK_TRANSFER_MODE); > client.setFileType(FTP.BINARY_FILE_TYPE); > client.setBufferSize(DEFAULT_BUFFER_SIZE); > } else { > throw new IOException("Login failed on server - " + host + ", port = - " > + port); > } > return client; > } > {code} > FTPFileSystem use conf to get host .port .username and password. > after the first two steps. the fs.ftp.host in conf is set to 10.27.178.20= 7. > thirdly,a source in ftp://10.27.89.13:21 come.cache find there is exit. > so ftpClient use cache to connect. But the fs.ftp.host in conf is 10.27.1= 78.207!!!! it's Confusing=EF=BC=81=EF=BC=81=EF=BC=81 -- This message was sent by Atlassian JIRA (v6.3.4#6332)