Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 30954 invoked from network); 21 Jul 2008 18:56:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 21 Jul 2008 18:56:23 -0000 Received: (qmail 93615 invoked by uid 500); 21 Jul 2008 18:56:23 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 93463 invoked by uid 500); 21 Jul 2008 18:56:23 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 93454 invoked by uid 99); 21 Jul 2008 18:56:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2008 11:56:23 -0700 X-ASF-Spam-Status: No, hits=-1998.9 required=10.0 tests=ALL_TRUSTED,FB_GET_MEDS X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2008 18:55:37 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 2CB79238888A; Mon, 21 Jul 2008 11:56:02 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r678527 - in /hadoop/core/branches/branch-0.18: CHANGES.txt src/core/org/apache/hadoop/fs/FileSystem.java src/hdfs/org/apache/hadoop/dfs/DistributedFileSystem.java src/test/org/apache/hadoop/fs/TestFileSystem.java Date: Mon, 21 Jul 2008 18:56:01 -0000 To: core-commits@hadoop.apache.org From: omalley@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20080721185602.2CB79238888A@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Mon Jul 21 11:56:01 2008 New Revision: 678527 URL: http://svn.apache.org/viewvc?rev=678527&view=rev Log: Merge -r 678510:678511 from trunk to branch 0.18 to fix HADOOP-3762. Modified: hadoop/core/branches/branch-0.18/CHANGES.txt hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/FileSystem.java hadoop/core/branches/branch-0.18/src/hdfs/org/apache/hadoop/dfs/DistributedFileSystem.java hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/fs/TestFileSystem.java Modified: hadoop/core/branches/branch-0.18/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/CHANGES.txt?rev=678527&r1=678526&r2=678527&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/CHANGES.txt (original) +++ hadoop/core/branches/branch-0.18/CHANGES.txt Mon Jul 21 11:56:01 2008 @@ -541,12 +541,14 @@ HADOOP-3454. Fix Text::find to search only valid byte ranges. (Chad Whipkey via cdouglas) - HADOOP-3417. Removes the static configuration variable, commandLineConfig from - JobClient. Moves the cli parsing from JobShell to GenericOptionsParser. - Thus removes the class org.apache.hadoop.mapred.JobShell. - (Amareshwari Sriramadasu via ddas) + HADOOP-3417. Removes the static configuration variable, + commandLineConfig from JobClient. Moves the cli parsing from + JobShell to GenericOptionsParser. Thus removes the class + org.apache.hadoop.mapred.JobShell. (Amareshwari Sriramadasu via + ddas) - HADOOP-2132. Only RUNNING/PREP jobs can be killed. (Jothi Padmanabhan via ddas) + HADOOP-2132. Only RUNNING/PREP jobs can be killed. (Jothi Padmanabhan + via ddas) HADOOP-3472 MapFile.Reader getClosest() function returns incorrect results when before is true (Todd Lipcon via Stack) @@ -579,8 +581,9 @@ HADOOP-3135. Get the system directory from the JobTracker instead of from the conf. (Subramaniam Krishnan via ddas) - HADOOP-3503. Fix a race condition when client and namenode start simultaneous - recovery of the same block. (dhruba & Tsz Wo (Nicholas), SZE) + HADOOP-3503. Fix a race condition when client and namenode start + simultaneous recovery of the same block. (dhruba & Tsz Wo + (Nicholas), SZE) HADOOP-3440. Fixes DistributedCache to not create symlinks for paths which don't have fragments even when createSymLink is true. @@ -590,7 +593,8 @@ HADOOP-3489. Fix NPE in SafeModeMonitor. (Lohit Vijayarenu via shv) - HADOOP-3509. Fix NPE in FSNamesystem.close. (Tsz Wo (Nicholas), SZE via shv) + HADOOP-3509. Fix NPE in FSNamesystem.close. (Tsz Wo (Nicholas), SZE via + shv) HADOOP-3491. Name-node shutdown causes InterruptedException in ResolutionMonitor. (Lohit Vijayarenu via shv) @@ -765,6 +769,9 @@ HADOOP-3774. Fix typos in shell output. (Tsz Wo (Nicholas), SZE via cdouglas) + HADOOP-3762. Fixed FileSystem cache to work with the default port. (cutting + via omalley) + HADOOP-3798. Fix tests compilation. (Mukund Madhugiri via omalley) Release 0.17.2 - Unreleased Modified: hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/FileSystem.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/FileSystem.java?rev=678527&r1=678526&r2=678527&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/FileSystem.java (original) +++ hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/fs/FileSystem.java Mon Jul 21 11:56:01 2008 @@ -60,6 +60,10 @@ /** FileSystem cache */ private static final Cache CACHE = new Cache(); + + /** The key this instance is stored under in the cache. */ + private Cache.Key key; + /** Recording statistics per a FileSystem class */ private static final Map, Statistics> statisticsTable = @@ -312,9 +316,9 @@ thisAuthority.equalsIgnoreCase(defaultUri.getAuthority())) return; } - throw new IllegalArgumentException("Wrong FS: "+path+ - ", expected: "+this.getUri()); } + throw new IllegalArgumentException("Wrong FS: "+path+ + ", expected: "+this.getUri()); } /** @@ -1237,7 +1241,7 @@ public void close() throws IOException { // delete all files that were marked as delete-on-exit. processDeleteOnExit(); - CACHE.remove(new Cache.Key(this), this); + CACHE.remove(this.key, this); } /** Return the total size of all files in the filesystem.*/ @@ -1341,13 +1345,15 @@ private final Map map = new HashMap(); synchronized FileSystem get(URI uri, Configuration conf) throws IOException{ - FileSystem fs = map.get(new Key(uri, conf)); + Key key = new Key(uri, conf); + FileSystem fs = map.get(key); if (fs == null) { fs = createFileSystem(uri, conf); if (map.isEmpty() && !clientFinalizer.isAlive()) { Runtime.getRuntime().addShutdownHook(clientFinalizer); } - map.put(new Key(fs), fs); + fs.key = key; + map.put(key, fs); } return fs; } @@ -1395,10 +1401,6 @@ final String authority; final String username; - Key(FileSystem fs) throws IOException { - this(fs.getUri(), fs.getConf()); - } - Key(URI uri, Configuration conf) throws IOException { scheme = uri.getScheme(); authority = uri.getAuthority(); Modified: hadoop/core/branches/branch-0.18/src/hdfs/org/apache/hadoop/dfs/DistributedFileSystem.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/hdfs/org/apache/hadoop/dfs/DistributedFileSystem.java?rev=678527&r1=678526&r2=678527&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/hdfs/org/apache/hadoop/dfs/DistributedFileSystem.java (original) +++ hadoop/core/branches/branch-0.18/src/hdfs/org/apache/hadoop/dfs/DistributedFileSystem.java Mon Jul 21 11:56:01 2008 @@ -69,6 +69,21 @@ this.workingDir = getHomeDirectory(); } + /** Permit paths which explicitly specify the default port. */ + protected void checkPath(Path path) { + URI thisUri = this.getUri(); + URI thatUri = path.toUri(); + String thatAuthority = thatUri.getAuthority(); + if (thatUri.getScheme() != null + && thatUri.getScheme().equalsIgnoreCase(thisUri.getScheme()) + && thatUri.getPort() == NameNode.DEFAULT_PORT + && thisUri.getPort() == -1 + && thatAuthority.substring(0,thatAuthority.indexOf(":")) + .equalsIgnoreCase(thisUri.getAuthority())) + return; + super.checkPath(path); + } + public Path getWorkingDirectory() { return workingDir; } Modified: hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/fs/TestFileSystem.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/fs/TestFileSystem.java?rev=678527&r1=678526&r2=678527&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/fs/TestFileSystem.java (original) +++ hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/fs/TestFileSystem.java Mon Jul 21 11:56:01 2008 @@ -32,6 +32,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.dfs.MiniDFSCluster; +import org.apache.hadoop.dfs.NameNode; import org.apache.hadoop.fs.shell.CommandFormat; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.SequenceFile; @@ -506,22 +507,51 @@ } { - MiniDFSCluster cluster = null; try { - cluster = new MiniDFSCluster(new Configuration(), 2, true, null); - URI uri = cluster.getFileSystem().getUri(); + runTestCache(NameNode.DEFAULT_PORT); + } catch(java.net.BindException be) { + LOG.warn("Cannot test NameNode.DEFAULT_PORT (=" + + NameNode.DEFAULT_PORT + ")", be); + } + + runTestCache(0); + } + } + + static void runTestCache(int port) throws Exception { + Configuration conf = new Configuration(); + MiniDFSCluster cluster = null; + try { + cluster = new MiniDFSCluster(port, conf, 2, true, true, null, null); + URI uri = cluster.getFileSystem().getUri(); + LOG.info("uri=" + uri); + + { FileSystem fs = FileSystem.get(uri, new Configuration()); checkPath(cluster, fs); for(int i = 0; i < 100; i++) { assertTrue(fs == FileSystem.get(uri, new Configuration())); } - } finally { - cluster.shutdown(); } + + if (port == NameNode.DEFAULT_PORT) { + //test explicit default port + URI uri2 = new URI(uri.getScheme(), uri.getUserInfo(), + uri.getHost(), NameNode.DEFAULT_PORT, uri.getPath(), + uri.getQuery(), uri.getFragment()); + LOG.info("uri2=" + uri2); + FileSystem fs = FileSystem.get(uri2, conf); + checkPath(cluster, fs); + for(int i = 0; i < 100; i++) { + assertTrue(fs == FileSystem.get(uri2, new Configuration())); + } + } + } finally { + if (cluster != null) cluster.shutdown(); } } - private void checkPath(MiniDFSCluster cluster, FileSystem fileSys) throws IOException { + static void checkPath(MiniDFSCluster cluster, FileSystem fileSys) throws IOException { InetSocketAddress add = cluster.getNameNode().getNameNodeAddress(); // Test upper/lower case fileSys.checkPath(new Path("hdfs://" + add.getHostName().toUpperCase() + ":" + add.getPort()));