Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 63312 invoked from network); 14 Mar 2008 02:53:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 14 Mar 2008 02:53:11 -0000 Received: (qmail 10730 invoked by uid 500); 14 Mar 2008 02:53:08 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 10706 invoked by uid 500); 14 Mar 2008 02:53:08 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 10697 invoked by uid 99); 14 Mar 2008 02:53:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Mar 2008 19:53:08 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO eris.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Mar 2008 02:52:27 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 01BDD1A9832; Thu, 13 Mar 2008 19:52:46 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r636967 - in /hadoop/core/trunk: ./ src/java/org/apache/hadoop/dfs/ src/java/org/apache/hadoop/dfs/namenode/metrics/ src/test/org/apache/hadoop/dfs/ Date: Fri, 14 Mar 2008 02:52:42 -0000 To: core-commits@hadoop.apache.org From: cdouglas@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20080314025247.01BDD1A9832@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: cdouglas Date: Thu Mar 13 19:52:40 2008 New Revision: 636967 URL: http://svn.apache.org/viewvc?rev=636967&view=rev Log: HADOOP-2470. Remove getContentLength(String), open(String, long, long) and isDir(String) from ClientProtocol. ClientProtocol version changed from 26 to 27. Contributed by Tsz Wo (Nicholas), SZE. Modified: hadoop/core/trunk/CHANGES.txt hadoop/core/trunk/src/java/org/apache/hadoop/dfs/ClientProtocol.java hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DFSClient.java hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSNamesystem.java hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNode.java hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNodeMetrics.java hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatistics.java hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatisticsMBean.java hadoop/core/trunk/src/test/org/apache/hadoop/dfs/NNThroughputBenchmark.java Modified: hadoop/core/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/CHANGES.txt (original) +++ hadoop/core/trunk/CHANGES.txt Thu Mar 13 19:52:40 2008 @@ -30,6 +30,10 @@ specifies whether a recursive delete is intended. (Mahadev Konar via dhruba) + HADOOP-2470. Remove getContentLength(String), open(String, long, long) + and isDir(String) from ClientProtocol. ClientProtocol version changed + from 26 to 27. (Tsz Wo (Nicholas), SZE via cdouglas) + NEW FEATURES HADOOP-1398. Add HBase in-memory block cache. (tomwhite) Modified: hadoop/core/trunk/src/java/org/apache/hadoop/dfs/ClientProtocol.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/dfs/ClientProtocol.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/dfs/ClientProtocol.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/dfs/ClientProtocol.java Thu Mar 13 19:52:40 2008 @@ -36,44 +36,25 @@ * Compared to the previous version the following changes have been introduced: * (Only the latest change is reflected. * The log of historical changes can be retrieved from the svn). - * 25 : added {@link #getContentSummary(String path)} - * 26 : added delete(src, boolean) for recursive deletes on the namenode + * 27 : removed getContentLength(String), open(String, long, long) and isDir(String) */ - public static final long versionID = 26L; + public static final long versionID = 27L; /////////////////////////////////////// // File contents /////////////////////////////////////// /** - * Open an existing file for read and get block locations within - * the specified range. + * Get locations of the blocks of the specified file within the specified range. + * DataNode locations for each block are sorted by + * the proximity to the client. *

* Return {@link LocatedBlocks} which contains * file length, blocks and their locations. * DataNode locations for each block are sorted by * the distance to the client's address. *

- * The client will then have to contact - * one of the indicated DataNodes to obtain the actual data. There - * is no need to call close() or any other function after - * calling open(). - * - * @param src file name - * @param offset range start offset - * @param length range length - * @return file length and array of blocks with their locations - * @throws IOException - */ - public LocatedBlocks open(String src, - long offset, - long length) throws IOException; - - /** - * Get locations of the blocks of the specified file within the specified range. - * DataNode locations for each block are sorted by - * the proximity to the client. - * - * @see #open(String, long, long) + * The client will then have to contact + * one of the indicated DataNodes to obtain the actual data. * * @param src file name * @param offset range start offset @@ -256,11 +237,6 @@ public boolean exists(String src) throws IOException; /** - * Check whether the given filename is a directory or not. - */ - public boolean isDir(String src) throws IOException; - - /** * Create a directory (or hierarchy of directories) with the given * name and permission. * @@ -445,16 +421,6 @@ * @return object containing information regarding the file */ public DFSFileInfo getFileInfo(String src) throws IOException; - - /** - * Get the total size of all files and directories rooted at - * the specified directory. - * @param src The string representation of the path - * @return size of directory subtree in bytes - * @deprecated use {@link #getContentSummary(String)} - */ - @Deprecated - public long getContentLength(String src) throws IOException; /** * Get {@link ContentSummary} rooted at the specified directory. Modified: hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DFSClient.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DFSClient.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DFSClient.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DFSClient.java Thu Mar 13 19:52:40 2008 @@ -28,7 +28,6 @@ import org.apache.hadoop.conf.*; import org.apache.hadoop.dfs.DistributedFileSystem.DiskStatus; import org.apache.hadoop.security.UnixUserGroupInformation; -import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.util.*; import org.apache.commons.logging.*; @@ -461,11 +460,14 @@ return namenode.exists(src); } - /** - */ + /** @deprecated Use getFileStatus() instead */ + @Deprecated public boolean isDirectory(String src) throws IOException { - checkOpen(); - return namenode.isDir(src); + try { + return getFileInfo(src).isDir(); + } catch (FileNotFoundException e) { + return false; // f does not exist + } } /** @@ -578,21 +580,6 @@ return namenode.mkdirs(src, masked); } - /** - * Retrieves the total size of all files and directories under - * the specified path. - * - * @param src - * @throws IOException - * @return the number of bytes in the subtree rooted at src - * @deprecated use {@link #getContentSummary(String)} - */ - @Deprecated - public long getContentLength(String src - ) throws IOException { - return getContentSummary(src).getLength(); - } - ContentSummary getContentSummary(String src) throws IOException { return namenode.getContentSummary(src); } @@ -1005,7 +992,10 @@ * Grab the open-file info from namenode */ synchronized void openInfo() throws IOException { - LocatedBlocks newInfo = namenode.open(src, 0, prefetchSize); + LocatedBlocks newInfo = namenode.getBlockLocations(src, 0, prefetchSize); + if (newInfo == null) { + throw new IOException("Cannot open filename " + src); + } if (locatedBlocks != null) { Iterator oldIter = locatedBlocks.getLocatedBlocks().iterator(); Modified: hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSNamesystem.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSNamesystem.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSNamesystem.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSNamesystem.java Thu Mar 13 19:52:40 2008 @@ -786,8 +786,6 @@ /** * Get block locations within the specified range. - * - * @see ClientProtocol#open(String, long, long) * @see ClientProtocol#getBlockLocations(String, long, long) */ LocatedBlocks getBlockLocations(String src, long offset, long length @@ -1468,7 +1466,7 @@ * is a directory (non empty) and recursive is set to false then throw exception. */ public boolean delete(String src, boolean recursive) throws IOException { - if (isDir(src) && (!recursive) && (!dir.isDirEmpty(src))) { + if ((!recursive) && (!dir.isDirEmpty(src))) { throw new IOException(src + " is non empty"); } boolean status = deleteInternal(src, true, true); @@ -1527,16 +1525,6 @@ checkTraverse(src); } return dir.exists(src); - } - - /** - * Whether the given name is a directory - */ - public boolean isDir(String src) throws AccessControlException { - if (isPermissionEnabled) { - checkTraverse(src); - } - return dir.isDir(src); } /** Get the file info for a specific file. Modified: hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNode.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNode.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNode.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNode.java Thu Mar 13 19:52:40 2008 @@ -233,25 +233,11 @@ ///////////////////////////////////////////////////// // ClientProtocol ///////////////////////////////////////////////////// - - /** - */ - public LocatedBlocks open(String src, - long offset, - long length) throws IOException { - LocatedBlocks result = getBlockLocations(src, offset, length); - if (result == null) { - throw new IOException("Cannot open filename " + src); - } - myMetrics.numFilesOpened.inc(); - return result; - } - - /** - */ + /** {@inheritDoc} */ public LocatedBlocks getBlockLocations(String src, long offset, long length) throws IOException { + myMetrics.numGetBlockLocations.inc(); return namesystem.getBlockLocations(getClientMachine(), src, offset, length); } @@ -402,12 +388,6 @@ } /** - */ - public boolean isDir(String src) throws IOException { - return namesystem.isDir(src); - } - - /** * Check path length does not exceed maximum. Returns true if * length and depth are okay. Returns false if length is too long * or depth is too great. @@ -530,12 +510,6 @@ */ public void metaSave(String filename) throws IOException { namesystem.metaSave(filename); - } - - /** {@inheritDoc} */ - @Deprecated - public long getContentLength(String src) throws IOException { - return getContentSummary(src).getLength(); } /** {@inheritDoc} */ Modified: hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNodeMetrics.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNodeMetrics.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNodeMetrics.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNodeMetrics.java Thu Mar 13 19:52:40 2008 @@ -46,7 +46,7 @@ private NameNodeStatistics namenodeStats; public MetricsTimeVaryingInt numFilesCreated = new MetricsTimeVaryingInt("FilesCreated"); - public MetricsTimeVaryingInt numFilesOpened = new MetricsTimeVaryingInt("FilesOpened"); + public MetricsTimeVaryingInt numGetBlockLocations = new MetricsTimeVaryingInt("GetBlockLocations"); public MetricsTimeVaryingInt numFilesRenamed = new MetricsTimeVaryingInt("FilesRenamed"); public MetricsTimeVaryingInt numFilesListed = new MetricsTimeVaryingInt("FilesListed"); @@ -91,7 +91,7 @@ public void doUpdates(MetricsContext unused) { synchronized (this) { numFilesCreated.pushMetric(metricsRecord); - numFilesOpened.pushMetric(metricsRecord); + numGetBlockLocations.pushMetric(metricsRecord); numFilesRenamed.pushMetric(metricsRecord); numFilesListed.pushMetric(metricsRecord); Modified: hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatistics.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatistics.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatistics.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatistics.java Thu Mar 13 19:52:40 2008 @@ -170,11 +170,9 @@ return myMetrics.numFilesListed.getPreviousIntervalValue(); } - /** - * @inheritDoc - */ - public int getNumFilesOpened() { - return myMetrics.numFilesOpened.getPreviousIntervalValue(); + /** @inheritDoc */ + public int getNumGetBlockLocations() { + return myMetrics.numGetBlockLocations.getPreviousIntervalValue(); } /** Modified: hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatisticsMBean.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatisticsMBean.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatisticsMBean.java (original) +++ hadoop/core/trunk/src/java/org/apache/hadoop/dfs/namenode/metrics/NameNodeStatisticsMBean.java Thu Mar 13 19:52:40 2008 @@ -141,11 +141,12 @@ int getNumFilesCreated(); /** - * Number of files opened for reading in the last interval + * Number of + * {@link org.apache.hadoop.dfs.NameNode#getBlockLocations(String,long,long)} * @return number of operations */ - int getNumFilesOpened(); - + int getNumGetBlockLocations(); + /** * Number of files renamed in the last interval * @return number of operations Modified: hadoop/core/trunk/src/test/org/apache/hadoop/dfs/NNThroughputBenchmark.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/dfs/NNThroughputBenchmark.java?rev=636967&r1=636966&r2=636967&view=diff ============================================================================== --- hadoop/core/trunk/src/test/org/apache/hadoop/dfs/NNThroughputBenchmark.java (original) +++ hadoop/core/trunk/src/test/org/apache/hadoop/dfs/NNThroughputBenchmark.java Thu Mar 13 19:52:40 2008 @@ -526,7 +526,7 @@ long executeOp(int daemonId, int inputIdx, String ignore) throws IOException { long start = System.currentTimeMillis(); - nameNode.open(fileNames[daemonId][inputIdx], 0L, BLOCK_SIZE); + nameNode.getBlockLocations(fileNames[daemonId][inputIdx], 0L, BLOCK_SIZE); long end = System.currentTimeMillis(); return end-start; }