Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BEF5410282 for ; Wed, 10 Apr 2013 20:18:17 +0000 (UTC) Received: (qmail 82219 invoked by uid 500); 10 Apr 2013 20:18:17 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 82175 invoked by uid 500); 10 Apr 2013 20:18:17 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 82167 invoked by uid 99); 10 Apr 2013 20:18:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Apr 2013 20:18:17 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Apr 2013 20:18:12 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 6975C23889FA; Wed, 10 Apr 2013 20:17:51 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1466658 - in /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common: ./ src/main/docs/ src/main/java/ src/main/java/org/apache/hadoop/metrics2/ src/main/native/src/org/apache/hadoop/io/nativeio/ src/test/core/ src/test/java/... Date: Wed, 10 Apr 2013 20:17:51 -0000 To: common-commits@hadoop.apache.org From: szetszwo@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20130410201751.6975C23889FA@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: szetszwo Date: Wed Apr 10 20:17:39 2013 New Revision: 1466658 URL: http://svn.apache.org/r1466658 Log: Merge r1464808 through r1466652 from trunk. Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt (contents, props changed) hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/docs/ (props changed) hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/ (props changed) hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsFilter.java hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/core/ (props changed) hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MockitoUtil.java Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1466658&r1=1466657&r2=1466658&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt (original) +++ hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt Wed Apr 10 20:17:39 2013 @@ -505,6 +505,10 @@ Trunk (Unreleased) HADOOP-9353. Activate native-win maven profile by default on Windows. (Arpit Agarwal via szetszwo) + + HADOOP-9437. TestNativeIO#testRenameTo fails on Windows due to assumption + that POSIX errno is embedded in NativeIOException. (Chris Nauroth via + suresh) Release 2.0.5-beta - UNRELEASED @@ -625,6 +629,9 @@ Release 2.0.4-alpha - UNRELEASED BUG FIXES + HADOOP-9467. Metrics2 record filter should check name as well as tags. + (Ganeshan Iyler via llu) + HADOOP-9406. hadoop-client leaks dependency on JDK tools jar. (tucu) HADOOP-9301. hadoop client servlet/jsp/jetty/tomcat JARs creating Propchange: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt:r1464808-1466652 Propchange: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/docs/ ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs:r1464808-1466652 Propchange: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/ ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java:r1464808-1466652 Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsFilter.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsFilter.java?rev=1466658&r1=1466657&r2=1466658&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsFilter.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsFilter.java Wed Apr 10 20:17:39 2013 @@ -57,7 +57,7 @@ public abstract class MetricsFilter impl * @return true to accept; false otherwise. */ public boolean accepts(MetricsRecord record) { - return accepts(record.tags()); + return accepts(record.name()) && accepts(record.tags()); } } Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c?rev=1466658&r1=1466657&r2=1466658&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c (original) +++ hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c Wed Apr 10 20:17:39 2013 @@ -816,6 +816,7 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_io_nativeio_NativeIO_renameTo0(JNIEnv *env, jclass clazz, jstring jsrc, jstring jdst) { +#ifdef UNIX const char *src = NULL, *dst = NULL; src = (*env)->GetStringUTFChars(env, jsrc, NULL); @@ -829,6 +830,23 @@ jclass clazz, jstring jsrc, jstring jdst done: if (src) (*env)->ReleaseStringUTFChars(env, jsrc, src); if (dst) (*env)->ReleaseStringUTFChars(env, jdst, dst); +#endif + +#ifdef WINDOWS + LPCWSTR src = NULL, dst = NULL; + + src = (LPCWSTR) (*env)->GetStringChars(env, jsrc, NULL); + if (!src) goto done; // exception was thrown + dst = (LPCWSTR) (*env)->GetStringChars(env, jdst, NULL); + if (!dst) goto done; // exception was thrown + if (!MoveFile(src, dst)) { + throw_ioe(env, GetLastError()); + } + +done: + if (src) (*env)->ReleaseStringChars(env, jsrc, src); + if (dst) (*env)->ReleaseStringChars(env, jdst, dst); +#endif } /** Propchange: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/core/ ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core:r1464808-1466652 Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java?rev=1466658&r1=1466657&r2=1466658&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java Wed Apr 10 20:17:39 2013 @@ -446,7 +446,13 @@ public class TestNativeIO { NativeIO.renameTo(nonExistentFile, targetFile); Assert.fail(); } catch (NativeIOException e) { - Assert.assertEquals(e.getErrno(), Errno.ENOENT); + if (Path.WINDOWS) { + Assert.assertEquals( + String.format("The system cannot find the file specified.%n"), + e.getMessage()); + } else { + Assert.assertEquals(Errno.ENOENT, e.getErrno()); + } } // Test renaming a file to itself. It should succeed and do nothing. @@ -465,7 +471,13 @@ public class TestNativeIO { NativeIO.renameTo(sourceFile, badTarget); Assert.fail(); } catch (NativeIOException e) { - Assert.assertEquals(e.getErrno(), Errno.ENOTDIR); + if (Path.WINDOWS) { + Assert.assertEquals( + String.format("The parameter is incorrect.%n"), + e.getMessage()); + } else { + Assert.assertEquals(Errno.ENOTDIR, e.getErrno()); + } } FileUtils.deleteQuietly(TEST_DIR); Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java?rev=1466658&r1=1466657&r2=1466658&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java Wed Apr 10 20:17:39 2013 @@ -24,7 +24,9 @@ import java.util.List; import org.apache.commons.configuration.SubsetConfiguration; import org.junit.Test; import static org.junit.Assert.*; +import static org.mockito.Mockito.*; +import org.apache.hadoop.metrics2.MetricsRecord; import org.apache.hadoop.metrics2.MetricsTag; import org.apache.hadoop.metrics2.impl.ConfigBuilder; import static org.apache.hadoop.metrics2.lib.Interns.*; @@ -38,6 +40,8 @@ public class TestPatternFilter { SubsetConfiguration empty = new ConfigBuilder().subset(""); shouldAccept(empty, "anything"); shouldAccept(empty, Arrays.asList(tag("key", "desc", "value"))); + shouldAccept(empty, mockMetricsRecord("anything", Arrays.asList( + tag("key", "desc", "value")))); } /** @@ -50,9 +54,15 @@ public class TestPatternFilter { shouldAccept(wl, "foo"); shouldAccept(wl, Arrays.asList(tag("bar", "", ""), tag("foo", "", "f"))); + shouldAccept(wl, mockMetricsRecord("foo", Arrays.asList( + tag("bar", "", ""), tag("foo", "", "f")))); shouldReject(wl, "bar"); shouldReject(wl, Arrays.asList(tag("bar", "", ""))); shouldReject(wl, Arrays.asList(tag("foo", "", "boo"))); + shouldReject(wl, mockMetricsRecord("bar", Arrays.asList( + tag("foo", "", "f")))); + shouldReject(wl, mockMetricsRecord("foo", Arrays.asList( + tag("bar", "", "")))); } /** @@ -64,9 +74,15 @@ public class TestPatternFilter { .add("p.exclude.tags", "foo:f").subset("p"); shouldAccept(bl, "bar"); shouldAccept(bl, Arrays.asList(tag("bar", "", ""))); + shouldAccept(bl, mockMetricsRecord("bar", Arrays.asList( + tag("bar", "", "")))); shouldReject(bl, "foo"); shouldReject(bl, Arrays.asList(tag("bar", "", ""), tag("foo", "", "f"))); + shouldReject(bl, mockMetricsRecord("foo", Arrays.asList( + tag("bar", "", "")))); + shouldReject(bl, mockMetricsRecord("bar", Arrays.asList( + tag("bar", "", ""), tag("foo", "", "f")))); } /** @@ -81,10 +97,18 @@ public class TestPatternFilter { .add("p.exclude.tags", "bar:b").subset("p"); shouldAccept(c, "foo"); shouldAccept(c, Arrays.asList(tag("foo", "", "f"))); + shouldAccept(c, mockMetricsRecord("foo", Arrays.asList( + tag("foo", "", "f")))); shouldReject(c, "bar"); shouldReject(c, Arrays.asList(tag("bar", "", "b"))); + shouldReject(c, mockMetricsRecord("bar", Arrays.asList( + tag("foo", "", "f")))); + shouldReject(c, mockMetricsRecord("foo", Arrays.asList( + tag("bar", "", "b")))); shouldAccept(c, "foobar"); shouldAccept(c, Arrays.asList(tag("foobar", "", ""))); + shouldAccept(c, mockMetricsRecord("foobar", Arrays.asList( + tag("foobar", "", "")))); } /** @@ -98,6 +122,8 @@ public class TestPatternFilter { .add("p.exclude.tags", "foo:f").subset("p"); shouldAccept(c, "foo"); shouldAccept(c, Arrays.asList(tag("foo", "", "f"))); + shouldAccept(c, mockMetricsRecord("foo", Arrays.asList( + tag("foo", "", "f")))); } static void shouldAccept(SubsetConfiguration conf, String s) { @@ -110,6 +136,17 @@ public class TestPatternFilter { assertTrue("accepts "+ tags, newRegexFilter(conf).accepts(tags)); } + /** + * Asserts that filters with the given configuration accept the given record. + * + * @param conf SubsetConfiguration containing filter configuration + * @param record MetricsRecord to check + */ + static void shouldAccept(SubsetConfiguration conf, MetricsRecord record) { + assertTrue("accepts " + record, newGlobFilter(conf).accepts(record)); + assertTrue("accepts " + record, newRegexFilter(conf).accepts(record)); + } + static void shouldReject(SubsetConfiguration conf, String s) { assertTrue("rejects "+ s, !newGlobFilter(conf).accepts(s)); assertTrue("rejects "+ s, !newRegexFilter(conf).accepts(s)); @@ -121,6 +158,17 @@ public class TestPatternFilter { } /** + * Asserts that filters with the given configuration reject the given record. + * + * @param conf SubsetConfiguration containing filter configuration + * @param record MetricsRecord to check + */ + static void shouldReject(SubsetConfiguration conf, MetricsRecord record) { + assertTrue("rejects " + record, !newGlobFilter(conf).accepts(record)); + assertTrue("rejects " + record, !newRegexFilter(conf).accepts(record)); + } + + /** * Create a new glob filter with a config object * @param conf the config object * @return the filter @@ -141,4 +189,19 @@ public class TestPatternFilter { f.init(conf); return f; } + + /** + * Creates a mock MetricsRecord with the given name and tags. + * + * @param name String name + * @param tags List tags + * @return MetricsRecord newly created mock + */ + private static MetricsRecord mockMetricsRecord(String name, + List tags) { + MetricsRecord record = mock(MetricsRecord.class); + when(record.name()).thenReturn(name); + when(record.tags()).thenReturn(tags); + return record; + } } Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java?rev=1466658&r1=1466657&r2=1466658&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java Wed Apr 10 20:17:39 2013 @@ -20,6 +20,9 @@ package org.apache.hadoop.test; import java.io.File; import java.io.IOException; import java.io.StringWriter; +import java.lang.management.ManagementFactory; +import java.lang.management.ThreadInfo; +import java.lang.management.ThreadMXBean; import java.lang.reflect.InvocationTargetException; import java.util.Arrays; import java.util.Random; @@ -330,4 +333,33 @@ public abstract class GenericTestUtils { " but got:\n" + output, Pattern.compile(pattern).matcher(output).find()); } + + public static void assertValueNear(long expected, long actual, long allowedError) { + assertValueWithinRange(expected - allowedError, expected + allowedError, actual); + } + + public static void assertValueWithinRange(long expectedMin, long expectedMax, + long actual) { + Assert.assertTrue("Expected " + actual + " to be in range (" + expectedMin + "," + + expectedMax + ")", expectedMin <= actual && actual <= expectedMax); + } + + /** + * Assert that there are no threads running whose name matches the + * given regular expression. + * @param regex the regex to match against + */ + public static void assertNoThreadsMatching(String regex) { + Pattern pattern = Pattern.compile(regex); + ThreadMXBean threadBean = ManagementFactory.getThreadMXBean(); + + ThreadInfo[] infos = threadBean.getThreadInfo(threadBean.getAllThreadIds(), 20); + for (ThreadInfo info : infos) { + if (info == null) continue; + if (pattern.matcher(info.getThreadName()).matches()) { + Assert.fail("Leaked thread: " + info + "\n" + + Joiner.on("\n").join(info.getStackTrace())); + } + } + } } Modified: hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MockitoUtil.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MockitoUtil.java?rev=1466658&r1=1466657&r2=1466658&view=diff ============================================================================== --- hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MockitoUtil.java (original) +++ hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MockitoUtil.java Wed Apr 10 20:17:39 2013 @@ -20,6 +20,9 @@ package org.apache.hadoop.test; import java.io.Closeable; import org.mockito.Mockito; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; +import org.mockito.stubbing.Stubber; public abstract class MockitoUtil { @@ -33,4 +36,29 @@ public abstract class MockitoUtil { return Mockito.mock(clazz, Mockito.withSettings().extraInterfaces(Closeable.class)); } + + /** + * Throw an exception from the mock/spy only in the case that the + * call stack at the time the method has a line which matches the given + * pattern. + * + * @param t the Throwable to throw + * @param pattern the pattern against which to match the call stack trace + * @return the stub in progress + */ + public static Stubber doThrowWhenCallStackMatches( + final Throwable t, final String pattern) { + return Mockito.doAnswer(new Answer() { + @Override + public Object answer(InvocationOnMock invocation) throws Throwable { + t.setStackTrace(Thread.currentThread().getStackTrace()); + for (StackTraceElement elem : t.getStackTrace()) { + if (elem.toString().matches(pattern)) { + throw t; + } + } + return invocation.callRealMethod(); + } + }); + } }