Return-Path: X-Original-To: apmail-incubator-chukwa-commits-archive@www.apache.org Delivered-To: apmail-incubator-chukwa-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 00942717C for ; Mon, 21 Nov 2011 06:40:06 +0000 (UTC) Received: (qmail 99481 invoked by uid 500); 21 Nov 2011 06:40:05 -0000 Delivered-To: apmail-incubator-chukwa-commits-archive@incubator.apache.org Received: (qmail 99456 invoked by uid 500); 21 Nov 2011 06:40:04 -0000 Mailing-List: contact chukwa-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-dev@incubator.apache.org Delivered-To: mailing list chukwa-commits@incubator.apache.org Received: (qmail 99448 invoked by uid 99); 21 Nov 2011 06:40:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Nov 2011 06:40:03 +0000 X-ASF-Spam-Status: No, hits=-1996.8 required=5.0 tests=ALL_TRUSTED,FRT_ROLEX,FUZZY_ROLEX X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Nov 2011 06:40:00 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 2CC012388860; Mon, 21 Nov 2011 06:39:40 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r1204365 - in /incubator/chukwa/trunk: ./ ivy/ src/docs/src/documentation/resources/images/ src/java/ src/main/ src/main/java/ src/main/java/org/apache/hadoop/chukwa/inputtools/ src/main/java/org/apache/hadoop/chukwa/inputtools/log4j/ src/m... Date: Mon, 21 Nov 2011 06:39:39 -0000 To: chukwa-commits@incubator.apache.org From: eyang@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111121063940.2CC012388860@eris.apache.org> Author: eyang Date: Mon Nov 21 06:39:38 2011 New Revision: 1204365 URL: http://svn.apache.org/viewvc?rev=1204365&view=rev Log: CHUKWA-590. Migrate from ant to maven. (Eric Yang) Added: incubator/chukwa/trunk/pom.xml incubator/chukwa/trunk/src/main/ incubator/chukwa/trunk/src/main/java/ - copied from r1202951, incubator/chukwa/trunk/src/java/ incubator/chukwa/trunk/src/main/web/ - copied from r1202951, incubator/chukwa/trunk/src/web/ incubator/chukwa/trunk/src/packages/ incubator/chukwa/trunk/src/packages/tarball/ incubator/chukwa/trunk/src/packages/tarball/all.xml incubator/chukwa/trunk/src/site/ incubator/chukwa/trunk/src/site/apt/ incubator/chukwa/trunk/src/site/resources/ incubator/chukwa/trunk/src/site/resources/images/ - copied from r1202951, incubator/chukwa/trunk/src/docs/src/documentation/resources/images/ incubator/chukwa/trunk/src/test/java/ incubator/chukwa/trunk/src/test/java/org/ - copied from r1202951, incubator/chukwa/trunk/src/test/org/ incubator/chukwa/trunk/src/test/resources/ incubator/chukwa/trunk/src/test/resources/Hadoop18JobHistoryLog.txt - copied unchanged from r1202951, incubator/chukwa/trunk/src/test/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/Hadoop18JobHistoryLog.txt incubator/chukwa/trunk/src/test/resources/TestJobLog.txt - copied unchanged from r1202951, incubator/chukwa/trunk/src/test/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLog.txt incubator/chukwa/trunk/src/test/resources/hbase-site.xml - copied unchanged from r1202951, incubator/chukwa/trunk/src/test/hbase-site.xml incubator/chukwa/trunk/src/test/resources/late-log4j.properties - copied unchanged from r1202951, incubator/chukwa/trunk/src/test/org/apache/hadoop/chukwa/inputtools/log4j/late-log4j.properties incubator/chukwa/trunk/src/test/resources/log4j.properties incubator/chukwa/trunk/src/test/resources/tasklog-log4j.properties - copied unchanged from r1202951, incubator/chukwa/trunk/src/test/org/apache/hadoop/chukwa/inputtools/log4j/tasklog-log4j.properties Removed: incubator/chukwa/trunk/ivy/ incubator/chukwa/trunk/ivy.xml incubator/chukwa/trunk/src/docs/src/documentation/resources/images/ incubator/chukwa/trunk/src/java/ incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/inputtools/ChukwaTTInstru.java incubator/chukwa/trunk/src/test/hbase-site.xml incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/database/TestDatabaseIostat.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/Hadoop18JobHistoryLog.txt incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLog.txt incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/late-log4j.properties incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/tasklog-log4j.properties incubator/chukwa/trunk/src/test/org/ incubator/chukwa/trunk/src/web/ Modified: incubator/chukwa/trunk/CHANGES.txt incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/inputtools/log4j/Log4jMetricsSink.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/analysis/salsa/fsm/TestFSMBuilder.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/TestOffsetStatsManager.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestDirTailingAdaptor.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxManager.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLogEntry.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestChukwaDailyRollingFileAppender.java incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestTaskLogAppender.java Modified: incubator/chukwa/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/CHANGES.txt?rev=1204365&r1=1204364&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/CHANGES.txt (original) +++ incubator/chukwa/trunk/CHANGES.txt Mon Nov 21 06:39:38 2011 @@ -4,6 +4,8 @@ Trunk (unreleased changes) NEW FEATURES + CHUKWA-590. Migrate from ant to maven. (Eric Yang) + CHUKWA-599. Added Metrics 2 support for Hadoop 0.20.203+. (Eric Yang) CHUKWA-575. Added cluster summary pig script. (Eric Yang) Added: incubator/chukwa/trunk/pom.xml URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/pom.xml?rev=1204365&view=auto ============================================================================== --- incubator/chukwa/trunk/pom.xml (added) +++ incubator/chukwa/trunk/pom.xml Mon Nov 21 06:39:38 2011 @@ -0,0 +1,573 @@ + + + + + + + test + ${env.BUILD_NUMBER} + 0.5.0-SNAPSHOT + UTF-8 + /usr + /etc/chukwa + /var/log/chukwa + /var/run/chukwa + 1 + 0.5.0 + ${project.artifactId}-incubating-${package.version} + ${basedir}/target/test + ${test.build.dir}/data + ${test.build.dir}/cache + ${test.build.dir}/debug + ${test.build.dir}/logs + + + Apache Chukwa + Chukwa is an open source data collection system for monitoring large distributed systems. + Chukwa is built on top of the Hadoop Distributed File System (HDFS) and Map/Reduce framework and + inherits Hadoop’s scalability and robustness. Chukwa also includes a flexible and powerful toolkit + for displaying, monitoring and analyzing results to make the best use of the collected data. + + http://incubator.apache.org/chukwa + 4.0.0 + + org.apache.incubator + 0.5.0 + chukwa + jar + + + Chukwa JIRA + http://issues.apache.org/jira/browse/CHUKWA + + + + scm:svn:http://svn.apache.org/repos/asf/incubator/chukwa + scm:svn:https://svn.apache.org/repos/asf/incubator/chukwa + + + + + + chukwa + chukwa-dev-subscribe@incubator.apache.org + chukwa-dev-unsubscribe@incubator.apache.org + mailto:chukwa-user@incubator.apache.org + + + + + + + com.josephoconnell.html + html-filter + 1.0 + system + ${basedir}/lib/html-filter-1.0.jar + + + edu.berkeley.confspell + confspellcheck + 1.0 + system + ${basedir}/lib/confspellcheck.jar + + + com.mdimension + jchronic + 0.2.3 + system + ${basedir}/lib/jchronic-0.2.3.jar + + + org.apache.log4j + NagiosAppender + 1.5.0 + system + ${basedir}/lib/NagiosAppender-1.5.0.jar + + + org.hyperic + sigar + 1.6.4 + system + ${basedir}/lib/sigar.jar + + + org.json + json + 20090211 + + + org.apache.activemq + activemq-core + 5.5.1 + + + org.prefuse + prefuse + beta-20071021 + + + commons-configuration + commons-configuration + 1.7 + + + commons-fileupload + commons-fileupload + 1.2 + + + commons-httpclient + commons-httpclient + 3.1 + + + commons-lang + commons-lang + 2.4 + + + commons-logging + commons-logging + 1.1 + + + commons-io + commons-io + 1.4 + + + commons-codec + commons-codec + 1.3 + + + commons-collections + commons-collections + 3.1 + + + commons-net + commons-net + 1.4.1 + + + commons-logging + commons-logging-api + 1.1 + + + commons-beanutils + commons-beanutils + 1.8.0 + + + org.apache.tomcat + jasper + 6.0.33 + + + org.mortbay.jetty + jetty + 6.1.26 + + + org.mortbay.jetty + jetty-util + 6.1.26 + + + junit + junit + 4.10 + + + log4j + log4j + 1.2.16 + + + org.springframework + spring-mock + 2.0.8 + + + jdiff + jdiff + 1.0.9 + + + xmlenc + xmlenc + 0.52 + + + xerces + xercesImpl + 2.10.0 + + + org.apache.hbase + hbase + 0.90.4 + + + org.apache.hbase + hbase + tests + 0.90.4 + + + org.apache.hadoop + libthrift + 0.5.0 + + + org.apache.hadoop + zookeeper + 3.3.1 + + + com.sun.jersey + jersey-bundle + 1.10 + + + com.google.guava + guava + 10.0.1 + + + com.googlecode.json-simple + json-simple + 1.1 + + + org.apache.hadoop + hadoop-core + 0.20.205.0 + + + org.apache.hadoop + hadoop-test + 0.20.205.0 + + + + + + eyang + Eric Yang + eric818@gmail.com + (GMT-08:00) Pacific Time(US & Canada) + + + + + + + + Apache Software Foundation + http://www.apache.org/ + + + + + + src/main/resources + true + + + + + org.apache.maven.plugins + maven-resources-plugin + 2.4.3 + + UTF-8 + + + + org.apache.maven.plugins + maven-jar-plugin + 2.3.1 + + + + test-jar + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 2.3.2 + + 1.5 + 1.6 + 1.6 + + + + + org.apache.maven.plugins + maven-source-plugin + 2.1.1 + + + prepare-package + + jar-no-fork + + + + + + org.apache.maven.plugins + maven-surefire-plugin + 2.10 + + true + + + + default-test + test + + test + + + false + -Xmx1024m + ${project.build.directory}/test-reports + pertest + true + + **/Test*.java + + + **/TestFSMBuilder.java + **/TestDatabase*.java + **/TestArchive.java + **/*$* + + + ${project.build.directory}/test/classes + ${project.build.directory}/logs + ${test.build.dir}/conf + ${test.build.dir}/var + ${test.build.dir}/var/run + /chukwa + ${basedir}/target + + + ${project.build.directory}/logs + ${test.build.dir}/conf + ${test.build.dir}/var + ${test.build.dir}/var/run + /chukwa + ${basedir}/target + + ${project.build.directory}/test/classes + + + + integration-test + + test + + + false + -Xmx1024m + ${project.build.directory}/test-reports + pertest + true + + **/TestFSMBuilder.java + + + **/TestDatabase*.java + **/TestArchive.java + **/*$* + + + ${project.build.directory}/logs + ${test.build.dir}/conf + ${test.build.dir}/var + ${test.build.dir}/var/run + /chukwa + ${basedir}/target + + + ${project.build.directory}/logs + ${test.build.dir}/conf + ${test.build.dir}/var + ${test.build.dir}/var/run + /chukwa + ${basedir}/target + + + + + + + maven-assembly-plugin + + ${final.name} + gnu + + src/packages/tarball/all.xml + + + + + build-tarball + package + + single + + + + + + org.apache.maven.plugins + maven-antrun-plugin + 1.4 + + + setup + validate + + + + + + + + + + run + + + + test-setup + generate-test-resources + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + run + + + + + + + + + + hudson + + + !env.BUILD_NUMBER + + + + 0 + + + + + + + maven2-repository.dev.java.net + Java.net Repository for Maven + http://download.java.net/maven/2/ + default + + + codehaus + http://repository.codehaus.org/ + + + Sonatype-public + SnakeYAML repository + http://oss.sonatype.org/content/groups/public/ + + + clojars + http://clojars.org/repo/ + + + + + + + + + + Modified: incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/inputtools/log4j/Log4jMetricsSink.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/inputtools/log4j/Log4jMetricsSink.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/inputtools/log4j/Log4jMetricsSink.java (original) +++ incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/inputtools/log4j/Log4jMetricsSink.java Mon Nov 21 06:39:38 2011 @@ -94,99 +94,3 @@ public class Log4jMetricsSink implements public void flush() { } } -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.chukwa.inputtools.log4j; - -import org.json.simple.JSONObject; -import org.apache.log4j.Logger; -import org.apache.log4j.PatternLayout; -import java.io.BufferedOutputStream; -import java.io.File; -import java.io.FileWriter; -import java.io.PrintWriter; -import org.apache.commons.configuration.SubsetConfiguration; -import org.apache.hadoop.metrics2.Metric; -import org.apache.hadoop.metrics2.MetricsException; -import org.apache.hadoop.metrics2.MetricsRecord; -import org.apache.hadoop.metrics2.MetricsSink; -import org.apache.hadoop.metrics2.MetricsTag; - -public class Log4jMetricsSink implements MetricsSink { - /* Configuration attribute names */ - private static final String HOST_PROPERTY = "host"; - private static final String PORT_PROPERTY = "port"; - private static final String TIMESTAMP = "timestamp"; - private static String CONTEXT = "context"; - private static final String CONTEXT_NAME = "contextName"; - private static final String RECORD_NAME = "recordName"; - protected String context = "HadoopMetrics"; - protected String host = "localhost"; - protected int port = 9095; - protected Logger out = null; - - @Override - public void init(SubsetConfiguration conf) { - String host = conf.getString(HOST_PROPERTY); - if (host != null) { - this.host = host; - } - String port = conf.getString(PORT_PROPERTY); - if (port != null) { - this.port = Integer.parseInt(port); - } - String context = conf.getString(CONTEXT); - if (context != null) { - this.context = context; - } - - PatternLayout layout = new PatternLayout("%d{ISO8601} %p %c: %m%n"); - - org.apache.log4j.net.SocketAppender appender = new org.apache.log4j.net.SocketAppender(this.host, this.port); - - appender.setName("chukwa.metrics." + this.context); - appender.setLayout(layout); - - Logger logger = Logger.getLogger("chukwa.metrics." + this.context); - logger.setAdditivity(false); - logger.addAppender(appender); - appender.activateOptions(); - out = logger; - } - - @Override - @SuppressWarnings("unchecked") - public void putMetrics(MetricsRecord record) { - JSONObject json = new JSONObject(); - json.put(TIMESTAMP, Long.valueOf(record.timestamp())); - json.put(CONTEXT_NAME, record.context()); - json.put(RECORD_NAME, record.name()); - for (MetricsTag tag : record.tags()) { - json.put(tag.name(), tag.value()); - } - for (Metric metric : record.metrics()) { - json.put(metric.name(), metric.value()); - } - out.info(json); - } - - @Override - public void flush() { - } -} Added: incubator/chukwa/trunk/src/packages/tarball/all.xml URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/packages/tarball/all.xml?rev=1204365&view=auto ============================================================================== --- incubator/chukwa/trunk/src/packages/tarball/all.xml (added) +++ incubator/chukwa/trunk/src/packages/tarball/all.xml Mon Nov 21 06:39:38 2011 @@ -0,0 +1,62 @@ + + + + + tar.gz + + + + + ${basedir}/*.txt + + + + + pom.xml + + + + src + + + conf + + + bin + bin + 755 + + + target + / + + ${artifactId}-${project.version}.jar + ${artifactId}-${project.version}-tests.jar + + + + target/site + docs + + + src/packages + sbin + 755 + + update-${artifactId}-env.sh + + + + + + false + /lib + false + runtime + + + Modified: incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/analysis/salsa/fsm/TestFSMBuilder.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/analysis/salsa/fsm/TestFSMBuilder.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/analysis/salsa/fsm/TestFSMBuilder.java (original) +++ incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/analysis/salsa/fsm/TestFSMBuilder.java Mon Nov 21 06:39:38 2011 @@ -236,13 +236,13 @@ public class TestFSMBuilder extends Test return name.endsWith(".log"); } }); - + System.out.println(System.getenv("CHUKWA_DATA_DIR") + File.separator + "log"); for(String fname : source) { if (!(fname.equals("JobHistory.log") || fname.equals("ClientTrace.log"))) { continue; } StringBuilder fullPath = new StringBuilder(); - fullPath.append(System.getenv("CHUKWA_DATA_DIR")); + fullPath.append(System.getProperty("CHUKWA_DATA_DIR")); fullPath.append(File.separator); fullPath.append("log"); fullPath.append(File.separator); @@ -309,7 +309,7 @@ public class TestFSMBuilder extends Test //assertEquals(ToolRunner.run(job, new Demux(), sortArgs), 0); JobClient.runJob(job); } catch (Exception e) { - fail(e.toString()); + fail(ExceptionUtil.getStackTrace(e)); } System.out.println("Done initialTasks()"); Modified: incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/TestOffsetStatsManager.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/TestOffsetStatsManager.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/TestOffsetStatsManager.java (original) +++ incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/TestOffsetStatsManager.java Mon Nov 21 06:39:38 2011 @@ -74,5 +74,5 @@ public class TestOffsetStatsManager exte assertEquals("Should have gotten a stale data response", -1.0, rate); } - private class DummyKey {} + private static class DummyKey {} } Modified: incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestDirTailingAdaptor.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestDirTailingAdaptor.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestDirTailingAdaptor.java (original) +++ incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestDirTailingAdaptor.java Mon Nov 21 06:39:38 2011 @@ -84,7 +84,7 @@ public class TestDirTailingAdaptor exten agent.shutdown(); conf.setBoolean("chukwaAgent.checkpoint.enabled", true); - Thread.sleep(500); //wait a little bit to make sure new file ts is > last checkpoint time. + Thread.sleep(1500); //wait a little bit to make sure new file ts is > last checkpoint time. File anOldFile = File.createTempFile("oldXYZ","file", dirWithFile); File aNewFile = File.createTempFile("new", "file", dirWithFile); anOldFile.deleteOnExit(); @@ -100,10 +100,12 @@ public class TestDirTailingAdaptor exten System.out.println(adaptors.getKey() +": " + adaptors.getValue()); assertFalse(adaptors.getValue().contains("oldXYZ")); } + Thread.sleep(3 * SCAN_INTERVAL); //wait a bit for the new file to be detected. //should be four adaptors: the DirTailer on emptyDir, the DirTailer on the full dir, //and FileTailers for File inDir and file newfile and not the noread file. assertEquals(4, agent.adaptorCount()); agent.shutdown(); + Thread.sleep(1500); //wait a little bit to make sure new file ts is > last checkpoint time. nukeDirContents(checkpointDir);//nuke dir checkpointDir.delete(); @@ -128,8 +130,16 @@ public class TestDirTailingAdaptor exten conf.set("chukwaAgent.checkpoint.name", "checkpoint_"); conf.setInt("chukwaAgent.control.port", 0); conf.setBoolean("chukwaAgent.checkpoint.enabled", false); - - agent = new ChukwaAgent(conf); + + boolean retry = true; + while(retry) { + try { + retry = false; + agent = new ChukwaAgent(conf); + } catch(Exception e) { + retry = true; + } + } File emptyDir = new File(baseDir, "emptyDir"); createEmptyDir(emptyDir); @@ -151,7 +161,7 @@ public class TestDirTailingAdaptor exten agent.shutdown(); conf.setBoolean("chukwaAgent.checkpoint.enabled", true); - Thread.sleep(500); //wait a little bit to make sure new file ts is > last checkpoint time. + Thread.sleep(1500); //wait a little bit to make sure new file ts is > last checkpoint time. File anOldFile = File.createTempFile("oldXYZ","file", dirWithFile); File aNewFile = File.createTempFile("new", "file", dirWithFile); anOldFile.deleteOnExit(); @@ -169,6 +179,7 @@ public class TestDirTailingAdaptor exten } //should be four adaptors: the DirTailer on emptyDir, the DirTailer on the full dir, //and FileTailers for File inDir and file newfile + Thread.sleep(3 * SCAN_INTERVAL); //wait a bit for the new file to be detected. assertEquals(4, agent.adaptorCount()); agent.shutdown(); Modified: incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxManager.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxManager.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxManager.java (original) +++ incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxManager.java Mon Nov 21 06:39:38 2011 @@ -31,7 +31,6 @@ import org.apache.hadoop.chukwa.extracti public class TestDemuxManager extends TestCase { - /** * Standard workflow */ @@ -430,7 +429,7 @@ public class TestDemuxManager extends Te //// HELPER CLASSES ///// \ /////////////////////////____\ - class DemuxManagerScenario6 extends DemuxManagerScenario { + private static class DemuxManagerScenario6 extends DemuxManagerScenario { int count = 0; public DemuxManagerScenario6(ChukwaConfiguration conf, int count) throws Exception { super(conf); @@ -467,7 +466,7 @@ public class TestDemuxManager extends Te } - class DemuxManagerScenario5 extends DemuxManagerScenario { + private static class DemuxManagerScenario5 extends DemuxManagerScenario { public DemuxManagerScenario5(ChukwaConfiguration conf) throws Exception { super(conf); } @@ -508,7 +507,7 @@ public class TestDemuxManager extends Te } } - class DemuxManagerScenario2 extends DemuxManagerScenario { + private static class DemuxManagerScenario2 extends DemuxManagerScenario { public DemuxManagerScenario2(ChukwaConfiguration conf) throws Exception { super(conf); } @@ -542,7 +541,7 @@ public class TestDemuxManager extends Te } } - class DemuxManagerScenario1 extends DemuxManagerScenario { + private static class DemuxManagerScenario1 extends DemuxManagerScenario { int count = 0; public DemuxManagerScenario1(ChukwaConfiguration conf, int count) throws Exception { super(conf); @@ -560,7 +559,7 @@ public class TestDemuxManager extends Te } } - class DemuxManagerScenario extends DemuxManager { + private static class DemuxManagerScenario extends DemuxManager { public Listactions = new ArrayList(); public DemuxManagerScenario(ChukwaConfiguration conf) throws Exception { Modified: incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLogEntry.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLogEntry.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLogEntry.java (original) +++ incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLogEntry.java Mon Nov 21 06:39:38 2011 @@ -38,7 +38,7 @@ public class TestJobLogEntry extends Tes protected void setUp() throws Exception { super.setUp(); - InputStream stream = this.getClass().getResourceAsStream("/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/TestJobLog.txt"); + InputStream stream = this.getClass().getResourceAsStream("/TestJobLog.txt"); BufferedReader br = new BufferedReader(new InputStreamReader(stream)); while(true) { String line = br.readLine(); @@ -47,7 +47,7 @@ public class TestJobLogEntry extends Tes testLogList.add(line); } - stream = this.getClass().getResourceAsStream("/org/apache/hadoop/chukwa/extraction/demux/processor/mapper/Hadoop18JobHistoryLog.txt"); + stream = this.getClass().getResourceAsStream("/Hadoop18JobHistoryLog.txt"); br = new BufferedReader(new InputStreamReader(stream)); while(true) { String line = br.readLine(); Modified: incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestChukwaDailyRollingFileAppender.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestChukwaDailyRollingFileAppender.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestChukwaDailyRollingFileAppender.java (original) +++ incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestChukwaDailyRollingFileAppender.java Mon Nov 21 06:39:38 2011 @@ -45,7 +45,7 @@ public class TestChukwaDailyRollingFileA LogManager.CONFIGURATOR_CLASS_KEY, null); URL url = TestChukwaDailyRollingFileAppender.class - .getResource("late-log4j.properties"); + .getResource("/late-log4j.properties"); System.getProperties().setProperty("CHUKWA_TEST_LOG_LATE_INIT", folder); Modified: incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestTaskLogAppender.java URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestTaskLogAppender.java?rev=1204365&r1=1202951&r2=1204365&view=diff ============================================================================== --- incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestTaskLogAppender.java (original) +++ incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/inputtools/log4j/TestTaskLogAppender.java Mon Nov 21 06:39:38 2011 @@ -37,7 +37,7 @@ public class TestTaskLogAppender extends public void testTaskLogAppender() { String folder = System.getProperty("test.build.classes"); - File logFile = new File(folder + "/userlogs/attempt_200905220200_13470_r_000000_0/syslog"); + File logFile = new File(folder + "/userlogs/job_200905220200_13470/attempt_200905220200_13470_r_000000_0/syslog"); if (logFile.exists()) { logFile.delete(); } @@ -51,7 +51,7 @@ public class TestTaskLogAppender extends LogManager.CONFIGURATOR_CLASS_KEY, null); URL url = TestTaskLogAppender.class - .getResource("tasklog-log4j.properties"); + .getResource("/tasklog-log4j.properties"); System.getProperties().setProperty("CHUKWA_LOG_DIR", folder); Added: incubator/chukwa/trunk/src/test/resources/log4j.properties URL: http://svn.apache.org/viewvc/incubator/chukwa/trunk/src/test/resources/log4j.properties?rev=1204365&view=auto ============================================================================== --- incubator/chukwa/trunk/src/test/resources/log4j.properties (added) +++ incubator/chukwa/trunk/src/test/resources/log4j.properties Mon Nov 21 06:39:38 2011 @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +log4j.rootLogger=INFO, R +log4j.appender.R=org.apache.log4j.RollingFileAppender +log4j.appender.R.File=${CHUKWA_LOG_DIR}/chukwa.log +log4j.appender.R.MaxFileSize=10MB +log4j.appender.R.MaxBackupIndex=10 +log4j.appender.R.layout=org.apache.log4j.PatternLayout +log4j.appender.R.layout.ConversionPattern=%d{ISO8601} %p %t %c{1} - %m%n + +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.follow=true +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %p %t %c{1} - %m%n +