Return-Path: X-Original-To: apmail-giraph-dev-archive@www.apache.org Delivered-To: apmail-giraph-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BF8D5992B for ; Mon, 18 Jun 2012 22:57:44 +0000 (UTC) Received: (qmail 67591 invoked by uid 500); 18 Jun 2012 22:57:44 -0000 Delivered-To: apmail-giraph-dev-archive@giraph.apache.org Received: (qmail 67557 invoked by uid 500); 18 Jun 2012 22:57:44 -0000 Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@giraph.apache.org Delivered-To: mailing list dev@giraph.apache.org Received: (qmail 67544 invoked by uid 500); 18 Jun 2012 22:57:44 -0000 Delivered-To: apmail-incubator-giraph-dev@incubator.apache.org Received: (qmail 67540 invoked by uid 99); 18 Jun 2012 22:57:44 -0000 Received: from issues-vm.apache.org (HELO issues-vm) (140.211.11.160) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jun 2012 22:57:44 +0000 Received: from isssues-vm.apache.org (localhost [127.0.0.1]) by issues-vm (Postfix) with ESMTP id 68E10141F4C for ; Mon, 18 Jun 2012 22:57:44 +0000 (UTC) Date: Mon, 18 Jun 2012 22:57:44 +0000 (UTC) From: "Eugene Koontz (JIRA)" To: giraph-dev@incubator.apache.org Message-ID: <76740168.27257.1340060264431.JavaMail.jiratomcat@issues-vm> In-Reply-To: <1484908161.27233.1340059603754.JavaMail.jiratomcat@issues-vm> Subject: [jira] [Updated] (GIRAPH-208) LocalTestMode's zookeeper directory is not being cleaned up after job runs MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/GIRAPH-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-208: --------------------------------- Attachment: GIRAPH-208.patch > LocalTestMode's zookeeper directory is not being cleaned up after job runs > -------------------------------------------------------------------------- > > Key: GIRAPH-208 > URL: https://issues.apache.org/jira/browse/GIRAPH-208 > Project: Giraph > Issue Type: Bug > Affects Versions: 0.2.0 > Reporter: Eugene Koontz > Assignee: Eugene Koontz > Priority: Minor > Attachments: GIRAPH-208.patch > > > Discovered bug while running PageRankBenchmark in localTestMode, with -Phadoop_1.0, with the following giraph-site.xml: > {code} > > > giraph.SplitMasterWorker > false > > > giraph.localTestMode > true > > > giraph.zkJar > /home/eugene/giraph/target/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar > > > {code} > With this configuration, I ran PageRankBenchmark as follows: > {code} > java -cp (all the jars..) org.apache.giraph.benchmark.PageRankBenchmark -c 0 -e 3 -s 5 -v -w 1 -V 10 > {code} > This worked the first time: > {code} > 12/06/18 15:33:51 INFO mapred.JobClient: Job complete: job_local_0001 > 12/06/18 15:33:51 INFO mapred.JobClient: Counters: 31 > 12/06/18 15:33:51 INFO mapred.JobClient: Giraph Timers > 12/06/18 15:33:51 INFO mapred.JobClient: Total (milliseconds)=5361 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 3 (milliseconds)=305 > 12/06/18 15:33:51 INFO mapred.JobClient: Vertex input superstep (milliseconds)=207 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 4 (milliseconds)=317 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 10 (milliseconds)=297 > 12/06/18 15:33:51 INFO mapred.JobClient: Setup (milliseconds)=459 > 12/06/18 15:33:51 INFO mapred.JobClient: Shutdown (milliseconds)=875 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 7 (milliseconds)=305 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 0 (milliseconds)=553 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 8 (milliseconds)=304 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 9 (milliseconds)=306 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 6 (milliseconds)=339 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 5 (milliseconds)=268 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 2 (milliseconds)=313 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep 1 (milliseconds)=503 > 12/06/18 15:33:51 INFO mapred.JobClient: File Output Format Counters > 12/06/18 15:33:51 INFO mapred.JobClient: Bytes Written=0 > 12/06/18 15:33:51 INFO mapred.JobClient: Giraph Stats > 12/06/18 15:33:51 INFO mapred.JobClient: Aggregate edges=100 > 12/06/18 15:33:51 INFO mapred.JobClient: Superstep=11 > 12/06/18 15:33:51 INFO mapred.JobClient: Current workers=1 > 12/06/18 15:33:51 INFO mapred.JobClient: Last checkpointed superstep=0 > 12/06/18 15:33:51 INFO mapred.JobClient: Current master task partition=0 > 12/06/18 15:33:51 INFO mapred.JobClient: Sent messages=0 > 12/06/18 15:33:51 INFO mapred.JobClient: Aggregate finished vertices=10 > 12/06/18 15:33:51 INFO mapred.JobClient: Aggregate vertices=10 > 12/06/18 15:33:51 INFO mapred.JobClient: File Input Format Counters > 12/06/18 15:33:51 INFO mapred.JobClient: Bytes Read=0 > 12/06/18 15:33:51 INFO mapred.JobClient: FileSystemCounters > 12/06/18 15:33:51 INFO mapred.JobClient: FILE_BYTES_READ=88 > 12/06/18 15:33:51 INFO mapred.JobClient: FILE_BYTES_WRITTEN=32525 > 12/06/18 15:33:51 INFO mapred.JobClient: Map-Reduce Framework > 12/06/18 15:33:51 INFO mapred.JobClient: Map input records=1 > 12/06/18 15:33:51 INFO mapred.JobClient: Spilled Records=0 > 12/06/18 15:33:51 INFO mapred.JobClient: SPLIT_RAW_BYTES=44 > 12/06/18 15:33:51 INFO mapred.JobClient: Map output records=0 > {code} > but trying to run it again yields the following: > {code} > 12/06/18 15:35:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done. > 12/06/18 15:35:01 WARN mapred.FileOutputCommitter: Output path is null in cleanup > 12/06/18 15:35:02 INFO mapred.JobClient: map 100% reduce 0% > 12/06/18 15:35:02 INFO mapred.JobClient: Job complete: job_local_0001 > 12/06/18 15:35:02 INFO mapred.JobClient: Counters: 8 > 12/06/18 15:35:02 INFO mapred.JobClient: File Output Format Counters > 12/06/18 15:35:02 INFO mapred.JobClient: Bytes Written=0 > 12/06/18 15:35:02 INFO mapred.JobClient: File Input Format Counters > 12/06/18 15:35:02 INFO mapred.JobClient: Bytes Read=0 > 12/06/18 15:35:02 INFO mapred.JobClient: FileSystemCounters > 12/06/18 15:35:02 INFO mapred.JobClient: FILE_BYTES_READ=88 > 12/06/18 15:35:02 INFO mapred.JobClient: FILE_BYTES_WRITTEN=32493 > 12/06/18 15:35:02 INFO mapred.JobClient: Map-Reduce Framework > 12/06/18 15:35:02 INFO mapred.JobClient: Map input records=1 > 12/06/18 15:35:02 INFO mapred.JobClient: Spilled Records=0 > 12/06/18 15:35:02 INFO mapred.JobClient: SPLIT_RAW_BYTES=44 > 12/06/18 15:35:02 INFO mapred.JobClient: Map output records=0 > Disconnected from the target VM, address: '127.0.0.1:33268', transport: 'socket' > {code} > which is wrong because the Giraph mapper never got called (note the lack of Superstep timers and "Giraph Stats" section in the above) > A workaround for this bug is to run "rm -rf ~/giraph/_bsp/_defaultZkManagerDir" before re-running PageRankBenchmark - then it will run correctly afterwards. > The problem in the code is that the ZookeeperManager's directory is not being removed as it should be. This is because the zkDirDefault in ZooKeeperManager.java is not being set correctly - it is currently: > {code} > System.getProperty("user.dir") + "/_bspZooKeeper"; > {code} > but it should be: > {code} > System.getProperty("user.dir") + GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT; > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira