giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-208) LocalTestMode's zookeeper directory is not being cleaned up after job runs
Date Mon, 18 Jun 2012 23:44:43 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396374#comment-13396374
] 

Hudson commented on GIRAPH-208:
-------------------------------

Integrated in giraph-trunk-hadoop-0.20.203 #7 (See [https://builds.apache.org/job/giraph-trunk-hadoop-0.20.203/7/])
    GIRAPH-208: LocalTestMode's zookeeper directory is not being cleaned up after job runs.
(Revision 1351533)

     Result = SUCCESS
ekoontz : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351533
Files : 
* /giraph/trunk/src/main/java/org/apache/giraph/zk/ZooKeeperManager.java

                
> LocalTestMode's zookeeper directory is not being cleaned up after job runs
> --------------------------------------------------------------------------
>
>                 Key: GIRAPH-208
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-208
>             Project: Giraph
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: Eugene Koontz
>            Assignee: Eugene Koontz
>            Priority: Minor
>         Attachments: GIRAPH-208.patch, GIRAPH-208.patch
>
>
> Discovered bug while running PageRankBenchmark in localTestMode, with -Phadoop_1.0, with
the following giraph-site.xml:
> {code}
> <configuration>
>   <property>
>     <name>giraph.SplitMasterWorker</name>
>     <value>false</value>
>   </property>
>   <property>
>     <name>giraph.localTestMode</name>
>     <value>true</value>
>   </property>
>   <property>
>     <name>giraph.zkJar</name>
>     <value>/home/eugene/giraph/target/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar</value>
>   </property>
> </configuration>
> {code}
> With this configuration, I ran PageRankBenchmark as follows:
> {code}
> java -cp (all the jars..) org.apache.giraph.benchmark.PageRankBenchmark -c 0 -e 3 -s
5 -v -w 1 -V 10
> {code}
> This worked the first time:
> {code}
> 12/06/18 15:33:51 INFO mapred.JobClient: Job complete: job_local_0001
> 12/06/18 15:33:51 INFO mapred.JobClient: Counters: 31
> 12/06/18 15:33:51 INFO mapred.JobClient:   Giraph Timers
> 12/06/18 15:33:51 INFO mapred.JobClient:     Total (milliseconds)=5361
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 3 (milliseconds)=305
> 12/06/18 15:33:51 INFO mapred.JobClient:     Vertex input superstep (milliseconds)=207
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 4 (milliseconds)=317
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 10 (milliseconds)=297
> 12/06/18 15:33:51 INFO mapred.JobClient:     Setup (milliseconds)=459
> 12/06/18 15:33:51 INFO mapred.JobClient:     Shutdown (milliseconds)=875
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 7 (milliseconds)=305
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 0 (milliseconds)=553
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 8 (milliseconds)=304
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 9 (milliseconds)=306
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 6 (milliseconds)=339
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 5 (milliseconds)=268
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 2 (milliseconds)=313
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep 1 (milliseconds)=503
> 12/06/18 15:33:51 INFO mapred.JobClient:   File Output Format Counters 
> 12/06/18 15:33:51 INFO mapred.JobClient:     Bytes Written=0
> 12/06/18 15:33:51 INFO mapred.JobClient:   Giraph Stats
> 12/06/18 15:33:51 INFO mapred.JobClient:     Aggregate edges=100
> 12/06/18 15:33:51 INFO mapred.JobClient:     Superstep=11
> 12/06/18 15:33:51 INFO mapred.JobClient:     Current workers=1
> 12/06/18 15:33:51 INFO mapred.JobClient:     Last checkpointed superstep=0
> 12/06/18 15:33:51 INFO mapred.JobClient:     Current master task partition=0
> 12/06/18 15:33:51 INFO mapred.JobClient:     Sent messages=0
> 12/06/18 15:33:51 INFO mapred.JobClient:     Aggregate finished vertices=10
> 12/06/18 15:33:51 INFO mapred.JobClient:     Aggregate vertices=10
> 12/06/18 15:33:51 INFO mapred.JobClient:   File Input Format Counters 
> 12/06/18 15:33:51 INFO mapred.JobClient:     Bytes Read=0
> 12/06/18 15:33:51 INFO mapred.JobClient:   FileSystemCounters
> 12/06/18 15:33:51 INFO mapred.JobClient:     FILE_BYTES_READ=88
> 12/06/18 15:33:51 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=32525
> 12/06/18 15:33:51 INFO mapred.JobClient:   Map-Reduce Framework
> 12/06/18 15:33:51 INFO mapred.JobClient:     Map input records=1
> 12/06/18 15:33:51 INFO mapred.JobClient:     Spilled Records=0
> 12/06/18 15:33:51 INFO mapred.JobClient:     SPLIT_RAW_BYTES=44
> 12/06/18 15:33:51 INFO mapred.JobClient:     Map output records=0
> {code}
>  but trying to run it again yields the following:
> {code}
> 12/06/18 15:35:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
> 12/06/18 15:35:01 WARN mapred.FileOutputCommitter: Output path is null in cleanup
> 12/06/18 15:35:02 INFO mapred.JobClient:  map 100% reduce 0%
> 12/06/18 15:35:02 INFO mapred.JobClient: Job complete: job_local_0001
> 12/06/18 15:35:02 INFO mapred.JobClient: Counters: 8
> 12/06/18 15:35:02 INFO mapred.JobClient:   File Output Format Counters 
> 12/06/18 15:35:02 INFO mapred.JobClient:     Bytes Written=0
> 12/06/18 15:35:02 INFO mapred.JobClient:   File Input Format Counters 
> 12/06/18 15:35:02 INFO mapred.JobClient:     Bytes Read=0
> 12/06/18 15:35:02 INFO mapred.JobClient:   FileSystemCounters
> 12/06/18 15:35:02 INFO mapred.JobClient:     FILE_BYTES_READ=88
> 12/06/18 15:35:02 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=32493
> 12/06/18 15:35:02 INFO mapred.JobClient:   Map-Reduce Framework
> 12/06/18 15:35:02 INFO mapred.JobClient:     Map input records=1
> 12/06/18 15:35:02 INFO mapred.JobClient:     Spilled Records=0
> 12/06/18 15:35:02 INFO mapred.JobClient:     SPLIT_RAW_BYTES=44
> 12/06/18 15:35:02 INFO mapred.JobClient:     Map output records=0
> Disconnected from the target VM, address: '127.0.0.1:33268', transport: 'socket'
> {code}
> which is wrong because the Giraph mapper never got called (note the lack of Superstep
timers and "Giraph Stats" section in the above)
> A workaround for this bug is to run "rm -rf ~/giraph/_bsp/_defaultZkManagerDir" before
re-running PageRankBenchmark - then it will run correctly afterwards.
> The problem in the code is that the ZookeeperManager's directory is not being removed
as it should be. This is because the zkDirDefault in ZooKeeperManager.java is not being set
correctly - it is currently: 
> {code}
> System.getProperty("user.dir") + "/_bspZooKeeper";
> {code}
> but it should be:
> {code}
> System.getProperty("user.dir") + GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT;
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message