hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (HAMA-750) Determine the path of partition files
Date Mon, 29 Apr 2013 07:06:16 GMT

     [ https://issues.apache.org/jira/browse/HAMA-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Edward J. Yoon reopened HAMA-750:
---------------------------------


This error seems related with HAMA-750 patch. I'll look at it.

{code}
edward@udanax:~/workspace/hama-trunk$ bin/hama jar examples/target/hama-examples-0.7.0-SNAPSHOT.jar
pagerank randomgraph pagerankresult 2
13/04/29 15:58:18 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
13/04/29 15:58:18 INFO bsp.FileInputFormat: Total input paths to process : 2
13/04/29 15:58:19 INFO bsp.BSPJobClient: Running job: job_201304291546_0006
13/04/29 15:58:22 INFO bsp.BSPJobClient: Current supersteps number: 0
attempt_201304291546_0006_000000_0: 13/04/29 15:58:19 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO sync.ZKSyncClient: Initializing
ZK Sync Client
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO sync.ZooKeeperSyncClientImpl: Start
connecting to Zookeeper! At localhost/127.0.0.1:61002
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 ERROR sync.ZooKeeperSyncClientImpl:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /bsp/job_201304291546_0006/peers
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 WARN message.MessageTransferQueueFactory:
Message queue is configured on deprecated parameter:hama.messenger.queue.class
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 WARN message.MessageTransferQueueFactory:
Message queue is configured on deprecated parameter:hama.messenger.queue.class
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: Starting SocketReader
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: IPC Server Responder:
starting
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: IPC Server listener
on 61002: starting
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO message.HadoopMessageManagerImpl:
 BSPPeer address:localhost port:61002
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: IPC Server handler
0 on 61002: starting
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 ERROR bsp.BSPTask: Error running bsp
setup and bsp function.
attempt_201304291546_0006_000000_0: java.io.IOException: wrong key class: org.apache.hama.examples.PageRank$PageRankVertex
is not class org.apache.hadoop.io.Text
attempt_201304291546_0006_000000_0: 	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2113)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.SequenceFileRecordReader.next(SequenceFileRecordReader.java:85)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.TrackedRecordReader.moveToNext(TrackedRecordReader.java:63)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.TrackedRecordReader.next(TrackedRecordReader.java:49)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPPeerImpl.readNext(BSPPeerImpl.java:598)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:387)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:99)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 WARN message.MessageTransferQueueFactory:
Message queue is configured on deprecated parameter:hama.messenger.queue.class
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: Stopping server on
61002
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: IPC Server handler
0 on 61002: exiting
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: Stopping IPC Server
listener on 61002
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO metrics.RpcInstrumentation: shut
down
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 INFO ipc.Server: Stopping IPC Server
Responder
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 ERROR bsp.BSPTask: Shutting down ping
service.
attempt_201304291546_0006_000000_0: 13/04/29 15:58:20 FATAL bsp.GroomServer: Error running
child
attempt_201304291546_0006_000000_0: java.io.IOException: wrong key class: org.apache.hama.examples.PageRank$PageRankVertex
is not class org.apache.hadoop.io.Text
attempt_201304291546_0006_000000_0: 	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2113)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.SequenceFileRecordReader.next(SequenceFileRecordReader.java:85)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.TrackedRecordReader.moveToNext(TrackedRecordReader.java:63)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.TrackedRecordReader.next(TrackedRecordReader.java:49)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPPeerImpl.readNext(BSPPeerImpl.java:598)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:387)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:99)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
attempt_201304291546_0006_000000_0: java.io.IOException: wrong key class: org.apache.hama.examples.PageRank$PageRankVertex
is not class org.apache.hadoop.io.Text
attempt_201304291546_0006_000000_0: 	at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2113)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.SequenceFileRecordReader.next(SequenceFileRecordReader.java:85)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.TrackedRecordReader.moveToNext(TrackedRecordReader.java:63)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.TrackedRecordReader.next(TrackedRecordReader.java:49)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPPeerImpl.readNext(BSPPeerImpl.java:598)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:387)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:99)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
attempt_201304291546_0006_000000_0: 	at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
13/04/29 15:59:25 INFO bsp.BSPJobClient: Job failed.
edward@udanax:~/workspace/hama-trunk$ 
{code}
                
> Determine the path of partition files
> -------------------------------------
>
>                 Key: HAMA-750
>                 URL: https://issues.apache.org/jira/browse/HAMA-750
>             Project: Hama
>          Issue Type: Bug
>          Components: bsp core
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.7.0
>
>         Attachments: HAMA-750.patch, HAMA-750_v02.patch
>
>
> The parent directory of input file was used to determine the path of base directory for
partition files. The problem is when input is multiple files.
> {code}
>   protected BSPJob partition(BSPJob job, int maxTasks) throws IOException {
>     String inputPath = job.getConfiguration().get(Constants.JOB_INPUT_DIR);
>     Path inputDir = new Path(inputPath);
>     if (fs.isFile(inputDir)) {
>       inputDir = inputDir.getParent();
>     }
>     Path partitionDir = new Path(inputDir + "/partitions");
>     if (fs.exists(partitionDir)) {
>       fs.delete(partitionDir, true);
>     }
> {code}
> Simply we can create partitions on temp directory. For example, /tmp/hama-partitions/{$JOB_NAME}/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message