hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lyu Xuedong <lxd.1...@gmail.com>
Subject Re: Why my distributed mode does not work?
Date Sat, 20 Apr 2013 10:26:52 GMT
Hi, Edward,

Thank you.
I updated the version. There comes a new problem.

In the /Apache Hama BSP Programming Model 
(http://people.apache.org/~tjungblut/downloads/hamadocs/ApacheHamaBSPProgrammingmodel_06.pdf)/

there is a paragraph describes 'Graph repair' : "Hama requires a graph 
to be completed before feeding it to an algorithm. By complete we mean 
that every vertex that is referenced by an edge must somewhere be a 
vertex in the graph. In many cases of leafs this is not always the case, 
therefore we have added a repair functionality which is traversing the 
whole graph for leafs and adding them to the vertex structure to prevent 
algorithms from breaking with NullPointerExceptions when it does not 
find a referenced vertex. You can turn this feature on by setting it in 
your configuration like this: 
conf.setBoolean(GraphJobRunner.GRAPH_REPAIR, true);"

I followed the guidance but got a hint 'GRAPH_REPAIR cannot be resolved 
or is not a field' programming in Eclipse. I read the source code later 
and there is really no variable named GRAPH_REPAIR in 
GraphJobRunner.java which exists in the version 0.5.0. Is this function 
not supported any longer ? How can I make a 'repaired graph'?
Thank you.

On 04/17/2013 12:51 PM, Edward J. Yoon wrote:
> Please use 0.6.1 and try your application with small data again.
>
> See also http://hama.apache.org/run_examples.html
>
> On Wed, Apr 17, 2013 at 10:27 AM, Lyu Xuedong <lxd.1990@gmail.com> wrote:
>> hama: 0.6.0
>> hadoop : 1.0.4
>> JDK : 1.6
>> OS : ubuntu 12.04
>>
>>
>> On 04/17/2013 05:47 AM, Edward J. Yoon wrote:
>>> Your version?
>>>
>>> On Wed, Apr 17, 2013 at 12:07 AM, Lvxuedong <lxd.1990@gmail.com> wrote:
>>>> Hi, Edward, thank you. But your suggestion seems do not work, do you have
>>>> some other advice ?
>>>>
>>>> Is java.lang.NullPointerException related with heap size ?
>>>>
>>>>
>>>>
>>>> 在 2013-4-16,22:13,"Edward J. Yoon" <edwardyoon@apache.org> 写道:
>>>>
>>>>> I guess you need to increase the child processor JVM heap size.
>>>>>
>>>>> - conf/hama-site.xml:
>>>>>
>>>>>    <property>
>>>>>      <name>bsp.child.java.opts</name>
>>>>>      <value>-Xmx2048m</value>
>>>>>    </property>
>>>>>
>>>>> On Tue, Apr 16, 2013 at 10:55 PM, Lyu Xuedong <lxd.1990@gmail.com>
>>>>> wrote:
>>>>>> My project can work well in eclipse, but when I export it as a jar
file
>>>>>> and
>>>>>> submit it to a two-node hama cluster, some errors and fatal occur
if
>>>>>> the
>>>>>> input file is over 64M.
>>>>>> I suspect that my configure files of hadoop or hama is somewhere
not
>>>>>> right,
>>>>>> but a pi estimator can run normally on my cluster and
>>>>>> GroomServer$BSPPerChild can be seen on each nodes. I debug for a
whole
>>>>>> day,
>>>>>> nothing improved.
>>>>>> What in my input file are a large number of RDF triples: "<subject>
>>>>>> <predicate> <object> ." My task is to create vertices
for subjects and
>>>>>> objects. Predicates are subjects' edges.
>>>>>> What should I do ?
>>>>>>
>>>>>> Terminal output:
>>>>>>
>>>>>> 13/04/16 21:13:36 INFO bgp.HamaBgpComplete: Job begain.
>>>>>> 13/04/16 21:13:37 INFO bsp.FileInputFormat: Total input paths to
>>>>>> process : 2
>>>>>> 13/04/16 21:13:38 INFO bsp.BSPJobClient: Running job:
>>>>>> job_201304161357_0015
>>>>>> 13/04/16 21:13:41 INFO bsp.BSPJobClient: Current supersteps number:
0
>>>>>> 13/04/16 21:13:47 INFO bsp.BSPJobClient: Current supersteps number:
2
>>>>>> 13/04/16 21:13:53 INFO bsp.BSPJobClient: Current supersteps number:
3
>>>>>> 13/04/16 21:13:59 INFO bsp.BSPJobClient: Current supersteps number:
4
>>>>>> 13/04/16 21:14:05 INFO bsp.BSPJobClient: Current supersteps number:
5
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:13:43 INFO
>>>>>> sync.ZKSyncClient: Initializing ZK Sync Client
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:13:43 INFO
>>>>>> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
>>>>>> hadoop1/1.2.3.4:61002
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:13:43 INFO ipc.Server:
>>>>>> Starting SocketReader
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:13:43 INFO ipc.Server:
>>>>>> IPC
>>>>>> Server Responder: starting
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:13:43 INFO ipc.Server:
>>>>>> IPC
>>>>>> Server handler 0 on 61002: starting
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:13:43 INFO
>>>>>> message.HadoopMessageManagerImpl:  BSPPeer address:hadoop1 port:61002
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:13:43 INFO ipc.Server:
>>>>>> IPC
>>>>>> Server listener on 61002: starting
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:03 ERROR
>>>>>> bsp.BSPTask:
>>>>>> Error running bsp setup and bsp function.
>>>>>> attempt_201304161357_0015_000000_0: java.lang.NullPointerException
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:04 INFO ipc.Server:
>>>>>> Stopping server on 61002
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:04 INFO ipc.Server:
>>>>>> IPC
>>>>>> Server handler 0 on 61002: exiting
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:04 INFO ipc.Server:
>>>>>> Stopping IPC Server listener on 61002
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:04 INFO ipc.Server:
>>>>>> Stopping IPC Server Responder
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:04 INFO
>>>>>> metrics.RpcInstrumentation: shut down
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:04 ERROR
>>>>>> bsp.BSPTask:
>>>>>> Shutting down ping service.
>>>>>> attempt_201304161357_0015_000000_0: 13/04/16 21:14:04 FATAL
>>>>>> bsp.GroomServer:
>>>>>> Error running child
>>>>>> attempt_201304161357_0015_000000_0: java.lang.NullPointerException
>>>>>> attempt_201304161357_0015_000000_0: java.lang.NullPointerException
>>>>>> 13/04/16 21:15:11 INFO bsp.BSPJobClient: Job failed.
>>>>>>
>>>>>> tasklogs:
>>>>>> attempt_201304161357_0015_000000_0.log
>>>>>> 13/04/16 21:13:43 INFO sync.ZKSyncClient: Initializing ZK Sync Client
>>>>>> 13/04/16 21:13:43 INFO sync.ZooKeeperSyncClientImpl: Start connecting
>>>>>> to
>>>>>> Zookeeper! At hadoop1/1.2.3.4:61002
>>>>>> 13/04/16 21:13:43 INFO ipc.Server: Starting SocketReader
>>>>>> 13/04/16 21:13:43 INFO ipc.Server: IPC Server Responder: starting
>>>>>> 13/04/16 21:13:43 INFO ipc.Server: IPC Server handler 0 on 61002:
>>>>>> starting
>>>>>> 13/04/16 21:13:43 INFO message.HadoopMessageManagerImpl:  BSPPeer
>>>>>> address:hadoop1 port:61002
>>>>>> 13/04/16 21:13:43 INFO ipc.Server: IPC Server listener on 61002:
>>>>>> starting
>>>>>> 13/04/16 21:14:03 ERROR bsp.BSPTask: Error running bsp setup and
bsp
>>>>>> function.
>>>>>> java.lang.NullPointerException
>>>>>> 13/04/16 21:14:04 INFO ipc.Server: Stopping server on 61002
>>>>>> 13/04/16 21:14:04 INFO ipc.Server: IPC Server handler 0 on 61002:
>>>>>> exiting
>>>>>> 13/04/16 21:14:04 INFO ipc.Server: Stopping IPC Server listener on
>>>>>> 61002
>>>>>> 13/04/16 21:14:04 INFO ipc.Server: Stopping IPC Server Responder
>>>>>> 13/04/16 21:14:04 INFO metrics.RpcInstrumentation: shut down
>>>>>> 13/04/16 21:14:04 ERROR bsp.BSPTask: Shutting down ping service.
>>>>>> 13/04/16 21:14:04 FATAL bsp.GroomServer: Error running child
>>>>>> java.lang.NullPointerException
>>>>>> java.lang.NullPointerException
>>>>>>
>>>>>> attempt_201304161357_0015_000001_0.log
>>>>>> 13/04/16 21:13:42 INFO sync.ZKSyncClient: Initializing ZK Sync Client
>>>>>> 13/04/16 21:13:42 INFO sync.ZooKeeperSyncClientImpl: Start connecting
>>>>>> to
>>>>>> Zookeeper! At hadoop1/1.2.3.4:61001
>>>>>> 13/04/16 21:13:42 ERROR sync.ZooKeeperSyncClientImpl:
>>>>>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
=
>>>>>> NoNode for /bsp/job_201304161357_0015/peers
>>>>>> 13/04/16 21:13:42 INFO ipc.Server: Starting SocketReader
>>>>>> 13/04/16 21:13:42 INFO ipc.Server: IPC Server Responder: starting
>>>>>> 13/04/16 21:13:42 INFO message.HadoopMessageManagerImpl:  BSPPeer
>>>>>> address:hadoop1 port:61001
>>>>>> 13/04/16 21:13:42 INFO ipc.Server: IPC Server listener on 61001:
>>>>>> starting
>>>>>> 13/04/16 21:13:42 INFO ipc.Server: IPC Server handler 0 on 61001:
>>>>>> starting
>>>>>> 13/04/16 21:14:06 ERROR bsp.BSPPeerImpl: Error while sending messages
>>>>>> java.io.IOException: Call to hadoop1/1.2.3.4:61002 failed on local
>>>>>> exception: java.io.EOFException
>>>>>>      at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
>>>>>>      at org.apache.hadoop.ipc.Client.call(Client.java:1071)
>>>>>>      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>>>>>>      at $Proxy3.put(Unknown Source)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hama.bsp.message.HadoopMessageManagerImpl.transfer(HadoopMessageManagerImpl.java:108)
>>>>>>      at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:410)
>>>>>>      at
>>>>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:118)
>>>>>>      at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166)
>>>>>>      at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1266)
>>>>>> Caused by: java.io.EOFException
>>>>>>      at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:800)
>>>>>>      at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>
>>>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message