giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maja Kabiljo <majakabi...@fb.com>
Subject Re: Problem with Giraph (please help me)
Date Thu, 09 Jan 2014 18:30:53 GMT
Hi Chadi,

That does seem like a serialization issue. Which OutEdges class are you using, is it something
you implemented?

Regards,
Maja

From: chadi jaber <chadijaber986@hotmail.com<mailto:chadijaber986@hotmail.com>>
Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@giraph.apache.org<mailto:user@giraph.apache.org>>
Date: Thursday, January 9, 2014 2:08 AM
To: Lukas Nalezenec <lukas.nalezenec@firma.seznam.cz<mailto:lukas.nalezenec@firma.seznam.cz>>,
"user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@giraph.apache.org<mailto:user@giraph.apache.org>>
Subject: RE: Problem with Giraph (please help me)

Hello Lukas
I have enclosed in my previous emails the exception. It seems to be a serialization issue
(This occurs only when workers > 1)

...
2013-12-31 16:27:33,494 INFO org.apache.giraph.comm.netty.NettyClient: connectAllAddresses:
Successfully added 4 connections, (4 total connected) 0 failed, 0 failures total.
2013-12-31 16:27:33,501 INFO org.apache.giraph.worker.BspServiceWorker: loadInputSplits: Using
1 thread(s), originally 1 threads(s) for 1 total splits.
2013-12-31 16:27:33,508 INFO org.apache.giraph.comm.SendPartitionCache: SendPartitionCache:
maxVerticesPerTransfer = 10000
2013-12-31 16:27:33,508 INFO org.apache.giraph.comm.SendPartitionCache: SendPartitionCache:
maxEdgesPerTransfer = 80000
2013-12-31 16:27:33,524 INFO org.apache.giraph.worker.InputSplitsCallable: call: Loaded 0
input splits in 0.020270009 secs, (v=0, e=0) 0.0 vertices/sec, 0.0 edges/sec
2013-12-31 16:27:33,527 INFO org.apache.giraph.comm.netty.NettyClient: waitAllRequests: Finished
all requests. MBytes/sec sent = 0, MBytes/sec received = 0, MBytesSent = 0, MBytesReceived
= 0, ave sent req MBytes = 0, ave received req MBytes = 0, secs waited = 0.656
2013-12-31 16:27:33,527 INFO org.apache.giraph.worker.BspServiceWorker: setup: Finally loaded
a total of (v=0, e=0)
2013-12-31 16:27:33,598 INFO org.apache.giraph.comm.netty.handler.RequestDecoder: decode:
Server window metrics MBytes/sec sent = 0, MBytes/sec received = 0, MBytesSent = 0, MBytesReceived
= 0, ave sent req MBytes = 0, ave received req MBytes = 0, secs waited = 0.816
2013-12-31 16:27:33,605 WARN org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
Channel failed with remote address /172.16.45.53:59257
java.io.EOFException
at org.jboss.netty.buffer.ChannelBufferInputStream.checkAvailable(ChannelBufferInputStream.java:231)
at org.jboss.netty.buffer.ChannelBufferInputStream.readInt(ChannelBufferInputStream.java:174)
at org.apache.giraph.edge.ByteArrayEdges.readFields(ByteArrayEdges.java:172)
at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInput(WritableUtils.java:480)
at org.apache.giraph.utils.WritableUtils.readVertexFromDataInput(WritableUtils.java:511)
at org.apache.giraph.partition.SimplePartition.readFields(SimplePartition.java:126)
at org.apache.giraph.comm.requests.SendVertexRequest.readFieldsRequest(SendVertexRequest.java:66)
at org.apache.giraph.comm.requests.WritableRequest.readFields(WritableRequest.java:120)
at org.apache.giraph.comm.netty.handler.RequestDecoder.decode(RequestDecoder.java:92)
at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:72)
at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

the code for my vertex compute function :

public class MergeVertex extends
Vertex<LongWritable,DoubleWritable, DoubleWritable, NodeMessage> {

...

/***
 * Convert a Vertex Id from its LongWritable format to Point format (2 Element Array Format)
 * @param lng LongWritable Format of the VertexId
 * @return Alignment point Array
 */
public static int[] cvtLongToPoint(LongWritable lng){
int[] point={0,0};

point[0]=(int) (lng.get()/1000);
point[1]=(int) (lng.get()% 1000);

return point;
}

@Override
public void compute(Iterable<NodeMessage> messages) throws IOException {

int currentId[]= cvtLongToPoint(getId());

if (getSuperstep()==0) {

//NodeValue nv=new NodeValue();
setValue(new DoubleWritable(0d));
}


_signallength=getContext().getConfiguration().getInt("SignalLength",0);


if((getSuperstep() < _signallength && getId().get()!=0L) || (getSuperstep()== 0
&& getId().get()==0L)){

LongWritable dstId=new LongWritable();

//Nodes which are on Graph "Spine" //Remaining Edges Construction
if(currentId[0]== currentId[1]){

//right Side
for (int i=currentId[1]+1;i<_signallength;i++){
dstId=cvtPointToLong(currentId[0]+1,i);
addVertexRequest(dstId,new DoubleWritable(Double.MAX_VALUE));
addEdgeRequest(getId(),EdgeFactory.create(dstId, new DoubleWritable(computeCost(getId(),dstId))));
}

//Left Side
for (int i=currentId[0]+2;i<_signallength;i++){
dstId=cvtPointToLong(i,currentId[1]+1);
addVertexRequest(dstId,new DoubleWritable(Double.MAX_VALUE));
addEdgeRequest(getId(),EdgeFactory.create(dstId, new DoubleWritable(computeCost(getId(),dstId))));
}

//Nodes which are not on Graph "Spine" //Remaining Edges Construction

}else{

//right Side
if(currentId[0]+1<_signallength){
for (int i=currentId[1]+1;i<_signallength;i++){
dstId=cvtPointToLong(currentId[0]+1,i);
addEdgeRequest(getId(),EdgeFactory.create(dstId, new DoubleWritable(computeCost(getId(),dstId))));
}
}

//Left Side
if(currentId[1]+1<_signallength){
for (int i=currentId[0]+2;i<_signallength;i++){
dstId=cvtPointToLong(i,currentId[1]+1);
addEdgeRequest(getId(),EdgeFactory.create(dstId, new DoubleWritable(computeCost(getId(),dstId))));
}
}

}

//No need to other vertex than source to be active
if(getId().get() != 0L){
voteToHalt();
}

}else if (getSuperstep() >= _signallength && getSuperstep() < 2*_signallength){

double minDist;
long minSource=0L;


if(getId().get() == 0L){
minDist=0;
}else{
minDist=Double.MAX_VALUE;
}

for(NodeMessage message : messages){
if(minDist > message.get()){
minDist=message.get();
minSource=message.getSourceID();
}
}


if (minDist < getValue().get()){
setValue(new DoubleWritable(minDist));

for (Edge<LongWritable, DoubleWritable> edge : getEdges()) {
double distance = minDist + edge.getValue().get();
sendMessage(edge.getTargetVertexId(),
new NodeMessage(distance,getId().get()));
}
}



//Only last Node is active
if(currentId[0] != _signallength-1 || currentId[1] != _signallength-1){
voteToHalt();
}



}else if(getSuperstep() >= 2*_signallength){
voteToHalt();
}

}

If you need more details please don't hesitate.

Thanks in advance,
Chadi

________________________________
Date: Thu, 9 Jan 2014 10:49:54 +0100
From: lukas.nalezenec@firma.seznam.cz<mailto:lukas.nalezenec@firma.seznam.cz>
To: chadijaber986@hotmail.com<mailto:chadijaber986@hotmail.com>
Subject: Re: Problem with Giraph (please help me)

Hi,
Find the mapper running on the remote address and check what happened. Maybe there will be
exception.
Lukas

On 9.1.2014 09:38, chadi jaber wrote:
exceptionCaught: Channel failed with remote address /172.16.45.53:59257


Mime
View raw message