Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 69E5F10893 for ; Thu, 9 Jan 2014 08:39:15 +0000 (UTC) Received: (qmail 77296 invoked by uid 500); 9 Jan 2014 08:39:14 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 77053 invoked by uid 500); 9 Jan 2014 08:39:13 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 77044 invoked by uid 99); 9 Jan 2014 08:39:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 08:39:12 +0000 X-ASF-Spam-Status: No, hits=2.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of chadijaber986@hotmail.com designates 157.55.2.89 as permitted sender) Received: from [157.55.2.89] (HELO dub0-omc4-s14.dub0.hotmail.com) (157.55.2.89) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 08:39:06 +0000 Received: from DUB111-W44 ([157.55.2.73]) by dub0-omc4-s14.dub0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 9 Jan 2014 00:38:44 -0800 X-TMN: [rbMMJ5aXCLnYJ4zoJOnrxTFSm3FeCJBx] X-Originating-Email: [chadijaber986@hotmail.com] Message-ID: Content-Type: multipart/alternative; boundary="_0ebe0b67-4fad-457d-8e44-6c97c61c280e_" From: chadi jaber To: "user@giraph.apache.org" Subject: Problem with Giraph (please help me) Date: Thu, 9 Jan 2014 09:38:44 +0100 Importance: Normal In-Reply-To: References: , MIME-Version: 1.0 X-OriginalArrivalTime: 09 Jan 2014 08:38:44.0571 (UTC) FILETIME=[34F162B0:01CF0D16] X-Virus-Checked: Checked by ClamAV on apache.org --_0ebe0b67-4fad-457d-8e44-6c97c61c280e_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi everybody=2C I'm really sorry to put this problem on the table again but this is blockin= g for my project=2C Please help me to solve it.If you need more information= please don't hesitateThanks in advancechadi From: chadijaber986@hotmail.com To: user@giraph.apache.org Subject: Problem with Giraph Date: Tue=2C 31 Dec 2013 17:09:39 +0100 =0A= =0A= =0A= Hello=2C I am not sure this is the right way to submit my problem with giraph otherw= ise I'm sorry.I have developped an algorithm inspired from the giraph short= est path example but where the graphis constructed during the first superst= eps and before the shortest path search. My application works fine when use= d with one worker (on one machine). However when more workers (4 on one mac= hine or on a cluster) are used the following error often appears: ...2013-12-31 16:27:33=2C472 INFO org.apache.giraph.comm.netty.NettyClient:= Using Netty without authentication.2013-12-31 16:27:33=2C478 INFO org.apac= he.giraph.comm.netty.NettyServer: start: Using Netty without authentication= .2013-12-31 16:27:33=2C480 INFO org.apache.giraph.comm.netty.NettyServer: s= tart: Using Netty without authentication.2013-12-31 16:27:33=2C482 INFO org= .apache.giraph.comm.netty.NettyServer: start: Using Netty without authentic= ation.2013-12-31 16:27:33=2C484 INFO org.apache.giraph.comm.netty.NettyClie= nt: Using Netty without authentication.2013-12-31 16:27:33=2C485 INFO org.a= pache.giraph.comm.netty.NettyClient: Using Netty without authentication.201= 3-12-31 16:27:33=2C487 INFO org.apache.giraph.comm.netty.NettyClient: Using= Netty without authentication.2013-12-31 16:27:33=2C494 INFO org.apache.gir= aph.comm.netty.NettyClient: connectAllAddresses: Successfully added 4 conne= ctions=2C (4 total connected) 0 failed=2C 0 failures total.2013-12-31 16:27= :33=2C501 INFO org.apache.giraph.worker.BspServiceWorker: loadInputSplits: = Using 1 thread(s)=2C originally 1 threads(s) for 1 total splits.2013-12-31 = 16:27:33=2C508 INFO org.apache.giraph.comm.SendPartitionCache: SendPartitio= nCache: maxVerticesPerTransfer =3D 100002013-12-31 16:27:33=2C508 INFO org.= apache.giraph.comm.SendPartitionCache: SendPartitionCache: maxEdgesPerTrans= fer =3D 800002013-12-31 16:27:33=2C524 INFO org.apache.giraph.worker.InputS= plitsCallable: call: Loaded 0 input splits in 0.020270009 secs=2C (v=3D0=2C= e=3D0) 0.0 vertices/sec=2C 0.0 edges/sec2013-12-31 16:27:33=2C527 INFO org= .apache.giraph.comm.netty.NettyClient: waitAllRequests: Finished all reques= ts. MBytes/sec sent =3D 0=2C MBytes/sec received =3D 0=2C MBytesSent =3D 0= =2C MBytesReceived =3D 0=2C ave sent req MBytes =3D 0=2C ave received req M= Bytes =3D 0=2C secs waited =3D 0.6562013-12-31 16:27:33=2C527 INFO org.apac= he.giraph.worker.BspServiceWorker: setup: Finally loaded a total of (v=3D0= =2C e=3D0)2013-12-31 16:27:33=2C598 INFO org.apache.giraph.comm.netty.handl= er.RequestDecoder: decode: Server window metrics MBytes/sec sent =3D 0=2C M= Bytes/sec received =3D 0=2C MBytesSent =3D 0=2C MBytesReceived =3D 0=2C ave= sent req MBytes =3D 0=2C ave received req MBytes =3D 0=2C secs waited =3D = 0.8162013-12-31 16:27:33=2C605 WARN org.apache.giraph.comm.netty.handler.Re= questServerHandler: exceptionCaught: Channel failed with remote address /17= 2.16.45.53:59257java.io.EOFException at org.jboss.netty.buffer.ChannelBuffe= rInputStream.checkAvailable(ChannelBufferInputStream.java:231) at org.jboss= .netty.buffer.ChannelBufferInputStream.readInt(ChannelBufferInputStream.jav= a:174) at org.apache.giraph.edge.ByteArrayEdges.readFields(ByteArrayEdges.j= ava:172) at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDat= aInput(WritableUtils.java:480) at org.apache.giraph.utils.WritableUtils.rea= dVertexFromDataInput(WritableUtils.java:511) at org.apache.giraph.partition= .SimplePartition.readFields(SimplePartition.java:126) at org.apache.giraph.= comm.requests.SendVertexRequest.readFieldsRequest(SendVertexRequest.java:66= ) at org.apache.giraph.comm.requests.WritableRequest.readFields(WritableReq= uest.java:120) at org.apache.giraph.comm.netty.handler.RequestDecoder.decod= e(RequestDecoder.java:92) at org.jboss.netty.handler.codec.oneone.OneToOneD= ecoder.handleUpstream(OneToOneDecoder.java:72) at org.jboss.netty.handler.e= xecution.ChannelEventRunnable.run(ChannelEventRunnable.java:69) at java.uti= l.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 615) at java.lang.Thread.run(Thread.java:724) the code for my vertex compute function : public class MergeVertex extendsVertex { ... /*** * Convert a Vertex Id from its LongWritable format to Point format (2= Element Array Format) * @param lng LongWritable Format of the VertexId *= @return Alignment point Array */ public static int[] cvtLongToPoint(LongW= ritable lng){ int[] point=3D{0=2C0}=3B point[0]=3D(int) (lng.get()/1000)=3B point[1]=3D(int) (lng.get()% 1000)= =3B return point=3B } @Override public void compute(Iterable messages) throws IOExc= eption { int currentId[]=3D cvtLongToPoint(getId())=3B if (getSuperstep()=3D=3D0) { //NodeValue nv=3Dnew NodeValue()=3B setValue(new DoubleWritable(0d))= =3B } _signallength=3DgetContext().getConfiguration().getInt("SignalLength"=2C0= )=3B if((getSuperstep() < _signallength && getId().get()!=3D0L) || (getSuperst= ep()=3D=3D 0 && getId().get()=3D=3D0L)){ LongWritable dstId=3Dnew LongWritable()=3B //Nodes which are on Graph "Spine" //Remaining Edges Construction if(c= urrentId[0]=3D=3D currentId[1]){ //right Side for (int i=3DcurrentId[1]+1=3Bi<_signallength=3Bi++){ = dstId=3DcvtPointToLong(currentId[0]+1=2Ci)=3B addVertexRequest(dstId= =2Cnew DoubleWritable(Double.MAX_VALUE))=3B addEdgeRequest(getId()=2CEd= geFactory.create(dstId=2C new DoubleWritable(computeCost(getId()=2CdstId)))= )=3B } //Left Side for (int i=3DcurrentId[0]+2=3Bi<_signallength=3Bi++){ = dstId=3DcvtPointToLong(i=2CcurrentId[1]+1)=3B addVertexRequest(dstId= =2Cnew DoubleWritable(Double.MAX_VALUE))=3B addEdgeRequest(getId()=2CEd= geFactory.create(dstId=2C new DoubleWritable(computeCost(getId()=2CdstId)))= )=3B } //Nodes which are not on Graph "Spine" //Remaining Edges Construction }else{ //right Side if(currentId[0]+1<_signallength){ for (int i=3Dcurr= entId[1]+1=3Bi<_signallength=3Bi++){ dstId=3DcvtPointToLong(currentId[= 0]+1=2Ci)=3B addEdgeRequest(getId()=2CEdgeFactory.create(dstId=2C new = DoubleWritable(computeCost(getId()=2CdstId))))=3B } } //Left Side if(currentId[1]+1<_signallength){ for (int i=3Dcurre= ntId[0]+2=3Bi<_signallength=3Bi++){ dstId=3DcvtPointToLong(i=2Ccurrent= Id[1]+1)=3B addEdgeRequest(getId()=2CEdgeFactory.create(dstId=2C new D= oubleWritable(computeCost(getId()=2CdstId))))=3B } } } //No need to other vertex than source to be active if(getId().get() != =3D 0L){ voteToHalt()=3B } }else if (getSuperstep() >=3D _signallength && getSuperstep() < 2*_signal= length){ double minDist=3B long minSource=3D0L=3B if(getId().get() =3D=3D 0L){ minDist=3D0=3B }else{ minDist=3DDou= ble.MAX_VALUE=3B } for(NodeMessage message : messages){ if(minDist > message.get()){ = minDist=3Dmessage.get()=3B minSource=3Dmessage.getSourceID()=3B } = } if (minDist < getValue().get()){ setValue(new DoubleWritable(minDist)= )=3B =09 for (Edge edge : getEdges()) { doub= le distance =3D minDist + edge.getValue().get()=3B sendMessage(edge.get= TargetVertexId()=2C new NodeMessage(distance=2CgetId().get()))=3B = } } //Only last Node is active if(currentId[0] !=3D _signallength-1 || cur= rentId[1] !=3D _signallength-1){ voteToHalt()=3B } }else if(getSuperstep() >=3D 2*_signallength){ voteToHalt()=3B } } If you need more details please don't hesitate. Thanks in advance=2CChadi = --_0ebe0b67-4fad-457d-8e44-6c97c61c280e_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi everybody=2C

I'm really sorry to put this problem on the table again but this is bloc= king for my project=2C Please help me to solve it.
If you need mo= re information please don't hesitate
Thanks in advance
= chadi


From: chadijaber986@ho= tmail.com
To: user@giraph.apache.org
Subject: Problem with Giraph
= Date: Tue=2C 31 Dec 2013 17:09:39 +0100

=0A= =0A= =0A=
Hello=2C

I am no= t sure this is the right way to submit my problem with giraph otherwise I'm= sorry.
I have developped an algorithm in= spired from the giraph shortest path example but where the graph
is constructed during the first supersteps and befo= re the shortest path search. My application works fine when used with one w= orker (on one machine). However when more workers (4 on one machine or on a= cluster) are used the following error often appears:

=
...
2013-12-31 16:27:33=2C472 INFO org.apache.giraph.comm.netty.= NettyClient: Using Netty without authentication.
2013-12-31 16:27:33=2C478 INFO org.apache.giraph.comm= .netty.NettyServer: start: Using Netty without authentication.
2013-12-31 16:27:33=2C480 INFO org.apac= he.giraph.comm.netty.NettyServer: start: Using Netty without authentication= .
2013-12-31 16:27:33=2C482= INFO org.apache.giraph.comm.netty.NettyServer: start: Using Netty without = authentication.
2013-12-31 = 16:27:33=2C484 INFO org.apache.giraph.comm.netty.NettyClient: Using Netty w= ithout authentication.
2013= -12-31 16:27:33=2C485 INFO org.apache.giraph.comm.netty.NettyClient: Using = Netty without authentication.
2013-12-31 16:27:33=2C487 INFO org.apache.giraph.comm.netty.NettyClient:= Using Netty without authentication.
2013-12-31 16:27:33=2C494 INFO org.apache.giraph.comm.netty.Netty= Client: connectAllAddresses: Successfully added 4 connections=2C (4 total c= onnected) 0 failed=2C 0 failures total.
2013-12-31 16:27:33=2C501 INFO org.apache.giraph.worker.BspSer= viceWorker: loadInputSplits: Using 1 thread(s)=2C originally 1 threads(s) f= or 1 total splits.
2013-12-= 31 16:27:33=2C508 INFO org.apache.giraph.comm.SendPartitionCache: SendParti= tionCache: maxVerticesPerTransfer =3D 10000
2013-12-31 16:27:33=2C508 INFO org.apache.giraph.comm.Send= PartitionCache: SendPartitionCache: maxEdgesPerTransfer =3D 80000
2013-12-31 16:27:33=2C524 INFO org.a= pache.giraph.worker.InputSplitsCallable: call: Loaded 0 input splits in 0.0= 20270009 secs=2C (v=3D0=2C e=3D0) 0.0 vertices/sec=2C 0.0 edges/sec<= /i>
2013-12-31 16:27:33=2C527 INFO org= .apache.giraph.comm.netty.NettyClient: waitAllRequests: Finished all reques= ts. MBytes/sec sent =3D 0=2C MBytes/sec received =3D 0=2C MBytesSent =3D 0= =2C MBytesReceived =3D 0=2C ave sent req MBytes =3D 0=2C ave received req M= Bytes =3D 0=2C secs waited =3D 0.656
2013-12-31 16:27:33=2C527 INFO org.apache.giraph.worker.BspServic= eWorker: setup: Finally loaded a total of (v=3D0=2C e=3D0)
=
2013-12-31 16:27:33=2C598 INFO org.apache.g= iraph.comm.netty.handler.RequestDecoder: decode: Server window metrics MByt= es/sec sent =3D 0=2C MBytes/sec received =3D 0=2C MBytesSent =3D 0=2C MByte= sReceived =3D 0=2C ave sent req MBytes =3D 0=2C ave received req MBytes =3D= 0=2C secs waited =3D 0.816
2013-12-31 16:27:33=2C605 WARN org.apache.giraph.comm.netty.handler.Reques= tServerHandler: exceptionCaught: Channel failed with remote address /172.16= .45.53:59257
java.io.EOFExc= eption
at org.jboss.netty.buffe= r.ChannelBufferInputStream.checkAvailable(ChannelBufferInputStream.java:231= )
at org.jboss.netty.buffer.Cha= nnelBufferInputStream.readInt(ChannelBufferInputStream.java:174)=
at org.apache.giraph.edge.ByteArrayEdges.= readFields(ByteArrayEdges.java:172)
at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInp= ut(WritableUtils.java:480)
= at o= rg.apache.giraph.utils.WritableUtils.readVertexFromDataInput(WritableUtils.= java:511)
at org.apache.giraph.= partition.SimplePartition.readFields(SimplePartition.java:126)
at org.apache.giraph.comm.requests.SendVert= exRequest.readFieldsRequest(SendVertexRequest.java:66)
at org.apache.giraph.comm.requests.WritableRequest.= readFields(WritableRequest.java:120)
<= /span>at org.apache.giraph.comm.netty.handler.RequestDecoder.decode(Request= Decoder.java:92)
at org.jboss.n= etty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.ja= va:72)
at org.jboss.netty.handl= er.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)<= /i>
at java.util.concurrent.ThreadPoolExec= utor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP= oolExecutor.java:615)
at java.l= ang.Thread.run(Thread.java:724)

the code for my = vertex compute function :

public class MergeVertex extends
Vertex<=3BLongWritable=2CD= oubleWritable=2C DoubleWritable=2C NodeMessage>=3B {

...

/***
* Convert a Vertex Id from its LongWritable format to Point form= at (2 Element Array Format)
* = @param lng LongWritable Format of the VertexId
* @return Alignment point Array
*/
public st= atic int[] cvtLongToPoint(LongWritable lng){
int[] point=3D{0=2C0}=3B

point[0]=3D(= int) (lng.get()/1000)=3B
point= [1]=3D(int) (lng.get()% 1000)=3B

return point=3B
}

@Override<= /font>
public void compute(Iterable<= =3BNodeMessage>=3B messages) throws IOException {

= int currentId[]=3D cvtLongToPoint(getId())=3B

if (ge= tSuperstep()=3D=3D0) {

=
//NodeValue nv=3Dnew NodeVal= ue()=3B
setValue(new DoubleWr= itable(0d))=3B
}


_signalleng= th=3DgetContext().getConfiguration().getInt("SignalLength"=2C0)=3B


if((get= Superstep() <=3B _signallength &=3B&=3B getId().get()!=3D0L) || (ge= tSuperstep()=3D=3D 0 &=3B&=3B getId().get()=3D=3D0L)){

LongWritable dstId=3Dnew LongWritable()=3B
=
//Nodes which are on Graph "Spine" //Remaining Edges Construction=
if(currentId[0]=3D=3D currentId[1])= {

//right Side
for (int i=3DcurrentId[1]+1=3Bi<=3B_signallength=3Bi++){
dstId=3DcvtPointToLong(currentI= d[0]+1=2Ci)=3B
addVertexRe= quest(dstId=2Cnew DoubleWritable(Double.MAX_VALUE))=3B
addEdgeRequest(getId()=2CEdgeFactory.create(dst= Id=2C new DoubleWritable(computeCost(getId()=2CdstId))))=3B
}

//Left Side
for (int i=3DcurrentId[0]+2=3Bi&= lt=3B_signallength=3Bi++){
= = dstId=3DcvtPointToLong(i=2CcurrentId[1]+1)=3B
addVertexRequest(dstId=2Cnew DoubleWritable(Double.MAX_V= ALUE))=3B
addEdgeRequest(ge= tId()=2CEdgeFactory.create(dstId=2C new DoubleWritable(computeCost(getId()= =2CdstId))))=3B
}=

//Nodes which are not on Graph "Spine" //Remaining Edges Co= nstruction

<= /div>
}else{

//right = Side
if(currentId[0]+1<=3B= _signallength){
for (int i= =3DcurrentId[1]+1=3Bi<=3B_signallength=3Bi++){
dstId=3DcvtPointToLong(currentId[0]+1=2Ci)=3B=
addEdgeRequest(getId()=2CEdgeFac= tory.create(dstId=2C new DoubleWritable(computeCost(getId()=2CdstId))))=3B<= /font>
}
<= font color=3D"#d24726"> }
//Left Side
if(currentId[1]+1<=3B_signallength){
for (int i=3DcurrentId[0]+2= =3Bi<=3B_signallength=3Bi++){
<= /span>dstId=3DcvtPointToLong(i=2CcurrentId[1]+1)=3B
addEdgeRequest(getId()=2CEdgeFactory.create(dstId= =2C new DoubleWritable(computeCost(getId()=2CdstId))))=3B
<= div> }
}

}

//No need to other= vertex than source to be active
if(getId().get() !=3D 0L){
voteToHalt()=3B
}

}else if (getSuperstep() >=3B=3D _signallength &=3B&= =3B getSuperstep() <=3B 2*_signallength){

double = minDist=3B
long minSource=3D0= L=3B

<= div>
= if(getId().get() =3D=3D 0L){
= minDist=3D0=3B
}else{=
minDist=3DDouble.MAX_VALUE= =3B
}

for(NodeMessage message : messages){
if(minDist >=3B message.get()){
minDist=3Dmessage.get()=3B
minSource=3Dmessage.getSourceID()=3B
}
}


if (minDist <=3B getValue().get()){
setValue(new DoubleWritable(minDist))=3B

for (Edge<=3BLongW= ritable=2C DoubleWritable>=3B edge : getEdges()) {
<= i> double distance =3D minDist + edge.getValue().get= ()=3B
sendMessage(edge.getT= argetVertexId()=2C
new No= deMessage(distance=2CgetId().get()))=3B
}
}<= /div>



//Only last Node is active
if(currentId[0] !=3D _signallengt= h-1 || currentId[1] !=3D _signallength-1){
voteToHalt()=3B
}

<= div>

}else if(getSu= perstep() >=3B=3D 2*_signallength){
voteToHalt()=3B
= }

}
 =3B
If you need more details please don't hesitate.

Thanks in advance=2C
Chadi

=
= --_0ebe0b67-4fad-457d-8e44-6c97c61c280e_--