giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ghufran malik <ghufran1ma...@gmail.com>
Subject Re: Master/Agreggators
Date Thu, 03 Apr 2014 16:34:48 GMT
I just tested the compute method with just:

public void compute(Vertex<IntWritable, IntWritable, NullWritable> vertex,
Iterable<IntWritable> messages) throws IOException
{
//check if its the first superstep
 if (getSuperstep() == 0)
 {

 if(isStart(vertex))
 {
 vertex.setValue(new IntWritable((int) getSuperstep()));
 for (Edge<IntWritable, NullWritable> edge : vertex.getEdges())
 {
 idQueue.addInt(edge.getTargetVertexId().get());
 sendMessage(edge.getTargetVertexId(), new IntWritable(1));
 }
Collections.sort(idQueue.getArrayList());
aggregate(ID_AGG, idQueue);
 }
 else
 {
 vertex.setValue(new IntWritable(Integer.MAX_VALUE));
 }
 }

 else{}
 vertex.voteToHalt();
}

On Thu, Apr 3, 2014 at 5:24 PM, ghufran malik <ghufran1malik@gmail.com>wrote:

> After a while of those info jobs been printed this is printed out after:
>
> 14/04/03 17:01:25 INFO zookeeper.ClientCnxn: Unable to read additional
> data from server sessionid 0x145284966610002, likely server has closed
> socket, closing socket connection and attempting reconnect
> 14/04/03 17:01:26 INFO mapred.JobClient:  map 50% reduce 0%
> 14/04/03 17:01:27 INFO zookeeper.ClientCnxn: Opening socket connection to
> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using
> SASL (unknown error)
> 14/04/03 17:01:27 WARN zookeeper.ClientCnxn: Session 0x145284966610002 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>  at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 14/04/03 17:01:27 WARN zk.ZooKeeperExt: exists: Connection loss on attempt
> 0, waiting 5000 msecs before retrying.
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for
> /_hadoopBsp/job_201404031649_0001/_workerProgresses
>  at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>  at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1069)
>  at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:360)
> at
> org.apache.giraph.job.JobProgressTracker$2.run(JobProgressTracker.java:87)
>  at java.lang.Thread.run(Thread.java:744)
> 14/04/03 17:01:29 INFO zookeeper.ClientCnxn: Opening socket connection to
> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using
> SASL (unknown error)
> 14/04/03 17:01:29 WARN zookeeper.ClientCnxn: Session 0x145284966610002 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>  at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 14/04/03 17:01:30 INFO zookeeper.ClientCnxn: Opening socket connection to
> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using
> SASL (unknown error)
> 14/04/03 17:01:30 WARN zookeeper.ClientCnxn: Session 0x145284966610002 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>  at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 14/04/03 17:01:31 INFO zookeeper.ClientCnxn: Opening socket connection to
> server ghufran/127.0.1.1:22181. Will not attempt to authenticate using
> SASL (unknown error)
> 14/04/03 17:01:31 WARN zookeeper.ClientCnxn: Session 0x145284966610002 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>  at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 14/04/03 17:01:31 INFO mapred.JobClient: Job complete:
> job_201404031649_0001
> 14/04/03 17:01:31 INFO mapred.JobClient: Counters: 6
> 14/04/03 17:01:31 INFO mapred.JobClient:   Job Counters
> 14/04/03 17:01:31 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=1235189
> 14/04/03 17:01:31 INFO mapred.JobClient:     Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 14/04/03 17:01:31 INFO mapred.JobClient:     Total time spent by all maps
> waiting after reserving slots (ms)=0
> 14/04/03 17:01:31 INFO mapred.JobClient:     Launched map tasks=2
> 14/04/03 17:01:31 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
> 14/04/03 17:01:31 INFO mapred.JobClient:     Failed map tasks=1
>
>
>
> On Thu, Apr 3, 2014 at 5:05 PM, ghufran malik <ghufran1malik@gmail.com>wrote:
>
>> My Giraph job gets stuck at this point and will not go any further, that
>> log is what is continually printed out every 5 seconds. This log is coming
>> from the CombinedWorkerProgress class:
>>
>>  else if (isComputeSuperstep()) {
>>       sb.append("Compute superstep ").append(currentSuperstep).append(":
>> ");
>>       sb.append(verticesComputed).append(" out of ").append(
>>           verticesToCompute).append(" vertices computed; ");
>>       sb.append(partitionsComputed).append(" out of ").append(
>>           partitionsToCompute).append(" partitions computed");
>>
>> So the data is loaded in fine and then the Giraph job gets stuck in
>> superstep 0 for some reason?
>>
>> public void compute(Vertex<IntWritable, IntWritable, NullWritable>
>> vertex, Iterable<IntWritable> messages) throws IOException
>> {
>>  //check if its the first superstep
>>  if (getSuperstep() == 0)
>>  {
>>
>>  if(isStart(vertex))
>>  {
>>  vertex.setValue(new IntWritable((int) getSuperstep()));
>>   for (Edge<IntWritable, NullWritable> edge : vertex.getEdges())
>>  {
>>  idQueue.addInt(edge.getTargetVertexId().get());
>>  sendMessage(edge.getTargetVertexId(), new IntWritable(1));
>>  }
>> Collections.sort(idQueue.getArrayList());
>> aggregate(ID_AGG, idQueue);
>>  }
>>  else
>>  {
>>  vertex.setValue(new IntWritable(Integer.MAX_VALUE));
>>  }
>>  }
>>
>> Thats the code I wrote for the first superstep. I ran this code before
>> without the aggregate and it worked, so I think my problem is related the
>> the aggregator/master.
>>
>> Kind regards,
>>
>> Ghufran
>>
>>
>> On Thu, Apr 3, 2014 at 4:40 PM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
>>
>>> How is that an error?
>>>
>>> That's just some informational log statements from Giraph, you'll need
>>> to provide the actual error message/describe the issue to get help with
>>> your problem
>>>
>>> Rob
>>>
>>> From: ghufran malik <ghufran1malik@gmail.com>
>>> Reply-To: <user@giraph.apache.org>
>>> Date: Thursday, 3 April 2014 16:09
>>> To: <user@giraph.apache.org>
>>> Subject: Master/Agreggators
>>>
>>> Hi,
>>>
>>> I received the error:
>>>
>>> 14/04/03 16:01:07 INFO mapred.JobClient:  map 100% reduce 0%
>>> 14/04/03 16:01:11 INFO job.JobProgressTracker: Data from 1 workers -
>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions
>>> computed; min free memory on worker 1 - 106.6MB, average 106.6MB
>>> 14/04/03 16:01:16 INFO job.JobProgressTracker: Data from 1 workers -
>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions
>>> computed; min free memory on worker 1 - 106.6MB, average 106.6MB
>>> 14/04/03 16:01:21 INFO job.JobProgressTracker: Data from 1 workers -
>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions
>>> computed; min free memory on worker 1 - 106.6MB, average 106.6MB
>>> 14/04/03 16:01:26 INFO job.JobProgressTracker: Data from 1 workers -
>>> Compute superstep 0: 0 out of 4 vertices computed; 0 out of 1 partitions
>>> computed; min free memory on worker 1 - 106.59MB, average 106.59MB
>>>
>>>
>>> After trying to run a computation class I made that makes use of an
>>> agreggator and master. I remember getting a similar error when I tried
>>> SimplePageRank which also makes uses of a master and a agreggator.
>>>
>>> Does anyone know why I receive this error and how to fix it?
>>>
>>> Kind regards,
>>>
>>> Ghufran
>>>
>>>
>>
>

Mime
View raw message