giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Neumann <mneum...@spotify.com>
Subject Re: Different num supersteps
Date Mon, 03 Mar 2014 16:09:20 GMT
Hi,

I managed to fix it even if I'm still not entirely sure what happened.
The fix is to make a new Text object every time a Text is required as input
(Text does not implement Cloneable). I guess it

So instead of:
  Text candidate = e.getTargetVertexId();
  ...
  vertex.setValue(candidate))

The working version is:
  Text candidate = e.getTargetVertexId();
  ...
  vertex.setValue(new Text(candidate.toString()))
















@Override
    public void compute(Vertex<Text, Text, NullWritable> vertex,
            Iterable<Text> inmessage) throws IOException {

        boolean changed = false;

        // first superstep && setup
        if (getSuperstep() == 0) {
            //initialize value
            vertex.setValue(new Text(vertex.getId().toString()));
            //cheating by checking the neighbors ID's (cuts down 1
iteration)
            for (Edge<Text, NullWritable> e : vertex.getEdges()) {
                Text candidate = e.getTargetVertexId();
                if (candidate.compareTo(vertex.getValue()) < 0) {
                    changed = true;
                    vertex.setValue(new Text(candidate.toString()));
                }
            }
        }

        // other superstep
        else {
            // read all messages and compare with own state
            for (Text message : inmessage) {
                if (message.compareTo(vertex.getValue()) < 0) {
                    changed = true;
                    vertex.setValue(new Text(message.toString()));
                }
            }
        }

        // if state has changed send a message to all neighbors
        if (changed && getSuperstep() < limiter) {
            sendMessageToAllEdges(vertex, new
Text(vertex.getValue().toString()));
        }

        vertex.voteToHalt();
    }




On Mon, Mar 3, 2014 at 2:19 PM, Sebastian Schelter <ssc@apache.org> wrote:

> Hi Martin,
>
> I'm not sure wether we require InputFormats to be threadsafe. Can someone
> answer that question?
>
> Maybe thats the reason you see this behavior.
>
> --sebastian
>
>
>
> On 03/03/2014 10:05 AM, Martin Neumann wrote:
>
>> I checked the input just creating the graph and comparing it. While I cant
>> say the graph is correct (its to big) its at least consistent.
>>
>> So the only things where the different output can come from is the
>> connected component part (see code further down). I'm completely stomped,
>> the code is basically the example code Giraph ships with all I changed was
>> replacing the IntWriteable Id's with Text.
>>
>> Anyone has any Idea what the problem could be, I'm running out of Idea's.
>>
>> cheers Martin
>>
>>
>>
>> @Override
>>      public void compute(Vertex<Text, Text, NullWritable> vertex,
>>              Iterable<Text> inmessage) throws IOException {
>>          boolean changed = false;
>>
>> // first superstep && setup
>>          if (getSuperstep() == 0) {
>>              //initialize value
>>              vertex.setValue(vertex.getId()
>> );
>>              //cheating by checking the neighbors ID's (cuts down 1
>> iteration)
>>              for (Edge<Text, NullWritable> e : vertex.getEdges()) {
>>                  Text candidate = e.getTargetVertexId();
>>                  if (candidate.compareTo(vertex.getValue()) < 0) {
>>                      changed = true;
>>                      vertex.setValue(candidate);
>>                  }
>>              }
>>          }
>>
>>          // other superstep
>>          else {
>>              // read all messages and compare with own state
>>              for (Text message : inmessage) {
>>                  if (message.compareTo(vertex.getValue()) < 0) {
>>                      changed = true;
>>                      vertex.setValue(message);
>>                  }
>>              }
>>          }
>>
>>          // if state has changed send a message to all neighbors
>>          if (changed && getSuperstep() < limiter) {
>>              sendMessageToAllEdges(vertex, vertex.getValue());
>>          }
>>          vertex.voteToHalt();
>>      }
>>
>>
>>
>> On Mon, Mar 3, 2014 at 9:56 AM, Martin Neumann <mneumann@spotify.com>
>> wrote:
>>
>>  Hey,
>>>
>>> sorry, yes I intended to send it to the mailing-list, thanks for the hint
>>> (I repost it to the list). Guess I should stop working in the middle of
>>> the
>>> night. :-)
>>>
>>> I tested the input format and its output is consistent, I always get the
>>> same graph when running it. So the bug has to be in the connected
>>> component
>>> part since that one gives me different outputs for the same input graph.
>>>
>>> cheers Martin
>>>
>>>
>>> On Mon, Mar 3, 2014 at 8:06 AM, Sebastian Schelter <ssc@apache.org>
>>> wrote:
>>>
>>>  Martin,
>>>>
>>>> can you write a MapReduce job that creates your graph and run it with a
>>>> simpler inputformat?
>>>>
>>>> I really suspect that the bug lies somewhere in your input format.
>>>>
>>>> --sebastian
>>>>
>>>>
>>>> On 03/02/2014 09:48 PM, Martin Neumann wrote:
>>>>
>>>>  I checked the input just creating the graph and comparing it. While I
>>>>> cant
>>>>> say the graph is correct (for its to big) its at least consistent.
>>>>>
>>>>> So the only things where the different output can come from is the
>>>>> connected component part (see code in the first mail). I'm completely
>>>>> stomped, the code is basically the example code Giraph ships with all
I
>>>>> changed was replacing the IntWriteable Id's with Text.
>>>>>
>>>>> Anyone has any Idea what the problem could be, I'm running out of
>>>>> Idea's.
>>>>>
>>>>> cheers Martin
>>>>>
>>>>>
>>>>> On Thu, Feb 27, 2014 at 4:59 PM, Sebastian Schelter <ssc@apache.org>
>>>>> wrote:
>>>>>
>>>>>   Hi Martin
>>>>>
>>>>>>
>>>>>> I don't think that there are problems with comparing and sorting
Text
>>>>>> writables as Hadoop is basically a big external sorting system.
>>>>>>
>>>>>> I'm not sure I understand your edge input reader, it looks very
>>>>>> complex,
>>>>>> maybe there's a bug somewhere. You could try to preprocess your data
>>>>>> using
>>>>>> Hadoop so that you can use a simple VertexInputFormat and see if
your
>>>>>> problems still occur.
>>>>>>
>>>>>> --sebastian
>>>>>>
>>>>>>
>>>>>> On 02/27/2014 04:41 PM, Martin Neumann wrote:
>>>>>>
>>>>>>   Hm
>>>>>>
>>>>>>>
>>>>>>> I ran the job 5 times and made a diff between the outputs and
they
>>>>>>> are
>>>>>>> not
>>>>>>> the same. I cant find anything in the code that could lead to
this
>>>>>>> behaviour.
>>>>>>>
>>>>>>> The only idea where to look a the moment would be the identifier.
Has
>>>>>>> anyone experience with String identifier?
>>>>>>> Is a possible that there are problems with comparing and sorting
>>>>>>> TextWritables?
>>>>>>>
>>>>>>> cheers Martin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message