giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Schelter <...@apache.org>
Subject Re: Different num supersteps
Date Mon, 03 Mar 2014 13:19:33 GMT
Hi Martin,

I'm not sure wether we require InputFormats to be threadsafe. Can 
someone answer that question?

Maybe thats the reason you see this behavior.

--sebastian


On 03/03/2014 10:05 AM, Martin Neumann wrote:
> I checked the input just creating the graph and comparing it. While I cant
> say the graph is correct (its to big) its at least consistent.
>
> So the only things where the different output can come from is the
> connected component part (see code further down). I'm completely stomped,
> the code is basically the example code Giraph ships with all I changed was
> replacing the IntWriteable Id's with Text.
>
> Anyone has any Idea what the problem could be, I'm running out of Idea's.
>
> cheers Martin
>
>
>
> @Override
>      public void compute(Vertex<Text, Text, NullWritable> vertex,
>              Iterable<Text> inmessage) throws IOException {
>          boolean changed = false;
>
> // first superstep && setup
>          if (getSuperstep() == 0) {
>              //initialize value
>              vertex.setValue(vertex.getId()
> );
>              //cheating by checking the neighbors ID's (cuts down 1
> iteration)
>              for (Edge<Text, NullWritable> e : vertex.getEdges()) {
>                  Text candidate = e.getTargetVertexId();
>                  if (candidate.compareTo(vertex.getValue()) < 0) {
>                      changed = true;
>                      vertex.setValue(candidate);
>                  }
>              }
>          }
>
>          // other superstep
>          else {
>              // read all messages and compare with own state
>              for (Text message : inmessage) {
>                  if (message.compareTo(vertex.getValue()) < 0) {
>                      changed = true;
>                      vertex.setValue(message);
>                  }
>              }
>          }
>
>          // if state has changed send a message to all neighbors
>          if (changed && getSuperstep() < limiter) {
>              sendMessageToAllEdges(vertex, vertex.getValue());
>          }
>          vertex.voteToHalt();
>      }
>
>
>
> On Mon, Mar 3, 2014 at 9:56 AM, Martin Neumann <mneumann@spotify.com> wrote:
>
>> Hey,
>>
>> sorry, yes I intended to send it to the mailing-list, thanks for the hint
>> (I repost it to the list). Guess I should stop working in the middle of the
>> night. :-)
>>
>> I tested the input format and its output is consistent, I always get the
>> same graph when running it. So the bug has to be in the connected component
>> part since that one gives me different outputs for the same input graph.
>>
>> cheers Martin
>>
>>
>> On Mon, Mar 3, 2014 at 8:06 AM, Sebastian Schelter <ssc@apache.org> wrote:
>>
>>> Martin,
>>>
>>> can you write a MapReduce job that creates your graph and run it with a
>>> simpler inputformat?
>>>
>>> I really suspect that the bug lies somewhere in your input format.
>>>
>>> --sebastian
>>>
>>>
>>> On 03/02/2014 09:48 PM, Martin Neumann wrote:
>>>
>>>> I checked the input just creating the graph and comparing it. While I
>>>> cant
>>>> say the graph is correct (for its to big) its at least consistent.
>>>>
>>>> So the only things where the different output can come from is the
>>>> connected component part (see code in the first mail). I'm completely
>>>> stomped, the code is basically the example code Giraph ships with all I
>>>> changed was replacing the IntWriteable Id's with Text.
>>>>
>>>> Anyone has any Idea what the problem could be, I'm running out of Idea's.
>>>>
>>>> cheers Martin
>>>>
>>>>
>>>> On Thu, Feb 27, 2014 at 4:59 PM, Sebastian Schelter <ssc@apache.org>
>>>> wrote:
>>>>
>>>>   Hi Martin
>>>>>
>>>>> I don't think that there are problems with comparing and sorting Text
>>>>> writables as Hadoop is basically a big external sorting system.
>>>>>
>>>>> I'm not sure I understand your edge input reader, it looks very complex,
>>>>> maybe there's a bug somewhere. You could try to preprocess your data
>>>>> using
>>>>> Hadoop so that you can use a simple VertexInputFormat and see if your
>>>>> problems still occur.
>>>>>
>>>>> --sebastian
>>>>>
>>>>>
>>>>> On 02/27/2014 04:41 PM, Martin Neumann wrote:
>>>>>
>>>>>   Hm
>>>>>>
>>>>>> I ran the job 5 times and made a diff between the outputs and they
are
>>>>>> not
>>>>>> the same. I cant find anything in the code that could lead to this
>>>>>> behaviour.
>>>>>>
>>>>>> The only idea where to look a the moment would be the identifier.
Has
>>>>>> anyone experience with String identifier?
>>>>>> Is a possible that there are problems with comparing and sorting
>>>>>> TextWritables?
>>>>>>
>>>>>> cheers Martin
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>


Mime
View raw message