hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahesh Babu <jmb...@gmail.com>
Subject Re: Error - Messages must never be behind the vertex in ID! Current Message ID: 2 vs. 20
Date Sun, 11 Aug 2013 09:59:53 GMT
Hi,

I am trying to understand below code. i.e. see the bolded comment below.


  /**
   * Do the main logic of a superstep, namely checking if vertices are
active,
   * feeding compute with messages and controlling combiners/aggregators.
   */
  @SuppressWarnings("unchecked")
  private void doSuperstep(GraphJobMessage currentMessage,
      BSPPeer<Writable, Writable, Writable, Writable, GraphJobMessage> peer)
      throws IOException {
    int activeVertices = 0;
    vertices.startSuperstep();
*    /*
     * We iterate over our messages and vertices in sorted order. That means
     * that we need to seek the first vertex that has the same ID as the
     * currentMessage or the first vertex that is active.
     */
*    IDSkippingIterator<V, E, M> iterator = vertices.skippingIterator();
    // note that can't skip inactive vertices because we have to rewrite the
    // complete vertex file in each iteration
    while (iterator.hasNext(
        currentMessage == null ? null : (V) currentMessage.getVertexId(),
        Strategy.ALL)) {

      Vertex<V, E, M> vertex = iterator.next();
      VertexMessageIterable<V, M> iterable = null;
      if (currentMessage != null) {
*        iterable = iterate(currentMessage, (V)
currentMessage.getVertexId(), <<=== EXCEPTION InvalidArgumentException
occurs here.*
            vertex, peer);
      }


I am unable to understand this. I want to know how/where the messages
sorted ? is there a way to influence sorting by coding something at the
high level or at the graph job level ?

My observation is that all the vertices are already created when sync
happens post partition  files creation i.e. all the vertices are created is
my understanding.

My graph data file is really big something like below:
0       1:41    2:18    3:83    4:58    5:97    6:53    7:45    8:51
9:21    10:34
3       540:29  581:72  1009:46 1356:62 1544:15 2044:70 2045:78 2046:44
2047:53 2048:50
7       671:36  1063:17 1159:91 1562:22 1572:67 2004:2  2034:3  2035:91
2036:90 2037:42
9       92:23   95:39   96:62   350:9   530:61  703:3   751:93  822:61
823:98  837:67
10      237:90  430:65  1164:92 1574:61 2038:19 2039:4  2040:49 2041:8
2042:22 2043:93
11      4:4     12:63   13:89   14:25   15:11   16:34   17:0    18:81
19:40   20:1
14      1373:76 2049:67 2050:42 2051:80 2052:11
15      177:84  2355:11 2356:43
19      347:7   589:16  1459:7  1557:48 1571:90 2060:42 2061:92 2062:98
2063:42 2064:14
20      90:89   356:32  531:69  703:68  1436:82 1909:91 1910:96 1911:35
1912:30 1913:89
21      22:82   23:72   24:84   25:1    26:86   27:58   28:88   29:49
30:53   31:96
25      90:58   95:54   534:22  752:82  824:56  1237:50 1643:95 2091:36
2092:49 2093:70
28      90:52   531:67  564:35  707:67  837:24  1042:50 1073:94 1309:5
1365:24 1656:98
31      270:43  1765:56 1975:28 2053:58 2054:7  2055:99 2056:85 2057:75
2058:14 2059:58
....... more lines,, expect it to be in 10k's


Thanks,
Mahesh Babu


On Sun, Aug 11, 2013 at 9:11 AM, Mahesh Babu <jmbabu@gmail.com> wrote:

> Hi Francisco,
>
> Thanks for the detailed reply. I will try this and update back.
>
> In between, is there a way to defer sending messages until all the graph
> is created ?
>
> Regards,
> Mahesh Babu
>
>
>
> On Sun, Aug 11, 2013 at 1:04 AM, Francisco Sanches <
> sanchesbanhos@gmail.com> wrote:
>
>> Hi Mahesh Babu
>>
>> You are trying to send a mesagem for vertex that has not been created yet.
>> Note the graph:
>>
>> # source destination
>> 1                   5
>> 1                   7
>> 2                   1
>>
>> When the vertex 1 to send a message to the 5 it will give this error
>> because the vertex 5 has not yet been created. To fix this would have to
>> change to the graph:
>>
>> # source destination
>> 1                  2
>> 1                  5
>> 1                  7
>> 2                  1
>> 5                  1
>> 7                  1
>>
>> This should fix the problem. At least when I had this problem I solved
>> well. I would recommend you run the example sometimes SSSP hama, and begin
>> their work by her.
>>
>>
>> 2013/8/10 Mahesh Babu <jmbabu@gmail.com>
>>
>> > Hi,
>> >
>> > My graph definition is as follows:
>> >    Vertex<Text, IntWritable, StringIntPairWritable>
>> >    VertexInputReader<LongWritable, Text, Text, IntWritable,
>> > StringIntPairWritable>
>> >           with parseVertex(LongWritable key, Text value, Vertex<Text,
>> > IntWritable, StringIntPairWritable> vertex)
>> >
>> > and hence
>> >     setInputKeyClass(LongWritable.class);
>> >     setInputValueClass(Text.class);
>> >     setOutputKeyClass(Text.class);
>> >     setOutputValueClass(StringIntPairWritable.class);
>> >
>> > I am not sure why I decided InputKey to be LongWritable, just went by
>> one
>> > of the examples and this parameter is unused in parseVertex
>> implementation.
>> >
>> > Is there something wrong with my definitions ?
>> >
>> > StringIntPairWritable is similar to IntIntPairWritable where I wanted
>> the
>> > predecessor to be string type.
>> >
>> > My latest error is:
>> > 13/08/10 23:23:53 ERROR bsp.LocalBSPRunner: Exception during BSP
>> execution!
>> > java.lang.IllegalArgumentException: Messages must never be behind the
>> > vertex in ID! Current Message ID: five vs. fivehundredeighty
>> >     at
>> > org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:284)
>> >     at
>> >
>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:229)
>> >
>> >
>> > Regards,
>> > Mahesh Babu
>> >
>> >
>> >
>> > On Fri, Aug 9, 2013 at 8:35 PM, Mahesh Babu <jmbabu@gmail.com> wrote:
>> >
>> > > Hi,
>> > >
>> > > Can you help me understand:
>> > > 1) How to check input graph type
>> > > 2) How to debug partition jobs
>> > >
>> > > Insights would be helpful.
>> > >
>> > > Regards,
>> > > Mahesh Babu
>> > >
>> > >
>> > > On Fri, Aug 9, 2013 at 5:35 PM, Edward J. Yoon <edwardyoon@apache.org
>> > >wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> Integer ID (1, 2, ..) should be fine. As other people said, I think
>> > >> you need to check the input graph type.
>> > >>
>> > >> On Fri, Aug 9, 2013 at 8:50 PM, Mahesh Babu <jmbabu@gmail.com>
>> wrote:
>> > >> > Hi Edward,
>> > >> >
>> > >> > I replaced node names from 1 to "one", 6288 to
>> > >> > "sixthousandtwohundredeightyeight", etc.
>> > >> > I am not getting the error now. so it may be something to do with
>> > number
>> > >> > itself being the name.
>> > >> >
>> > >> > Will monitor if I hit at this error once again. For now it is
>> working.
>> > >> > Proceeding with some more code changes.
>> > >> >
>> > >> > Thanks,
>> > >> > Mahesh Babu
>> > >> >
>> > >> >
>> > >> > On Fri, Aug 9, 2013 at 2:38 PM, Mahesh Babu <jmbabu@gmail.com>
>> wrote:
>> > >> >
>> > >> >> Hi Edward,
>> > >> >>
>> > >> >> In my case vertices are named with numbers. like 1,2,3...etc
>> itself.
>> > No
>> > >> >> alphabets in the vertex name, it is just 1 to 10000+ nodes
>> identified
>> > >> by
>> > >> >> the number itself.
>> > >> >>
>> > >> >> You think that could be reason ?
>> > >> >>
>> > >> >> Regards,
>> > >> >> Mahesh Babu
>> > >> >>
>> > >> >>
>> > >> >> On Fri, Aug 9, 2013 at 1:58 PM, Edward J. Yoon <
>> > edwardyoon@apache.org
>> > >> >wrote:
>> > >> >>
>> > >> >>> It occurs when the vertices are unsorted.
>> > >> >>>
>> > >> >>> Please check whether your vertex parser reads correctly,
and
>> > >> >>> partitioning job was run before the graph job.
>> > >> >>>
>> > >> >>> On Fri, Aug 9, 2013 at 3:43 PM, Mahesh Babu <jmbabu@gmail.com>
>> > wrote:
>> > >> >>> > Hi,
>> > >> >>> >
>> > >> >>> > adding more details about the environment.
>> > >> >>> >
>> > >> >>> > When I run hama-examples-0.6.2.jar with tutorial
example/input
>> > >> (berlin
>> > >> >>> > example) file it works without any error.
>> > >> >>> >
>> > >> >>> > Now when I try my own input file which has approximately
10k
>> nodes
>> > >>  and
>> > >> >>> 39k
>> > >> >>> > edges but it fails with:
>> > >> >>> >
>> > >> >>> >        java.lang.IllegalArgumentException: Messages
must never
>> be
>> > >> >>> behind the
>> > >> >>> > vertex in ID! Current Message ID: 2 vs. 20
>> > >> >>> >
>> > >> >>> > What am I doing wrong ?
>> > >> >>> >
>> > >> >>> > Any help/explanation is appreciated.
>> > >> >>> >
>> > >> >>> > Regards,
>> > >> >>> > Mahesh Babu
>> > >> >>> >
>> > >> >>> >
>> > >> >>> >
>> > >> >>> >
>> > >> >>> > On Thu, Aug 8, 2013 at 4:25 PM, Mahesh Babu <jmbabu@gmail.com>
>> > >> wrote:
>> > >> >>> >>
>> > >> >>> >> Hi,
>> > >> >>> >>
>> > >> >>> >> I followed the SSSP example, GraphJob example
and getting
>> > following
>> > >> >>> error:
>> > >> >>> >>
>> > >> >>> >> 13/08/08 16:01:57 INFO bsp.BSPJobClient: Current
supersteps
>> > >> number: 0
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> mortbay.log:
>> > >> >>> >> Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
>> via
>> > >> >>> >> org.mortbay.log.Slf4jLog
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> >>> >> sync.ZKSyncClient: Initializing ZK Sync Client
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> >>> >> sync.ZooKeeperSyncClientImpl: Start connecting
to Zookeeper!
>> At
>> > >> >>> >> localhost/127.0.0.1:61001
>> > >> >>> >>
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> ipc.Server:
>> > >> >>> >> Starting SocketReader
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> ipc.Server:
>> > >> >>> IPC
>> > >> >>> >> Server Responder: starting
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> ipc.Server:
>> > >> >>> IPC
>> > >> >>> >> Server listener on 61001: starting
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> >>> >> message.HadoopMessageManagerImpl:  BSPPeer address:localhost
>> > >> port:61001
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> ipc.Server:
>> > >> >>> IPC
>> > >> >>> >> Server handler 0 on 61001: starting
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> >>> >> graph.GraphJobRunner: 4935 vertices are loaded
into
>> > localhost:61001
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 ERROR
>> > >> >>> bsp.BSPTask:
>> > >> >>> >> Error running bsp setup and bsp function.
>> > >> >>> >> attempt_201308081548_0005_000000_0:
>> > >> java.lang.IllegalArgumentException:
>> > >> >>> >> Messages must never be behind the vertex in ID!
Current
>> Message
>> > >> ID: 2
>> > >> >>> vs. 20
>> > >> >>> >> attempt_201308081548_0005_000000_0:     at
>> > >> >>> >>
>> > >> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:284)
>> > >> >>> >> attempt_201308081548_0005_000000_0:     at
>> > >> >>> >>
>> > >> >>>
>> > >>
>> >
>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:229)
>> > >> >>> >> attempt_201308081548_0005_000000_0:     at
>> > >> >>> >>
>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:133)
>> > >> >>> >> attempt_201308081548_0005_000000_0:     at
>> > >> >>> >> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
>> > >> >>> >> attempt_201308081548_0005_000000_0:     at
>> > >> >>> >> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> > >> >>> >> attempt_201308081548_0005_000000_0:     at
>> > >> >>> >>
>> > >> >>>
>> > >>
>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 WARN
>> > >> >>> >> message.MessageTransferQueueFactory: Message
queue is
>> configured
>> > on
>> > >> >>> >> deprecated parameter:hama.messenger.queue.class
>> > >> >>> >> attempt_201308081548_0005_000000_0: 13/08/08
16:00:49 INFO
>> > >> ipc.Server:
>> > >> >>> >> Stopping server on 61001
>> > >> >>> >>
>> > >> >>> >>
>> > >> >>> >> What could be the reason I am getting this error.
>> > >> >>> >>
>> > >> >>> >> I am able to run the code in eclipse/standalong
environment.
>> > >> However,
>> > >> >>> when
>> > >> >>> >> I try to run with hama/hadoop pseudo distribution
mode it
>> fails.
>> > >> >>> >>
>> > >> >>> >> Thanks,
>> > >> >>> >> Mahesh Babu
>> > >> >>> >>
>> > >> >>> >
>> > >> >>>
>> > >> >>>
>> > >> >>>
>> > >> >>> --
>> > >> >>> Best Regards, Edward J. Yoon
>> > >> >>> @eddieyoon
>> > >> >>>
>> > >> >>
>> > >> >>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Best Regards, Edward J. Yoon
>> > >> @eddieyoon
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Francisco Sanches
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message