hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: RE: RE: RE: RE: Do Hama support member member variable?
Date Thu, 28 Jan 2016 11:11:27 GMT
Wow, you find the bug. Thanks ;)

>  at org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)

When the framework assign vertices to the proper machine at initial
phase, vertex objects are transferred in serialized form. At this step
 user defined code won't work correctly. I'll fix soon.

Anyway, you should able to manage an array of TextPair objects like below:

    private TextPair[] test = new TextPair[1];

    public void readState(DataInput in) throws IOException {
      int size = in.readInt();
      test = new TextPair[size];
      for(int i = 0; i < size; i++) {
        test[i].readFields(in);
      }
    }

    public void writeState(DataOutput out) throws IOException {
      out.writeInt(test.length);
      for(int i = 0; i < test.length; i++) {
        test[i].write(out);
      }
    }

Thanks.


On Thu, Jan 28, 2016 at 6:45 PM, 步青云 <mailliuping@qq.com> wrote:
> Thanks for your reply. You do help me a lot.
> I have tried to use two methods. Some problem are still bothering me.
> When I use the first method of using Hadoop built-in writable classes, I get a NullPointerException
meaning the parents is null, even though I have initialize parents. The code is like this.
> static ArrayWritable parents= new ArrayWritable(TextPair.class);
>
>
> public void writeState(DataOutput out) throws IOException {
>     out.writeBoolean(match);
>     parents.write(out);
> }
>
>
> public void readState(DataInput in) throws IOException {
>    match = in.readBoolean();
>    parents.readFields(in);
> }
>
>
>
> And the error message is as follow:
> Exception in thread "pool-6-thread-2" java.lang.RuntimeException: java.lang.NullPointerException
>         at org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:562)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.io.ArrayWritable.write(ArrayWritable.java:103)
>         at ProbMatch$ProbMatchVertex.writeState(ProbMatch.java:154)
>         at org.apache.hama.graph.Vertex.write(Vertex.java:311)
>         at org.apache.hama.util.WritableUtils.unsafeSerialize(WritableUtils.java:55)
>         at org.apache.hama.graph.MapVerticesInfo.put(MapVerticesInfo.java:64)
>         at org.apache.hama.graph.GraphJobRunner.addVertex(GraphJobRunner.java:577)
>         at org.apache.hama.graph.GraphJobRunner.access$300(GraphJobRunner.java:64)
>         at org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)
>
>
>
> When I'm trying use the second method, I don't know how to read and write an object array.
I can use out.writeInt() method to write int. But when I need to write an object. How can
I do this? I'm so sorry that I'm not good at Java. Here is the some code I'm trying to write.
>
>
> static TextPair[] arr = new TextPair[1];
>
>
> public void writeState(DataOutput out) throws IOException {
>     out.writeBoolean(match);
>     out.writeInt(arr.length);
>     for(int i = 0; i < arr.length; i++) {
>       out.writeInt(arr[i].toString());  // Is this right?
>     }}
>
>
> public void readState(DataInput in) throws IOException {
>    match = in.readBoolean();
>    int length = in.readInt();
>    for(int i=0;i<length;i++){
>       arr[i] = in.read    // How can I read the textpair here?
>    }
> }
>
>
>
> I'm very grateful to your help. Thanks again.
> Best Regards, Ping Liu.
>
>
>
>
> ------------------ Original ------------------
> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
> Date:  Thu, Jan 28, 2016 11:51 AM
> To:  "user"<user@hama.apache.org>;
>
> Subject:  RE: RE: RE: RE: Do Hama support member member variable?
>
>
>
> Hi,
>
> You can use Hadoop built-in writable classes or own custom Writable.
>
> static ArrayWritable arr = new ArrayWritable(DoubleWritable.class);
>
>   public void writeState(DataOutput out) throws IOException {
>     arr.write(out);
>   }
>
> Or,
>
> static int[] arr2 = new int[3];
>
>   public void writeState(DataOutput out) throws IOException {
>     out.writeInt(arr2.length);
>     for(int i = 0; i < arr2.length; i++) {
>       out.writeInt(arr2[i]);
>     }
>   }
>
>   public void readState(DataInput in) throws IOException {
>     int size = in.readInt();
>     for(int i = 0; i < size; i++) {
>       arr2[i] = in.readInt();
>     }
>   }
>
> --
> Best Regards, Edward J. Yoon
>
>
> -----Original Message-----
> From: 步?云 [mailto:mailliuping@qq.com]
> Sent: Thursday, January 28, 2016 11:16 AM
> To: user
> Subject: Re: RE: RE: RE: Do Hama support member member variable?
>
> Hi,
>      I still don't know how to use member variable. What's the input parameter
> DataOutput out in writeState() method? Could you please give me a example to
> use this method? Thanks a lot.
>      By the way, how can I deal with object member variable? For example, the
> member variable "private List parents" store the parents of the vertex. How
> can I read and write this variable?
>      If you can take time to reply me, I will be very grateful to you.
>      Best Regard, Ping Liu.
>
>
> ------------------ Original ------------------
> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
> Date:  Wed, Jan 27, 2016 03:12 PM
> To:  "user"<user@hama.apache.org>;
>
> Subject:  RE: RE: RE: Do Hama support member member variable?
>
>
>
> Just FYI, Hama 0.7.1 release candidate is now available. Please feel free to
> use this, and it'd nice if you can let me know whether it works well with you.
>
> Release tarball: http://people.apache.org/~edwardyoon/dist/0.7.1-RC1
>
> --
> Best Regards, Edward J. Yoon
>
>
> -----Original Message-----
> From: 步?云 [mailto:mailliuping@qq.com]
> Sent: Wednesday, January 27, 2016 11:01 AM
> To: user
> Subject: Re: RE: RE: Do Hama support member member variable?
>
> I know. If I use the static variable like "private static boolean match", I
> could get the right value of match too. Could I use static variable?
> Thanks a lot.
> Best regards. Ping Liu.
>
>
>
>
> ------------------ Original ------------------
> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
> Date:  Wed, Jan 27, 2016 09:39 AM
> To:  "user"<user@hama.apache.org>;
>
> Subject:  RE: RE: Do Hama support member member variable?
>
>
>
> Basically Vertex object is writable, and we store the vertex objects in
> serialized form. There are two purposes: 1) to reduce memory usage 2) to write
> on file system (checkpoint and recovery).
>
> So, you should use readState() and writeState() methods to save object member
> variables.
>
> Thanks!
>
> --
> Best Regards, Edward J. Yoon
>
>
> -----Original Message-----
> From: 步?云 [mailto:mailliuping@qq.com]
> Sent: Wednesday, January 27, 2016 10:27 AM
> To: user
> Subject: Re: RE: Do Hama support member member variable?
>
> Hi,
> Thank you very much. You helped me a lot. But I still don't know how to use
> readState() and writeState() methods. What is the input parameter DataInput in
> and DataOutput out?  And as far as I know, I can assign the value of member
> variable directly in hama-0.6.3, such as match = true. Why I can't do this in
> Hama-0.7.0?
> Waiting for your reply. Thanks.
> Best Regards, Ping Liu.
>
>
> ------------------ Original ------------------
> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
> Date:  Wed, Jan 27, 2016 07:15 AM
> To:  "user"<user@hama.apache.org>;
>
> Subject:  RE: Do Hama support member member variable?
>
>
>
> Hi,
>
> You should use readState() and writeState() methods like below:
>
> public static class ProbMatchVertex extends Vertex<Text, NullWritable,
> TriTextPair> {
>   private boolean match = false;
>
>   public void readState(DataInput in) throws IOException {
>     match = in.readBoolean();
>   }
>
>   public void writeState(DataOutput out) throws IOException {
>     out.writeBoolean(match);
>   }
>
>   ..
>
> }
>
> --
> Best Regards, Edward J. Yoon
>
> -----Original Message-----
> From: 꼍행暾 [mailto:mailliuping@qq.com]
> Sent: Tuesday, January 26, 2016 10:16 PM
> To: user
> Subject: Do Hama support member member variable?
>
> Hello,
>       I'm trying to run a graph job. But i have got some problems.
>       I want to use member variable in vertex class, the code is as follow.
> I have changed the value of the member variable in one superstep. But when I
> use this member variable in next superstep, the value of this member
> variable is still the defalult value. For example, "match" is the member
> variable. I have changed the value of "match" to be ture in superstep 0, but
> when I print "match" in superstep 1, the result was "match: false".
>       Could anyone tell me why the value of member variable is changed?
> Thanks very much.
>       Best wishes.
>
>
>       public static class ProbMatchVertex extends Vertex<Text, NullWritable,
> TriTextPair> {
>                 private boolean match = false;
>
>
>                 @Override
>                 public void compute(Iterable<TriTextPair> messages) throws
> IOException {
>                         if (getSuperstepCount() == 0) {
>                                 match = true;
>                                 sendMessageToNeighbors(new
> TriTextPair(getVertexID(),
>                                                 getVertexLabel(), new
> Text("")));
>                         } else if(getSuperstepCount() == 1){
>                                 System.out.println("match:" +  match);
>                                 parents = new ArrayList<TriTextPair>();
>                                 for(TriTextPair msg : messages){
>                                         parents.add(msg);
>                                         sendMessage(msg.getFirst(), new
> TriTextPair(getVertexID(), getVertexLabel(), new Text("")));
>                                 }
>                         }
>                 }
>
>
>
>
> Ping Liu



-- 
Best Regards, Edward J. Yoon

Mime
View raw message