hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: RE: RE: RE: RE: Do Hama support member member variable?
Date Thu, 28 Jan 2016 11:33:09 GMT
I filed on https://issues.apache.org/jira/browse/HAMA-982

Once it's fixed, I'll let you know.

On Thu, Jan 28, 2016 at 8:11 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Wow, you find the bug. Thanks ;)
>
>>  at org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)
>
> When the framework assign vertices to the proper machine at initial
> phase, vertex objects are transferred in serialized form. At this step
>  user defined code won't work correctly. I'll fix soon.
>
> Anyway, you should able to manage an array of TextPair objects like below:
>
>     private TextPair[] test = new TextPair[1];
>
>     public void readState(DataInput in) throws IOException {
>       int size = in.readInt();
>       test = new TextPair[size];
>       for(int i = 0; i < size; i++) {
>         test[i].readFields(in);
>       }
>     }
>
>     public void writeState(DataOutput out) throws IOException {
>       out.writeInt(test.length);
>       for(int i = 0; i < test.length; i++) {
>         test[i].write(out);
>       }
>     }
>
> Thanks.
>
>
> On Thu, Jan 28, 2016 at 6:45 PM, 步青云 <mailliuping@qq.com> wrote:
>> Thanks for your reply. You do help me a lot.
>> I have tried to use two methods. Some problem are still bothering me.
>> When I use the first method of using Hadoop built-in writable classes, I get a NullPointerException
meaning the parents is null, even though I have initialize parents. The code is like this.
>> static ArrayWritable parents= new ArrayWritable(TextPair.class);
>>
>>
>> public void writeState(DataOutput out) throws IOException {
>>     out.writeBoolean(match);
>>     parents.write(out);
>> }
>>
>>
>> public void readState(DataInput in) throws IOException {
>>    match = in.readBoolean();
>>    parents.readFields(in);
>> }
>>
>>
>>
>> And the error message is as follow:
>> Exception in thread "pool-6-thread-2" java.lang.RuntimeException: java.lang.NullPointerException
>>         at org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:562)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.NullPointerException
>>         at org.apache.hadoop.io.ArrayWritable.write(ArrayWritable.java:103)
>>         at ProbMatch$ProbMatchVertex.writeState(ProbMatch.java:154)
>>         at org.apache.hama.graph.Vertex.write(Vertex.java:311)
>>         at org.apache.hama.util.WritableUtils.unsafeSerialize(WritableUtils.java:55)
>>         at org.apache.hama.graph.MapVerticesInfo.put(MapVerticesInfo.java:64)
>>         at org.apache.hama.graph.GraphJobRunner.addVertex(GraphJobRunner.java:577)
>>         at org.apache.hama.graph.GraphJobRunner.access$300(GraphJobRunner.java:64)
>>         at org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)
>>
>>
>>
>> When I'm trying use the second method, I don't know how to read and write an object
array. I can use out.writeInt() method to write int. But when I need to write an object. How
can I do this? I'm so sorry that I'm not good at Java. Here is the some code I'm trying to
write.
>>
>>
>> static TextPair[] arr = new TextPair[1];
>>
>>
>> public void writeState(DataOutput out) throws IOException {
>>     out.writeBoolean(match);
>>     out.writeInt(arr.length);
>>     for(int i = 0; i < arr.length; i++) {
>>       out.writeInt(arr[i].toString());  // Is this right?
>>     }}
>>
>>
>> public void readState(DataInput in) throws IOException {
>>    match = in.readBoolean();
>>    int length = in.readInt();
>>    for(int i=0;i<length;i++){
>>       arr[i] = in.read    // How can I read the textpair here?
>>    }
>> }
>>
>>
>>
>> I'm very grateful to your help. Thanks again.
>> Best Regards, Ping Liu.
>>
>>
>>
>>
>> ------------------ Original ------------------
>> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
>> Date:  Thu, Jan 28, 2016 11:51 AM
>> To:  "user"<user@hama.apache.org>;
>>
>> Subject:  RE: RE: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Hi,
>>
>> You can use Hadoop built-in writable classes or own custom Writable.
>>
>> static ArrayWritable arr = new ArrayWritable(DoubleWritable.class);
>>
>>   public void writeState(DataOutput out) throws IOException {
>>     arr.write(out);
>>   }
>>
>> Or,
>>
>> static int[] arr2 = new int[3];
>>
>>   public void writeState(DataOutput out) throws IOException {
>>     out.writeInt(arr2.length);
>>     for(int i = 0; i < arr2.length; i++) {
>>       out.writeInt(arr2[i]);
>>     }
>>   }
>>
>>   public void readState(DataInput in) throws IOException {
>>     int size = in.readInt();
>>     for(int i = 0; i < size; i++) {
>>       arr2[i] = in.readInt();
>>     }
>>   }
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Thursday, January 28, 2016 11:16 AM
>> To: user
>> Subject: Re: RE: RE: RE: Do Hama support member member variable?
>>
>> Hi,
>>      I still don't know how to use member variable. What's the input parameter
>> DataOutput out in writeState() method? Could you please give me a example to
>> use this method? Thanks a lot.
>>      By the way, how can I deal with object member variable? For example, the
>> member variable "private List parents" store the parents of the vertex. How
>> can I read and write this variable?
>>      If you can take time to reply me, I will be very grateful to you.
>>      Best Regard, Ping Liu.
>>
>>
>> ------------------ Original ------------------
>> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
>> Date:  Wed, Jan 27, 2016 03:12 PM
>> To:  "user"<user@hama.apache.org>;
>>
>> Subject:  RE: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Just FYI, Hama 0.7.1 release candidate is now available. Please feel free to
>> use this, and it'd nice if you can let me know whether it works well with you.
>>
>> Release tarball: http://people.apache.org/~edwardyoon/dist/0.7.1-RC1
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Wednesday, January 27, 2016 11:01 AM
>> To: user
>> Subject: Re: RE: RE: Do Hama support member member variable?
>>
>> I know. If I use the static variable like "private static boolean match", I
>> could get the right value of match too. Could I use static variable?
>> Thanks a lot.
>> Best regards. Ping Liu.
>>
>>
>>
>>
>> ------------------ Original ------------------
>> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
>> Date:  Wed, Jan 27, 2016 09:39 AM
>> To:  "user"<user@hama.apache.org>;
>>
>> Subject:  RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Basically Vertex object is writable, and we store the vertex objects in
>> serialized form. There are two purposes: 1) to reduce memory usage 2) to write
>> on file system (checkpoint and recovery).
>>
>> So, you should use readState() and writeState() methods to save object member
>> variables.
>>
>> Thanks!
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Wednesday, January 27, 2016 10:27 AM
>> To: user
>> Subject: Re: RE: Do Hama support member member variable?
>>
>> Hi,
>> Thank you very much. You helped me a lot. But I still don't know how to use
>> readState() and writeState() methods. What is the input parameter DataInput in
>> and DataOutput out?  And as far as I know, I can assign the value of member
>> variable directly in hama-0.6.3, such as match = true. Why I can't do this in
>> Hama-0.7.0?
>> Waiting for your reply. Thanks.
>> Best Regards, Ping Liu.
>>
>>
>> ------------------ Original ------------------
>> From:  "Edward J. Yoon";<edward.yoon@samsung.com>;
>> Date:  Wed, Jan 27, 2016 07:15 AM
>> To:  "user"<user@hama.apache.org>;
>>
>> Subject:  RE: Do Hama support member member variable?
>>
>>
>>
>> Hi,
>>
>> You should use readState() and writeState() methods like below:
>>
>> public static class ProbMatchVertex extends Vertex<Text, NullWritable,
>> TriTextPair> {
>>   private boolean match = false;
>>
>>   public void readState(DataInput in) throws IOException {
>>     match = in.readBoolean();
>>   }
>>
>>   public void writeState(DataOutput out) throws IOException {
>>     out.writeBoolean(match);
>>   }
>>
>>   ..
>>
>> }
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>> -----Original Message-----
>> From: 꼍행暾 [mailto:mailliuping@qq.com]
>> Sent: Tuesday, January 26, 2016 10:16 PM
>> To: user
>> Subject: Do Hama support member member variable?
>>
>> Hello,
>>       I'm trying to run a graph job. But i have got some problems.
>>       I want to use member variable in vertex class, the code is as follow.
>> I have changed the value of the member variable in one superstep. But when I
>> use this member variable in next superstep, the value of this member
>> variable is still the defalult value. For example, "match" is the member
>> variable. I have changed the value of "match" to be ture in superstep 0, but
>> when I print "match" in superstep 1, the result was "match: false".
>>       Could anyone tell me why the value of member variable is changed?
>> Thanks very much.
>>       Best wishes.
>>
>>
>>       public static class ProbMatchVertex extends Vertex<Text, NullWritable,
>> TriTextPair> {
>>                 private boolean match = false;
>>
>>
>>                 @Override
>>                 public void compute(Iterable<TriTextPair> messages) throws
>> IOException {
>>                         if (getSuperstepCount() == 0) {
>>                                 match = true;
>>                                 sendMessageToNeighbors(new
>> TriTextPair(getVertexID(),
>>                                                 getVertexLabel(), new
>> Text("")));
>>                         } else if(getSuperstepCount() == 1){
>>                                 System.out.println("match:" +  match);
>>                                 parents = new ArrayList<TriTextPair>();
>>                                 for(TriTextPair msg : messages){
>>                                         parents.add(msg);
>>                                         sendMessage(msg.getFirst(), new
>> TriTextPair(getVertexID(), getVertexLabel(), new Text("")));
>>                                 }
>>                         }
>>                 }
>>
>>
>>
>>
>> Ping Liu
>
>
>
> --
> Best Regards, Edward J. Yoon



-- 
Best Regards, Edward J. Yoon

Mime
View raw message