giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman" <initialcont...@gmail.com>
Subject Re: Review Request: GIRAPH-372: Write worker addresses to Zookeeper; move addresses and resolution to NettyClient
Date Wed, 17 Oct 2012 15:52:53 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7610/#review12513
-----------------------------------------------------------


This is great work Maja. Now that everyone's addresses are in the same place (master and clients),
perhaps objects like InetSocketAddress or especially WorkerInfo could do with a name change,
or possibly even reconfiguration into a simpler form? Also, just for my understanding: all
host addresses are now on the same ZK children list, and are read by NettyClient's service
object from this list, each superstep?

Is there ever a time when a given worker task would not have both a PartitionId -> PartitionOwners
map and also now a taskId -> WorkerInfo map? Perhaps these two related things could be
wrapped in an object that further cleans up interactions from calling code (various master
and worker code like Netty* classes)
I feel like at this point a lot of the tangles in the addressing/messaging setup code is cleaned
up in this patch. anything else you can do while you're in there to modularize these interactions
would be a big added bonus.

Anyway, this is great. I love when the right side of the diff has lots of empty space! Looks
like you shaved lots of lines off those classes!


- Eli Reisman


On Oct. 16, 2012, 12:24 a.m., Maja Kabiljo wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7610/
> -----------------------------------------------------------
> 
> (Updated Oct. 16, 2012, 12:24 a.m.)
> 
> 
> Review request for giraph.
> 
> 
> Description
> -------
> 
> In preparation for GIRAPH-273, we need to have addresses of all workers available, so
I write them to Zookeeper along with the master address.
> 
> Since address resoultion is needed in both NettyWorkerClient and NettyMasterClient, I
moved that to NettyClient, and also added a map taskId->address in there. Now NettyMasterClient
and NettyWorkerClient don't need to take care of InetSocketAddresses, just task ids which
they want to send messages to.
> 
> 
> This addresses bug GIRAPH-372.
>     https://issues.apache.org/jira/browse/GIRAPH-372
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/MasterClient.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/WorkerClient.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyClient.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClientServer.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/WorkerInfo.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/WritableUtils.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java
1398379 
>   http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
1398379 
> 
> Diff: https://reviews.apache.org/r/7610/diff/
> 
> 
> Testing
> -------
> 
> Passes mvn verify
> 
> 
> Thanks,
> 
> Maja Kabiljo
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message