giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Resource Allocation Model Of Apache Giraph
Date Thu, 23 Oct 2014 21:26:03 GMT
Regarding your second point, partitions are decoupled from workers.  A 
worker can handle zero or more partitions.  You can make each worker 
handle one partition, but we typically like multiple partitions since we 
can use multi-threading per machine.

On 10/23/14, 9:04 AM, Claudio Martella wrote:
> the way mappers (or containers) and hence workers are assigned to 
> machines is not under the control of giraph, but of the underlying 
> hadoop environment (with different responsibilities that depend on the 
> hadoop version, e.g. YARN). You'll have to tweak your hadoop 
> configuration to control the maximum number of workers assigned to one 
> machine (optimally one with multiple threads).
>
> On Thu, Oct 23, 2014 at 5:53 PM, Charith Wickramarachchi 
> <charith.dhanushka@gmail.com <mailto:charith.dhanushka@gmail.com>> wrote:
>
>     Hi Folks,
>
>     I'm wondering what is the resource allocation model for Apache
>     Giraph.
>
>     As I understand each worker is one to one Mapped with a Mapper and
>     a worker can process multiple partitions with a user defined
>     number of threads.
>
>     Is it possible to make sure that one worker, only process a single
>     partition? Also is it possible to control the worker assignment in
>     the cluster nodes? (Ex: Make sure only N  workers runs on a single
>     machine, assuming we have enough resources)
>
>     Thanks,
>     Charith
>
>
>
>
>
>
>
>
>     -- 
>     Charith Dhanushka Wickramaarachchi
>
>     Tel +1 213 447 4253
>     Web http://apache.org/~charith <http://www-scf.usc.edu/%7Ecwickram/>
>     Blog http://charith.wickramaarachchi.org/
>     <http://charithwiki.blogspot.com/>
>     Twitter @charithwiki <https://twitter.com/charithwiki>
>
>     This communication may contain privileged or other confidential
>     information and is intended exclusively for the addressee/s. If
>     you are not the intended recipient/s, or believe that you may have
>     received this communication in error, please reply to the sender
>     indicating that fact and delete the copy you received and in
>     addition, you should not print, copy, retransmit, disseminate, or
>     otherwise use the information contained in this communication.
>     Internet communications cannot be guaranteed to be timely, secure,
>     error or virus-free. The sender does not accept liability for any
>     errors or omissions
>
>
>
>
> -- 
>    Claudio Martella


Mime
View raw message