reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Yang (JIRA)" <>
Subject [jira] [Commented] (REEF-6) Improve allocation performance
Date Wed, 29 Oct 2014 06:26:35 GMT


John Yang commented on REEF-6:

Hi Yingda,

unfortunately I could not reproduce it, even using * as the rack-preference. I have several

1) Could you send me the driver log?
2) Could you instruct me how you run .NET REEF to test the code?
3) Is there a particular reason for using * as the rack-preference for all requests? When
a rack-preference is specified, AMRMClient sends to RM not only the resourceName=rack requests
but also the equivalent requests with resourceName=ResourceRequest.ANY(which is *) together.
Therefore when the rack(resourceName) is *, we end up asking for 2 times the number of containers
we intended to ask. YarnContainerManager does only take the right amount by looking at the
outstanding request queue. Another problem is that when we add reef-runtime-mesos, Mesos doesn't
understand *. It may be better to not specify the rack-preference at all until we support
locality-specific requests for YARN.


> Improve allocation performance
> ------------------------------
>                 Key: REEF-6
>                 URL:
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF-Runtime-YARN
>            Reporter: Markus Weimer
>            Assignee: Yingda Chen
>            Priority: Critical
> REEF today uses a very conservative protocol when interacting with YARN, which only allows
for one allocation request to be in flight at a time. 
> We can improve upon this by allowing one kind of allocation request to be in flight at
a time without giving up the safety of the current approach.

This message was sent by Atlassian JIRA

View raw message