reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobin Baker <tdba...@cs.washington.edu>
Subject Re: question on proper ConfigurationModule for configuring YARN runtime
Date Thu, 17 Dec 2015 04:20:39 GMT
Thanks for the explanation! I think I understand now that d
river.YarnDriverConfiguration is only intended for internal use, while both
YarnClientConfiguration and client.YarnDriverConfiguration are intended for
use by the application, but I'm not entirely clear when an application
would e.g.  specify YarnClientConfiguration.YARN_QUEUE_NAME vs. c
lient.YarnDriverConfiguration.QUEUE. What am I missing? Also, I can't see
anywhere that client.YarnDriverConfiguration.CONF is used, so I'm a bit
confused. Is the client expected to manually merge the configuration built
from client.YarnDriverConfiguration.CONF into the Driver configuration that
they pass to REEF.submit(), so client.YarnDriverConfiguration is just a
convenience class that isn't directly consumed by any REEF code (although
the NamedParameter classes it references are of course used internally)? If
this is the right interpretation, then it seems like I should be putting
the
JOB_SUBMISSION_DIRECTORY_PREFIX parameter into c
lient.YarnDriverConfiguration. Then (after the client manually merges c
lient.YarnDriverConfiguration.CONF.build() into its Driver configuration),
JobSubmissionDirectoryProviderImpl will have the value of c
lient.YarnDriverConfiguration.JOB_SUBMISSION_DIRECTORY_PREFIX (bound to
   JobSubmissionDirectoryPrefix) properly injected into its constructor.
Does that sound like the right approach?

Thanks,
Tobin


On Wed, Dec 16, 2015 at 7:10 AM, Markus Weimer <markus@weimo.de> wrote:

> On 2015-12-15 20:50, Tobin Baker wrote:
>
>> Hi, I've found the following ConfigurationModule implementations for
>> configuring the YARN runtime:
>>
>> org.apache.reef.runtime.yarn.client.YarnClientConfiguration
>> org.apache.reef.runtime.yarn.client.YarnDriverConfiguration
>> org.apache.reef.runtime.yarn.driver.YarnDriverConfiguration
>>
>
> Sorry about that! I came up with these lovely confusing names. Let me
> unroll by first stepping back:
>
> REEF abstracts the resource manager. That abstraction holds true in
> basically two places:
>
>
>   * The client: the default implementation of the `REEF` interface depends
> on a bunch of runtime specific implementations to actually submit and
> monitor a job. Those need to be bound before an instance of `REEF` can be
> obtained.
>
>   * The driver: in order to instantiate a Driver, we need to bind the
> runtime specific implementations as well. Typically, this is done behind
> the scenes by the implementations of the client side runtime adapters.
>
>
> Now, the role of the classes you mentioned in this scheme is as follows:
>
> org.apache.reef.runtime.yarn.client.YarnClientConfiguration: This binds
> all the needed implementations such that an instance of `REEF` can be
> obtained and can successfully submit jobs. Most importantly, it binds
>
>     org.apache.reef.runtime.common.client.api.JobSubmissionHandler
> to
>     org.apache.reef.runtime.yarn.client.YarnJobSubmissionHandler.
>
> YarnJobSubmissionHandler then uses the third class in your list,
>     org.apache.reef.runtime.yarn.driver.YarnDriverConfiguration,
> to bind the driver-side of the runtime adapter.
>
> Now, about org.apache.reef.runtime.yarn.client.YarnDriverConfiguration:
> This class is to be used by the app on the client side to specify options
> for the Driver-side runtime adapter. In this case, the queue.
>
> It is probably a good idea to document all of this better, and make less
> of it `public`...
>
> Hope this helps,
>
> Markus
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message