reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <>
Subject [jira] [Created] (REEF-1383) Improve MultiRuntimeConfigurationBuilder API
Date Sat, 07 May 2016 00:30:13 GMT
Markus Weimer created REEF-1383:

             Summary: Improve MultiRuntimeConfigurationBuilder API
                 Key: REEF-1383
             Project: REEF
          Issue Type: Sub-task
            Reporter: Markus Weimer

I had another look at {{MultiRuntimeConfigurationBuilder}}. It now seems very counter-intuitive
to me. Look at the code in {{HelloREEFMultiYarn.getHybridYarnSubmissionRuntimeConfiguration}}:

return new MultiRuntimeConfigurationBuilder()

At no point does the user actually configure the runtimes in question. For instance, what
if they want to submit to a YARN instance which needs a correct user name to work? There doesn't
seem to be a way to specify that. What if I want to support Mesos next? This tight coupling
of the multi-runtime with the (small) set of supported participants creates an explosion in
the complexity of all the code associated with this change. What prevents us from creating
an API that makes the client code look more like this:

return new MultiRuntimeConfigurationBuilder()
  .addRuntime(LOCAL, LocalDriverConfiguration.CONF.set(NUMBER_OF_EVALUATORS, 1).build())

This would allow an arbitrary set of runtimes to take part in the multi-runtime. Also, it
would remove all special-casing in the multi-runtime for specific (combinations of) runtimes.
This should simplify the code in the multi-runtime tremendously.

On the flip side, runtimes now need to implement those APIs, namely {{ConfigurationBuilder}}s
for the Driver side and the Client side separately. That should be doable and has been introduced
via the {{Extensible*}} builders done as part of this work.


This message was sent by Atlassian JIRA

View raw message