reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobin Baker <tdba...@cs.washington.edu>
Subject question on proper ConfigurationModule for configuring YARN runtime
Date Wed, 16 Dec 2015 04:50:46 GMT
Hi, I've found the following ConfigurationModule implementations for
configuring the YARN runtime:

org.apache.reef.runtime.yarn.client.YarnClientConfiguration
org.apache.reef.runtime.yarn.client.YarnDriverConfiguration
org.apache.reef.runtime.yarn.driver.YarnDriverConfiguration

Besides the duplicated class name between the last two, there seems to be
some duplication in functionality as well: both
org.apache.reef.runtime.yarn.client.YarnClientConfiguration.YARN_QUEUE_NAME
and org.apache.reef.runtime.yarn.client.YarnDriverConfiguration.QUEUE
presumably are equivalent, right? (Since the latter class contains the
javadoc "Additional YARN-Specific configuration options to be merged with
DriverConfiguration.", I assume it's meant to be passed to REEF.submit()
rather than to TANG.newInjector().) I guess this duplication would allow
the queue name to be configured at either the runtime or Driver level, but
was that really the intention? Also, the Mesos runtime has no corresponding
bifurcation of MesosDriverConfiguration into different namespaces, so was
this a deliberate design decision for YARN or just a historical accident?

Context: I'm preparing to submit a patch for REEF-1034, which is a trivial
fix, but it's not clear to me whether I should add the new parameter
JOB_SUBMISSION_DIRECTORY_PREFIX to
org.apache.reef.runtime.yarn.client.YarnDriverConfiguration or
org.apache.reef.runtime.yarn.driver.YarnDriverConfiguration. The fact that
org.apache.reef.runtime.yarn.driver.YarnDriverConfiguration.JOB_SUBMISSION_DIRECTORY
is computed from the value of
org.apache.reef.runtime.yarn.driver.parameters.JobSubmissionDirectoryPrefix
(injected into the constructor of
org.apache.reef.runtime.yarn.driver.JobSubmissionDirectoryProviderImpl)
suggests that JOB_SUBMISSION_DIRECTORY_PREFIX shouldn't be defined in the
same class, but the fact that the Mesos runtime defines
JOB_SUBMISSION_DIRECTORY_PREFIX under the namespace
org.apache.reef.runtime.mesos.driver.MesosDriverConfiguration suggests that
it should.

If it helps, here's what I could find from git log:

git log
/Users/tdbaker/reef/lang/java/reef-runtime-yarn/src/main/java/org/apache/reef/runtime/yarn/client/YarnDriverConfiguration.java

commit ac1555b19c715f3ebb3cb6c59561b2df1ff22a4a
Author: Markus Weimer <weimer@apache.org>
Date:   Mon Jul 6 17:44:59 2015 -0700

    [REEF-454] Allow for per-job queues

    This adds `YarnDriverConfiguration` that can be filled out by an
application and
    merged into their DriverConfiguration to set the queue to submit the
job to.

    This also makes sure the `YarnClientConfiguration.YARN_QUEUE_NAME` is
actually
    used (it wasn't). It now provides the default for when the new per-job
setting
    isn't used.

    Lastly, this change deprecates `JobSubmissionEvent.getQueue()` which
was never
    used in the first place. Its purpose is now served by the mechanism
introduced
    here.

    JIRA:
      [REEF-454](https://issues.apache.org/jira/browse/REEF-454)

    Pull Request:
      This closes #279

    Author:    Markus Weimer <weimer@apache.org>

git log
/Users/tdbaker/reef/lang/java/reef-runtime-yarn/src/main/java/org/apache/reef/runtime/yarn/driver/YarnDriverConfiguration.java

commit 2515c5377ee97e3dfcde54ef49cf3446ea209881
Author: Dongjoon Hyun <dongjoon@apache.org>
Date:   Mon Nov 2 02:04:10 2015 +0900

    [REEF-899] Make JobSubmissionDirectory in REEF-Runtime-YARN

    To move JobSubmissionDirectory from 'org.apache.reef.driver.parameters'
    into 'org.apache.reef.runtime.yarn.driver.parameters' package, this PR
      * Rename the package of existing one in reef-runtime-yarn
      * Make a new *deprecated* o.a.r.d.p.JobSubmissionDirectory in
reef-common
      * Add 'TODO[JIRA REEF-904]'s

    JIRA:
      [REEF-899](https://issues.apache.org/jira/browse/REEF-899)

    Pull request:
      This closes #606

commit 46d9bc058d8ad37ff0958f7f32bf45d333abc995
Author: Markus Weimer <github@weimo.de>
Date:   Mon Jul 7 20:26:10 2014 -0700

    Added a Heap-Slack parameter

    This adds the option for users to define a fraction of the container
memory that is not used for the JVM heap.

    Also, some named parameters were moved out of being inner classes.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message