reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Chung (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (REEF-1180) Split REEFClient submission parameter file into per-application and per-job
Date Fri, 05 Feb 2016 00:18:39 GMT

     [ https://issues.apache.org/jira/browse/REEF-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Chung updated REEF-1180:
-------------------------------
    Description: 
Loosely defining the terms used here:
Application: A REEF application.
Job: Submissions of REEF applications to the RM framework.

In other words, there can be many REEF jobs, but there is only one REEF application, from
the RM framework point of view.

Currently, we package both application and job parameters into the same Avro parameter file.
We should break the file down into two parameter files -- one for application and another
for job, such that we can generate a package once and re-use it many times as long as the
user supplies the job parameters and the package (or the location of the package).

In the YARN case, the zip file will be uploaded to YARN as an ARCHIVE resource. The zip file
will contain the Avro application parameter file, while the Avro job parameter file will be
uploaded to YARN as a FILE resource. This will allow the user to generate only the Avro file
for the job, and will be able to reuse the package, provided that their general application
parameters do not change.
On YARN resource localization, the Avro job file will be localized outside of the "reef" folder,
while the zip file will create the "reef" folder.

  was:
Loosely defining the terms used here:
Application: A REEF application.
Job: Submissions of REEF applications to the RM framework.

In other words, there can be many REEF jobs, but there is only one REEF application, from
the RM framework point of view.

Currently, we package both application and job parameters into the same Avro parameter file.
We should break the file down into two parameter files -- one for application and another
for job, such that we can generate a package once and re-use it many times as long as the
user supplies the job parameters and the package (or the location of the package).


> Split REEFClient submission parameter file into per-application and per-job
> ---------------------------------------------------------------------------
>
>                 Key: REEF-1180
>                 URL: https://issues.apache.org/jira/browse/REEF-1180
>             Project: REEF
>          Issue Type: New Feature
>          Components: REEF.NET, REEF.NET Client
>            Reporter: Andrew Chung
>            Assignee: Andrew Chung
>
> Loosely defining the terms used here:
> Application: A REEF application.
> Job: Submissions of REEF applications to the RM framework.
> In other words, there can be many REEF jobs, but there is only one REEF application,
from the RM framework point of view.
> Currently, we package both application and job parameters into the same Avro parameter
file. We should break the file down into two parameter files -- one for application and another
for job, such that we can generate a package once and re-use it many times as long as the
user supplies the job parameters and the package (or the location of the package).
> In the YARN case, the zip file will be uploaded to YARN as an ARCHIVE resource. The zip
file will contain the Avro application parameter file, while the Avro job parameter file will
be uploaded to YARN as a FILE resource. This will allow the user to generate only the Avro
file for the job, and will be able to reuse the package, provided that their general application
parameters do not change.
> On YARN resource localization, the Avro job file will be localized outside of the "reef"
folder, while the zip file will create the "reef" folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message