tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitesh Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-1233) Allow configuration of framework parameters per vertex
Date Tue, 03 Feb 2015 19:32:35 GMT

    [ https://issues.apache.org/jira/browse/TEZ-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303820#comment-14303820
] 

Hitesh Shah commented on TEZ-1233:
----------------------------------

As part of the TezClient re-factor some of this support was dropped. There is no api support
currently to modify parameters per DAG. There are a couple of reasons for this: 
   - the main one being distinguishing what parameters are AM/session specific and which ones
are only DAG specific. What configs can be overridden per DAG is also something that needs
to be looked at and restricted accordingly. 
   - also how this ties into container re-use across DAGs also needs to be looked at. 
   
My gut feeling is that we probably need to look at first support DAG-level configs and verifying
that they work in a clean manner before looking at introducing per vertex configurations as
vertex configs will overlay on top of DAG-specific configs. Also, how are vertex specific
configs meant to be passed to Tez from the user? Does an API make sense or will it be a backdoor
channel via a config ( which will need dag specific config support in any case )? 


> Allow configuration of framework parameters per vertex
> ------------------------------------------------------
>
>                 Key: TEZ-1233
>                 URL: https://issues.apache.org/jira/browse/TEZ-1233
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>
> Currently, configuration properties specific in AMConfiguration are used to configure
tasks - e.g. memory configs, process tree, etc. These should be configurable at a Vertex level.
> Also, tasks end up reading these configs from the dist cache each time - would be simpler
to send over the wire as part of the TaskSpec.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message