hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Omkar Vinit Joshi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1019) YarnConfiguration validation for local disk path and http addresses.
Date Wed, 07 Aug 2013 01:34:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731551#comment-13731551
] 

Omkar Vinit Joshi commented on YARN-1019:
-----------------------------------------

[~josephkniest] to give you more insight into how it is used (General configuration read in
hadoop).

bq. Ok so this module YarnConfiguration, does other portions of the codebase access this for
the config info like directories and what not and I need to find all those places?
Probably not. If you are using eclipse for hadoop development then just do call hierarchy
for the variable under consideration. say, for YarnConfiguration#RM_ADDRESS. You will see
where it is getting used. That way your search is reduced. You should here ignore places where
it is getting used inside test code. You don't need to validate test code. But you will have
to add unit test case later to verify your changes.

bq. How does that information get passed to this object?
Probably you don't need to worry about this. You can trace
{code}
new YarnConfiguration()
{code}
call. It will read from configuration files. for YARN : yarn-site.xml, HDFS :hdfs-site.xml,
CORE : core-site.xml

bq. Ultimately, we want to find where this object gets instantiated and ensure that it doesn't
get relative paths correct?
Yes for all the places where we are getting file paths we need to ensure this. Make sure it
is not OS specific. i.e. it works for WINDOWS/LINUX/MAC.

bq. What exactly do we want with number 2 of this issue? I'm confused about that one
when we expect for example RM_ADDRESS then we expect it to be <host>:<port>. just
validate it.

Finally once you have done the changes you need to create patch and upload it via "More Actions
--> Attach Files" and then Submit Patch.
                
> YarnConfiguration validation for local disk path and http addresses.
> --------------------------------------------------------------------
>
>                 Key: YARN-1019
>                 URL: https://issues.apache.org/jira/browse/YARN-1019
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Omkar Vinit Joshi
>            Priority: Minor
>              Labels: newbie
>
> Today we are not validating certain configuration parameters set in yarn-site.xml. 1)
Configurations related to paths... such as local-dirs, log-dirs.. Our NM crashes during startup
if they are set to relative paths rather than absolute paths. To avoid such failures we can
enforce checks (absolute paths) before startup . i.e. before we actually startup...( i.e.
directory handler creating directories).
> 2) Also for all the parameters using "hostname:port" unless we are ok with default port.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message