hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2866) JobConf should validate key names in well-defined namespaces and warn on misspelling
Date Thu, 21 Feb 2008 07:28:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570967#action_12570967
] 

Joydeep Sen Sarma commented on HADOOP-2866:
-------------------------------------------

+1. But ..:

the mechanism proposed here - by itself - will not prevent regressions. we can create a 'registry'
of legal options - but that will not prevent a software change that adds a new option, moves
semantics of old option to the new option - and then merrily leaves the old option behind
in the registry.

(exactly what happened here. the tree has good looking code with references to deprecated
options).

we would need isolated regression tests against each registered option (tall order indeed)
for this to suffice.

besides - even the Java interfaces were obsoleted or their semantics changed without removing
old interfaces. and that's not a 'lack of mechanism' issue - it's a pure software engineering
issue.

> JobConf should validate key names in well-defined namespaces and warn on misspelling
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2866
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2866
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.16.0
>            Reporter: Aaron Kimball
>            Priority: Minor
>             Fix For: 0.16.1, 0.17.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A discussion on the mailing list reveals that some configuration strings in the JobConf
are deprecated over time and new configuration names replace them:
> e.g., "mapred.output.compression.type" is now replaced with "mapred.map.output.compression.type"
> Programmers who have been manually specifying the former string, however, receive no
diagnostic output during testing to suggest that their compression type is being silently
ignored.
> It would be desirable to notify developers of this change by printing a warning message
when deprecated configuration names are used in a newer version of Hadoop. More generally,
when any configuration string in the mapred.\*, fs.\*, dfs.\*, etc namespaces are provided
by a user and are not recognized by Hadoop, it is desirable to print a warning, to indicate
malformed configurations. No warnings should be printed when configuration keys are in user-defined
namespaces (e.g., "myprogram.mytask.myvalue").

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message