hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5708) Configuration should provide a way to write only properties that have been set
Date Mon, 20 Apr 2009 21:36:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700952#action_12700952
] 

Doug Cutting commented on HADOOP-5708:
--------------------------------------

> The configuration would contain all of the defaults picked up from my non-grid machine,
and would completely overwrite all the defaults on that grid.

The defaults come from the jar file, and the jar must currently have the same version of the
code in the cluster, so in practice we overwrite things with the same values.

Long-term, if we permit users with one version of Hadoop installed to submit jobs to a cluster
running a different version, then sending the defaults could perhaps cause problems.  But
anything that should not be overridden should be declared final in the cluster's configuration,
and otherwise the user's configuration, including defaults, should be observed, no?


> Configuration should provide a way to write only properties that have been set
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-5708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5708
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.19.1
>            Reporter: Topher ZiCornell
>            Priority: Minor
>
> The Configuration.write and .writeXml methods always output all properties, whether they
came from a default source, a loaded resource file, or an "overlay" set call.  There should
be a way to write only the properties that were set, leaving out the properties that came
from a default source.
> Why?  Suppose I build a configuration on a machine that is not associated with a grid,
write it out to XML, then try to load it on a grid gateway.  The configuration would contain
all of the defaults picked up from my non-grid machine, and would completely overwrite all
the defaults on that grid.
> I propose to add methods to write out only the overlay values in Object and XML formats.
> I see two options for implementing this:
> 1) Either completely new methods could be crafted (writeOverlay(DataOutput) and writeOverlayXml(OutputStream),
or 
> 2) The existing write() and writeXml() methods could be adjusted to take an additional
parameter indicating whether the full properties or overlay properties should be written.
 (Of course, the existing write() and writeXml() methods would remain, defaulting to the current
behavior.)
> Option 1 has less impact to existing code.  Option 2 is a cleaner implementation with
less code-duplication involved.  I would much prefer to do option 2.
> Oh, and in case it's not clear, I'm offering to make this change and submit it.
> Thoughts?
> .  Topher

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message