hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1620) Hadoop should serialize the Configration after the call to getSplits() to the backend such that any changes to the Configuration in getSplits() is serialized to the backend
Date Wed, 24 Mar 2010 17:53:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849311#action_12849311
] 

Tom White commented on MAPREDUCE-1620:
--------------------------------------

Is this related to MAPREDUCE-1486?

> Hadoop should serialize the Configration after the call to getSplits() to the backend
such that any changes to the Configuration in getSplits() is serialized to the backend
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1620
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1620
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Pradeep Kamath
>
> In 0.20.1 and 0.20.2, when using the new API, while working on the next pig release we
discovered that the hadoop code makes a copy of the Configuration and hands a copy to the
getSplits() call. Any changes to the Configuration made in getSplits() are on that copy. However
the original Configuraiton is the one which gets serialized to the backend - hence any changes
made to the Configuration in the getSplits() implementation does not get serialized to the
backend. In a framework like Pig, there are usecases for writing information into the Configuration
during getSplits - it would be helpful if Hadoop would ensure that these changes get serialized
to the backend.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message