hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ying He (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-961) Integration with Hadoop 21
Date Tue, 15 Sep 2009 22:31:57 GMT

    [ https://issues.apache.org/jira/browse/PIG-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755748#action_12755748
] 

Ying He commented on PIG-961:
-----------------------------

there are a few problems while porting pig to hadoop 21 new API.

1.  When running a Task (map or reduce), Two context are created, a JobContext and a TaskAttempContext.
Each context has its own copy of job config. So a property set from one context can't be accessed
from another context.  I have to make the following change to JobContext class to make pig
work:

the original code makes a copy of input config:
public JobContext(Configuration conf, JobID jobId) {
       this.conf = new org.apache.hadoop.mapred.JobConf(conf);	
       this.jobId = jobId;
  }

I changed it to share the config object
public JobContext(Configuration conf, JobID jobId) {
	  if (conf instanceof org.apache.hadoop.mapred.JobConf) {
		  this.conf = (org.apache.hadoop.mapred.JobConf)conf;
	  }else{
		  this.conf = new org.apache.hadoop.mapred.JobConf(conf);
	  }
    this.jobId = jobId;
  }

2. The "Reporter" object is not visible from JobContext. There is no access to reporter.incrCounter()

3. JobConf is obsolete. There is not access to some convenient methods such as getUser(),
 Instead, I have to use config.get("user.name")

> Integration with Hadoop 21
> --------------------------
>
>                 Key: PIG-961
>                 URL: https://issues.apache.org/jira/browse/PIG-961
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: Olga Natkovich
>            Assignee: Ying He
>         Attachments: hadoop21.jar, PIG-961.patch
>
>
> Hadoop 21 is not yet released but we know that switch to new MR API is coming there.
This JIRA is for early integration with this API

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message