hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ying He (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-961) Integration with Hadoop 21
Date Tue, 15 Sep 2009 22:31:57 GMT

    [ https://issues.apache.org/jira/browse/PIG-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755748#action_12755748

Ying He commented on PIG-961:

there are a few problems while porting pig to hadoop 21 new API.

1.  When running a Task (map or reduce), Two context are created, a JobContext and a TaskAttempContext.
Each context has its own copy of job config. So a property set from one context can't be accessed
from another context.  I have to make the following change to JobContext class to make pig

the original code makes a copy of input config:
public JobContext(Configuration conf, JobID jobId) {
       this.conf = new org.apache.hadoop.mapred.JobConf(conf);	
       this.jobId = jobId;

I changed it to share the config object
public JobContext(Configuration conf, JobID jobId) {
	  if (conf instanceof org.apache.hadoop.mapred.JobConf) {
		  this.conf = (org.apache.hadoop.mapred.JobConf)conf;
		  this.conf = new org.apache.hadoop.mapred.JobConf(conf);
    this.jobId = jobId;

2. The "Reporter" object is not visible from JobContext. There is no access to reporter.incrCounter()

3. JobConf is obsolete. There is not access to some convenient methods such as getUser(),
 Instead, I have to use config.get("user.name")

> Integration with Hadoop 21
> --------------------------
>                 Key: PIG-961
>                 URL: https://issues.apache.org/jira/browse/PIG-961
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: Olga Natkovich
>            Assignee: Ying He
>         Attachments: hadoop21.jar, PIG-961.patch
> Hadoop 21 is not yet released but we know that switch to new MR API is coming there.
This JIRA is for early integration with this API

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message