pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Santhosh Srinivasan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
Date Fri, 24 Jul 2009 19:11:14 GMT

    [ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735129#action_12735129
] 

Santhosh Srinivasan commented on PIG-889:
-----------------------------------------

The issue here is the lack of support for counters within Pig.

The intention of warn method in the PigLogger interface was to allow sources within Pig and
UDFs  for warning aggregation. Your use of the reporter within the logger is not supported.
An implementation detail prevents the correct use of this interface for load functions. The
Hadoop reporter object is provided in the getRecordReader, map and reduce calls. For load
functions, Pig provides an interface and for UDFs, an abstract class. As a result, the logger
instance cannot be initialized in the loaders till we decide to add a method to support it.


Will having the code from PigMapBase.map()  in PigInputFormat.java.getRecordReader work for
you? 

{code}
            PigHadoopLogger pigHadoopLogger = PigHadoopLogger.getInstance();
            pigHadoopLogger.setAggregate(aggregateWarning);
            pigHadoopLogger.setReporter(reporter);
            PhysicalOperator.setPigLogger(pigHadoopLogger);
{code}

Note that this is a workaround for your situation. I would highly recommend that you move
to the use of counters when they are supported.

> Pig can not access reporter of PigHadoopLog in Load Func
> --------------------------------------------------------
>
>                 Key: PIG-889
>                 URL: https://issues.apache.org/jira/browse/PIG-889
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>    Affects Versions: 0.4.0
>            Reporter: Jeff Zhang
>            Assignee: Jeff Zhang
>             Fix For: 0.4.0
>
>         Attachments: Pig_889_Patch.txt
>
>
> I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException.
It seems that the reporter is not initialized.  
> I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter)
in PigInputFormat.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message