pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Howard (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4609) PIG : How to use counter in pig script
Date Fri, 19 Jun 2015 02:06:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-4609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14592881#comment-14592881

Michael Howard commented on PIG-4609:

I am also interested in using counters in pig. 
I posted the message below to user@pig.apache.org on 2015-06-18. 

Thus far I have only received one response, saying that the PigStatusReporter does not asynchronously
update the counters. 

accessing custom counters during execution of pig job
My questions relate to custom "counters" in order to dynamically monitor progress during execution
of a pig job. 

I would like to increment some custom counters in the context of pig UDFs.
I would like to access these counters during execution under differnt ExecTypes ... local/mapreduce/tez

I see that PigStatusReporter.getInstance().incrCounter(...) is the recommended way to increment
counters so that they are accessible under both MR and tez execution models. 

I do not see how one can fetch the current value of a counter during execution via PigRunner
or PigServer. 
I expected to find access to counters under PigStats, but have been unable to locate a way
to access counters. 

Q: How does one retrieve current values of counters during execution of a pig script? 


> PIG : How to use counter in pig script
> --------------------------------------
>                 Key: PIG-4609
>                 URL: https://issues.apache.org/jira/browse/PIG-4609
>             Project: Pig
>          Issue Type: Task
>          Components: internal-udfs, parser, piggybank
>    Affects Versions: 0.8.0, 0.11
>            Reporter: Sudhir Ranjan
> Hi There,
> How can I use counter to count the total #of records processed through pig script. 
> I know that it can be achieved by writing & invoking a UDF from pig script or by
using GROUPBY & COUNT(), but here I don't want to go this way.
> I need the way in PIG like the way we are accessing the counter as 
> context.getCounter(MyCounters.Counter).increment(1);

This message was sent by Atlassian JIRA

View raw message