pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kexianda (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4634) Fix records count issues in output statistics
Date Thu, 23 Jul 2015 08:43:05 GMT

    [ https://issues.apache.org/jira/browse/PIG-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638486#comment-14638486

kexianda commented on PIG-4634:

Hi [~xuefuz],

latest patch(PIG-4634_2.patch) is attached. Please help review the code. Thanks.

1. I threw the exception just following the behavior of function SparkJobStats.collectStats().
 I agree with you, it is too harsh. Now, logging message instead of throwing an exception.
The exception is also removed in SparkJobStats.collectStats().

2. From my understanding, we should only aggregate the OutputMetrics of the Result stages,
and ignore the OutputMetrics of ShuffleMap stages. The JobMetricsListener just collect all
the taskMetrics of the stages. We don't know the taskMetrics is from ShuffleMap stage or Result
There is a tricky detail: the OutputMetrics is null in the ShuffleMapTask. And previous patch
file use this tricky detail. It is not intuitive, hard to maintain.
To improve the robustness, the ID of Result stages are collected in a set in JobMetricsListener.java.
Then, in getRecordCount(), only the Result stages's OutputMetrics are aggregated, the OutputMetrics
of ShuffleMap stages are ignored.

3. How to test this?
Here are the two cases for testing this:
ant -Dhadoopversion=23 -Dexectype=spark -Dtestcase=TestPigRunner test


> Fix records count issues in output statistics
> ---------------------------------------------
>                 Key: PIG-4634
>                 URL: https://issues.apache.org/jira/browse/PIG-4634
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: kexianda
>            Assignee: kexianda
>             Fix For: spark-branch
>         Attachments: PIG-4634.patch, PIG-4634_2.patch
> Test cases simpleTest() and simpleTest2()  in TestPigRunner failed, caused by following
> 1. pig context in SparkPigStats isn't initialized.
> 2. the records count logic hasn't been implemented.
> 3. getOutpugAlias(), getPigProperties(), getBytesWritten() and getRecordWritten() have
not been implemented.

This message was sent by Atlassian JIRA

View raw message