pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mohit Sabharwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4633) Update hadoop version to enable Spark output statistics
Date Tue, 14 Jul 2015 02:33:05 GMT

    [ https://issues.apache.org/jira/browse/PIG-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625712#comment-14625712
] 

Mohit Sabharwal commented on PIG-4633:
--------------------------------------

Thanks, [~kexianda], +1 (non-binding).

Could you please paste the exception you saw on this jira ? Thanks!

> Update hadoop version to enable Spark output statistics
> -------------------------------------------------------
>
>                 Key: PIG-4633
>                 URL: https://issues.apache.org/jira/browse/PIG-4633
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: kexianda
>            Assignee: kexianda
>             Fix For: spark-branch
>
>         Attachments: PIG-4633.patch
>
>
> Spark support output statistics from 1.3.0 ([SPARK-3179. Add task OutputMetrics|https://issues.apache.org/jira/browse/SPARK-3179])
> {code:title=SparkHadoopUtil.scala|borderStyle=solid}
> stats.map(Utils.invoke(classOf[Statistics], _, "getThreadStatistics"))
> {code}
> Spark invoke hadoop's function getThreadStatistics. But, this method was added into hadoop
from version 2.5.0 ([HADOOP-10688|https://issues.apache.org/jira/browse/HADOOP-10688])
> The version of hadoop in ivy/libraries.properties should be 2.5.0 +



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message