hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-18690) Integrate with Spark OutputMetrics
Date Mon, 16 Apr 2018 02:19:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-18690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sahil Takiar updated HIVE-18690:
--------------------------------
    Attachment: HIVE-18690.1.patch

> Integrate with Spark OutputMetrics
> ----------------------------------
>
>                 Key: HIVE-18690
>                 URL: https://issues.apache.org/jira/browse/HIVE-18690
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
>         Attachments: HIVE-18690.1.patch
>
>
> Spark has an {{OutputMetrics}} it uses to expose records / bytes written. We currently
don't integrate with it and the Spark UI shows a blank value for output records / bytes. We
have our own customer accumulators instead (like {{HIVE_RECORDS_OUT}}).
> Spark exposes the {{OutputMetrics}} object inside individual tasks via the {{TaskContext.get()}}
method. We can use this method to access the {{OutputMetrics}} object and update it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message