pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-5316) Initialize mapred.task.id property for PoS jobs
Date Tue, 28 Nov 2017 18:42:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269241#comment-16269241

Xuefu Zhang commented on PIG-5316:

[~nkollar], Sorry for the late reply.
Though MRConfiguration is not intended for public use in Pig, should Hive use MRConfiguration#TASK_ID
instead of referring to the taskId as a string?
Your concern is valid. However, Hive has a lot of legacy MR1 code (more than just mapred.task.id).
It probably takes a lot of effort to clean up this. Before that happens, Yes, the risk is
going to be there.

> Initialize mapred.task.id property for PoS jobs
> -----------------------------------------------
>                 Key: PIG-5316
>                 URL: https://issues.apache.org/jira/browse/PIG-5316
>             Project: Pig
>          Issue Type: Improvement
>          Components: spark
>            Reporter: Adam Szita
>            Assignee: Nandor Kollar
>             Fix For: 0.18.0
>         Attachments: PIG-5316_1.patch, PIG-5316_2.patch
> Some downstream systems may require the presence of {{mapred.task.id}} property (e.g.
HCatalog). This is currently not set when Pig On Spark jobs are started. Let's initialise

This message was sent by Atlassian JIRA

View raw message