pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3585) Make union work with tez
Date Sun, 01 Dec 2013 17:29:35 GMT

    [ https://issues.apache.org/jira/browse/PIG-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836074#comment-13836074
] 

Rohini Palaniswamy commented on PIG-3585:
-----------------------------------------

+1. The code is good if we have union after some processing. With this patch a simple union
takes 3 vertices (2 load and 1 union) instead of 1 vertex. We should make it optimal to handle
in a single vertex.

bq. Tez doesn't allow multiple inputs for the root vertex. We probably can't implement UNION
with a single vertex.
   With MR, we just load both inputs as two different PigSplit in PigInputFormat. That should
be possible here too right?

> Make union work with tez
> ------------------------
>
>                 Key: PIG-3585
>                 URL: https://issues.apache.org/jira/browse/PIG-3585
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>    Affects Versions: tez-branch
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: tez-branch
>
>         Attachments: PIG-3585-1.patch
>
>
> Unlike MR, union needs to be implemented as multiple vertices because Tez doesn't allow
multiple inputs in root vertex.
> I propose that we implement union by connecting load vertices to union vertex with broadcasting
edges.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message