pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4135) Fetch optimization should be disabled if plan contains no limit
Date Thu, 21 Aug 2014 21:26:11 GMT

     [ https://issues.apache.org/jira/browse/PIG-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Cheolsoo Park updated PIG-4135:
-------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Committed to trunk.

Thank you Daniel and Lorand for the review!

> Fetch optimization should be disabled if plan contains no limit
> ---------------------------------------------------------------
>
>                 Key: PIG-4135
>                 URL: https://issues.apache.org/jira/browse/PIG-4135
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: 0.14.0
>
>         Attachments: PIG-4135-1.patch
>
>
> After deploying fetch optimization in production, a couple of users ran into this situation.
They had fairly large input data, but after filtering it by a regular expression, it becomes
small. So they didn't add limit to the query. 
> The problem is that even though the output is small, processing the input must be done
in the cluster not in the client. However, fetch optimization blindly fetches the entire input
into the client since the plan is map-only job and finishes with dump.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message