hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-932) Required fields projection in Loader: nested fields in bag, map key lookup more than two levels
Date Tue, 25 Aug 2009 23:31:59 GMT
Required fields projection in Loader: nested fields in bag, map key lookup more than two levels
-----------------------------------------------------------------------------------------------

                 Key: PIG-932
                 URL: https://issues.apache.org/jira/browse/PIG-932
             Project: Pig
          Issue Type: Improvement
          Components: impl
    Affects Versions: 0.3.0
            Reporter: Daniel Dai


To leverage the performance features provided by Zebra, Pig should be able to figure out which
input fields are actually used in Pig script, and prune unnecessary inputs. This feature is
being implementing in [PIG-922|https://issues.apache.org/jira/browse/PIG-922]. However, there
are two limitations currently:

1. Pruning nested fields only apply to map and record. We do not prune sub-field inside a
collection
2. For map, currently we only go one level deep. Eg, if in Pig script, user uses a#'key0'#'key1',
a#'key0' will be asked

These two limitations are in line with current limitation of Zebra loader. Once Zebra loader
can handle this, we need to work to lift these limitations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message