hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-966) Proposed rework for LoadFunc, StoreFunc, and Slice/r interfaces
Date Tue, 13 Oct 2009 18:07:31 GMT

    [ https://issues.apache.org/jira/browse/PIG-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765172#action_12765172

Dmitriy V. Ryaboy commented on PIG-966:

Alan, thanks for the explanation on the kinds of pushdowns you are envisioning. This makes
sense, although I have a feeling that if we get this complex with pushdowns, it may be more
appropriate to start thinking of interfaces that expose different access paths, rather than
"pushdownable operations".

Starting to think perhaps you are right in wanting to make this a single interface instead
of multiple ones like I suggested.

A couple more thoughts on the LoadPushdown interface.

getFeatures() should probably return a Set, not a List, as duplicates don't really make sense
and we want fast contains() calls on the returned object.

The new idea is just a small tweak on your design that aims to avoid the OperatorPlan issue.

Maintain a Set of LogicalOperator classes (as in, LOProject.class) to indicate acceptable
operators, and provide an pushOperator(LogicalOperator op) method, which can be called multiple
times.  If the order of operators matters, it should be up to whoever is calling this method
to do so in the right order. 

This does force LoadFunc implementations to understand Pig operator classes, and in the case
of Filter it does have to deal with an inner LogicalPlan, but I think those classes are mostly
ok. If someone is advanced enough to want to implement pushdowns, they can handle those interfaces.
There is the danger of the interfaces changing, of course, but, well, that consideration hasn't
stopped Hadoop... and we are setting a precedent by breaking the LoadFunc interface right
now anyway :-).

Too simple?

> Proposed rework for LoadFunc, StoreFunc, and Slice/r interfaces
> ---------------------------------------------------------------
>                 Key: PIG-966
>                 URL: https://issues.apache.org/jira/browse/PIG-966
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Alan Gates
>            Assignee: Alan Gates
> I propose that we rework the LoadFunc, StoreFunc, and Slice/r interfaces significantly.
 See http://wiki.apache.org/pig/LoadStoreRedesignProposal for full details

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message