hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-934) Merge join implementation currently does not seek to right point on the right side input based on the offset provided by the index
Date Sat, 29 Aug 2009 00:22:32 GMT

    [ https://issues.apache.org/jira/browse/PIG-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749042#action_12749042
] 

Pradeep Kamath commented on PIG-934:
------------------------------------

The reason I thought a separate function with a "singleFile" in the name was needed was because
the current FileLocalizer.open() can handle directories and hence returns a DataStorageInputStreamIterator
which internally iterates over the underlying multiple streams of the files in the directory.
Keeping the same name may give the impression that the same capability is present even for
the version which seeks to an offset. Seeking to an offset would only work for a single file
- hence maybe have a separate function where the name implies this restriction might be cleaner.

> Merge join implementation currently does not seek to right point on the right side input
based on the offset provided by the index
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-934
>                 URL: https://issues.apache.org/jira/browse/PIG-934
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.3.1
>            Reporter: Pradeep Kamath
>            Assignee: Ashutosh Chauhan
>         Attachments: pig-934.patch
>
>
> We use POLoad to seek into right file which has the following code: 
> {noformat}
>    public void setUp() throws IOException{
>         String filename = lFile.getFileName();
>         loader = (LoadFunc)PigContext.instantiateFuncFromSpec(lFile.getFuncSpec()); 
      
>         is = FileLocalizer.open(filename, pc);
>         loader.bindTo(filename , new BufferedPositionedInputStream(is), this.offset,
Long.MAX_VALUE);
>     }
> {noformat}
> Between opening the stream and bindTo we do not seek to the right offset. bindTo itself
does not perform any seek.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message