pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1684) Inconsistent usage of store func.
Date Fri, 22 Oct 2010 21:52:19 GMT

     [ https://issues.apache.org/jira/browse/PIG-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Thejas M Nair updated PIG-1684:

    Attachment: PIG-1684.1.patch

In patch, POStore creates only one instance of StoreFunc. PigOutputFormat, PigOutputCommitter
classes create only one instance of POStore for each store.

> Inconsistent usage of store func.
> ---------------------------------
>                 Key: PIG-1684
>                 URL: https://issues.apache.org/jira/browse/PIG-1684
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.7.0
>         Environment: A custom StoreFuncInterface used to store data at the reducer.
> (Output of a group )
>            Reporter: Mridul Muralidharan
>         Attachments: PIG-1684.1.patch
> Pig seems to be using multiple instances of StoreFuncInterface in the reducer inconsistently.
> Some hadoop api calls are made to one instance and others made to other : which makes
state management very inconsistent and is requiring hacks on our part to deal with it.
> The call snippet below should hopefully indicate the issue.
> The format is :
> Instance.toString()   method_call.
> com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 getOutputFormat()
> com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 getOutputCommitter
> com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 setupTask
> com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 init
> com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 getOutputFormat()
> com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 getRecordWriter
> com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 init
> com.yahoo.psox.fish.pig.IndexJoinStore@1429cb2 putNext()
> ... 
> com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 needsTaskCommit
> com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 commitTask
> com.yahoo.psox.fish.pig.IndexJoinStore@1be4777 finish()
> As is obvious, two instances are used for different purposes - one to get the record
writer and do the actual write, and another to call the OutputCommitter and its methods.
> Since they are from different instances (StoreFuncInterface), the output committer is
unable to gracefully commit and cleanup.
> I am not attaching the StoreFunc, but any user defined StoreFunc will exhibit this behavior.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message