hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Reed <br...@yahoo-inc.com>
Subject Re: [jira] Commented: (PIG-71) Support for Hadoop Speculative Execution
Date Thu, 24 Jan 2008 16:16:53 GMT
So we should ignore both . and _ ? Isn't . enough?

ben

On Wednesday 23 January 2008 20:32:34 eric baldeschwieler (JIRA) wrote:
>     [
> https://issues.apache.org/jira/browse/PIG-71?page=com.atlassian.jira.plugin
>.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561934#action_125
>61934 ]
>
> eric baldeschwieler commented on PIG-71:
> ----------------------------------------
>
> There are two different issues confused here.
>
> 1) metadata files in output directories.  Hadoop reserves the right to put
> metadata in the output directory in files and directories that start with
> '_'.  Pig should behave consistently with this.  We should document this in
> Hadoop and Pig
>
> 2) speculative execution seems to be leaving task directories about.  If
> this is happening on successful runs, something is broken and should be
> fixed.
>
> > Support for Hadoop Speculative Execution
> > ----------------------------------------
> >
> >                 Key: PIG-71
> >                 URL: https://issues.apache.org/jira/browse/PIG-71
> >             Project: Pig
> >          Issue Type: New Feature
> >         Environment: Hadoop
> >            Reporter: Amir Youssefi
> >            Priority: Minor
> >
> > If Speculative Execution is used in Hadoop while creating a data-set then
> > Pig scripts loading this data-set may fail. Reason is temp directories
> > generated in the process. Pig can filter out these temp directories and
> > problem gets solved. Here is sample error: [main] ERROR org.apache.pig -
> > Error message from task (map) tip_..._0001_m_002735 java.io.EOFException
> > at java.io.DataInputStream.readFully(DataInputStream.java:180) at
> > org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:
> >56) at
> > org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90) at
> > org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:152
> >4) at
> > org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.
> >java:1590) at
> > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.jav
> >a:1626) at
> > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712) at
> > org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecord
> >Reader.java:79) ...



Mime
View raw message