hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amir Youssefi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-71) Support for Hadoop Speculative Execution
Date Wed, 23 Jan 2008 22:44:33 GMT

    [ https://issues.apache.org/jira/browse/PIG-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561850#action_12561850
] 

Amir Youssefi commented on PIG-71:
----------------------------------

Here is naming convention: temp directories start with _  e.g. _task

-Amir

> Support for Hadoop Speculative Execution
> ----------------------------------------
>
>                 Key: PIG-71
>                 URL: https://issues.apache.org/jira/browse/PIG-71
>             Project: Pig
>          Issue Type: New Feature
>         Environment: Hadoop
>            Reporter: Amir Youssefi
>            Priority: Minor
>
> If Speculative Execution is used in Hadoop while creating a data-set then Pig scripts
loading this data-set may fail. Reason is temp directories generated in the process. 
> Pig can filter out these temp directories and problem gets solved. Here is sample error:
> [main] ERROR org.apache.pig - Error message from task (map) tip_..._0001_m_002735 java.io.EOFException
>         at java.io.DataInputStream.readFully(DataInputStream.java:180)
>         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
>         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
>         at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1524)
>         at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1590)
>         at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1626)
>         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
>         at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
>         ...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message