pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-71) Support for Hadoop Speculative Execution
Date Tue, 26 Feb 2008 22:14:51 GMT

    [ https://issues.apache.org/jira/browse/PIG-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572692#action_12572692

Olga Natkovich commented on PIG-71:

One thought here is that it would be better for hadoop API to ingore non-data files and to
have special APIs to access them if needed. This seems a better abstraction then asking everybody
to skip the files starting with "_".

> Support for Hadoop Speculative Execution
> ----------------------------------------
>                 Key: PIG-71
>                 URL: https://issues.apache.org/jira/browse/PIG-71
>             Project: Pig
>          Issue Type: New Feature
>         Environment: Hadoop
>            Reporter: Amir Youssefi
>            Priority: Minor
> If Speculative Execution is used in Hadoop while creating a data-set then Pig scripts
loading this data-set may fail. Reason is temp directories generated in the process. 
> Pig can filter out these temp directories and problem gets solved. Here is sample error:
> [main] ERROR org.apache.pig - Error message from task (map) tip_..._0001_m_002735 java.io.EOFException
>         at java.io.DataInputStream.readFully(DataInputStream.java:180)
>         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
>         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
>         at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1524)
>         at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1590)
>         at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1626)
>         at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
>         at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
>         ...

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message