pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Szita (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-5277) Spark mode is writing nulls among tuples to the output
Date Mon, 31 Jul 2017 11:44:02 GMT

     [ https://issues.apache.org/jira/browse/PIG-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Adam Szita updated PIG-5277:
----------------------------
    Description: 
After committing PIG-3655 a couple of Spark mode tests (e.g. org.apache.pig.test.TestEvalPipeline.testCogroupAfterDistinct)
started failing on:
{code}
java.lang.Error: java.io.IOException: Corrupt data file, expected tuple type byte, but seen
27
	at org.apache.pig.backend.hadoop.executionengine.HJob$1.hasNext(HJob.java:122)
	at org.apache.pig.test.TestEvalPipeline.testCogroupAfterDistinct(TestEvalPipeline.java:1052)
Caused by: java.io.IOException: Corrupt data file, expected tuple type byte, but seen 27
	at org.apache.pig.impl.io.InterRecordReader.readDataOrEOF(InterRecordReader.java:158)
	at org.apache.pig.impl.io.InterRecordReader.nextKeyValue(InterRecordReader.java:194)
	at org.apache.pig.impl.io.InterStorage.getNext(InterStorage.java:79)
	at org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:238)
	at org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:218)
	at org.apache.pig.backend.hadoop.executionengine.HJob$1.hasNext(HJob.java:115)
{code}

This is because InterRecordReader became much stricter after PIG-3655. Before it just simply
skipped these bytes thinking that they are just garbage on the split beginning. Now when we
expect a [proper tuple with a tuple type byte|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/impl/io/InterRecordReader.java#L153]
we see these nulls and throw an Exception.

> Spark mode is writing nulls among tuples to the output 
> -------------------------------------------------------
>
>                 Key: PIG-5277
>                 URL: https://issues.apache.org/jira/browse/PIG-5277
>             Project: Pig
>          Issue Type: Bug
>          Components: spark
>            Reporter: Adam Szita
>            Assignee: Adam Szita
>
> After committing PIG-3655 a couple of Spark mode tests (e.g. org.apache.pig.test.TestEvalPipeline.testCogroupAfterDistinct)
started failing on:
> {code}
> java.lang.Error: java.io.IOException: Corrupt data file, expected tuple type byte, but
seen 27
> 	at org.apache.pig.backend.hadoop.executionengine.HJob$1.hasNext(HJob.java:122)
> 	at org.apache.pig.test.TestEvalPipeline.testCogroupAfterDistinct(TestEvalPipeline.java:1052)
> Caused by: java.io.IOException: Corrupt data file, expected tuple type byte, but seen
27
> 	at org.apache.pig.impl.io.InterRecordReader.readDataOrEOF(InterRecordReader.java:158)
> 	at org.apache.pig.impl.io.InterRecordReader.nextKeyValue(InterRecordReader.java:194)
> 	at org.apache.pig.impl.io.InterStorage.getNext(InterStorage.java:79)
> 	at org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:238)
> 	at org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:218)
> 	at org.apache.pig.backend.hadoop.executionengine.HJob$1.hasNext(HJob.java:115)
> {code}
> This is because InterRecordReader became much stricter after PIG-3655. Before it just
simply skipped these bytes thinking that they are just garbage on the split beginning. Now
when we expect a [proper tuple with a tuple type byte|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/impl/io/InterRecordReader.java#L153]
we see these nulls and throw an Exception.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message