hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <>
Subject [jira] [Commented] (HIVE-11977) Hive should handle an external avro table with zero length files present
Date Fri, 02 Oct 2015 18:42:26 GMT


Ashutosh Chauhan commented on HIVE-11977:

Thanks for patch [~dossett] 
A 0-length file is an invalid Avro file, as in Avro's {{DataFileWriter}} will always write
MAGIC header for version. Thats the reason {{DataFileReader}} expects it and throws up when
it doesn't get one.
It seems these 0 length files got there because of some faulty generator process. Isn't it
better to just not generate those 0 length files. Or, alternatively, delete these faulty files.

> Hive should handle an external avro table with zero length files present
> ------------------------------------------------------------------------
>                 Key: HIVE-11977
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Aaron Dossett
>            Assignee: Aaron Dossett
>         Attachments: HIVE-11977-2.patch, HIVE-11977.patch
> If a zero length file is in the top level directory housing an external avro table, 
all hive queries on the table fail.
> This issue is that creates
a new org.apache.avro.file.DataFileReader and DataFileReader throws an exception when trying
to read an empty file (because the empty file lacks the magic number marking it as avro).
> AvroGenericRecordReader should detect an empty file and then behave reasonably.
> Caused by: Not a data file.
> at org.apache.avro.file.DataFileStream.initialize(
> at org.apache.avro.file.DataFileReader.<init>(
> at<init>(
> at
> at
> ... 25 more

This message was sent by Atlassian JIRA

View raw message