hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Valluri, Sathish" <>
Subject Any sugesstions Not a data file error
Date Wed, 30 Oct 2013 08:50:37 GMT
Resending after disabling security signing..

From: Valluri, Sathish []
Sent: Wednesday, October 30, 2013 2:17 PM
Subject: Any sugesstions Not a data file error

Hi All,

Hive Mapreduce jobs failing with the following Not a data file error
if there are files other than avro in the HDFS.

I have created a Hive external table as shown below,

CREATE EXTERNAL TABLE testTable ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
WITH SERDEPROPERTIES ('avro.schema.literal'='{ <schema json literal>') STORED AS INPUTFORMAT
LOCATION '/testdata/';

Running select count(*) from testable;

When /testdata contains avro files the query works fine and gives the results properly.

If the /testdata have some other format files let's say /testdata/test.txt the query is failing
with the following error. java.lang.reflect.InvocationTargetException at
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(
at org.apache.hadoop.hive.shims.HadoopShimsSecure$
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext( at org.apache.hadoop.mapred.MapTask$
at at org.apache.hadoop.mapred.MapTask.runOldMapper(
at at org.apache.hadoop.mapred.Child$
at Method) at
at at
org.apache.hadoop.mapred.Child.main( Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
at java.lang.reflect.Constructor.newInstance( at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(
... 11 more Caused by: Not a data file. at org.apache.avro.file.DataFileStream.initialize(
at org.apache.avro.file.DataFileReader.<init>( at<init>(
... 16 more

Can anyone suggest any parameter or any changes needs to be made for the query to be successful.
Basically Hive should skip the other format files and load only the avro files when processing
data on the HDFS.

Waiting for any suggestions to resolve this issue.


Sathish Valluri

View raw message