hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harshit Mathur <mathursh...@gmail.com>
Subject Mapreduce combinefileinputformat java.lang.reflect.InvocationTargetException while two jobs access same data
Date Tue, 25 Nov 2014 05:36:54 GMT
Hi,

The Hadoop Mapreduce CombineFileInputFormat works great when it comes to
read a lot of small size files, however i have been noticing that sometimes
the job gets failed with the following exception,

java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException at
org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.java:164)
at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.nextKeyValue(CombineFileRecordReader.java:67)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at
org.apache.hadoop.mapred.Child$4.run(Child.java:268) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.

I have noticed that this is happening only when some other mapreduce job is
running on the same data at the same time otherwise it works as expected.

Even the same exception is generated when i run hive query under the
similar condition.

I have been searching for the solution or probable cause for this.


-- 
Harshit Mathur

Mime
View raw message