hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shashi Vishwakarma <shashi.vish...@gmail.com>
Subject Map Reduce job failing - Hbase Bulk load
Date Wed, 03 Jun 2015 06:13:56 GMT
Hi

I have map reduce job for hbase bulk load. Job is converting data into
Hfiles and loading into hbase but after certain map % job is failing. Below
is the exception that I am getting.

Error: java.io.FileNotFoundException:
/var/mapr/local/tm4/mapred/nodeManager/spill/job_1433110149357_0005/attempt_1433110149357_0005_m_000000_0/spill83.out.index
    at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:198)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:800)
    at
org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:156)
    at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:74)
    at
org.apache.hadoop.mapred.MapRFsOutputBuffer.mergeParts(MapRFsOutputBuffer.java:1382)
    at
org.apache.hadoop.mapred.MapRFsOutputBuffer.flush(MapRFsOutputBuffer.java:1627)
    at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:709)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:345)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Above error says that file not found exception but I was able to locate
that particular spill on disk.

Only thing that i noticed in job that for small set of data it is working
fine but as data grows job starts failing.

Let me know if anyone has faced this issue.

Thanks

Shashi

Mime
View raw message