hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niranjan Subramanian <niran...@webaction.com>
Subject Exception while appending to an existing file in HDFS
Date Fri, 30 Oct 2015 16:20:33 GMT
Hi guys, 

I'm basically trying to append data to an already exisiting file in HDFS. This is the exception
I get

03:49:54,456WARN org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run:628 DataStreamer
Exception
java.lang.NullPointerException
at com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
at com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$UpdatePipelineRequestProto$Builder.addAllStorageIDs(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updatePipeline(ClientNamenodeProtocolTranslatorPB.java:842)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1238)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:532)
My replication factor is 1. I'm using 2.5.0 of Apache's Hadoop distribution. This is the code
snippet which I'm using for creating a file if it doesn't exist or create in append mode if
it exists

String url = getHadoopUrl()+fileName;
    Path file = new Path(url);
    try {
        if(append) {
            if(hadoopFileSystem.exists(file))
                fsDataOutputStream = hadoopFileSystem.append(file);
            else
                fsDataOutputStream = hadoopFileSystem.create(file);
        }
        else
            fsDataOutputStream = hadoopFileSystem.create(file); 
Appending stack trace that I found in datanode's log

2015-10-30 16:19:54,435 INFO  org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1012136337-192.168.123.103-1411103100884:blk_1073742239_1421
src: /127.0.0.1:54160  dest: /127.0.0.1:50010
 2015-10-30 16:19:54,435 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
Appending to FinalizedReplica, blk_1073742239_1421, FINALIZED
 getNumBytes()     = 812
 getBytesOnDisk()  = 812
 getVisibleLength()= 812
 getVolume()       = /Users/niranjan/hadoop/hdfs/datanode/current
 getBlockFile()    = /Users/niranjan/hadoop/hdfs/datanode/current/BP-        1012136337-192.168.123.103-1411103100884/current/finalized/blk_1073742239
 unlinked          =false
2015-10-30 16:19:54,461 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for
BP-1012136337-192.168.123.103-1411103100884:blk_1073742239_1422
java.io.IOException: Premature EOF from inputStream
    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:435)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:693)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:569)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
    at java.lang.Thread.run(Thread.java:745)

It's not quite clear what is causing this exception. Also I'm quite confused after reading
various sources whether appending is supported in HDFS or not. Let me know what I'm missing
here

Regards,
Niranjan
Mime
View raw message