Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 27515 invoked from network); 10 Dec 2008 00:41:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 10 Dec 2008 00:41:55 -0000 Received: (qmail 78477 invoked by uid 500); 10 Dec 2008 00:42:02 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 78421 invoked by uid 500); 10 Dec 2008 00:42:02 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 78410 invoked by uid 99); 10 Dec 2008 00:42:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Dec 2008 16:42:02 -0800 X-ASF-Spam-Status: No, hits=4.9 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 74.125.44.29 is neither permitted nor denied by domain of jonathanc@rockyou.com) Received: from [74.125.44.29] (HELO yx-out-2324.google.com) (74.125.44.29) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Dec 2008 00:41:47 +0000 Received: by yx-out-2324.google.com with SMTP id 31so126060yxl.29 for ; Tue, 09 Dec 2008 16:41:26 -0800 (PST) Received: by 10.114.124.12 with SMTP id w12mr524397wac.131.1228869685372; Tue, 09 Dec 2008 16:41:25 -0800 (PST) Received: by 10.142.48.18 with HTTP; Tue, 9 Dec 2008 16:41:25 -0800 (PST) Message-ID: Date: Tue, 9 Dec 2008 16:41:25 -0800 From: "Jonathan Cao" To: core-user@hadoop.apache.org Subject: File append corrupts file w/ smaill size file Cc: "james warren" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_74948_355456.1228869685365" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_74948_355456.1228869685365 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline We are evaluating the file append feature on 0.19.0. I got the following error while trying to append one small size file to another one (both are less than 1 block). While the file size check indicated the file size has changed to reflect the new size, but the file itself has been apparently corrupted. Same append works fine for large file size. This issue could be related to bug https://issues.apache.org/jira/browse/HADOOP-4423. [hadoop@cloud-1 ~]$ hadoop dfs -cat hdfs:///user/hadoop/test1.txt abcd [hadoop@cloud-1 ~]$ hadoop dfs -cat hdfs:///user/hadoop/test2.txt 08/12/09 16:35:05 INFO hdfs.DFSClient: Could not obtain block blk_-7713572143166377177_1010 from any node: java.io.IOException: No live nodes contain current block 08/12/09 16:35:08 INFO hdfs.DFSClient: Could not obtain block blk_-7713572143166377177_1010 from any node: java.io.IOException: No live nodes contain current block ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 08/12/09 16:28:50 WARN hdfs.DFSClient: Error Recovery for block blk_-7713572143166377177_1009 bad datanode[0] 192.168.1.10:50010 08/12/09 16:28:50 WARN hdfs.DFSClient: Error Recovery for block blk_-7713572143166377177_1009 in pipeline 192.168.1.10:50010, 192.168.1.8:50010: bad datanode 192.168.1.10:50010 08/12/09 16:28:50 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_-7713572143166377177_1010java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readLong(DataInputStream.java:399) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2318) 08/12/09 16:28:50 WARN hdfs.DFSClient: Error Recovery for block blk_-7713572143166377177_1010 bad datanode[0] 192.168.1.8:50010 Exception in thread "main" java.io.IOException: All datanodes 192.168.1.8:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2442) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1997) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160) ------=_Part_74948_355456.1228869685365--