Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E546110F3C for ; Fri, 13 Dec 2013 01:22:36 +0000 (UTC) Received: (qmail 29641 invoked by uid 500); 13 Dec 2013 01:22:31 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 29554 invoked by uid 500); 13 Dec 2013 01:22:31 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 29547 invoked by uid 99); 13 Dec 2013 01:22:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Dec 2013 01:22:31 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of xelllee@outlook.com designates 65.54.190.81 as permitted sender) Received: from [65.54.190.81] (HELO bay0-omc2-s6.bay0.hotmail.com) (65.54.190.81) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Dec 2013 01:22:25 +0000 Received: from BAY176-W32 ([65.54.190.123]) by bay0-omc2-s6.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 12 Dec 2013 17:22:04 -0800 X-TMN: [sg4SA9jaIB40cnrsCup+JFEQP6ovf8Zn] X-Originating-Email: [xelllee@outlook.com] Message-ID: Content-Type: multipart/alternative; boundary="_8f2204cd-6edd-4ffd-9dc6-7a2fae706381_" From: xiao li To: "user@hadoop.apache.org" Subject: File size 0 bytes while open for write Date: Thu, 12 Dec 2013 20:22:04 -0500 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 13 Dec 2013 01:22:04.0842 (UTC) FILETIME=[BB88C0A0:01CEF7A1] X-Virus-Checked: Checked by ClamAV on apache.org --_8f2204cd-6edd-4ffd-9dc6-7a2fae706381_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi All=2C I am writing data from java thread=2C while it is writing to the file.(fsDa= taOutputStream =3D fs.append(pt)=3B). It shows 0 bytes for that file . while the file is actually has content. i = guess the reason is it is still open.=20 But the question is what if the Thread got killed without closing the file?= what should be done in this case? the fille will keep showing 'open for wr= ite=2C size 0'. hadoop fs -ls /test/-rw-r--r-- 3 storm supergroup 0 2013-12-12 1= 6:44 /test/SinkToHDFS-ip-.us-west-2.compute.internal-6703-22-20131212-0.sna= ppy hadoop fs -cat /test/SinkToHDFS-i.us-west-2.compute.internal-6703-22-20131= 212-0.snappy | wc -l243 hdfs fsck /test/ -openforwrite Connecting to namenode via http://i.us-west-2.compute.internal:50070FSCK st= arted by xiao (auth:SIMPLE) from for path /test/ at Thu Dec 12 16:52:01 PS= T 2013/test/SinkToHDFS-ip.us-west-2.compute.internal-6703-22-20131212-0.sna= ppy 0 bytes=2C 1 block(s)=2C OPENFORWRITE: Status: HEALTHY Total size: 0 B = Total dirs: 1 Total files: 1 Total blocks (validated): 1 (avg. block size 0= B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0= %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) De= fault replication factor: 3 Average block replication: 3.0 Corrupt blocks: = 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: = 1FSCK ended at Thu Dec 12 16:52:01 PST 2013 in 1 milliseconds = --_8f2204cd-6edd-4ffd-9dc6-7a2fae706381_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi All= =2C


<= /div>
I am writing data from java thread=2C while i= t is writing to the file.(fsDataOutputStream =3D fs.append(pt)=3B).

It shows 0 byte= s for that file . while the file is actually has content. i guess the reaso= n is it is still open. =3B


But the question is= what if the Thread got killed without closing the file? what should be don= e in this case? the fille will keep showing 'open for write=2C size 0'.




hadoop fs -ls /test/
-rw-r--r--  =3B 3 storm= supergroup  =3B  =3B  =3B  =3B  =3B0 2013-12-12 16:44 = /test/SinkToHDFS-ip-.us-west-2.compute.internal-6703-22-20131212-= 0.snappy

 =3Bhadoop fs -cat /test/SinkToHDFS-i.us-west-2.compute.= internal-6703-22-20131212-0.snappy | wc -l
243

hdfs fsck /test= / -openforwrite
Connecting to nameno= de via =3Bhttp://i.us-west-2.com= pute.internal:50070
FSCK started by xiao (auth:SIMPLE) f= rom  =3Bfor path /test/ at Thu Dec 12 16:52:01 PST 2013
/test= /SinkToHDFS-ip.us-west-2.compute.internal-6703-22-20131212-0.snap= py 0 bytes=2C 1 block(s)=2C OPENFORWRITE: Status: HEALTHY
 = =3BTotal size: 0 B
 =3BTotal dirs: 1
 =3BTotal files: 1
 =3BTotal blocks (validated): 1 (avg. block size 0 B)
 =3BMinimally rep= licated blocks: 1 (100.0 %)=
 =3BOver-replicated blocks: 0 (0.0 %)
 =3BUnder-replicated blocks: 0 (0.0 %)
 =3BMis-r= eplicated blocks: 0 (0.0 %= )
 =3BDefault replication factor: 3
 =3BAverage block replication: 3.0
 =3BCorrupt bloc= ks: 0
 =3BMi= ssing replicas: 0 (0.0 %)<= /div>
 =3BNumber of data-nodes: 3
 =3BNumber of racks: 1
FSCK ended at Thu Dec 12 16:52:01 PST = 2013 in 1 milliseconds
= --_8f2204cd-6edd-4ffd-9dc6-7a2fae706381_--