Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C0E110F25 for ; Fri, 13 Dec 2013 18:38:27 +0000 (UTC) Received: (qmail 93918 invoked by uid 500); 13 Dec 2013 18:38:22 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 93759 invoked by uid 500); 13 Dec 2013 18:38:22 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 93752 invoked by uid 99); 13 Dec 2013 18:38:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Dec 2013 18:38:22 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of xelllee@outlook.com designates 65.54.190.96 as permitted sender) Received: from [65.54.190.96] (HELO bay0-omc2-s21.bay0.hotmail.com) (65.54.190.96) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Dec 2013 18:38:17 +0000 Received: from BAY176-W25 ([65.54.190.123]) by bay0-omc2-s21.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Fri, 13 Dec 2013 10:37:56 -0800 X-TMN: [MuND+zbGCQrTqaaNwdVNzSEvBaMK/Aym] X-Originating-Email: [xelllee@outlook.com] Message-ID: Content-Type: multipart/alternative; boundary="_d8c7ce72-e4b4-4095-88ca-0568d67e860e_" From: xiao li To: "user@hadoop.apache.org" Subject: RE: File size 0 bytes while open for write Date: Fri, 13 Dec 2013 13:37:56 -0500 Importance: Normal In-Reply-To: References: , MIME-Version: 1.0 X-OriginalArrivalTime: 13 Dec 2013 18:37:56.0930 (UTC) FILETIME=[7110C220:01CEF832] X-Virus-Checked: Checked by ClamAV on apache.org --_d8c7ce72-e4b4-4095-88ca-0568d67e860e_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi=2C Yong Thanks for you reply first. It is a really good explanation. But I am wonde= ring if this happened=2C is there any thing i could do to close the file ot= her than waiting for the limit(soft or hard)?((DistributedFileSystem) fs).r= ecoverLease(file)=3B this seems works=2C how to do it from the command line= ? BTW=2C like you avatar. Xiao From: java8964@hotmail.com To: user@hadoop.apache.org Subject: RE: File size 0 bytes while open for write Date: Fri=2C 13 Dec 2013 09:31:18 -0500 =0A= =0A= =0A= The HDFS client that opens a file for writing is granted a lease for the fi= le=3B no other client can write to the file. The writing client periodicall= y renews the lease by sending a heartbeat to the NameNode. When the file is= closed=2C the lease is revoked. The lease duration is bound by a soft limi= t and a hard limit. Until the soft limit expires=2C the writer is certain o= f exclusive access to the file. If the soft limit expires and the client fa= ils to close the file or renew the lease=2C another client can preempt the = lease. If after the hard limit expires (one hour) and the client has failed= to renew the lease=2C HDFS assumes that the client has quit and will autom= atically close the file on behalf of the writer=2C and recover the lease. T= he writer's lease does not prevent other clients from reading the file=3B a= file may have many concurrent readers. Yong From: xelllee@outlook.com To: user@hadoop.apache.org Subject: File size 0 bytes while open for write Date: Thu=2C 12 Dec 2013 20:22:04 -0500 =0A= =0A= =0A= Hi All=2C I am writing data from java thread=2C while it is writing to the file.(fsDa= taOutputStream =3D fs.append(pt)=3B). It shows 0 bytes for that file . while the file is actually has content. i = guess the reason is it is still open.=20 But the question is what if the Thread got killed without closing the file?= what should be done in this case? the fille will keep showing 'open for wr= ite=2C size 0'. hadoop fs -ls /test/-rw-r--r-- 3 storm supergroup 0 2013-12-12 1= 6:44 /test/SinkToHDFS-ip-.us-west-2.compute.internal-6703-22-20131212-0.sna= ppy hadoop fs -cat /test/SinkToHDFS-i.us-west-2.compute.internal-6703-22-20131= 212-0.snappy | wc -l243 hdfs fsck /test/ -openforwrite Connecting to namenode via http://i.us-west-2.compute.internal:50070FSCK st= arted by xiao (auth:SIMPLE) from for path /test/ at Thu Dec 12 16:52:01 PS= T 2013/test/SinkToHDFS-ip.us-west-2.compute.internal-6703-22-20131212-0.sna= ppy 0 bytes=2C 1 block(s)=2C OPENFORWRITE: Status: HEALTHY Total size: 0 B = Total dirs: 1 Total files: 1 Total blocks (validated): 1 (avg. block size 0= B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0= %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) De= fault replication factor: 3 Average block replication: 3.0 Corrupt blocks: = 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 3 Number of racks: = 1FSCK ended at Thu Dec 12 16:52:01 PST 2013 in 1 milliseconds = = --_d8c7ce72-e4b4-4095-88ca-0568d67e860e_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi=2C Yong

Th= anks for you reply first. It is a really good explanation. =3B
But I am wondering if this happened=2C is there any thing i could do to c= lose the file other than waiting for the limit(soft or hard)?
((D= istributedFileSystem) fs).recoverLease(file)=3B this seems works=2C how to = do it from the command line?

BTW=2C like you avata= r.

Xiao



<= hr id=3D"stopSpelling">From: java8964@hotmail.com
To: user@hadoop.apache= .org
Subject: RE: File size 0 bytes while open for write
Date: Fri=2C= 13 Dec 2013 09:31:18 -0500

=0A= =0A= =0A=
The HDFS client that opens a file for writing is granted a l= ease for the file=3B no other client can write to the file. The writing cli= ent periodically renews the lease by sending a heartbeat to the NameNode. W= hen the file is closed=2C the lease is revoked. The lease duration is bound= by a soft limit and a hard limit. Until the soft limit expires=2C the writ= er is certain of exclusive access to the file. If the soft limit expires an= d the client fails to close the file or renew the lease=2C another client c= an preempt the lease. If after the hard limit expires (one hour) and the cl= ient has failed to renew the lease=2C HDFS assumes that the client has quit= and will automatically close the file on behalf of the writer=2C and recov= er the lease. The writer's lease does not prevent other clients from readin= g the file=3B a file may have many concurrent readers.

Yong

From: xelllee@outlook.com
To: us= er@hadoop.apache.org
Subject: File size 0 bytes while open for write
= Date: Thu=2C 12 Dec 2013 20:22:04 -0500

=0A= =0A= =0A=
Hi All=2C
<= br>

I am writing dat= a from java thread=2C while it is writing to the file.(fsDataOutputStream = =3D fs.append(pt)=3B).

It shows 0 bytes for that file . while the file is actually has conten= t. i guess the reason is it is still open. =3B


But the ques= tion is what if the Thread got killed without closing the file? what should= be done in this case? the fille will keep showing 'open for write=2C size = 0'.



<= div>
hadoop fs -ls /test/
-rw-r--r--  =3B 3 sto= rm supergroup  =3B  =3B  =3B  =3B  =3B0 2013-12-12 16:4= 4 /test/SinkToHDFS-ip-.us-west-2.compute.internal-6703-22-2013121= 2-0.snappy

 =3Bhadoop fs -cat /test/SinkToHDFS-i.us-west-2.compute.internal= -6703-22-20131212-0.snappy | wc -l
243

hdfs fsck /test/ -openforwrite
<= /div>
Connecting to namenode via =3Bhttp://i.us-west-2.compute.internal:50070
=
 =3BUnder-replicated blocks: 0 (0.0 %)
 =3BMis-replicated blocks: 0 (0.0 %)
 =3BDefault repli= cation factor: 3
&= nbsp=3BAverage block replication: <= /span>3.0
 =3BCorrupt blocks: 0
 =3BMissing replicas: 0 (0.0 %)
 =3BNumber of data-node= s: 3
 =3BNumb= er of racks: 1
FS= CK ended at Thu Dec 12 16:52:01 PST 2013 in 1 milliseconds
=
= = --_d8c7ce72-e4b4-4095-88ca-0568d67e860e_--