hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chenchun <chenchun.f...@gmail.com>
Subject Re: empty file
Date Thu, 12 Dec 2013 07:04:04 GMT
Thanks, all of you. The file is indeed an 'open' state. Is there any way I can read it ?

$ /opt/local/hadoop/bin/hadoop fsck /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo -openforwrite
 
FSCK started by sankuai from /10.64.10.102 for path /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo
at Thu Dec 12 14:57:41 CST 2013
/tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo 1123927 bytes, 1 block(s), OPENFORWRITE:  
/tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo:  Under replicated blk_-5402857470524491959_58312275.
Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
 Total size:    1123927 B
 Total dirs:    0
 Total files:   1
 Total blocks (validated):      1 (avg. block size 1123927 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       1 (100.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              1 (50.0 %)
 Number of data-nodes:          38
 Number of racks:               6
FSCK ended at Thu Dec 12 14:57:41 CST 2013 in 1 milliseconds


The filesystem under path '/tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo' is HEALTHY  

--  
chenchun


On Thursday, 12 December, 2013 at 12:09 PM, Harsh J wrote:

> That file is still in an 'open' state. Running the below may show it up:
>  
> /opt/local/hadoop/bin/hadoop fsck
> /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo -openforwrite
>  
> On Thu, Dec 12, 2013 at 9:22 AM, chenchun <chenchun.feed@gmail.com (mailto:chenchun.feed@gmail.com)>
wrote:
> > Nothing is still writing to it. I can't read that file.
> > I'm using hadoop 1.0.1.
> >  
> > $ hadoop fs -text /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo
> > 13/12/12 11:48:56 WARN hdfs.DFSClient: Failed to connect to
> > /10.64.32.14:50010, add to deadNodes and continuejava.io.IOException: Got
> > error for OP_READ_BLOCK, self=/10.64.10.102:51390,
> > remote=/10.64.32.14:50010, for file
> > /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo, for block
> > -5402857470524491959_58312275
> > 13/12/12 11:48:56 WARN hdfs.DFSClient: Failed to connect to
> > /10.64.32.36:50010, add to deadNodes and continuejava.io.IOException: Got
> > error for OP_READ_BLOCK, self=/10.64.10.102:41277,
> > remote=/10.64.32.36:50010, for file
> > /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo, for block
> > -5402857470524491959_58312275
> > 13/12/12 11:48:56 INFO hdfs.DFSClient: Could not obtain block
> > blk_-5402857470524491959_58312275 from any node: java.io.IOException: No
> > live nodes contain current block. Will get new block locations from namenode
> > and retry...
> > 13/12/12 11:48:59 WARN hdfs.DFSClient: Failed to connect to
> > /10.64.32.14:50010, add to deadNodes and continuejava.io.IOException: Got
> > error for OP_READ_BLOCK, self=/10.64.10.102:51403,
> > remote=/10.64.32.14:50010, for file
> > /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo, for block
> > -5402857470524491959_58312275
> > 13/12/12 11:48:59 WARN hdfs.DFSClient: Failed to connect to
> > /10.64.32.36:50010, add to deadNodes and continuejava.io.IOException: Got
> > error for OP_READ_BLOCK, self=/10.64.10.102:41290,
> > remote=/10.64.32.36:50010, for file
> > /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo, for block
> > -5402857470524491959_58312275
> > 13/12/12 11:48:59 INFO hdfs.DFSClient: Could not obtain block
> > blk_-5402857470524491959_58312275 from any node: java.io.IOException: No
> > live nodes contain current block. Will get new block locations from namenode
> > and retry...
> >  
> > --
> > chenchun
> >  
> > On Thursday, 12 December, 2013 at 5:51 AM, John Meagher wrote:
> >  
> > Is something still writing to it?
> > ...
> > Total files: 0 (Files currently being written: 1)
> > Total blocks (validated): 0 (Total open file blocks (not validated): 1)
> >  
> >  
> >  
> > On Wed, Dec 11, 2013 at 2:37 PM, Adam Kawa <kawa.adam@gmail.com (mailto:kawa.adam@gmail.com)>
wrote:
> >  
> > i have never seen something like that.
> >  
> > Can you read that file?
> >  
> > $ hadoop fs -text /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo
> >  
> >  
> > 2013/12/11 chenchun <chenchun.feed@gmail.com (mailto:chenchun.feed@gmail.com)>
> >  
> >  
> > Hi,
> > I find some files on hdfs which command “hadoop fs -ls” tells they are not
> > empty. But command “fsck” tells that these files have no replications. Is
> > it normal?
> >  
> > $ hadoop fs -ls /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo;
> > Found 1 items
> > -rw-r--r-- 3 sankuai supergroup 1123927 2013-12-06 03:22
> > /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo
> >  
> > $ /opt/local/hadoop/bin/hadoop fsck
> > /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo -files -blocks -locations
> > -racks
> > FSCK started by sankuai from /10.64.10.102 for path
> > /tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo at Wed Dec 11 20:35:30 CST
> > 2013
> > Status: HEALTHY
> > Total size: 0 B (Total open files size: 1123927 B)
> > Total dirs: 0
> > Total files: 0 (Files currently being written: 1)
> > Total blocks (validated): 0 (Total open file blocks (not validated): 1)
> > Minimally replicated blocks: 0
> > Over-replicated blocks: 0
> > Under-replicated blocks: 0
> > Mis-replicated blocks: 0
> > Default replication factor: 3
> > Average block replication: 0.0
> > Corrupt blocks: 0
> > Missing replicas: 0
> > Number of data-nodes: 38
> > Number of racks: 6
> > FSCK ended at Wed Dec 11 20:35:30 CST 2013 in 1 milliseconds
> >  
> >  
> > The filesystem under path '/tmp/corrupt_lzo/lc_hadoop16.1386270004881.lzo'
> > is HEALTHY
> >  
> > --
> > chenchun
> >  
>  
>  
>  
>  
> --  
> Harsh J
>  
>  



Mime
View raw message