hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ulul <had...@ulul.org>
Subject Re: HDFS openforwrite CORRUPT -> HEALTHY
Date Sat, 11 Oct 2014 12:33:06 GMT
Hi Vinayak,
Sorry this is beyond my understanding. I would need to test furthet to 
try and understand the problem.
Hope you'll find help from someone else
Ulul

Le 08/10/2014 07:18, Vinayak Borkar a écrit :
> Hi Ulul,
>
> I think I can explain why the sizes differ and the block names vary. 
> There is no client interaction. My client writes data and calls hsync, 
> and then writes more data to the same file. My understanding is that 
> under such circumstances, the file size is not reflected accurately in 
> HDFS until the file is actually closed. So the namenode's view of the 
> file size will be lower than the actual size of the data in the block. 
> If you look at the block closely, you will see that the block number 
> is the same for the two blocks. The part that is different is the 
> version number - this is consistent with HDFS's behavior when hsyncing 
> the output stream and then continuing to write more. It looks like the 
> name node is informed much later about the last block that the 
> datanode actually wrote.
>
> My client was not started when the machine came back up. So all 
> changes seen in the FSCK output were owing to HDFS.
>
>
> Vinayak
>
>
> On 10/7/14, 2:37 PM, Ulul wrote:
>>
>> Hi Vinayak
>>
>> I find strange that the file should have a different size and the block
>> a different name.
>> Are you sure your writing client wasn't interfering ?
>>
>> Ulul
>>
>> Le 07/10/2014 19:41, Vinayak Borkar a écrit :
>>> Trying again since I did not get a reply. Please let me know if I
>>> should use a different forum to ask this question.
>>>
>>> Thanks,
>>> Vinayak
>>>
>>>
>>>
>>> On 10/4/14, 8:45 PM, Vinayak Borkar wrote:
>>>> Hi,
>>>>
>>>>
>>>> I was experimenting with HDFS to push its boundaries on fault 
>>>> tolerance.
>>>> Here is what I observed.
>>>>
>>>> I am using HDFS from Hadoop 2.2. I started the NameNode and then a
>>>> single DataNode. I started writing to a DFS file from a Java client
>>>> periodically calling hsync(). After some time, I powered off the 
>>>> machine
>>>> that was running this test (not shutdown, just abruptly powered off).
>>>>
>>>> When the system came back up, and HDFS processes were up and HDFS was
>>>> out of safe mode, I ran fsck on the DFS filesystem (with -openforwrite
>>>> -files -blocks) options and here is the output:
>>>>
>>>>
>>>> /test/test.log 388970 bytes, 1 block(s), OPENFORWRITE: MISSING 1 
>>>> blocks
>>>> of total size 388970 B
>>>> 0.
>>>> BP-1471648347-10.211.55.100-1412458980748:blk_1073743243_2420{blockUCState=UNDER_CONSTRUCTION,

>>>>
>>>>
>>>> primaryNodeIndex=-1,
>>>> replicas=[ReplicaUnderConstruction[[DISK]DS-e5bed5ae-1fa9-45ed-8d4c-8006919b4d9c:NORMAL|RWR]]}

>>>>
>>>>
>>>> len=388970 MISSING!
>>>>
>>>> Status: CORRUPT
>>>>   Total size:    7214119 B
>>>>   Total dirs:    54
>>>>   Total files:    232
>>>>   Total symlinks:        0
>>>>   Total blocks (validated):    214 (avg. block size 33710 B)
>>>>    ********************************
>>>>    CORRUPT FILES:    1
>>>>    MISSING BLOCKS:    1
>>>>    MISSING SIZE:        388970 B
>>>>    ********************************
>>>>   Minimally replicated blocks:    213 (99.53271 %)
>>>>   Over-replicated blocks:    0 (0.0 %)
>>>>   Under-replicated blocks:    213 (99.53271 %)
>>>>   Mis-replicated blocks:        0 (0.0 %)
>>>>   Default replication factor:    3
>>>>   Average block replication:    0.9953271
>>>>   Corrupt blocks:        0
>>>>   Missing replicas:        426 (66.35514 %)
>>>>   Number of data-nodes:        1
>>>>   Number of racks:        1
>>>> FSCK ended at Sat Oct 04 23:09:40 EDT 2014 in 47 milliseconds
>>>>
>>>>
>>>> I just let the system sit for some time and reran fsck (after about
>>>> 15-20 mins) and surprisingly the output was very different. The
>>>> corruption was magically gone:
>>>>
>>>> /test/test.log 1859584 bytes, 1 block(s):  Under replicated
>>>> BP-1471648347-10.211.55.100-1412458980748:blk_1073743243_2421. Target
>>>> Replicas is 3 but found 1 replica(s).
>>>> 0. BP-1471648347-10.211.55.100-1412458980748:blk_1073743243_2421
>>>> len=1859584 repl=1
>>>>
>>>> Status: HEALTHY
>>>>   Total size:    8684733 B
>>>>   Total dirs:    54
>>>>   Total files:    232
>>>>   Total symlinks:        0
>>>>   Total blocks (validated):    214 (avg. block size 40582 B)
>>>>   Minimally replicated blocks:    214 (100.0 %)
>>>>   Over-replicated blocks:    0 (0.0 %)
>>>>   Under-replicated blocks:    214 (100.0 %)
>>>>   Mis-replicated blocks:        0 (0.0 %)
>>>>   Default replication factor:    3
>>>>   Average block replication:    1.0
>>>>   Corrupt blocks:        0
>>>>   Missing replicas:        428 (66.666664 %)
>>>>   Number of data-nodes:        1
>>>>   Number of racks:        1
>>>> FSCK ended at Sat Oct 04 23:24:23 EDT 2014 in 63 milliseconds
>>>>
>>>>
>>>> The filesystem under path '/' is HEALTHY
>>>>
>>>>
>>>>
>>>> So my question is this: What just happened? How did the NameNode 
>>>> recover
>>>> that missing block and why did it take 15 mins or so? Is there some 
>>>> kind
>>>> of a lease on the file (because of the open nature) that expired after
>>>> the 15-20 mins? Can someone with knowledge of HDFS internals please 
>>>> shed
>>>> some light on what could possibly be going on or point me to 
>>>> sections of
>>>> the code that could answer my questions? Also is there a way to speed
>>>> this process up? Like say trigger the expiration of the lease 
>>>> (assuming
>>>> it is a lease).
>>>>
>>>> Thanks,
>>>> Vinayak
>>>
>>
>>
>


Mime
View raw message