hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rita <rmorgan...@gmail.com>
Subject Re: directory scan issue
Date Tue, 29 Mar 2011 12:30:36 GMT
Thankyou. Switching to ext4

On Tue, Mar 29, 2011 at 8:23 AM, Eric <eric.xkcd@gmail.com> wrote:

> Rita, another issue I've seen is that when you have lots of XFS filesystems
> that are heavily used, the Linux kernel will at some point crash. So the XFS
> driver seems to have problems that only appear with large volumes of data. I
> will switch to EXT4 soon because of this.
>
>
> 2011/3/29 Rita <rmorgan466@gmail.com>
>
>> Thanks with ext4 i created 2 16TB volumes and they are seen. I think it
>> maybe a issue with XFS.
>>
>>
>>
>> On Mon, Mar 28, 2011 at 3:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>>> On Fri, Mar 25, 2011 at 9:06 PM, Rita <rmorgan466@gmail.com> wrote:
>>>
>>>> Using 0.21
>>>>
>>>> When I have a filesystem (XFS) with 1TB it detects the
>>>> datanode detects it immediately. When I create 3 identical file systems all
>>>> 3TB are visible immediately.
>>>>
>>>> If I create a 6TB filesystem (XFS) and I add it to dfs.data.dir and I
>>>> restart the datanode, "hdfs dfsadmin -report" does not see the new 6TB
>>>> filesystem.
>>>>
>>>> In all of these occasions the datanode does create a 'finalized' file
>>>> structure in respective directories?
>>>>
>>>>
>>>> My questions are:
>>>> Is there a limitation in the size of dfs.data.dir? What is the largest
>>>> filesystem that can be part of it?
>>>>
>>>
>>> I've heard that there is a 4T limit, but I've never tried to replicate.
>>> Given that single disks aren't this large, it indicates you might be running
>>> RAID or SPAN rather than recommended JBOD.
>>>
>>>
>>>> Could this be a block scanner issue? Is it possible to make my block
>>>> scanning more aggressive?
>>>>
>>>
>>> Unrelated to block scanning most likely.
>>>
>>> -Todd
>>>  --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> --- Get your facts first, then you can distort them as you please.--
>>
>
>


-- 
--- Get your facts first, then you can distort them as you please.--

Mime
View raw message