Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.223.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <FDEB8A0E-10B1-4294-93C9-3D495474A1E7@gmail.com>
References: <522E52B1-497C-4D8D-9014-0182E8B9AABB@gmail.com>
 <CAO6W-2ew+j7NbVpd=vSEKp9zA+PnGbzVQvSKqxudmz9NUjTXRg@mail.gmail.com>
 <7ED0F250-9815-4262-BFD9-C743AE30F32E@gmail.com>
 <CAO6W-2cSQ-RBJe9zmfsxoYZEVBn9ccs6BWDq8FUxzihUisrKQQ@mail.gmail.com>
 <BLU0-SMTP271D6DBA8B373382531C419DAE90@phx.gbl>
 <C37D3D52-0479-4BBC-A83E-6478870CE981@gmail.com>
 <CAOcnVr3m7-abutYHSs3ekUNR2+C2w_SpWDdxd-zdO33ygVXARw@mail.gmail.com>
 <7387AA18-03E1-4234-BAE7-039083228241@gmail.com>
 <FDEB8A0E-10B1-4294-93C9-3D495474A1E7@gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Wed, 20 Mar 2013 09:35:15 +0530
Message-ID: 
 <CAOcnVr1JfwjJhroMsS82CNL4v1JqbnEVcOXCyOekor=fF8cdeQ@mail.gmail.com>
Subject: Re: disk used percentage is not symmetric on datanodes (balancer)
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

If your balancer does not exit, then it means its heavily working in
iterations trying to balance your cluster. The default bandwidth
allows only for limited transfer speed (10 Mbps) to not affect the
cluster's RW performance while moving blocks between DNs for
balancing, so the operation may be slow unless you raise the allowed
bandwidth.

On Wed, Mar 20, 2013 at 7:37 AM, Tapas Sarangi <tapas.sarangi@gmail.com> wr=
ote:
> Any more follow ups ?
>
> Thanks
> -Tapas
>
> On Mar 19, 2013, at 9:55 AM, Tapas Sarangi <tapas.sarangi@gmail.com> wrot=
e:
>
>>
>> On Mar 18, 2013, at 11:50 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>>> What do you mean that the balancer is always active?
>>
>> meaning, the same process is active for a long time. The process that st=
arts may not be exiting at all. We have a cron job set to run it every 10 m=
inutes, but that's not in effect because the process may never exit.
>>
>>
>>> It is to be used
>>> as a tool and it exits once it balances in a specific run (loops until
>>> it does, but always exits at end). The balancer does balance based on
>>> usage percentage so that is what you're probably looking for/missing.
>>>
>>
>> May be. How does the balancer look for the usage percentage ?
>>
>> -Tapas
>>
>>
>>> On Tue, Mar 19, 2013 at 6:56 AM, Tapas Sarangi <tapas.sarangi@gmail.com=
> wrote:
>>>> Hi,
>>>>
>>>> On Mar 18, 2013, at 8:21 PM, =C0=EE=BA=E9=D6=D2 <lhztop@hotmail.com> w=
rote:
>>>>
>>>> Maybe you need to modify the rackware script to make the rack balance,=
 ie,
>>>> all the racks are the same size,  on rack by 6 small nodes, one rack b=
y 1
>>>> large nodes.
>>>> P.S.
>>>> you need to reboot the cluster for rackware script modify.
>>>>
>>>>
>>>> Like I mentioned earlier in my reply to Bertrand, we haven't considere=
d rack
>>>> awareness for the cluster, currently it is considered as just one rack=
. Can
>>>> that be the problem ? I don't know=A1=AD
>>>>
>>>> -Tapas
>>>>
>>>>
>>>>
>>>> =D3=DA 2013/3/19 7:17, Bertrand Dechoux =D0=B4=B5=C0:
>>>>
>>>> And by active, it means that it does actually stops by itself? Else it=
 might
>>>> mean that the throttling/limit might be an issue with regard to the da=
ta
>>>> volume or velocity.
>>>>
>>>> What threshold is used?
>>>>
>>>> About the small and big datanodes, how are they distributed with regar=
ds to
>>>> racks?
>>>> About files, how is used the replication factor(s) and block size(s)?
>>>>
>>>> Surely trivial questions again.
>>>>
>>>> Bertrand
>>>>
>>>> On Mon, Mar 18, 2013 at 10:46 PM, Tapas Sarangi <tapas.sarangi@gmail.c=
om>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry about that, had it written, but thought it was obvious.
>>>>> Yes, balancer is active and running on the namenode.
>>>>>
>>>>> -Tapas
>>>>>
>>>>> On Mar 18, 2013, at 4:43 PM, Bertrand Dechoux <dechouxb@gmail.com> wr=
ote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> It is not explicitly said but did you use the balancer?
>>>>> http://hadoop.apache.org/docs/r1.0.4/commands_manual.html#balancer
>>>>>
>>>>> Regards
>>>>>
>>>>> Bertrand
>>>>>
>>>>> On Mon, Mar 18, 2013 at 10:01 PM, Tapas Sarangi <tapas.sarangi@gmail.=
com>
>>>>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am using one of the old legacy version (0.20) of hadoop for our
>>>>>> cluster. We have scheduled for an upgrade to the newer version withi=
n a
>>>>>> couple of months, but I would like to understand a couple of things =
before
>>>>>> moving towards the upgrade plan.
>>>>>>
>>>>>> We have about 200 datanodes and some of them have larger storage tha=
n
>>>>>> others. The storage for the datanodes varies between 12 TB to 72 TB.
>>>>>>
>>>>>> We found that the disk-used percentage is not symmetric through all =
the
>>>>>> datanodes. For larger storage nodes the percentage of disk-space use=
d is
>>>>>> much lower than that of other nodes with smaller storage space. In l=
arger
>>>>>> storage nodes the percentage of used disk space varies, but on avera=
ge about
>>>>>> 30-50%. For the smaller storage nodes this number is as high as 99.9=
%. Is
>>>>>> this expected ? If so, then we are not using a lot of the disk space
>>>>>> effectively. Is this solved in a future release ?
>>>>>>
>>>>>> If no, I would like to know  if there are any checks/debugs that one=
 can
>>>>>> do to find an improvement with the current version or upgrading hado=
op
>>>>>> should solve this problem.
>>>>>>
>>>>>> I am happy to provide additional information if needed.
>>>>>>
>>>>>> Thanks for any help.
>>>>>>
>>>>>> -Tapas
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Bertrand Dechoux
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>


--=20
Harsh J