Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of bradfordstephens@gmail.com
 designates 74.125.82.193 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=bVX0xWbbkxwz5T6P8sq6XEwveAnuPPxkuN+nCyzshnKKTBe7snsGtZw17NXXU0gdAO
         /lthCk2E9pwGXhKrAcH5tyyb0tBVUxmxFEub06xBXXO6AmhfAmfeC5CbBqdMTm83VIaE
         KpreSyMKqHVxOFCEWXwhbhDdBYjcq5KbDBIU8=
MIME-Version: 1.0
In-Reply-To: <AANLkTikXDpRqZUhFsb9qha_jAf1d0wAkHSV0MB6JaStA@mail.gmail.com>
References: <AANLkTin2HfTxVtm=x6c+Kz1xiqN_eSprVCkhW8mZbbPK@mail.gmail.com>
 <463596.68997.qm@web65514.mail.ac4.yahoo.com>
 <AANLkTimUZw+1Zt46a6_LtmBcC=8JYnp9r+TusyNHAeBD@mail.gmail.com>
 <AANLkTikXDpRqZUhFsb9qha_jAf1d0wAkHSV0MB6JaStA@mail.gmail.com>
From: Bradford Stephens <bradfordstephens@gmail.com>
Date: Wed, 22 Dec 2010 12:27:39 -0800
Message-ID: <AANLkTikc3-zbgDc6SRjKpxHJHf4uK2XjhwSxiCpszdfK@mail.gmail.com>
Subject: Re: Slow MR data load to table
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Very good points, I'm thinking it's environmental as well, but I
wanted to do a 'sanity check'. I don't really have any control over
this cluster. Upgrading to .89 reduced the load time from 24 hours to
8, but I was expecting 2hrs based on past tests. I can live with it
since it's an initial bulk import, but I want HBase to look awesome
for these customers (since they have a lot of pull).

Another cluster with nearly identical setup for HBase is blazing fast
(for EC2).

(I'm not much of a sysadmin).

Cheers,
B


On Wed, Dec 22, 2010 at 11:43 AM, Stack <stack@duboce.net> wrote:
> I took a look at your regionserver log. =A0As per your above comment,
> boring. =A0More importantly, no blocking going on.
>
> You have 4 column families going on. =A0Do you have to have this amount
> of CFs? =A0This might explain some slow down.
>
> If it was faster last week and this week its slow though 'nothing' has
> changed, it smells environmental.
>
> It looks like you have hooked your Map to TOF... so you should have a
> nice little write buffer in HTable going on (You might check).
>
> For sure, you are not swapping? =A0You have any monitoring of this
> cluster going on? =A0Setting swappyness to zero from 60 is probably a
> bit radical. =A0You want some swap if memory pressure. =A060 is too loose=
.
> =A0If you look at those killed map tasks... why they die? =A0Because
> processes were killed by the kernel?
>
> St.Ack
>
>
> On Tue, Dec 21, 2010 at 10:10 PM, Bradford Stephens
> <bradfordstephens@gmail.com> wrote:
>> Unfortunately, changing swappiness didn't seem to help.
>>
>> On Tue, Dec 21, 2010 at 4:28 PM, Andrew Purtell <apurtell@apache.org> wr=
ote:
>>>> Yes, a good point. Swappiness is set to 60 -- suppose I should set it =
to 0?
>>>
>>> Yes.
>>>
>>> Best regards,
>>>
>>> =A0 =A0- Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back.
>>> =A0- Piet Hein (via Tom White)
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Bradford Stephens,
>> Founder, Drawn to Scale
>> drawntoscalehq.com
>> 727.697.7528
>>
>> http://www.drawntoscalehq.com --=A0 The intuitive, cloud-scale data
>> solution. Process, store, query, search, and serve all your data.
>>
>> http://www.roadtofailure.com -- The Fringes of Scalability, Social
>> Media, and Computer Science
>>
>


--=20
Bradford Stephens,
Founder, Drawn to Scale
drawntoscalehq.com
727.697.7528

http://www.drawntoscalehq.com --=A0 The intuitive, cloud-scale data
solution. Process, store, query, search, and serve all your data.

http://www.roadtofailure.com -- The Fringes of Scalability, Social
Media, and Computer Science