Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: message received from 54.164.171.186
 which is an MX secondary for user@hbase.apache.org)
MIME-Version: 1.0
References: 
 <CAEf6Z5+cHi3BCoqajqo3xC4YgTZkh9oe-AxCGCXNwE2GCz43fg@mail.gmail.com>
 <BLU436-SMTP115B73325ED3A28353AAD0E8FD60@phx.gbl>
 <CAEf6Z5LW7H+d_Ob6QfsGZFeOEWJ_R+C=Bb+oAcs2FT9fNw1tjg@mail.gmail.com>
 <CALte62z8cK-f0Y2fZ5vSYFLgmBUsgY=owVbmeVO2gWGxYUyz1Q@mail.gmail.com>
 <CAEf6Z5KprhWG4PNBZ--L65UB=KEsiKhks8oMyKrDgK-LNmse8A@mail.gmail.com>
 <CAEf6Z5L0qTOOJtozhqWec_s62xLM2mxtsYxrBNvALZTrNrtvrQ@mail.gmail.com>
 <CALte62z1hSgUYCGyw8n+8bSbjg8+96NiEe1PaDQM1DLEE-dSUg@mail.gmail.com>
 <CAEf6Z5LZ+w9JZsaXLvVf8RAVF3J22vmOWGXvdxdNq6XBWU6vvA@mail.gmail.com>
In-Reply-To: 
 <CAEf6Z5LZ+w9JZsaXLvVf8RAVF3J22vmOWGXvdxdNq6XBWU6vvA@mail.gmail.com>
From: Dejan Menges <dejan.menges@gmail.com>
Date: Mon, 04 May 2015 15:29:04 +0000
Message-ID: 
 <CAEf6Z5+GXAP+sioubgN2nrzt8PLKGbKGVcCq77eKBnJjvU-s4A@mail.gmail.com>
Subject: Re: Right value for hbase.rpc.timeout
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7b3a88bcec23df05154338e0

--047d7b3a88bcec23df05154338e0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hm, one thing we did - going back through thread - we decided to manually
set hbase.master.loadbalance.bytable to true, no matter that by
documentation it's true by default. And on our 0.98.0 (2.1.10 by
Hortonworks) it finally really balanced regions evenly by tables we had :/

On Mon, May 4, 2015 at 4:24 PM Dejan Menges <dejan.menges@gmail.com> wrote:

> Cool, thanks a lot! :)
>
> On Mon, May 4, 2015 at 4:14 PM Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Please see https://issues.apache.org/jira/browse/HBASE-10501 and
>> https://issues.apache.org/jira/browse/HBASE-10716
>>
>> Cheers
>>
>> On Mon, May 4, 2015 at 1:43 AM, Dejan Menges <dejan.menges@gmail.com>
>> wrote:
>>
>> > Sorry, maybe I'm wrong: I made my conclusion based on what I read
>> > http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/
>> > that's default; now just found in HBase Book that:
>> >
>> > hbase.regionserver.region.split.policy
>> > Description
>> >
>> > A split policy determines when a region should be split. The various
>> other
>> > split policies that are available currently are
>> > ConstantSizeRegionSplitPolicy, DisabledRegionSplitPolicy,
>> > DelimitedKeyPrefixRegionSplitPolicy, KeyPrefixRegionSplitPolicy etc.
>> > Default
>> >
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPo=
licy
>> >
>> > Otherwise, I can not find any default set in any preinstalled config
>> files.
>> > So what's exactly the truth? :)
>> >
>> >
>> > On Mon, May 4, 2015 at 10:31 AM Dejan Menges <dejan.menges@gmail.com>
>> > wrote:
>> >
>> > > Hi Ted,
>> > >
>> > > Max filesize for region is set to 75G in our case. Regarding split
>> policy
>> > > we use most likely ConstantSizeRegionSplitPolicy
>> > > <
>> >
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/Con=
stantSizeRegionSplitPolicy.html
>> >
>> > (it's
>> > > 0.98.0 with bunch of patches and that should be default one).
>> > >
>> > > Also, regarding link you sent me in 98.3 - I can not find anywhere
>> what's
>> > > default value for hbase.regionserver.lease.period? Is this parameter
>> > still
>> > > called like this?
>> > >
>> > > On Thu, Apr 30, 2015 at 11:27 PM Ted Yu <yuzhihong@gmail.com> wrote:
>> > >
>> > >> Please take a look at 98.3 under
>> > >> http://hbase.apache.org/book.html#trouble.client
>> > >>
>> > >> BTW what's the value for hbase.hregion.max.filesize ?
>> > >> Which split policy do you use ?
>> > >>
>> > >> Cheers
>> > >>
>> > >> On Thu, Apr 30, 2015 at 6:59 AM, Dejan Menges <
>> dejan.menges@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Basically how I came to this question - this happened super rarel=
y,
>> > and
>> > >> we
>> > >> > narrowed it down to hotspotting. Map was timing out on three
>> regions
>> > >> which
>> > >> > were 4-5 times bigger then other regions for the same table, and
>> > region
>> > >> > split fixed this.
>> > >> >
>> > >> > However, was just thinking about if there are maybe some
>> > >> recommendations or
>> > >> > something about this, as it's also super hard to reproduce again
>> same
>> > >> > situation to retest it.
>> > >> >
>> > >> > On Thu, Apr 30, 2015 at 3:56 PM Michael Segel <
>> > >> michael_segel@hotmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > There is no single =E2=80=98right=E2=80=99 value.
>> > >> > >
>> > >> > > As you pointed out=E2=80=A6 some of your Mapper.map() iteration=
s are
>> taking
>> > >> > longer
>> > >> > > than 60 seconds.
>> > >> > >
>> > >> > > The first thing is to determine why that happens.  (It could be
>> > >> normal,
>> > >> > or
>> > >> > > it could be bad code on your developers part. We don=E2=80=99t =
know.)
>> > >> > >
>> > >> > > The other thing is that if you determine that your code is
>> perfect
>> > >> and it
>> > >> > > does what you want it to do=E2=80=A6 and its a major part of yo=
ur use
>> case=E2=80=A6
>> > >> you
>> > >> > > then increase your timeouts to 120 seconds.
>> > >> > >
>> > >> > > The reason why its a tough issue is that we don=E2=80=99t know =
what
>> hardware
>> > >> you
>> > >> > > are using. How many nodes=E2=80=A6 code quality.. etc =E2=80=A6=
 too many factors.
>> > >> > >
>> > >> > >
>> > >> > > > On Apr 30, 2015, at 6:51 AM, Dejan Menges <
>> dejan.menges@gmail.com
>> > >
>> > >> > > wrote:
>> > >> > > >
>> > >> > > > Hi,
>> > >> > > >
>> > >> > > > What's the best practice to calculate this value for your
>> cluster,
>> > >> if
>> > >> > > there
>> > >> > > > is some?
>> > >> > > >
>> > >> > > > In some situations we saw that some maps are taking more than
>> > >> default
>> > >> > 60
>> > >> > > > seconds which was failing specific map job (as if it failed
>> once,
>> > it
>> > >> > > failed
>> > >> > > > also every other time by number of configured retries).
>> > >> > > >
>> > >> > > > I would like to tune RPC parameters a bit, but googling and
>> > looking
>> > >> > into
>> > >> > > > HBase Book doesn't tell me how to calculate right values, and
>> what
>> > >> else
>> > >> > > to
>> > >> > > > take a look beside hbase.rpc.timeout.
>> > >> > > >
>> > >> > > > Thanks a lot,
>> > >> > > > Dejan
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

--047d7b3a88bcec23df05154338e0--