Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
References: 
 <CAO3OuAGOc=REVcHXpZyaLgFWvVAgirDieQaXLzAOsjKDp=K87A@mail.gmail.com>
 <1577500411.4316224.1434438809398.JavaMail.yahoo@mail.yahoo.com>
 <BLU436-SMTP16A168387D9F9940D20C0D8FA70@phx.gbl>
In-Reply-To: <BLU436-SMTP16A168387D9F9940D20C0D8FA70@phx.gbl>
From: Bryan Beaudreault <bbeaudreault@hubspot.com>
Date: Tue, 16 Jun 2015 20:08:10 +0000
Message-ID: 
 <CANZDn9uittRoMp5mLt-wFUpP76R+aMNOrRDVVi_ZghOXxWmkdA@mail.gmail.com>
Subject: Re: How to make the client fast fail
To: user@hbase.apache.org, lars hofhansl <larsh@apache.org>
Content-Type: multipart/alternative; boundary=001a113f8e42d78e640518a822fd

--001a113f8e42d78e640518a822fd
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I agree that more documentation would be better. However,


> Yet, there are some applications which require a faster time out than
> others. So, you tune some of the timers to have a fast fail, and you end =
up
> causing unintended problems for others.
>
> The simplest solution is to use threads in you client app. (Of course thi=
s
> assumes that you=E2=80=99re capable of writing clean multi-threaded code =
and I
> don=E2=80=99t want to assume anything.)
>
> Remember that HBase is a shared resource. So you need to consider the
> whole at the same time you consider the needs of one.
>
>
This is not really true, it assumes a very naive configuration solution in
which all processes are configured the same.  The default configs come from
xml files, but are easily customized.  Consider:

Application A:

Configuration conf =3D HBaseConfiguration.create();
conf.setInt("hbase.rpc.timeout", 8000);

// Operations on table have a timeout of 8s
Connection conn =3D ConnectionFactory.createConnection(conf);
conn.getTable("foo");

Application B:

// Operations on this table use default timeout
Connection conn =3D
ConnectionFactory.createConnection(HBaseConfiguration.create());
conn.getTable("foo");

// Operations on this table use a very short timeout of 500ms
Configuration conf =3D HBaseConfiguration.create();
conf.setInt("hbase.rpc.timeout", 500);
Connection shortConn =3D ConnectionFactory.createConnection(conf);
short
shortConn.getTable("foo");

Applications A and B are configured with different timeouts.  Further,
Application B has two separate table connections, each with a different
timeout.

The values are hardcoded above, but could easily be made configurable. The
simplest of solutions uses System.getProperty(), so you can pass
-Dmy.custom.timeout=3D500.  More complex solutions can utilize the various
open source live configuration solutions. Such as netflix archaius, or
hubspot's live-config -- both available on github.

Of course there can be unintended consequences if an application suddenly
> starts to drop connections before a result or timeout occurs too.  ;-)
>
>
> > On Jun 16, 2015, at 12:13 AM, lars hofhansl <larsh@apache.org> wrote:
> >
> > Please always tell us which version of HBase you are using. We have
> fixed a lot of issues in this area over time.Here's an _old_ blog post I
> wrote about this:
> http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
> >
> > Using yet more threads to monitor timeouts of another thread is a bad
> idea, especially when the timeout is configurable in the first place.
> >
> > -- Lars
> >      From: mukund murrali <mukundmurrali9@gmail.com>
> > To: user@hbase.apache.org
> > Sent: Sunday, June 14, 2015 10:22 PM
> > Subject: Re: How to make the client fast fail
> >
> > It would be great if there is a single timeout configuration from the
> > client end. All other parameters should fine tune based on that one
> > parameter. We have modified simple based on trail basis to suit our nee=
d.
> > Also not sure what side effect it would cause configuring those
> parameters.
> >
> >
> >
> > On Mon, Jun 15, 2015 at 10:38 AM, <Hariharan_Sethuraman@dell.com> wrote=
:
> >
> >> We are also interested on the solution for this. With
> >> hbase.client.retries.number =3D 7 and client.pause=3D400ms, it came do=
wn to
> >> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big
> number.
> >>
> >> Thanks,
> >> Hari
> >>
> >> -----Original Message-----
> >> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> >> Sent: Monday, June 15, 2015 10:33 AM
> >> To: user@hbase.apache.org
> >> Subject: Re: How to make the client fast fail
> >>
> >> Hi Michael,
> >>
> >> We can have a monitoring thread and interrupt the hbase client thread
> >> after time out instead of doing this I want the timeout or some
> exception
> >> to be thrown from the HBase client itself.
> >>
> >> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> >> wrote:
> >>
> >>> threads?
> >>>
> >>> So that regardless of your hadoop settings, if you want something
> >>> faster, you can use one thread for a timer and then the request is in
> >>> another. So if you hit your timeout before you get a response, you ca=
n
> >> stop your thread.
> >>> (YMMV depending on side effects... )
> >>>
> >>>> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> >>>>
> >>> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I have got the Connection object with default configuration, if the
> >>>> zookeeper or HMaster or Region server is down, the client didn't
> >>>> fast
> >>> fail
> >>>> and it took almost 20 mins to thrown an error.
> >>>>
> >>>> What is the best configuration to make the client fast fail.
> >>>>
> >>>> Also what is significance of changing the following parameters.
> >>>>
> >>>> hbase.client.retries.number
> >>>> zookeeper.recovery.retry
> >>>> zookeeper.session.timeout
> >>>> zookeeper.recovery.retry.intervalmill
> >>>> hbase.rpc.timeout
> >>>>
> >>>> Regards,
> >>>> Praneesh
> >>>
> >>>
> >>
> >
> >
>
>

--001a113f8e42d78e640518a822fd--