phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaoxiao Wang <xxw...@23andme.com.INVALID>
Subject Re: How Phoenix JDBC connection get hbase configuration
Date Thu, 21 Feb 2019 00:50:58 GMT
Pedro

to answer your question, I have verified that the configuration file has
loaded correctly from the classpath,

I have 300+ mappers try to make connections to the db at the same time, and
it still gives me the same error that timeout at after 60000 ms

On Wed, Feb 20, 2019 at 4:45 PM Xiaoxiao Wang <xxwang@23andme.com> wrote:

> Since I've known that the configuration have been loaded up correctly
> through the classpath
>
> I have tested on the real application, however, it still timed out with
> the same default value from the mappers
>
> Error: java.io.IOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions: Thu Feb 21 00:38:28 UTC 2019, null,
> java.net.SocketTimeoutException: callTimeout=60000, callDuration=60309
>
> On Wed, Feb 20, 2019 at 4:25 PM Xiaoxiao Wang <xxwang@23andme.com> wrote:
>
>> i made this work on my toy application, getConf() is not an issue, and
>> hbase conf can get the correct settings
>>
>> I'm trying out again on the real application
>>
>> On Wed, Feb 20, 2019 at 4:13 PM William Shen <willshen@marinsoftware.com>
>> wrote:
>>
>>> Whatever is in super.getConf() should get overriden by hbase-site.xml
>>> because addHbaseResources because will layer on hbase-site.xml last. The
>>> question is which one got picked up... (maybe there is another one on the
>>> classpath, is that possible?)
>>>
>>> On Wed, Feb 20, 2019 at 4:10 PM Xiaoxiao Wang <xxwang@23andme.com.invalid
>>> >
>>> wrote:
>>>
>>> > I'm trying out on the mapreduce application, I made it work on my toy
>>> > application
>>> >
>>> > On Wed, Feb 20, 2019 at 4:09 PM William Shen <
>>> willshen@marinsoftware.com>
>>> > wrote:
>>> >
>>> > > A bit of a long shot, but do you happen to have another
>>> hbase-site.xml
>>> > > bundled in your jar accidentally that might be overriding what is on
>>> the
>>> > > classpath?
>>> > >
>>> > > On Wed, Feb 20, 2019 at 3:58 PM Xiaoxiao Wang
>>> <xxwang@23andme.com.invalid
>>> > >
>>> > > wrote:
>>> > >
>>> > > > A bit more information, I feel the classpath didn't get passed
in
>>> > > correctly
>>> > > > by doing
>>> > > >
>>> > > > conf = HBaseConfiguration.addHbaseResources(super.getConf());
>>> > > >
>>> > > > and this conf also didn't pick up the expected properties
>>> > > >
>>> > > >
>>> > > > On Wed, Feb 20, 2019 at 3:56 PM Xiaoxiao Wang <xxwang@23andme.com>
>>> > > wrote:
>>> > > >
>>> > > > > Pedro
>>> > > > >
>>> > > > > thanks for your info, yes, I have tried both
>>> > > > > HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml and
>>> > > > > HADOOP_CLASSPATH=/etc/hbase/conf/ (without file), and yes
checked
>>> > > > > hadoop-env.sh as well to make sure it did
>>> > > > > HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/others
>>> > > > >
>>> > > > > And also for your second question, it is indeed a map reduce
>>> job, and
>>> > > it
>>> > > > > is trying to query phoenix from map function! (and we make
sure
>>> all
>>> > the
>>> > > > > nodes have hbase-site.xml installed properly )
>>> > > > >
>>> > > > > thanks
>>> > > > >
>>> > > > > On Wed, Feb 20, 2019 at 3:53 PM Pedro Boado <
>>> pedro.boado@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > >> Your classpath variable should be pointing to the folder
>>> containing
>>> > > your
>>> > > > >> hbase-site.xml and not directly to the file.
>>> > > > >>
>>> > > > >> But certain distributions tend to override that envvar
inside
>>> > > > >> hadoop-env.sh
>>> > > > >> or hadoop.sh .
>>> > > > >>
>>> > > > >> Out of curiosity, have you written a map-reduce application
and
>>> are
>>> > > you
>>> > > > >> querying phoenix from map functions?
>>> > > > >>
>>> > > > >> On Wed, 20 Feb 2019, 23:34 Xiaoxiao Wang,
>>> > <xxwang@23andme.com.invalid
>>> > > >
>>> > > > >> wrote:
>>> > > > >>
>>> > > > >> > HI Pedro
>>> > > > >> >
>>> > > > >> > thanks for your help, I think we know that we need
to set the
>>> > > > classpath
>>> > > > >> to
>>> > > > >> > the hadoop program, and what we tried was
>>> > > > >> > HADOOP_CLASSPATH=/etc/hbase/conf/hbase-site.xml
hadoop jar
>>> > $test_jar
>>> > > > >> but it
>>> > > > >> > didn't work
>>> > > > >> > So we are wondering if anything we did wrong?
>>> > > > >> >
>>> > > > >> > On Wed, Feb 20, 2019 at 3:24 PM Pedro Boado <
>>> pboado@apache.org>
>>> > > > wrote:
>>> > > > >> >
>>> > > > >> > > Hi,
>>> > > > >> > >
>>> > > > >> > > How many concurrent client connections are
we talking
>>> about? You
>>> > > > >> might be
>>> > > > >> > > opening more connections than the RS can handle
( under
>>> these
>>> > > > >> > circumstances
>>> > > > >> > > most of the client threads would end exhausting
their retry
>>> > count
>>> > > )
>>> > > > .
>>> > > > >> I
>>> > > > >> > > would bet that you've get a bottleneck in the
RS keeping
>>> > > > >> SYSTEM.CATALOG
>>> > > > >> > > table (this was an issue in 4.7 ) as every
new connection
>>> would
>>> > be
>>> > > > >> > querying
>>> > > > >> > > this table first.
>>> > > > >> > >
>>> > > > >> > > Try to update to our cloudera-compatible parcels
instead of
>>> > using
>>> > > > >> clabs -
>>> > > > >> > > which are discontinued by Cloudera and not
supported by the
>>> > Apache
>>> > > > >> > Phoenix
>>> > > > >> > > project - .
>>> > > > >> > >
>>> > > > >> > > Once updated to phoenix 4.14 you should be
able to use
>>> > > > >> > > UPDATE_CACHE_FREQUENCY
>>> > > > >> > > property in order to reduce pressure on system
tables.
>>> > > > >> > >
>>> > > > >> > > Adding an hbase-site.xml with the required
properties to the
>>> > > client
>>> > > > >> > > application classpath should just work.
>>> > > > >> > >
>>> > > > >> > > I hope it helps.
>>> > > > >> > >
>>> > > > >> > > On Wed, 20 Feb 2019, 22:50 Xiaoxiao Wang,
>>> > > > <xxwang@23andme.com.invalid
>>> > > > >> >
>>> > > > >> > > wrote:
>>> > > > >> > >
>>> > > > >> > > > Hi, who may help
>>> > > > >> > > >
>>> > > > >> > > > We are running a Hadoop application that
needs to use
>>> phoenix
>>> > > JDBC
>>> > > > >> > > > connection from the workers.
>>> > > > >> > > > The connection works, but when too many
connection
>>> established
>>> > > at
>>> > > > >> the
>>> > > > >> > > same
>>> > > > >> > > > time, it throws RPC timeouts
>>> > > > >> > > >
>>> > > > >> > > > Error: java.io.IOException:
>>> > > > >> > > > org.apache.phoenix.exception.PhoenixIOException:
Failed
>>> after
>>> > > > >> > > attempts=36,
>>> > > > >> > > > exceptions: Wed Feb 20 20:02:43 UTC 2019,
null, java.net
>>> > > > >> > > .SocketTimeoutException:
>>> > > > >> > > > callTimeout=60000, callDuration=60506.
...
>>> > > > >> > > >
>>> > > > >> > > > So we have figured we should probably
set a higher
>>> > > > >> hbase.rpc.timeout
>>> > > > >> > > > value, but then it comes to the issue:
>>> > > > >> > > >
>>> > > > >> > > > A little bit background on how we run
the application
>>> > > > >> > > >
>>> > > > >> > > > Here is how we get PhoenixConnection from
java program
>>> > > > >> > > > DriverManager.getConnection("jdbc:phoenix:host",
props)
>>> > > > >> > > > And we trigger the program by using
>>> > > > >> > > > hadoop jar $test_jar
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > > We have tried multiple approaches to load
hbase/phoenix
>>> > > > >> configuration,
>>> > > > >> > > but
>>> > > > >> > > > none of them get respected by PhoenixConnection,
here are
>>> the
>>> > > > >> methods
>>> > > > >> > we
>>> > > > >> > > > tried
>>> > > > >> > > > * Pass hbase_conf_dir through HADOOP_CLASSPATH,
so run the
>>> > > hadoop
>>> > > > >> > > > application like HADOOP_CLASSPATH=/etc/hbase/conf/
hadoop
>>> jar
>>> > > > >> > $test_jar .
>>> > > > >> > > > However, PhoenixConnection doesn’t respect
the parameters
>>> > > > >> > > > * Tried passing -Dhbase.rpc.timeout=1800,
which is picked
>>> up
>>> > by
>>> > > > >> hbase
>>> > > > >> > > conf
>>> > > > >> > > > object, but not PhoniexConnection
>>> > > > >> > > > * Explicitly set those parameters and
pass them to the
>>> > > > >> > PhoenixConnection
>>> > > > >> > > > props.setProperty("hbase.rpc.timeout",
"1800");
>>> > > > >> > > > props.setProperty(“phoenix.query.timeoutMs",
"1800");
>>> > > > >> > > > Also didn’t get respected by PhoenixConnection
>>> > > > >> > > > * also tried what is suggested by phoenix
here
>>> > > > >> > > > https://phoenix.apache.org/#connStr ,
use :longRunning
>>> > together
>>> > > > >> with
>>> > > > >> > > > those properties, still didn’t seem
to work
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > > Besides all those approaches we tried,
I have explicitly
>>> > output
>>> > > > >> those
>>> > > > >> > > > parameters we care from the connection,
>>> > > > >> > > > connection.getQueryServices().getProps()
>>> > > > >> > > > The default values I got are 60000 for
hbase.rpc.timeout,
>>> and
>>> > > 600k
>>> > > > >> for
>>> > > > >> > > > phoenix.query.timeoutMs , so I have tried
to run a query
>>> lthat
>>> > > > would
>>> > > > >> > run
>>> > > > >> > > > longer than 10 mins, Ideally it should
timeout, however,
>>> it
>>> > runs
>>> > > > >> over
>>> > > > >> > 20
>>> > > > >> > > > mins and didn’t timeout. So I’m wondering
how
>>> > PhoenixConnection
>>> > > > >> respect
>>> > > > >> > > > those properties?
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > > So with some of your help, we’d like
to know if there’s
>>> any
>>> > > thing
>>> > > > >> wrong
>>> > > > >> > > > with our approaches. And we’d like to
get rid of those
>>> > > > >> > > SocketTimeExceptions.
>>> > > > >> > > > We are using phoenix-core version is
>>> 4.7.0-clabs-phoenix1.3.0
>>> > ,
>>> > > > and
>>> > > > >> our
>>> > > > >> > > > phoenix-client version is
>>> phoenix-4.7.0-clabs-phoenix1.3.0.23
>>> > > (we
>>> > > > >> have
>>> > > > >> > > > tried phoenix-4.14.0-HBase-1.3 as well,
which didn’t work
>>> > > either).
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > > Thanks for your time
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >>
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message