hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Shah <amits...@gmail.com>
Subject Re: Region Server Crash On Upsert Query Execution
Date Thu, 31 Mar 2016 10:48:48 GMT
Thanks for the reply.

I already had both the above properties set to 1 hour. Do I need to set
them on the phoenix client or a different setting is causing the time out
errors?

On Thu, Mar 31, 2016 at 3:47 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> The attachments you mentioned did go through.
>
> For #2, please adjust:
>
> hbase.client.scanner.timeout.period
> hbase.rpc.timeout
>
> Both have default value of 60 seconds.
>
> If possible, please pastebin server log snippet before it crashed.
>
> Thanks
>
> On Thu, Mar 31, 2016 at 3:04 AM, Amit Shah <amits.84@gmail.com> wrote:
>
> > A couple of observations
> >
> > 1. I could see GC pauses in the logs but I do not think that could be
> > causing the jvm exists. I have configured the region server heap to be 2
> > GB. The jconsole indicates that it hardly reaches 1.5 GB. Kindly find
> some
> > graphs attached.
> >
> > 2. On another run the phoenix client failed with a socket timeout
> > operation. The error is pasted here - http://pastebin.com/mAWwiE2J
> > Are there client timeouts for each region server? It looks like the
> > default is 1 min. How can I increase them?
> >
> > Thanks,
> > Amit.
> >
> > On Thu, Mar 31, 2016 at 2:39 PM, Samir Ahmic <ahmic.samir@gmail.com>
> > wrote:
> >
> >> Hi Amit,
> >>
> >> Check regionserver logs, usual suspects, are log running GC and HDFS
> >> client
> >> related issues. Check for compaction queue.
> >>
> >> Regrads
> >> Samir
> >>
> >> On Thu, Mar 31, 2016 at 10:48 AM, Amit Shah <amits.84@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > We have been experimenting hbase (version 1.0) and phoenix (version
> 4.6)
> >> > for our OLAP workload. In order to precalculate aggregates we have
> been
> >> > executing an upsert phoenix query that aggregates raw data (over 10
> mil
> >> > records) to generate an OLAP cube.
> >> >
> >> > While executing the query, one of the region servers in a cluster of 3
> >> RS
> >> > crashes. I am trying to figure out what could be causing the region
> >> server
> >> > to crash.
> >> > The server shows high disk operations before the jvm crashed. Kindly
> >> find
> >> > the disk and other stats attached.
> >> >
> >> > Any suggestions on where could I look into would be helpful.
> >> >
> >> > The upsert query that was executed is
> >> >
> >> > upsert into AGENT_TER_PRO
> >> >
> >>
> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT)
> >> > select /*+ INDEX(TRANSACTIONS  AG_TER_PRO2) */
> >> >  AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES)
> >> > SUM_TOTAL_SALES,sum(TOTAL_EXPENSES)
> >> SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD)
> >> > SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT)  AVG_PRICE_PER_UNIT  from
> >> > TRANSACTIONS   group by AGENT_ID,TERRITORY_ID,PRODUCT_ID;
> >> >
> >> > Thanks,
> >> > Amit.
> >> >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message