hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Shah <amits...@gmail.com>
Subject Re: Region Server Crash On Upsert Query Execution
Date Thu, 31 Mar 2016 10:04:57 GMT
A couple of observations

1. I could see GC pauses in the logs but I do not think that could be
causing the jvm exists. I have configured the region server heap to be 2
GB. The jconsole indicates that it hardly reaches 1.5 GB. Kindly find some
graphs attached.

2. On another run the phoenix client failed with a socket timeout
operation. The error is pasted here - http://pastebin.com/mAWwiE2J
Are there client timeouts for each region server? It looks like the default
is 1 min. How can I increase them?

Thanks,
Amit.

On Thu, Mar 31, 2016 at 2:39 PM, Samir Ahmic <ahmic.samir@gmail.com> wrote:

> Hi Amit,
>
> Check regionserver logs, usual suspects, are log running GC and HDFS client
> related issues. Check for compaction queue.
>
> Regrads
> Samir
>
> On Thu, Mar 31, 2016 at 10:48 AM, Amit Shah <amits.84@gmail.com> wrote:
>
> > Hi,
> >
> > We have been experimenting hbase (version 1.0) and phoenix (version 4.6)
> > for our OLAP workload. In order to precalculate aggregates we have been
> > executing an upsert phoenix query that aggregates raw data (over 10 mil
> > records) to generate an OLAP cube.
> >
> > While executing the query, one of the region servers in a cluster of 3 RS
> > crashes. I am trying to figure out what could be causing the region
> server
> > to crash.
> > The server shows high disk operations before the jvm crashed. Kindly find
> > the disk and other stats attached.
> >
> > Any suggestions on where could I look into would be helpful.
> >
> > The upsert query that was executed is
> >
> > upsert into AGENT_TER_PRO
> >
> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT)
> > select /*+ INDEX(TRANSACTIONS  AG_TER_PRO2) */
> >  AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES)
> > SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD)
> > SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT)  AVG_PRICE_PER_UNIT  from
> > TRANSACTIONS   group by AGENT_ID,TERRITORY_ID,PRODUCT_ID;
> >
> > Thanks,
> > Amit.
> >
> >
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message