hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Region Server Crash On Upsert Query Execution
Date Thu, 31 Mar 2016 10:17:41 GMT
The attachments you mentioned did go through.

For #2, please adjust:

hbase.client.scanner.timeout.period
hbase.rpc.timeout

Both have default value of 60 seconds.

If possible, please pastebin server log snippet before it crashed.

Thanks

On Thu, Mar 31, 2016 at 3:04 AM, Amit Shah <amits.84@gmail.com> wrote:

> A couple of observations
>
> 1. I could see GC pauses in the logs but I do not think that could be
> causing the jvm exists. I have configured the region server heap to be 2
> GB. The jconsole indicates that it hardly reaches 1.5 GB. Kindly find some
> graphs attached.
>
> 2. On another run the phoenix client failed with a socket timeout
> operation. The error is pasted here - http://pastebin.com/mAWwiE2J
> Are there client timeouts for each region server? It looks like the
> default is 1 min. How can I increase them?
>
> Thanks,
> Amit.
>
> On Thu, Mar 31, 2016 at 2:39 PM, Samir Ahmic <ahmic.samir@gmail.com>
> wrote:
>
>> Hi Amit,
>>
>> Check regionserver logs, usual suspects, are log running GC and HDFS
>> client
>> related issues. Check for compaction queue.
>>
>> Regrads
>> Samir
>>
>> On Thu, Mar 31, 2016 at 10:48 AM, Amit Shah <amits.84@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > We have been experimenting hbase (version 1.0) and phoenix (version 4.6)
>> > for our OLAP workload. In order to precalculate aggregates we have been
>> > executing an upsert phoenix query that aggregates raw data (over 10 mil
>> > records) to generate an OLAP cube.
>> >
>> > While executing the query, one of the region servers in a cluster of 3
>> RS
>> > crashes. I am trying to figure out what could be causing the region
>> server
>> > to crash.
>> > The server shows high disk operations before the jvm crashed. Kindly
>> find
>> > the disk and other stats attached.
>> >
>> > Any suggestions on where could I look into would be helpful.
>> >
>> > The upsert query that was executed is
>> >
>> > upsert into AGENT_TER_PRO
>> >
>> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT)
>> > select /*+ INDEX(TRANSACTIONS  AG_TER_PRO2) */
>> >  AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES)
>> > SUM_TOTAL_SALES,sum(TOTAL_EXPENSES)
>> SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD)
>> > SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT)  AVG_PRICE_PER_UNIT  from
>> > TRANSACTIONS   group by AGENT_ID,TERRITORY_ID,PRODUCT_ID;
>> >
>> > Thanks,
>> > Amit.
>> >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message