phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sanket patel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3977) Region is not closed when moving region or balancing while writing to table with local index
Date Thu, 15 Mar 2018 07:44:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400044#comment-16400044
] 

sanket patel commented on PHOENIX-3977:
---------------------------------------

I hit the similar issue recently with 4.10. I find regions in PENDING_CLOSE state and same regions
would show up under `/hbase-unsecure/regions-in-transition`  Znode. Also on HMaster UI, under
Procedures tab, there are pending tasks for DisableTable (probably because region is stuck)

As a solution I remove the znode and restart hmaster. And then under same tables I see those
regions as offline region. Later I drop table.

Currently I am not able to reproduce the issue. I will post more symptoms and things I notice
if I happen to encounter it again.

> Region is not closed when moving region or balancing while writing to table with local
index
> --------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-3977
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3977
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.10.0
>            Reporter: JeongMin Ju
>            Priority: Major
>         Attachments: screenshot-1.png
>
>
> Region is not closed when moving region or balancing while writing to table with local
index.
> If the regionserver is forcibly killed and restart and then balancing is performed during
write operation perform using YCSB. The region is not moved properly, so it becomes jammed.
> This is also true when moving a specific region.
> {panel:title=RegionServer1}
> 2017-06-26 17:18:49,096 INFO org.apache.phoenix.hbase.index.util.IndexManagementUtil:
Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10):
Unable to find cached index metadata.  key=-4998016164816556219 region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
Index update failed
> 2017-06-26 17:18:49,531 INFO org.apache.phoenix.hbase.index.util.IndexManagementUtil:
Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10):
Unable to find cached index metadata.  key=6036765365681157624 region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
Index update failed
> 2017-06-26 17:18:49,536 INFO org.apache.phoenix.hbase.index.util.IndexManagementUtil:
Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10):
Unable to find cached index metadata.  key=-428088766346522675 region=PHOENIX_TABLE,&,1498464285648.df3f3da32306361ce5e38e5193f5e5d6.host=juke-cdh-36f531b4.s2.krane.9rum.cc,60020,1498464965204
Index update failed
> {panel}
> At other regionserver
> {panel:title=RegionServer2}
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1476, waiting
for 62  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1476, waiting
for 27  actions to finish on table: PHOENIX_TABLE
> 2017-06-26 17:19:49,755 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1476, waiting
for 52  actions to finish on table: PHOENIX_TABLE
> {panel}
> phoenix.coprocessor.maxServerCacheTimeToLiveMs & phoenix.coprocessor.maxMetaDataCacheSize
was not effected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message