hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region
Date Mon, 26 Dec 2016 12:16:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15778231#comment-15778231
] 

Hadoop QA commented on HBASE-17374:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} |
{color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 49s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 43s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 41s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red}
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 25m 24s {color}
| {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5
2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 53s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 27s {color} | {color:green}
hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s {color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 40s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12844670/0001-fix-for-HBASE-17374.patch
|
| JIRA Issue | HBASE-17374 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  |
| uname | Linux 9292af6424c8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8fa5b0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | https://builds.apache.org/job/PreCommit-HBASE-Build/5045/artifact/patchprocess/whitespace-eol.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/5045/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/5045/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> ZKPermissionWatcher crashed when grant after close region 
> ----------------------------------------------------------
>
>                 Key: HBASE-17374
>                 URL: https://issues.apache.org/jira/browse/HBASE-17374
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.98.15
>            Reporter: Liu Junhong
>            Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some regionservers
was not token effect and must be restart . When I look up logs,  I found that :
> 2016-12-08 15:00:26,238 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler
(CloseRegionHandler.java:process(128)) - Processing close of data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0]
regionserver.HRegion (HRegion.java:doClose(1163)) - Closing data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.:
disabling compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion
(HRegion.java:doClose(1190)) - Updates disabled for region data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion
(HRegion.java:internalFlushcache(1753)) - Started memstore flush for data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.,
current region memstore size 160
> 2016-12-08 15:00:26,284 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.DefaultStoreFlusher
(DefaultStoreFlusher.java:flushSnapshot(95)) - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true,
into tmp file hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegionFileSystem
(HRegionFileSystem.java:commitStoreFile(370)) - Committing store file hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
as hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore
(HStore.java:commitFile(877)) - Added hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion
(HRegion.java:internalFlushcache(1920)) - Finished memstore flush of ~160/160, currentsize=0/0
for region data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, sequenceid=6,
compaction requested=false
> 2016-12-08 15:00:26,323 INFO  [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost
(CoprocessorHost.java:shutdown(292)) - Stop coprocessor org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0]
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop coprocessor org.apache.hadoop.hbase.security.access.AccessController
 {color}
> 2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost
(CoprocessorHost.java:shutdown(292)) - Stop coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost
(CoprocessorHost.java:shutdown(292)) - Stop coprocessor org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion
(HRegion.java:doClose(1279)) - Closed data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red}  2016-12-08 15:00:27,590 ERROR [regionserver60020-EventThread] zookeeper.ClientCnxn
(ClientCnxn.java:processEvent(524)) - Error while calling watcher
> java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1851ab3a
rejected from java.util.concurrent.ThreadPoolExecutor@19c0794f[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 1]
>         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
>         at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
>         at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:638)
>         at org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeDataChanged(ZKPermissionWatcher.java:162)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:461)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-12-08 15:00:27,595 ERROR [regionserver60020-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:processEvent(524))
- Error while calling watcher
> java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@646fdb38
rejected from java.util.concurrent.ThreadPoolExecutor@19c0794f[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 1]
>         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
>         at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
>         at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:638)
>         at org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeChildrenChanged(ZKPermissionWatcher.java:201)
>         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:468)
>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {color}
> 2016-12-08 15:01:19,389 INFO  [PriorityRpcServer.handler=6,queue=0,port=60020] regionserver.HRegionServer
(HRegionServer.java:openRegion(3935)) - Open hbase:acl,,1480596080128.854f160ee9c3f2d1b591d6dbf211956e.
> 2016-12-08 15:01:19,443 DEBUG [RS_OPEN_REGION-dx-data-hbase-watcher05:60020-1] zookeeper.ZKAssign
(ZKAssign.java:transitionNode(832)) - regionserver:60020-0x358ba43097e1ada, quorum=dx-data-hbase-watcher02:2181,dx-data-hbase-watcher01:2181,dx-data-hbase-watcher03:2181,
baseZNode=/hbase Transitioning 854f160ee9c3f2d1b591d6dbf211956e from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
> {color:red}
> After look up source, I found that :
> 1 AccessController implements RegionObserver and  RegionServerObserver.
> 2 The reference of "ZooKeeperWatcher zk" is a same one object when the instance of AccessController
was be used as RegionObserver  and RegionServerObserver.
> 3 When closing region, org.apache.hadoop.hbase.security.access.AccessController.stop(CoprocessorEnvironment)
,  TableAuthManager.release(authManager),    org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.close()
will be called, it leads an ExecutorService of  ZKPermissionWatcher be shutdown few times
util a region was open.
> 4 org.apache.hadoop.hbase.security.access.TableAuthManager.getOrCreate(ZooKeeperWatcher,
Configuration) and org.apache.hadoop.hbase.security.access.TableAuthManager.release(TableAuthManager)
was not synchronized correct, it leads refCount has wrong value and the problem in 3 occurs.
> 5 If I grant a new permission when the ExecutorService is terminated, the exception before
whitch is a runtimeexcepetion will occurs, and at last the zookeeper watcher was crashed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message