phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (PHOENIX-1473) Connecting with Phoenix client when Phoenix is not deployed on region server(s) takes down region server(s).
Date Wed, 26 Nov 2014 06:13:12 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

James Taylor resolved PHOENIX-1473.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: 4.2.2
                   3.2.2
                   3.3
                   4.3

[~mujtaba] - would you mind verifying that when a server doesn't have the phoenix jar that
your cluster doesn't go down now?

> Connecting with Phoenix client when Phoenix is not deployed on region server(s) takes
down region server(s).
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-1473
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-1473
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.2
>            Reporter: Serhiy Bilousov
>             Fix For: 4.3, 3.3, 3.2.2, 4.2.2
>
>         Attachments: PHOENIX-1473.patch, PHOENIX-1473_4_2.patch
>
>
> When attempting to connect with Phoenix client when Phoenix server not deployed will
take down region servers. It looks like problem with creating SYSTEM.CATALOG table causing
it. It may be argued that there some improvements can be done on HBase side in regards to
not allow to create table if coprocessor jar files could not be found but from Phoenix side
I would think before doing anything it should check if required jar is in place (meaning Phoenix
deployed properly).
> here is log from region server
> 2014-11-20 14:29:48,411 ERROR [RS_OPEN_REGION-dn01:60020-1] handler.OpenRegionHandler:
Failed open of region=SYSTEM.CATALOG,,1416493787900.35b7b3a19f75688fce382e8f1323b4ae., starting
to roll back the global memstore size.
> java.io.IOException: Unable to load configured region split policy 'org.apache.phoenix.schema.MetaDataSplitPolicy'
for table 'SYSTEM.CATALOG'
>         at org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:121)
>         at org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.create(RegionSplitPolicy.java:101)
>         at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:727)
>         at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:684)
>        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4550)
>         at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4520)
>         at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4492)
>         at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4448)
>         at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4399)
>         at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465)
>         at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139)
>         at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.schema.MetaDataSplitPolicy
>         at java.net.URLClassLoader$1.run(Unknown Source)
>         at java.net.URLClassLoader$1.run(Unknown Source)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(Unknown Source)
>         at java.lang.ClassLoader.loadClass(Unknown Source)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
>         at java.lang.ClassLoader.loadClass(Unknown Source)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Unknown Source)
>         at org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPolicyClass(RegionSplitPolicy.java:117)
>         ... 14 more
> 2014-11-20 14:29:48,411 INFO  [RS_OPEN_REGION-dn01:60020-1] handler.OpenRegionHandler:
Opening of region {ENCODED => 35b7b3a19f75688fce382e8f1323b4ae, NAME => 'SYSTEM.CATALOG,,1416493787900.35b7b3a19f75688fce382e8f1323b4ae.',
STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
expecting version 4
> 2014-11-20 14:29:48,412 DEBUG [RS_OPEN_REGION-dn01:60020-1] zookeeper.ZKAssign: regionserver:60020-0x349cd39697a00e5,
quorum=nn02.abc.com:2181,nn01.abc.com:2181,jz01.abc.com:2181, baseZNode=/hbase Transitioning
35b7b3a19f75688fce382e8f1323b4ae from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 2014-11-20 14:29:48,414 DEBUG [RS_OPEN_REGION-dn01:60020-1] zookeeper.ZKAssign: regionserver:60020-0x349cd39697a00e5,
quorum=nn02.abc.com:2181,nn01.abc.com:2181,z01.abc.com:2181, baseZNode=/hbase Transitioned
node 35b7b3a19f75688fce382e8f1323b4ae from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 2014-11-20 14:29:48,528 INFO  [regionserver60020] regionserver.HRegionServer: stopping
server dn01.abc.com,60020,1416493546924; all regions closed.
> 2014-11-20 14:29:48,528 DEBUG [regionserver60020-WAL.AsyncNotifier] wal.FSHLog: regionserver60020-WAL.AsyncNotifier
interrupted while waiting for  notification from AsyncSyncer thread
> 2014-11-20 14:29:48,528 INFO  [regionserver60020-WAL.AsyncNotifier] wal.FSHLog: regionserver60020-WAL.AsyncNotifier
exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer0] wal.FSHLog: regionserver60020-WAL.AsyncSyncer0
interrupted while waiting for notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO  [regionserver60020-WAL.AsyncSyncer0] wal.FSHLog: regionserver60020-WAL.AsyncSyncer0
exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer1] wal.FSHLog: regionserver60020-WAL.AsyncSyncer1
interrupted while waiting for notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO  [regionserver60020-WAL.AsyncSyncer1] wal.FSHLog: regionserver60020-WAL.AsyncSyncer1
exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer2] wal.FSHLog: regionserver60020-WAL.AsyncSyncer2
interrupted while waiting for notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO  [regionserver60020-WAL.AsyncSyncer2] wal.FSHLog: regionserver60020-WAL.AsyncSyncer2
exiting
> 2014-11-20 14:29:48,529 DEBUG [regionserver60020-WAL.AsyncSyncer3] wal.FSHLog: regionserver60020-WAL.AsyncSyncer3
interrupted while waiting for notification from AsyncWriter thread
> 2014-11-20 14:29:48,529 INFO  [regionserver60020-WAL.AsyncSyncer3] wal.FSHLog: regionserver60020-WAL.AsyncSyncer3
exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020-WAL.AsyncSyncer4] wal.FSHLog: regionserver60020-WAL.AsyncSyncer4
interrupted while waiting for notification from AsyncWriter thread
> 2014-11-20 14:29:48,530 INFO  [regionserver60020-WAL.AsyncSyncer4] wal.FSHLog: regionserver60020-WAL.AsyncSyncer4
exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020-WAL.AsyncWriter] wal.FSHLog: regionserver60020-WAL.AsyncWriter
interrupted while waiting for newer writes added to local buffer
> 2014-11-20 14:29:48,530 INFO  [regionserver60020-WAL.AsyncWriter] wal.FSHLog: regionserver60020-WAL.AsyncWriter
exiting
> 2014-11-20 14:29:48,530 DEBUG [regionserver60020] wal.FSHLog: Closing WAL writer in hdfs://mycluster/hbase/WALs/dn01.abc.com,60020,1416493546924
> 2014-11-20 14:29:48,547 INFO  [regionserver60020] regionserver.Leases: regionserver60020
closing leases
> 2014-11-20 14:29:48,547 INFO  [regionserver60020] regionserver.Leases: regionserver60020
closed leases
> 2014-11-20 14:29:48,915 INFO  [regionserver60020.periodicFlusher] regionserver.HRegionServer$PeriodicMemstoreFlusher:
regionserver60020.periodicFlusher exiting
> 2014-11-20 14:29:48,915 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting
for Split Thread to finish...
> 2014-11-20 14:29:48,915 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting
for Merge Thread to finish...
> 2014-11-20 14:29:48,915 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting
for Large Compaction Thread to finish...
> 2014-11-20 14:29:48,916 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting
for Small Compaction Thread to finish...
> 2014-11-20 14:29:48,920 INFO  [regionserver60020] client.HConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x349cd39697a00e6
> 2014-11-20 14:29:48,922 INFO  [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread
shut down
> 2014-11-20 14:29:48,922 INFO  [regionserver60020] zookeeper.ZooKeeper: Session: 0x349cd39697a00e6
closed
> 2014-11-20 14:29:48,922 INFO  [regionserver60020.leaseChecker] regionserver.Leases: regionserver60020.leaseChecker
closing leases
> 2014-11-20 14:29:48,922 INFO  [regionserver60020.leaseChecker] regionserver.Leases: regionserver60020.leaseChecker
closed leases
> 2014-11-20 14:29:48,926 INFO  [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread
shut down
> 2014-11-20 14:29:48,926 INFO  [regionserver60020] zookeeper.ZooKeeper: Session: 0x349cd39697a00e5
closed
> 2014-11-20 14:29:48,926 INFO  [regionserver60020] regionserver.HRegionServer: stopping
server dn01.abc.com,60020,1416493546924; zookeeper connection closed.
> 2014-11-20 14:29:48,926 INFO  [regionserver60020] regionserver.HRegionServer: regionserver60020
exiting
> 2014-11-20 14:29:48,926 ERROR [main] regionserver.HRegionServerCommandLine: Region server
exiting
> java.lang.RuntimeException: HRegionServer Aborted
>         at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
>         at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2422)
> 2014-11-20 14:29:48,928 INFO  [Thread-9] regionserver.ShutdownHook: Shutdown hook starting;
hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@3e44f2a5
> 2014-11-20 14:29:48,928 INFO  [Thread-9] regionserver.ShutdownHook: Starting fs shutdown
hook thread.
> 2014-11-20 14:29:48,930 INFO  [Thread-9] regionserver.ShutdownHook: Shutdown hook finished.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message