phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wubiandelvbo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (PHOENIX-1735) Insert data in hbase through phoenix JDBC when restart hbase service
Date Sat, 14 Mar 2015 02:37:38 GMT
wubiandelvbo created PHOENIX-1735:
-------------------------------------

             Summary: Insert data in hbase through phoenix JDBC when restart hbase service
                 Key: PHOENIX-1735
                 URL: https://issues.apache.org/jira/browse/PHOENIX-1735
             Project: Phoenix
          Issue Type: Bug
    Affects Versions: 4.2
         Environment: linux os ( 64G ram,1.5T disk,16 cores) * 4
Hadoop 2.6.0
HBase 0.98.6
Phoenix 4.2.0
            Reporter: wubiandelvbo
            Priority: Critical


use four clients put data in hbase through phoenix-JDBC when restart hbase service。I discover
hbase break down.
use:
hbase hbck 

ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table CARD_INDEX_SALT
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table SYSTEM.CATALOG
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TZJT_CAMERA_SALT
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table T
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TT
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,026 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TT3
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,026 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table hbase:namespace
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TABLENAME
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table SYSTEM.SEQUENCE
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table SYSTEM.STATS
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table LOCATION_INDEX_SALT
2015-03-14 10:12:01,248 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181
sessionTimeout=30000 watcher=hbase Fsck, quorum=hadoop1:2181,hadoop2:2181,hadoop3:2181, baseZNode=/hbase-unsecure
2015-03-14 10:12:01,248 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase
Fsck connecting to ZooKeeper ensemble=hadoop1:2181,hadoop2:2181,hadoop3:2181
2015-03-14 10:12:01,249 INFO  [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Opening
socket connection to server hadoop1/10.165.97.7:2181. Will not attempt to authenticate using
SASL (unknown error)
2015-03-14 10:12:01,249 INFO  [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Socket
connection established to hadoop1/10.165.97.7:2181, initiating session
2015-03-14 10:12:01,252 INFO  [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Session
establishment complete on server hadoop1/10.165.97.7:2181, sessionid = 0x14c1337679e00f9,
negotiated timeout = 30000
Summary:
  IDX_CC is okay.
    Number of regions: 0
    Deployed on: 
  CARD_INDEX is okay.
    Number of regions: 0
    Deployed on: 
  US_POPULATION is okay.
    Number of regions: 1
    Deployed on:  hadoop4,60020,1426296975241
  EXAMPLE2 is okay.
    Number of regions: 1
    Deployed on:  hadoop2,60020,1426296977835
Table SKIPDEMO is inconsistent.
    Number of regions: 9
    Deployed on:  hadoop3,60020,1426296976617 hadoop4,60020,1426296975241
  STUDENT is okay.
    Number of regions: 0
    Deployed on: 
  LOCATION_INDEX is okay.
    Number of regions: 0
    Deployed on: 
  TZJT_CAMERA is okay.
    Number of regions: 0
    Deployed on: 
  hbase:meta is okay.
    Number of regions: 1
    Deployed on:  hadoop2,60020,1426296977835
  CARD_INDEX_SALT is okay.
    Number of regions: 0
    Deployed on: 
  SYSTEM.CATALOG is okay.
    Number of regions: 0
    Deployed on: 
  TZJT_CAMERA_SALT is okay.
    Number of regions: 0
    Deployed on: 
  T is okay.
    Number of regions: 0
    Deployed on: 
  TT is okay.
    Number of regions: 0
    Deployed on: 
  TT3 is okay.
    Number of regions: 0
    Deployed on: 
  hbase:namespace is okay.
    Number of regions: 0
    Deployed on: 
  TABLENAME is okay.
    Number of regions: 0
    Deployed on: 
  SYSTEM.SEQUENCE is okay.
    Number of regions: 0
    Deployed on: 
  SYSTEM.STATS is okay.
    Number of regions: 0
    Deployed on: 
  LOCATION_INDEX_SALT is okay.
    Number of regions: 0
    Deployed on: 
462 inconsistencies detected.
Status: INCONSISTENT
2015-03-14 10:12:01,255 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing
master protocol: MasterService

so,use 'hbase hbck -fixMeta -fixAssignments hbase:namespace ','hbase hbck -repair hbase:namespace'
 show:

2015-03-14 10:18:14,988 INFO  [main] util.HBaseFsckRepair: Region still in transition, waiting
for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.',
STARTKEY => '', ENDKEY => ''}
2015-03-14 10:18:15,994 INFO  [main] util.HBaseFsckRepair: Region still in transition, waiting
for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.',
STARTKEY => '', ENDKEY => ''}
2015-03-14 10:18:17,000 INFO  [main] util.HBaseFsckRepair: Region still in transition, waiting
for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.',
STARTKEY => '', ENDKEY => ''}
2015-03-14 10:18:18,005 INFO  [main] util.HBaseFsckRepair: Region still in transition, waiting
for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.',
STARTKEY => '', ENDKEY => ''}
2015-03-14 10:18:19,011 INFO  [main] util.HBaseFsckRepair: Region still in transition, waiting
for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.',
STARTKEY => '', ENDKEY => ''}
2015-03-14 10:18:24,041 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing
master protocol: MasterService
2015-03-14 10:18:24,041 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing
zookeeper sessionid=0x24c15ce67b80047
2015-03-14 10:18:24,045 INFO  [main] zookeeper.ZooKeeper: Session: 0x24c15ce67b80047 closed
2015-03-14 10:18:24,045 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-03-14 10:18:24,045 DEBUG [main] ipc.RpcClient: Stopping rpc client
Exception in thread "main" java.io.IOException: Region {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10,
NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY =>
'', ENDKEY => ''} failed to move out of transition within timeout 120000ms
        at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:141)
        at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1798)
        at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1969)
        at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1579)
        at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:473)
        at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:492)
        at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4213)
        at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4022)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4010)

------------------------------
the hbase still not normal.
I don't know why ?
Is bug ?
My operational problems?
Can someone help me
Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message