phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (PHOENIX-1735) Insert data in hbase through phoenix JDBC when restart hbase service
Date Sun, 24 Jan 2016 01:45:39 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

James Taylor resolved PHOENIX-1735.
-----------------------------------
    Resolution: Cannot Reproduce

Please reopen if the problem persists.

> Insert data in hbase through phoenix JDBC when restart hbase service
> --------------------------------------------------------------------
>
>                 Key: PHOENIX-1735
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-1735
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.2.0
>         Environment: linux os ( 64G ram,1.5T disk,16 cores) * 4
> Hadoop 2.6.0
> HBase 0.98.6
> Phoenix 4.2.0
>            Reporter: wubiandelvbo
>            Priority: Critical
>              Labels: Region, assigned:, become, for, in, it, still, to, transition,,
util.HBaseFsckRepair:, waiting
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> use four clients put data in hbase through phoenix-JDBC when restart hbase service。I
discover hbase break down.
> use:
> hbase hbck 
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table CARD_INDEX_SALT
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table SYSTEM.CATALOG
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TZJT_CAMERA_SALT
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table T
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:11:59,933 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TT
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,026 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TT3
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,026 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table hbase:namespace
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table TABLENAME
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table SYSTEM.SEQUENCE
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table SYSTEM.STATS
> ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo
and region dir in hdfs to plug the hole.
> 2015-03-14 10:12:00,027 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel.
set hbasefsck.overlap.merge.parallel to false to run serially.
> ERROR: Found inconsistency in table LOCATION_INDEX_SALT
> 2015-03-14 10:12:01,248 INFO  [main] zookeeper.ZooKeeper: Initiating client connection,
connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=30000 watcher=hbase Fsck,
quorum=hadoop1:2181,hadoop2:2181,hadoop3:2181, baseZNode=/hbase-unsecure
> 2015-03-14 10:12:01,248 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase
Fsck connecting to ZooKeeper ensemble=hadoop1:2181,hadoop2:2181,hadoop3:2181
> 2015-03-14 10:12:01,249 INFO  [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Opening
socket connection to server hadoop1/10.165.97.7:2181. Will not attempt to authenticate using
SASL (unknown error)
> 2015-03-14 10:12:01,249 INFO  [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Socket
connection established to hadoop1/10.165.97.7:2181, initiating session
> 2015-03-14 10:12:01,252 INFO  [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Session
establishment complete on server hadoop1/10.165.97.7:2181, sessionid = 0x14c1337679e00f9,
negotiated timeout = 30000
> Summary:
>   IDX_CC is okay.
>     Number of regions: 0
>     Deployed on: 
>   CARD_INDEX is okay.
>     Number of regions: 0
>     Deployed on: 
>   US_POPULATION is okay.
>     Number of regions: 1
>     Deployed on:  hadoop4,60020,1426296975241
>   EXAMPLE2 is okay.
>     Number of regions: 1
>     Deployed on:  hadoop2,60020,1426296977835
> Table SKIPDEMO is inconsistent.
>     Number of regions: 9
>     Deployed on:  hadoop3,60020,1426296976617 hadoop4,60020,1426296975241
>   STUDENT is okay.
>     Number of regions: 0
>     Deployed on: 
>   LOCATION_INDEX is okay.
>     Number of regions: 0
>     Deployed on: 
>   TZJT_CAMERA is okay.
>     Number of regions: 0
>     Deployed on: 
>   hbase:meta is okay.
>     Number of regions: 1
>     Deployed on:  hadoop2,60020,1426296977835
>   CARD_INDEX_SALT is okay.
>     Number of regions: 0
>     Deployed on: 
>   SYSTEM.CATALOG is okay.
>     Number of regions: 0
>     Deployed on: 
>   TZJT_CAMERA_SALT is okay.
>     Number of regions: 0
>     Deployed on: 
>   T is okay.
>     Number of regions: 0
>     Deployed on: 
>   TT is okay.
>     Number of regions: 0
>     Deployed on: 
>   TT3 is okay.
>     Number of regions: 0
>     Deployed on: 
>   hbase:namespace is okay.
>     Number of regions: 0
>     Deployed on: 
>   TABLENAME is okay.
>     Number of regions: 0
>     Deployed on: 
>   SYSTEM.SEQUENCE is okay.
>     Number of regions: 0
>     Deployed on: 
>   SYSTEM.STATS is okay.
>     Number of regions: 0
>     Deployed on: 
>   LOCATION_INDEX_SALT is okay.
>     Number of regions: 0
>     Deployed on: 
> 462 inconsistencies detected.
> Status: INCONSISTENT
> 2015-03-14 10:12:01,255 INFO  [main] client.ConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
> so,use 'hbase hbck -fixMeta -fixAssignments hbase:namespace ','hbase hbck -repair hbase:namespace'
 show:
> 2015-03-14 10:18:14,988 INFO  [main] util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME =>
'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY
=> ''}
> 2015-03-14 10:18:15,994 INFO  [main] util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME =>
'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY
=> ''}
> 2015-03-14 10:18:17,000 INFO  [main] util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME =>
'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY
=> ''}
> 2015-03-14 10:18:18,005 INFO  [main] util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME =>
'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY
=> ''}
> 2015-03-14 10:18:19,011 INFO  [main] util.HBaseFsckRepair: Region still in transition,
waiting for it to become assigned: {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME =>
'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY
=> ''}
> 2015-03-14 10:18:24,041 INFO  [main] client.ConnectionManager$HConnectionImplementation:
Closing master protocol: MasterService
> 2015-03-14 10:18:24,041 INFO  [main] client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x24c15ce67b80047
> 2015-03-14 10:18:24,045 INFO  [main] zookeeper.ZooKeeper: Session: 0x24c15ce67b80047
closed
> 2015-03-14 10:18:24,045 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut
down
> 2015-03-14 10:18:24,045 DEBUG [main] ipc.RpcClient: Stopping rpc client
> Exception in thread "main" java.io.IOException: Region {ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10,
NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY =>
'', ENDKEY => ''} failed to move out of transition within timeout 120000ms
>         at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:141)
>         at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1798)
>         at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1969)
>         at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1579)
>         at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:473)
>         at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:492)
>         at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4213)
>         at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4022)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>         at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4010)
> ------------------------------
> the hbase still not normal.
> I don't know why ?
> Is bug ?
> My operational problems?
> Can someone help me
> Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message