hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject IntegrationTestRecoveryRSLossBetweenInserts output
Date Tue, 09 Jul 2013 00:58:47 GMT
Here is the tail of output from IntegrationTestRecoveryRSLossBetweenInserts

2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/opt/HDP/hadoop-2.1.0-SNAPSHOT/lib/native
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-279.el6.x86_64
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.name=root
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.home=/root
2013-07-08 17:43:01,171 INFO  [main] zookeeper.ZooKeeper: Client
environment:user.dir=/opt/HDP/hbase-0.95.2-SNAPSHOT
2013-07-08 17:43:01,172 INFO  [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=hconnection-0x3ea86d12
2013-07-08 17:43:01,199 INFO  [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x3ea86d12 connecting to ZooKeeper ensemble=
lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:01,206 INFO  [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:01,212 INFO  [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, initiating session
2013-07-08 17:43:01,225 INFO  [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, sessionid =
0x13fbfcea423068e, negotiated timeout = 30000
2013-07-08 17:43:01,401 INFO  [main] mttr.AbstractIntegrationTestRecovery:
Creating table myTableName_1373330579708 with 10 splits.
2013-07-08 17:43:01,423 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,387 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,496 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,605 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,714 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,924 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:06,324 INFO  [main] mttr.AbstractIntegrationTestRecovery:
Pre-split table created successfully in 4923ms
2013-07-08 17:43:06,326 INFO  [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=hconnection-0x244e5056
2013-07-08 17:43:06,327 INFO  [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x244e5056 connecting to ZooKeeper ensemble=
lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:06,328 INFO  [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:06,330 INFO  [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, initiating session
2013-07-08 17:43:06,339 INFO  [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, sessionid =
0x33fbfcea77e069a, negotiated timeout = 30000
2013-07-08 17:43:06,354 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:06,401 INFO  [main] mttr.AbstractIntegrationTestRecovery:
Moving regions of table myTableName_1373330579708 away from
lvsdpehdc25dn0406.stratus.lvs.ebay.com,60020,1373316676163
2013-07-08 17:43:06,405 INFO  [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=catalogtracker-on-hconnection-0x3ea86d12
2013-07-08 17:43:06,406 INFO  [main] zookeeper.RecoverableZooKeeper:
Process identifier=catalogtracker-on-hconnection-0x3ea86d12 connecting to
ZooKeeper ensemble=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:06,406 DEBUG [main] catalog.CatalogTracker: Starting
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6af37a62
2013-07-08 17:43:06,407 INFO  [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:06,408 INFO  [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, initiating session
2013-07-08 17:43:06,416 INFO  [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, sessionid =
0x33fbfcea77e069b, negotiated timeout = 30000
2013-07-08 17:43:06,423 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,
2013-07-08 17:43:06,432 DEBUG [main] catalog.CatalogTracker: Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6af37a62
2013-07-08 17:43:06,441 INFO  [main] zookeeper.ZooKeeper: Session:
0x33fbfcea77e069b closed
2013-07-08 17:43:06,442 INFO  [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2013-07-08 17:43:06,734 INFO  [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=catalogtracker-on-hconnection-0x3ea86d12
2013-07-08 17:43:06,736 INFO  [main] zookeeper.RecoverableZooKeeper:
Process identifier=catalogtracker-on-hconnection-0x3ea86d12 connecting to
ZooKeeper ensemble=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:06,736 DEBUG [main] catalog.CatalogTracker: Starting
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1e406b09
2013-07-08 17:43:06,736 INFO  [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:06,737 INFO  [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, initiating session
2013-07-08 17:43:06,744 INFO  [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, sessionid =
0x13fbfcea423068f, negotiated timeout = 30000
2013-07-08 17:43:06,753 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,
2013-07-08 17:43:06,764 DEBUG [main] catalog.CatalogTracker: Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1e406b09
2013-07-08 17:43:06,772 INFO  [main] zookeeper.ZooKeeper: Session:
0x13fbfcea423068f closed
2013-07-08 17:43:06,772 INFO  [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2013-07-08 17:43:06,803 INFO  [main] mttr.AbstractIntegrationTestRecovery:
Moved 0 regions of table myTableName_1373330579708 away from
lvsdpehdc25dn0406.stratus.lvs.ebay.com,60020,1373316676163
2013-07-08 17:43:06,806 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:07,184 INFO  [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -A INPUT -i lo -j ACCEPT;sudo /sbin/iptables
-A OUTPUT -o lo -j ACCEPT;sudo /sbin/iptables -A INPUT -p tcp -s 0/0 -d
lvsdpehdc25dn0615.stratus.lvs.ebay.com --sport 513:65535 --dport 22 -m
state --state NEW,ESTABLISHED -j ACCEPT;sudo /sbin/iptables -A OUTPUT -p
tcp -s lvsdpehdc25dn0615.stratus.lvs.ebay.com -d 0/0 --sport 22 --dport
513:65535 -m state --state ESTABLISHED -j ACCEPT;sudo /sbin/iptables -P
INPUT DROP;sudo /sbin/iptables -P OUTPUT DROP;sudo /sbin/iptables -P
FORWARD DROP; , hostname:lvsdpehdc25dn0615.stratus.lvs.ebay.com
2013-07-08 17:43:07,185 INFO  [main] util.Shell: Executing full command
[/usr/bin/ssh  lvsdpehdc25dn0615.stratus.lvs.ebay.com "sudo /sbin/iptables
-A INPUT -i lo -j ACCEPT;sudo /sbin/iptables -A OUTPUT -o lo -j ACCEPT;sudo
/sbin/iptables -A INPUT -p tcp -s 0/0 -d
lvsdpehdc25dn0615.stratus.lvs.ebay.com --sport 513:65535 --dport 22 -m
state --state NEW,ESTABLISHED -j ACCEPT;sudo /sbin/iptables -A OUTPUT -p
tcp -s lvsdpehdc25dn0615.stratus.lvs.ebay.com -d 0/0 --sport 22 --dport
513:65535 -m state --state ESTABLISHED -j ACCEPT;sudo /sbin/iptables -P
INPUT DROP;sudo /sbin/iptables -P OUTPUT DROP;sudo /sbin/iptables -P
FORWARD DROP;"]
2013-07-08 17:50:38,229 INFO  [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:38,662 INFO  [main] mttr.AbstractIntegrationTestRecovery:
Detection took: 29
2013-07-08 17:50:38,907 INFO  [main] mttr.AbstractIntegrationTestRecovery:
Regions available after: 245
2013-07-08 17:50:38,909 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:50:38,985 INFO  [main] mttr.AbstractIntegrationTestRecovery:
Total failure fix took: 323
2013-07-08 17:50:38,986 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:50:39,037 INFO  [main]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x33fbfcea77e069a
2013-07-08 17:50:39,053 INFO  [main] zookeeper.ZooKeeper: Session:
0x33fbfcea77e069a closed
2013-07-08 17:50:39,053 INFO  [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2013-07-08 17:50:39,055 INFO  [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0614.stratus.lvs.ebay.com
2013-07-08 17:50:39,056 INFO  [main] util.Shell: Executing full command
[/usr/bin/ssh  lvsdpehdc25dn0614.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,176 INFO  [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,177 INFO  [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0615.stratus.lvs.ebay.com
2013-07-08 17:50:39,177 INFO  [main] util.Shell: Executing full command
[/usr/bin/ssh  lvsdpehdc25dn0615.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,293 INFO  [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,293 INFO  [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0616.stratus.lvs.ebay.com
2013-07-08 17:50:39,293 INFO  [main] util.Shell: Executing full command
[/usr/bin/ssh  lvsdpehdc25dn0616.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,416 INFO  [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,416 INFO  [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0617.stratus.lvs.ebay.com
2013-07-08 17:50:39,416 INFO  [main] util.Shell: Executing full command
[/usr/bin/ssh  lvsdpehdc25dn0617.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,538 INFO  [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,539 INFO  [main] util.PerformanceChecker:  max
expected=45000, actual=29, ok=true
2013-07-08 17:50:39,539 INFO  [main] util.PerformanceChecker:  max
expected=15000, actual=323, ok=true
2013-07-08 17:50:39,541 INFO  [main] client.HBaseAdmin: Started disable of
myTableName_1373330579708
2013-07-08 17:50:39,568 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:39,669 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:39,771 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:39,872 DEBUG [main] client.HBaseAdmin: Sleeping= 200ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:40,074 DEBUG [main] client.HBaseAdmin: Sleeping= 200ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:40,275 DEBUG [main] client.HBaseAdmin: Sleeping= 400ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:40,677 INFO  [main] client.HBaseAdmin: Disabled
myTableName_1373330579708
2013-07-08 17:50:41,232 INFO  [main] client.HBaseAdmin: Deleted
myTableName_1373330579708

Time: 461.624

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message