ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Onischuk" <aonis...@hortonworks.com>
Subject Review Request 26745: Decommission of RegionServer failed - HDP 1.3
Date Wed, 15 Oct 2014 12:38:27 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26745/
-----------------------------------------------------------

Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-7787
    https://issues.apache.org/jira/browse/AMBARI-7787


Repository: ambari


Description
-------

**STR:**

  1. Deploy 4-node cluster.
  2. Delete 4th host.
  3. Restart all required services.
  4. Apply 'Decommission' action for RegionServer.

**Result:** Task **Decommission RegionServer - Turn drain mode off** failed with such stderr:
    
    
    2014-10-13 17:37:30,279 - Error while executing command 'decommission':
    Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 122, in execute
        method(env)
      File "/var/lib/ambari-agent/cache/stacks/HDP/1.3.2/services/HBASE/package/scripts/hbase_master.py",
line 66, in decommission
        hbase_decommission(env)
      File "/var/lib/ambari-agent/cache/stacks/HDP/1.3.2/services/HBASE/package/scripts/hbase_decommission.py",
line 67, in hbase_decommission
        logoutput=True
      File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148,
in __init__
        self.env.run()
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
149, in run
        self.run_action(resource, action)
      File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line
115, in run_action
        provider_action()
      File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 237, in action_run
        raise ex
    Fail: Execution of ' /usr/lib/hbase/bin/hbase --config /etc/hbase/conf org.jruby.Main
/usr/lib/hbase/bin/region_mover.rb unload c6401.ambari.apache.org' returned 1. SLF4J: Class
path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/usr/lib/hbase/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-2--1,
built on 04/24/2014 20:09 GMT
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:host.name=c6401.ambari.apache.org
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/jdk64/jdk1.7.0_67/jre
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hbase/conf:/usr/jdk64/jdk1.7.0_67/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.94.6.1.3.7.0-2-security.jar:/usr/lib/hbase/bin/../hbase-0.94.6.1.3.7.0-2-security-tests.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.5.3.jar:/usr/lib/hbase/bin/../lib/avro-ipc-1.5.3.jar:/usr/lib/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/lib/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/lib/hbase/bin/../lib/commons-digester-1.8.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-io-2.1.jar:/usr/lib/hbase/bin/../lib/commons-la
 ng-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-math-2.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-11.0.2.jar:/usr/lib/hbase/bin/../lib/hadoop-core.jar:/usr/lib/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/lib/hbase/bin/../lib/httpclient-4.1.2.jar:/usr/lib/hbase/bin/../lib/httpcore-4.1.3.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.8.8.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.8.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.8.jar:/usr/lib/hbase/bin/../lib/j
 ersey-server-1.8.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6.5.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsr305-1.3.9.jar:/usr/lib/hbase/bin/../lib/junit-4.10-HBASE-1.jar:/usr/lib/hbase/bin/../lib/libthrift-0.8.0.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/metrics-core-2.1.2.jar:/usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.4.0a.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar:/usr/lib/hbase/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/velocity-1.7.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper.jar
 :/usr/lib/hadoop/libexec/../conf:/usr/jdk64/jdk1.7.0_67/lib/tools.jar:/usr/lib/hadoop/libexec/..:/usr/lib/hadoop/libexec/../hadoop-core-1.2.0.1.3.7.0-2.jar:/usr/lib/hadoop/libexec/../lib/ambari-log4j-1.7.0.106.jar:/usr/lib/hadoop/libexec/../lib/asm-3.2.jar:/usr/lib/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/usr/lib/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/usr/lib/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/lib/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/lib/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/lib/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/lib/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/lib/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/lib/had
 oop/libexec/../lib/commons-lang-2.4.jar:/usr/lib/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/lib/hadoop/libexec/../lib/commons-net-3.1.jar:/usr/lib/hadoop/libexec/../lib/core-3.1.1.jar:/usr/lib/hadoop/libexec/../lib/guava-11.0.2.jar:/usr/lib/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.0.1.3.7.0-2.jar:/usr/lib/hadoop/libexec/../lib/hadoop-fairscheduler-1.2.0.1.3.7.0-2.jar:/usr/lib/hadoop/libexec/../lib/hadoop-lzo-0.5.0.jar:/usr/lib/hadoop/libexec/../lib/hadoop-thriftfs-1.2.0.1.3.7.0-2.jar:/usr/lib/hadoop/libexec/../lib/hadoop-tools.jar:/usr/lib/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop/libexec/../lib/jdeb-0.8.ja
 r:/usr/lib/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/lib/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/lib/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/lib/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/lib/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/lib/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/lib/hadoop/libexec/../lib/junit-4.5.jar:/usr/lib/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/lib/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/lib/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/libexec/../lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/lib/hadoop/libexec/../lib/postgresql-9.1-901-1.jdbc4.jar:/usr/lib/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/lib/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop/libexec/../li
 b/jsp-2.1/jsp-api-2.1.jar::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/lib/hadoop-mapreduce/*
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/libexec/../lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.el6.x86_64
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:user.name=hbase
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hbase
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase
    14/10/13 17:37:27 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=c6404.ambari.apache.org:2181,c6403.ambari.apache.org:2181,c6402.ambari.apache.org:2181,c6401.ambari.apache.org:2181
sessionTimeout=60000 watcher=hconnection
    14/10/13 17:37:27 INFO zookeeper.RecoverableZooKeeper: The identifier of this process
is 6906@c6401.ambari.apache.org
    14/10/13 17:37:27 INFO zookeeper.ClientCnxn: Opening socket connection to server c6404.ambari.apache.org/192.168.64.104:2181.
Will not attempt to authenticate using SASL (unknown error)
    14/10/13 17:37:27 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error,
closing socket connection and attempting reconnect
    java.net.ConnectException: Connection refused
    	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
    14/10/13 17:37:27 INFO zookeeper.ClientCnxn: Opening socket connection to server c6401.ambari.apache.org/192.168.64.101:2181.
Will not attempt to authenticate using SASL (unknown error)
    14/10/13 17:37:27 INFO zookeeper.ClientCnxn: Socket connection established to c6401.ambari.apache.org/192.168.64.101:2181,
initiating session
    14/10/13 17:37:27 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
for /hbase-unsecure/hbaseid
    14/10/13 17:37:27 INFO util.RetryCounter: Sleeping 2000ms before retry #1...
    14/10/13 17:37:27 INFO zookeeper.ClientCnxn: Session establishment complete on server
c6401.ambari.apache.org/192.168.64.101:2181, sessionid = 0x1490a6e2d2f0003, negotiated timeout
= 40000
    RuntimeError: Server c6401.ambari.apache.org not online
        stripServer at /usr/lib/hbase/bin/region_mover.rb:212
      unloadRegions at /usr/lib/hbase/bin/region_mover.rb:321
             (root) at /usr/lib/hbase/bin/region_mover.rb:471
    


Diffs
-----

  ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HBASE/package/scripts/hbase_decommission.py
4f7ed9e 
  ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HBASE/package/scripts/params.py
8ab8aff 
  ambari-server/src/test/python/stacks/1.3.2/HBASE/test_hbase_master.py 5df3edb 

Diff: https://reviews.apache.org/r/26745/diff/


Testing
-------

mvn clean test


Thanks,

Andrew Onischuk


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message