hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Kennedy <james.kenn...@troove.net>
Subject How to handle data migration?
Date Thu, 13 Jan 2011 19:37:58 GMT
I'm currently validating the new 0.90.0 RC3 with the hbase-trx layer and our own application.

All seems well so far except for the fact that I now find that HBase doesn't adapt if I try
to run the same data on different machines.

1) I work from home and generated our seeded test data.
2) Run the test suite and all tests pass
3) I go to the office and re-run the tests.

Result: HMaster fails because the .ROOT data has the wrong ip address for locating the .META.
At least that is my understanding from the stacktrace below.  Note that the
IP address in that trace is the IP from my home network and is incorrect.

This wasn't an issue with previous versions of HBase as far as I've noticed.  And this seems
to be a big data portability fail.
Surely the HMaster should be able to absorb stale metadata and wait for new region-servers
to check in.
Instead it just keels over and dies.
But before logging a case I wanted to know if there was something I'm obviously missing or
doing wrong.

The seeded test data is on HDFS.


[13/01/11 10:58:42] 5939   [           main] INFO  ion.service.HBaseRegionService  - troove>
Starting region server thread.
[13/01/11 11:00:15] 98699  [        HMaster] FATAL he.hadoop.hbase.master.HMaster  - Unhandled
exception. Starting shutdown.
java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready
for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=]
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
	at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
	at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:258)
	at $Proxy15.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:384)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:283)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:478)
	at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:435)
	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:382)
	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
	at java.lang.Thread.run(Thread.java:680)

James Kennedy
Troove Inc.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message