hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Schless <patr...@tempo-db.com>
Subject Migrating Clusters - Broken Metadata
Date Thu, 05 Jul 2012 23:44:16 GMT
I have an existing hbase cluster (old.domain.com) and I am trying to
migrate the data to a new set of boxes (new.domain.com). Both are running
hbase 0.90.x.

I would like to minimize downtime, so I'm looking at the Backup tool from
mozilla (
http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/)
for doing incremental syncs, then a final sync and fail-over.

I am able to do the sync, but when I try to bring up the new hmaster, I get
errors along the lines of:
2012-07-05 23:26:27,674 ERROR org.apache.hadoop.hbase.HServerAddress: Could
not resolve the DNS name of data02.old.domain.com
2012-07-05 23:26:27,675 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.lang.IllegalArgumentException: hostname can't be null

        at java.net.InetSocketAddress.<init>(InetSocketAddress.java:139)

        at
org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)

        at
org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)

        at
org.apache.hadoop.hbase.catalog.MetaReader.readLocation(MetaReader.java:344)

        at
org.apache.hadoop.hbase.catalog.MetaReader.readMetaLocation(MetaReader.java:286)

        at
org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:338)

        at
org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:540)
        at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:474)

        at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:412)

        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)


Before that error, I do see the new.domain.com region servers register
successfully with the hmaster.

I tried (temporarily) adding the data*.old.domain.com to /etc/hosts, but
that did not fix the issue. [1]

In the comments, the post (linked above) says hbase will figure out that
the region servers have changed, but that doesn't seem to be the case. Is
there something I can do to fix the meta data after copying the files? Is
there some other tool/strategy I should be using for this? I would love to
use the replication, but it's documented as not-production-ready in 0.90 so
I'm avoiding it.

Thanks,
Patrick

[1] with /etc/hosts entries for data*.old.domain.com
2012-07-05 23:41:57,130 INFO org.apache.hadoop.hbase.master.HMaster: -ROOT-
assigned=0, rit=false, location=data02.new.domain.com:60020
2012-07-05 23:41:57,148 INFO org.apache.hadoop.hbase.master.HMaster: .META.
assigned=0, rit=false, location=data02.old.domain.com:60020
2012-07-05 23:41:57,148 INFO org.apache.hadoop.hbase.master.HMaster: Master
startup proceeding: master failover
2012-07-05 23:41:57,150 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.lang.NullPointerException

        at
org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:761)

        at
org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:227)

        at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:425)

        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message