hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Verkuylen <...@verkuylen.net>
Subject Re: distcp hbase-0.94.6(CDH4.5) hfiles to hbase1.2(CDH5.8)?
Date Tue, 16 Aug 2016 18:27:52 GMT
Thnx, yes confirmed and thats the way we recovered all of the non
calculated data. Thnx for the help.

As to the reason why: It seems to be a combination of network congestion
because of tcp ingress overflows leading to a lot of connections in the
cluster failing of hdfs and hbase. This and too many bulk loads generating
too small hfiles which triggering compactions leading to compaction queues
growing and not having time to settle down. We had fairly high network
traffic even on a cluster not under that much load. We've moved to a
different ETL flow on the new cluster, better optimizing loads into the
cluster. Some network improvements also seemed to help out on the frame
drops.

On Tue, Aug 16, 2016 at 1:25 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> HFile v2 would be saved in v3 format after major compaction in the 1.2
> cluster.
>
> > On Aug 16, 2016, at 12:55 AM, Rob V <robby.verkuylen@gmail.com> wrote:
> >
> > The hfile tool shows the hfiles from the 0.94 cluster are indeed v2.
> Does that allow a auto convert to v3 on the 1.2 cluster after a compaction?
> >
> > The Masters are down on the 0.94 so I can't use the hbase shell.
> >
> >
> >> On 15 Aug 2016, at 20:01, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >> Please verify that your 0.94 cluster is configured with hfile v2.
> >> Config hfile.format.version should have value of 2.
> >>
> >> To obtain region boundaries, you can use 'describe' hbase shell command.
> >>
> >> Cheers
> >>
> >>> On Mon, Aug 15, 2016 at 9:48 AM, Rob Verkuylen <rob@verkuylen.net>
> wrote:
> >>>
> >>> The 2nd cluster is an operational cluster, so option 2 seems to fit
> nicely.
> >>>
> >>> Will the LoadIncrementalHFiles on the Hbase1.2 cluster be able to
> process
> >>> the 0.94 version Hfiles?
> >>>
> >>> Also is there a convenient way to get the region splits from the old
> >>> hfiles, or is going though them the only option?
> >>>
> >>>> On Sun, Aug 14, 2016 at 9:00 PM, Jerry He <jerryjch@gmail.com>
wrote:
> >>>>
> >>>> If you distcp the raw hfiles, you have a couple of options to restore
> the
> >>>> data on the second cluster.
> >>>>
> >>>> 1. You an copy the entire hbase root.dir, you can set the hbase
> root.dir
> >>> to
> >>>> this directory and bootstrap the new cluster from there.
> >>>>   Before you start the new cluster, run the 'hbase upgrade' on the
> >>> hbase
> >>>> root.dir.
> >>>>   Refer to "13.1.3. Upgrading to 1.0 from 0.94"
> >>>> https://hbase.apache.org/book.html#_upgrade_paths
> >>>>
> >>>> 2.  You can run the bulk load tool to load the copied hfiles into the
> new
> >>>> cluster.
> >>>>    Refer to "130.12. CompleteBulkLoad" in the HBase Reference book.
> >>>>    You can write a program or script to go through all the region
> >>>> directories.
> >>>>
> >>>> Option 1 is more delicate, but as you said the old hdfs was fine, it
> >>> should
> >>>> work for you.
> >>>> For option 2, pre-split the tables on the new cluster to match the
> region
> >>>> boundaries of the old tables.
> >>>>
> >>>> Jerry
> >>>>
> >>>>> On Sun, Aug 14, 2016 at 8:08 AM, Ted Yu <yuzhihong@gmail.com>
wrote:
> >>>>>
> >>>>> For the Import tool, you can specify the following (quoted from
> usage):
> >>>>>
> >>>>>   System.err.println("To import data exported from HBase 0.94, use");
> >>>>>
> >>>>>   System.err.println("  -Dhbase.import.version=0.94");
> >>>>>
> >>>>> FYI
> >>>>>
> >>>>>> On Sun, Aug 14, 2016 at 12:09 AM, Rob Verkuylen <rob@verkuylen.net>
> >>>>> wrote:
> >>>>>
> >>>>>> We're recovering from a crash of hbase-0.94.6 where the master
> >>> refuses
> >>>> to
> >>>>>> come up, while hdfs is fine. Why this happened exactly is something
> >>>> we're
> >>>>>> looking into.
> >>>>>>
> >>>>>> In the meantime we're looking to migrate as fast as possible
to
> >>> another
> >>>>>> cluster running hbase1.2.
> >>>>>>
> >>>>>> Since we cannot use the default import tool anymore. If we distcp
> the
> >>>> raw
> >>>>>> hfiles from old to new cluster, is there a way to safely
> >>> load/bulkload
> >>>>>> these into the hbase1.2 cluster?
> >>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message