Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <A112ABB1-7CD2-4A87-AEEE-2CEE6DBC7D73@gmail.com>
References: <CADf0P4KYk4BWW9da3z+QbjiGM5gmLGGJneynSvg5LDkHEW+_ig@mail.gmail.com>
 <11C87341-D9C2-4331-A1F2-923E2680FAE1@gmail.com> <CADf0P4Jd9eN+L2k3n5bRX38RZink3apKJV-MUZ_mGQnJ9P+ZoA@mail.gmail.com>
 <CALte62z-J=99ipgB6C6CA0KDtoHubLw391B1QFG_JzDsa1r+Fg@mail.gmail.com>
 <CADf0P4KiNgoHirChF8eV77mYpHzDCZ__7WacXJkyJm+Ph-==jQ@mail.gmail.com> <A112ABB1-7CD2-4A87-AEEE-2CEE6DBC7D73@gmail.com>
From: Vasco Pinho <vasco@hotjar.com>
Date: Sun, 16 Apr 2017 09:25:52 +0200
Message-ID: <CADf0P4JAv9zJrqEs2W=Onx9iwSPpcA1nU3O0dTDwNB2Pk=6MTg@mail.gmail.com>
Subject: Re: Hbase exportSnapshot fails in HA missing hbase-prefix-tree.jar
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a11c019da0e1fe2054d439815
archived-at: Sun, 16 Apr 2017 07:26:41 -0000

--001a11c019da0e1fe2054d439815
Content-Type: text/plain; charset=UTF-8

Running Hadoop 2.7.2, Hbase 1.2.4.

On Fri, Apr 14, 2017 at 6:47 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> I can debug this next week when I have access to an HA cluster.
>
> Which release of Hadoop are you using ?
>
> Cheers
>
> > On Apr 14, 2017, at 9:21 AM, Vasco Pinho <vasco@hotjar.com> wrote:
> >
> > I'm not sure I follow. This file already exists under the install dir at:
> >
> > /home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/
> hbase-prefix-tree-1.2.4.jar
> >
> > The class is also present at:
> >
> > /home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/
> classes/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeCodec.class
> >
> > Why would I rename it? Is my classpath wrong for some reason? In any
> case,
> > renaming it to something like
> > "hbase-prefix-tree/target/hbase-prefix-tree-1.2.4.jar.bak " yields
> another
> > exception of the same sort (full trace: https://pastebin.com/mzcPthZR):
> >
> > java.io.FileNotFoundException: File does not exist:
> > hdfs://clusterID/root/.m2/repository/org/apache/htrace/
> htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar
> >
> > So it looks like something else is wrong with the installation or there
> is
> > a bug that is making these searches attempt to look in "hdfs://" instead
> of
> > the filesystem?
> >
> > Thanks,
> > Vasco Pinho
> >
> >> On Fri, Apr 14, 2017 at 4:32 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >> Here is how hbase-prefix-tree dependency is detected:
> >>
> >>    try {
> >>
> >>      prefixTreeCodecClass =
> >>
> >>          Class.forName(
> >> "org.apache.hadoop.hbase.codec.prefixtree.PrefixTreeCodec");
> >>
> >>    } catch (ClassNotFoundException e) {
> >>
> >>      // this will show up in unit tests but should not show in real
> >> deployments
> >>
> >>      LOG.warn("The hbase-prefix-tree module jar containing
> PrefixTreeCodec
> >> is not present." +
> >>
> >>          "  Continuing without it.");
> >>
> >>    }
> >>
> >> As a workaround, consider temporarily renaming hbase-prefix-tree jar on
> the
> >> node where ExportSnapshot is run.
> >>
> >> Remember to rename hbase-prefix-tree jar back after the job.
> >>
> >> FYI
> >>
> >>> On Fri, Apr 14, 2017 at 5:02 AM, Vasco Pinho <vasco@hotjar.com> wrote:
> >>>
> >>> Sure Ted,
> >>>
> >>> Here's the full trace: https://pastebin.com/AGimZ8wZ
> >>> And here's the hbase-site.xml: https://pastebin.com/yhRWCsaU
> >>>
> >>> This is a working Hbase HA setup which has been happily chugging
> along. 3
> >>> node HDFS JN, 3 node zk, 2 of them NN+backup NN. Also two of them
> >>> HMaster+HMaster backup. Then several data nodes.
> >>>
> >>> Thanks for taking a look!
> >>>
> >>> Vasco
> >>>
> >>>> On Fri, Apr 14, 2017 at 1:41 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>>>
> >>>> Can you show the complete stack trace ?
> >>>>
> >>>> Please pastebin contents of hbase/site.xml
> >>>>
> >>>> Thanks
> >>>>
> >>>>> On Apr 14, 2017, at 3:51 AM, Vasco Pinho <vasco@hotjar.com> wrote:
> >>>>>
> >>>>> When running:
> >>>>>
> >>>>> bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> >>>>> -Dfs.s3a.buffer.dir=/tmp/hbase_snap_tmp -snapshot
> >>>> TestTable-20170413-143020
> >>>>> -copy-to s3a://bucket-backup/hbase/snapshots/
> >>>>>
> >>>>>
> >>>>> The operation fails with:
> >>>>>
> >>>>>
> >>>>> 2017-04-13 15:03:24,947 ERROR [main] snapshot.ExportSnapshot:
> >> Snapshot
> >>>>> export failed
> >>>>> java.io.FileNotFoundException: File does not exist:
> >>>>> hdfs://clusterID/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/
> >>>> target/hbase-prefix-tree-1.2.4.jar
> >>>>>       at
> >>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.
> >>>> doCall(DistributedFileSystem.java:1072)
> >>>>>
> >>>>>
> >>>>> I assume it's looking for the
> >>>>> "/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/
> >>>> hbase-prefix-tree-1.2.4.jar"
> >>>>> which does exist but obviously not under hdfs://. I don't know why
> >> this
> >>>>> file is needed or why it's being looked for in the hdfs:// instead of
> >>> the
> >>>>> local filesystem, where it does exist. I confirmed that it looks
> >> under
> >>>>> "fs.defaultFS" + <hbase install path> + <relative path to
> >>>> hbase-prefix-tree
> >>>>> jar", although I have no idea how to fix this. Any ideas?
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Vasco Pinho
> >>
>


-- 
Vasco Pinho
DevOps Engineer
www.hotjar.com
Connect with me on LinkedIn
<https://pt.linkedin.com/in/vasco-pinho-58770534>
IMPORTANT CONFIDENTIALITY NOTICE: This message is confidential and intended
for the use only of the person to whom this message is addressed. If you
are not the intended recipient you are strictly prohibited from reading,
disseminating, copying or using this message, or its contents in any way
and must contact the sender immediately.

--001a11c019da0e1fe2054d439815--