hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: LocalJobRunner and HBASE-2669 woes
Date Fri, 22 Apr 2011 19:27:29 GMT
For HBASE-3777, Karthick and I finally nailed down issues related to
finalizer that made TestTableMapReduce fail.
A final patch would be put up for review :-).

In the end, we expect user to use (better tuned) API wisely.
We will add more javadoc for HTable and the new HConnectionKey class.

Take a look at the new TestTableMapReduce code for recommended usage of
HTable.

Cheers

On Wed, Apr 20, 2011 at 7:20 PM, Robert Mahfoud <robert.mahfoud@gmail.com>wrote:

> On Wed, Apr 20, 2011 at 12:38 PM, Stack <stack@duboce.net> wrote:
>
> > On Wed, Apr 20, 2011 at 12:18 PM, Robert Mahfoud
> > <rmahfoud@mediosystems.com> wrote:
> > > I'm upgrading our code base now to HBase 0.90.1 (CDH3-U0) from
> > 0.89.something.
> > > I'm facing a very hard time tracking failures in our unit tests and
> > working around them. Here's the situation:
> > >
> > > 1- Many of our unit tests run in-process M/R jobs using the
> > LocalJobRunner.
> > > 2- We often do things before the M/R jobs and things afte the M/R to
> > validate the outcome.
> > > 3- With HBASE-2669 adding the HCM.deleteAllConnections(true) to the
> > TableOutputFormat.close() method, any connections that were open before
> the
> > M/R job will be closed.
> >
> >
> > This came up last week.  There the user made a copy of TOF because
> > this.table is private preventing changing how TOF#close works.  If we
> > added an accessor for the table to TOF so you could override its close
> > behavior, would that work for you?
> >
> >
> Thanks for the suggestion. I'm going to try to do that and add a
> configuration flag to the copied class. I will look for the update that
> makes this.table protected.
> I think that this wasn't a wise design choice since one wouldn't expect
> using an "incidental" class (TOF) to have such a pervasive side effect.
>
> Adding a configuration flag could also be a quick and easy fix until a
> global solution emerges (maybe as part of HBASE-3766 or HBASE-37777).
>
>
> >
> > > Some issues I'm facing are much more subtle and harder to work around:
> > e.g. it's impossible to run 2 M/R jobs in parallel (at least one of them
> > using TableOutputFormat) because as soon as the first TableOutputFormat
> > calls HCM.deleteAllConnections(...) the other's connections will be
> closed.
> > >
> >
> > Yes.
> >
> > > The questions that I have are the following:
> > > 1- Is there any simpler way to do the same thing?
> > > 2- Are there any best-practices that we are missing, and that could
> help
> > with this issue?
> >
> > We changed TOF.  It looks like messed up the case where folks are
> > running multiple MR jobs all up in the one jvm context.
> >
> >
> > > 3- Nothing turns up when we google "HBASE-2669 AND LocalJobRunner" and
> > very few things come up when we google "HBASE-2669" alone. Do you know of
> > anyone else facing a similar problem?
> > > 4- Should there be a configuration flag that turns off this behavior
> when
> > using the LocalJobRunner?
> > >
> >
> > See above.
> >
> > Sorry for the inconvenience.
> >
> > St.Ack
> >
>
>
>
> --
> Robert Mahfoud
> http://www.1apple1day.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message