accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dlmarion <dlmar...@comcast.net>
Subject Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?
Date Thu, 31 Jul 2014 02:06:46 GMT
Thats fair. We could modify the upgrade code directly. 



<div>-------- Original message --------</div><div>From: Sean Busbey <busbey@cloudera.com>
</div><div>Date:07/30/2014  9:35 PM  (GMT-05:00) </div><div>To: "dev@accumulo
apache. org" <dev@accumulo.apache.org> </div><div>Subject: Re: [DISCUSS]
Should we support upgrading 1.4 -> 1.6 w/o going through 1.5? </div><div>
</div>On Wed, Jul 30, 2014 at 5:55 PM, <dlmarion@comcast.net> wrote:

> Please add a check that instance.dfs.uri, although deprecated, is
> configured in the upgrade and that instance.volumes is NOT configured, see
> the discussion in ACCUMULO-3006.
>
>

Dave, how is that change related to the changes in ACCUMULO-2988?

I'm all for adding that check. However, it's something we should be doing
for all 1.6 upgrades; it isn't a part of adding teh 1.4 -> 1.6 path.

-Sean




> -----Original Message-----
> From: Mike Drob [mailto:madrob@cloudera.com]
> Sent: Wednesday, July 30, 2014 6:12 PM
> To: Accumulo Dev List
> Subject: Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
> through 1.5?
>
> The upgrade test looks something like:
>
> * Start up 1.4 (or 1.5)
> * Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom
> Filter On/Off, Block Cache On/Off)
> * Load some data into these tables, enough such that flushes and
> compactions start to occur.
> * Abruptly kill all of the servers. This ensures that there are WALs
> around. Unfortunately, we can't have any compactions in progress at this
> point, since that causes a Fate Operation, and that would prevent the
> upgrade from completing.
> * Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
> * Check for data loss
> * Trigger some compactions and wait for those to finish. In a previous
> iteration, we discovered an issue where the relative path'd files and WALs
> were not being properly deleted from !0.
> * Check for data loss.
> * Restart the tablet servers to force tablets to reload (test for orphaned
> files).
> * Check for data loss.
>
> We did not try using custom iterators.
>
> Mike
>
>
> On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ctubbsii@apache.org> wrote:
>
> > Do you have a list of test cases you've tried?
> > (like with/without walogs, custom iterators on tables, delete markers
> > for the metadata table, etc.)?
> >
> > If we could consolidate some of those into a comprehensive list,
> > that'd be pretty useful, I'd think, to give confidence that we didn't
> > miss anything in the review. It might also help focus testing for
> > 1.6.1 release after you've pushed.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> >
> > On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <busbey@cloudera.com>
> wrote:
> >
> > > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> > >
> > > https://reviews.apache.org/r/23413/
> > >
> > > Pending any additional concerns, I'll be pushing this soon.
> > >
> > >
> > > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <busbey@cloudera.com>
> > wrote:
> > >
> > > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > > 1.6.1-SNAPSHOT branch is now up:
> > > >
> > > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > > * https://reviews.apache.org/r/23413/
> > > >
> > > >
> > > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ctubbsii@apache.org>
> > > wrote:
> > > >
> > > >> Cool. Thanks, Sean!
> > > >>
> > > >>
> > > >> --
> > > >> Christopher L Tubbs II
> > > >> http://gravatar.com/ctubbsii
> > > >>
> > > >>
> > > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey
> > > >> <busbey@cloudera.com>
> > > wrote:
> > > >>
> > > >> > I'll be creating a ticket and posting a patch this week.
> > > >> >
> > > >> >
> > > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher
> > > >> > <ctubbsii@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > So, just to revisit this conversation, it seems like there
is
> > > >> interest in
> > > >> > > supporting this. Is there already a ticket for it and/or
> > > >> > > somebody interested in doing the necessary work for 1.6.1?
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Christopher L Tubbs II
> > > >> > > http://gravatar.com/ctubbsii
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob
> > > >> > > <madrob@cloudera.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs
to HDFS
> > > >> > > > (ACCUMULO-2770), start 1.6
> > > >> > > >
> > > >> > > > Mike
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > > drew.farris@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Mike,
> > > >> > > > >
> > > >> > > > > So works just like upgrading from 1.5?
> > > >> > > > >
> > > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > >> > > > >
> > > >> > > > > That sounds entirely reasonable.
> > > >> > > > >
> > > >> > > > > Drew
> > > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob"
> > > >> > > > > <madrob@cloudera.com>
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > We initially tried to set it up as a stand-alone
> > > >> > > > > > utility but
> > > >> > > eventually
> > > >> > > > > > gave up. In order to properly do the upgrade,
you
> > concurrently
> > > >> need
> > > >> > > to
> > > >> > > > > run
> > > >> > > > > > whatever upgrade code concurrently with a
tablet server
> > > hosting
> > > >> > > > !METADATA
> > > >> > > > > > and a tablet server that can replay WALs.
We ended up
> > > >> duplicating a
> > > >> > > lot
> > > >> > > > > of
> > > >> > > > > > logic already present in master before scrapping
that
> plan.
> > An
> > > >> > > > > alternative
> > > >> > > > > > would have been to try to build on MAC, but
that was
> > > >> > > > > > also
> > > >> > non-trivial
> > > >> > > > to
> > > >> > > > > > deploy, so we spliced the code into the existing
> > > >> > > > > > upgrade
> > path.
> > > >> How
> > > >> > do
> > > >> > > > you
> > > >> > > > > > feel about that, Drew?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris
<
> > > >> > drew.farris@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I'm +1 for a utility that would allow
us to go
> > > >> > > > > > > directly
> > from
> > > >> 1.4
> > > >> > to
> > > >> > > > > 1.6.
> > > >> > > > > > >
> > > >> > > > > > > In terms of a general policy, I suggest
we make this
> > > >> > > > > > > sort
> > of
> > > >> > > decision
> > > >> > > > > on
> > > >> > > > > > a
> > > >> > > > > > > case by case basis. My unreasonably
self-centered
> > intuition
> > > >> > > suggests
> > > >> > > > > that
> > > >> > > > > > > there may be some folks that want to
go from 1.4 to
> > > >> > > > > > > 1.6
> > now
> > > >> due
> > > >> > to
> > > >> > > a
> > > >> > > > > > > relatively short 1.5 cycle. The need
to jump multiple
> > > versions
> > > >> > like
> > > >> > > > > might
> > > >> > > > > > > not exist in the future.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean
Busbey <
> > > >> > busbey@cloudera.com>
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > In an effort to get more users
off of our now
> > unsupported
> > > >> 1.4
> > > >> > > > > release,
> > > >> > > > > > > > should we support upgrading directly
to 1.6 without
> > going
> > > >> > > through a
> > > >> > > > > 1.5
> > > >> > > > > > > > upgrade?
> > > >> > > > > > > >
> > > >> > > > > > > > More directly for those on user@:
would you be more
> > > likely
> > > >> to
> > > >> > > > > upgrade
> > > >> > > > > > > off
> > > >> > > > > > > > of 1.4 if you could do so directly
to 1.6?
> > > >> > > > > > > >
> > > >> > > > > > > > We have this working locally at
Cloudera as a part
> > > >> > > > > > > > of
> > our
> > > >> CDH
> > > >> > > > > > integration
> > > >> > > > > > > > (we shipped 1.4 and we're planning
to ship 1.6 next).
> > > >> > > > > > > >
> > > >> > > > > > > > We can get into implementation
details on a jira if
> > > there's
> > > >> > > > positive
> > > >> > > > > > > > consensus, but the changes weren't
very complicated.
> > > They're
> > > >> > > mostly
> > > >> > > > > > > >
> > > >> > > > > > > > * forward porting and consolidating
some upgrade
> > > >> > > > > > > > code
> > > >> > > > > > > > * additions to the README for instructions
> > > >> > > > > > > >
> > > >> > > > > > > > Personally, I can see the both
sides of the
> > > >> > > > > > > > argument. On
> > > the
> > > >> > plus
> > > >> > > > > side,
> > > >> > > > > > > > anything to get more users off
of 1.4 is a good thing.
> > On
> > > >> the
> > > >> > > > > negative
> > > >> > > > > > > > side, it means we have the 1.4
related upgrade code
> > > sitting
> > > >> in
> > > >> > a
> > > >> > > > > > > supported
> > > >> > > > > > > > code branch longer.
> > > >> > > > > > > >
> > > >> > > > > > > > Thoughts?
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Sean
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Sean
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>


-- 
Sean
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message