Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAL5zq9YywjnXzxCk_Qt7kn=wFS0wgJh7CB3ViZ0jyOUPsXZaFw@mail.gmail.com>
References: <qhp6c85n4l15wip8m697bj2k.1418267001999@email.android.com>
	<CAL5zq9aYHMwFwF9XcrwiBs65KtoHpM8hgbzpuqVNG10UoUBX4g@mail.gmail.com>
	<999388760.2391882.1418303736974.JavaMail.zimbra@comcast.net>
	<CAL5zq9a=iFv_t2yy-QftiPFSLUjYK7DhVrPOUcqF8-wKNhzQVQ@mail.gmail.com>
	<1323315543.2591786.1418313150535.JavaMail.zimbra@comcast.net>
	<CAL5zq9YrT5p9S7tdSthMfoSA5Jwvj2didA5myFa0En=_EeKNbA@mail.gmail.com>
	<1461027396.2650793.1418315608834.JavaMail.zimbra@comcast.net>
	<CAL5zq9YywjnXzxCk_Qt7kn=wFS0wgJh7CB3ViZ0jyOUPsXZaFw@mail.gmail.com>
Date: Fri, 19 Dec 2014 17:37:30 -0500
Message-ID: 
 <CAL5zq9arX_AKdaqqce5x0HVBuxoh8DibkF5YvCAAm1zeLPMg7A@mail.gmail.com>
Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
From: Christopher <ctubbsii@apache.org>
To: Accumulo Dev List <dev@accumulo.apache.org>
Content-Type: multipart/alternative; boundary=089e013a1a209e3cfc050a995ad3

--089e013a1a209e3cfc050a995ad3
Content-Type: text/plain; charset=UTF-8

I spoke with Keith a bit offline, and he suggested a different path:

Have an intermediate release which requires use of instance.volumes,
instead of allowing users to continue using the old properties. Use the old
properties for resolving old relative paths, and re-write them on upgrade.

This solution creates a boundary for upgrades, though. One could only
(safely) upgrade to a version without these properties, from a version
where after this metadata upgrade had been forced. So, if we did this in
1.7.0, and dropped the old properties in 2.0 (for example), we could not
upgrade from 1.6 to 2.0 (unless a user was certain they didn't have any
relative paths left to resolve).


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii

On Thu, Dec 18, 2014 at 12:42 PM, Christopher <ctubbsii@apache.org> wrote:

> Talked with Dave a bit offline. I think I've convinced myself that
> whenever we do drop instance.dfs.{uri,dir}, what we can do when we see a
> relative path is resolve it with the first item in instance.volumes instead
> of all the complicated logic of resolving using instance.dfs.{uri,dir} with
> fallback to the Hadoop default fs. I think this will provide a reasonable
> user experience and resolve all the annoyingly confusing logic of relative
> path resolution today.
>
> We can also spit out a warning when we see a relative path, which displays
> instructions for compacting the file away. We could actually introduce this
> warning sooner, like say 1.7.0, rather than wait until we drop the old
> properties.
>
> So, in summary, I think:
>
> 1.7.0: continue current complicated resolution of old properties, warn
> about not using instance.volumes, warn with compact instructions when
> resolving relative paths
>
> 2.0.0: drop old properties and resolve using first item in
> instance.volumes, require instance.volumes to have at least one volume,
> warn with compact instructions when resolving relative paths
>
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
> On Thu, Dec 11, 2014 at 11:33 AM, <dlmarion@comcast.net> wrote:
>>
>> I think we crossed the streams. I'll talk to you offline.
>>
>>
>> ----- Original Message -----
>>
>> From: "Christopher" <ctubbsii@apache.org>
>> To: "Accumulo Dev List" <dev@accumulo.apache.org>
>> Sent: Thursday, December 11, 2014 11:27:48 AM
>> Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
>>
>> On Thu, Dec 11, 2014 at 10:52 AM, <dlmarion@comcast.net> wrote:
>>
>> > "Making it required is annoying if users don't have relative paths."
>> >
>> > But this property is used to determine the location of new files.....
>> >
>> >
>> No, it's not. Not if you're using instance.volumes. It should only be used
>> for resolving old files in that case.
>>
>>
>> > ----- Original Message -----
>> >
>> > From: "Christopher" <ctubbsii@apache.org>
>> > To: "Accumulo Dev List" <dev@accumulo.apache.org>
>> > Sent: Thursday, December 11, 2014 10:03:56 AM
>> > Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
>> >
>> > Well, no, the database will start if we rely on instance.volumes, but we
>> > may be unable to find files that have relative paths, if we incorrectly
>> > assume /accumulo. Making it required is annoying if users don't have
>> > relative paths.
>> >
>> >
>> > --
>> > Christopher L Tubbs II
>> > http://gravatar.com/ctubbsii
>> >
>> > On Thu, Dec 11, 2014 at 8:15 AM, <dlmarion@comcast.net> wrote:
>> >
>> > > How so? If someone upgrades from another version and is using a
>> different
>> > > dir, doesn't specify it in the configuration, and we assume /accumulo,
>> > then
>> > > their database won't start. The other option, which may be safer than
>> > > making any assumptions, is to make instance.volumes a required
>> parameter
>> > > with no defaults.
>> > >
>> > > ----- Original Message -----
>> > >
>> > > From: "Christopher" <ctubbsii@apache.org>
>> > > To: "Accumulo Dev List" <dev@accumulo.apache.org>
>> > > Sent: Wednesday, December 10, 2014 11:51:39 PM
>> > > Subject: Re: Planning for (eventual) removal of instance.dfs.{uri,dir}
>> > >
>> > > The URI is probably reasonable, but the dir is potentially
>> problematic if
>> > > you weren't using the default.
>> > >
>> > >
>> > > --
>> > > Christopher L Tubbs II
>> > > http://gravatar.com/ctubbsii
>> > >
>> > > On Wed, Dec 10, 2014 at 10:03 PM, dlmarion <dlmarion@comcast.net>
>> wrote:
>> > >
>> > > > Looks like VolumeConfiguration falls back to fs.defaultFS for the
>> uri
>> > and
>> > > > /accumulo for the dir. You could remove both properties and still
>> keep
>> > > this
>> > > > as the documented default behavior if instance.volumes is not
>> > specified.
>> > > >
>> > > >
>> > > >
>> > > > <div>-------- Original message --------</div><div>From: Christopher
>> <
>> > > > ctubbsii@apache.org> </div><div>Date:12/10/2014 9:13 PM (GMT-05:00)
>> > > > </div><div>To: Accumulo Dev List <dev@accumulo.apache.org>
>> > > > </div><div>Cc: </div><div>Subject: Re: Planning for (eventual)
>> removal
>> > of
>> > > > instance.dfs.{uri,dir} </div><div>
>> > > > </div>My ACCUMULO-2589 branch in github (
>> > > > https://github.com/ctubbsii/accumulo/tree/ACCUMULO-2589) does have
>> a
>> > > > commit
>> > > > that drops a bunch of stuff (which may or may not be accepted as is
>> for
>> > > > 2.0). The instance.dfs.{uri,dir} properties aren't so easy and
>> require
>> > > more
>> > > > planning, because it's not just removing a property... it's also
>> > dealing
>> > > > with updating internal data by making relative paths absolute.
>> > > >
>> > > > For what it's worth, I'm also looking at what changes if we drop
>> > Hadoop 1
>> > > > support.
>> > > >
>> > > > As for the validation of config, I think we do a sanity check on
>> > startup
>> > > > already, which we can always extend. Doesn't solve this issue,
>> though.
>> > > >
>> > > >
>> > > > --
>> > > > Christopher L Tubbs II
>> > > > http://gravatar.com/ctubbsii
>> > > >
>> > > > On Wed, Dec 10, 2014 at 8:59 PM, dlmarion <dlmarion@comcast.net>
>> > wrote:
>> > > >
>> > > > > We should schedule a bunch of deprecated things for removal in
>> 2.0 to
>> > > > ease
>> > > > > maintenance. Do we have a way to validate the site.xml and
>> zookeeper
>> > > > > settings before startup and fail with appropriate error message.
>> > > > >
>> > > > >
>> > > > >
>> > > > > <div>-------- Original message --------</div><div>From:
>> Christopher <
>> > > > > ctubbsii@apache.org> </div><div>Date:12/10/2014 8:44 PM
>> (GMT-05:00)
>> > > > > </div><div>To: Accumulo Dev List <dev@accumulo.apache.org>
>> > > > > </div><div>Cc: </div><div>Subject: Planning for (eventual)
>> removal of
>> > > > > instance.dfs.{uri,dir} </div><div>
>> > > > > </div>So,
>> > > > >
>> > > > > instance.volumes replaces instance.dfs.uri and instance.dfs.dir in
>> > 1.6.
>> > > > > But, what's our long-term plan for these? I ask, because we still
>> > have
>> > > > > internal code that uses instance.dfs.uri to resolve relative
>> paths.
>> > > > >
>> > > > > Should we force these to be re-written at some point (maybe on
>> > upgrade
>> > > to
>> > > > > 1.7)? Should we continue to support the deprecated properties
>> > > > indefinitely
>> > > > > and continue the lazy re-write-on-compact? Do we transition by
>> > > requiring
>> > > > > instance.volumes to specify the volumes, and only use the old
>> > > properties
>> > > > to
>> > > > > resolve relative paths?
>> > > > >
>> > > > > My personal view is that we should provide an upgrade-prep/check
>> tool
>> > > > prior
>> > > > > to upgrade to 2.0, which checks and/or re-writes paths and
>> verifies
>> > > that
>> > > > > instance.volumes is set.
>> > > > >
>> > > > > Does anybody have a different opinion on this?
>> > > > >
>> > > > > --
>> > > > > Christopher L Tubbs II
>> > > > > http://gravatar.com/ctubbsii
>> > > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>

--089e013a1a209e3cfc050a995ad3--