drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <tfar...@mapr.com>
Subject Re: Backwards Compatibility Policy
Date Fri, 22 Sep 2017 07:12:52 GMT
Hi Paul,

I've implemented the ZK fix. But I'm uncertain about how to make the options table backwards
compatible. The main issue with the options table is that formerly there was a "type" column.
The "type" column meant different things in different parts of the code, and the meaning was
overall inconsistent. Now the "type" column has been renamed to "accessibleScopes" and was
assigned consistent meaning throughout the code base. As part of that change the possible
values for "accessibleScopes" has changed and how it is assigned those values has changed
as well. Making it truly backward compatible would require emulating the previously bad behavior,
and I'm not sure how useful that would be. It would be possible to revert the name for the
column back to "type", but I think that would have limited usefulness since the value stored
for "type" will be different in almost every case from what it was before, so that would also
break any scripts that intimately depended on it as well. What are your thoughts on how to
handle this?


From: Paul Rogers <progers@mapr.com>
Sent: Thursday, September 21, 2017 5:50:26 PM
To: dev@drill.apache.org
Subject: Re: Backwards Compatibility Policy

Hi Tim,

Unfortunately, Drill has no version compatibility guidelines. We break compatibility all the
time — often by accident. As Drill matures, we should consider defining such a policy.

Experience with other systems suggests that provide an automatic way to handle the inevitable
evolution of APIs, data formats and the like. This is done for user convenience, and (in commercial
products) to avoid support calls.

Would have been great if we had a version number in ZK. Would solve problems not just with
options, but with storage plugins when we change their classes (and hence the JSON stored
in ZK.) But, we don’t…

Vitalii recently added versioning for the Parquet metadata file to avoid the need for users
to delete all the metadata each time we make a change. Would be great if we could follow that
example for other areas.

In the meantime, perhaps you can implement a way to read the old format ZK but write the new

I believe you also changed the layout of the system table for options. These were long-needed
improvements. Still, I wonder if anyone has a script that depends on the old format? Do we
need a way to support the old format, while offering a new table with the new format? Jyothsna
recently did that as part of her work in options; I wonder if something like that is needed
here also? In fact, you may be able to simply alter the table that she added: it hasn’t
see the light of a Drill release yet.


- Paul

> On Sep 21, 2017, at 4:16 PM, Timothy Farkas <tfarkas@mapr.com> wrote:
> Makes sense Abhishek, I'll work on making it backwards compatible then.
> Thanks,
> Tim
> ________________________________
> From: Abhishek Girish <agirish@apache.org>
> Sent: Thursday, September 21, 2017 3:58:55 PM
> To: dev@drill.apache.org
> Subject: Re: Backwards Compatibility Policy
> Hey Tim,
> Requiring users to purge Drill's ZK data is not advisable and we might not
> want to go that route. We need to have a seamless upgrade path - for
> instance modifying values found to be in an older format to the new one,
> without explicit user interaction.
> Regards,
> Abhishek
> On Thu, Sep 21, 2017 at 3:46 PM, Timothy Farkas <tfarkas@mapr.com> wrote:
>> Hi All,
>> I recently made a change to the option system which impacted the fields
>> contained in OptionValues and hence the format of option information we are
>> storing in zookeeper. So it is currently not backward compatible with old
>> system options stored in zookeeper. Two ways to resolve the issue are to
>> require old data to be purged from zookeeper when upgrading the cluster or
>> to attempt to allow backward compatibility by modifying the deserializer
>> for OptionValue. So my question is what is our stance on backwards
>> compatibility?
>> Thanks,
>> Tim

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message