Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@avro.apache.org
Subject: Re: [DISCUSS] Release and code management
To: dev@avro.apache.org
References: <563254F7.30304@apache.org>
 <CAGHyZ6+v4k4FB5jguK=YqXvzZqdfKRDThe7wKXWULQ-Jnb1qfg@mail.gmail.com>
 <56326347.4030604@cloudera.com>
 <CAGHyZ6J6AZjsOD-a6WtFOYw0wzdOo0oR_s+MTrbDAX+Y_h1j2w@mail.gmail.com>
 <563265E4.6050107@cloudera.com>
 <CAETy=eVRGHvyVKDfj+vox9A+Df-JV8diG4PxC=i-hs_SYAz-QA@mail.gmail.com>
 <56339CDA.5060200@cloudera.com> <563B954C.2050503@cloudera.com>
 <CAETy=eWcoFDU=f73KWRuJob_wxr2vmTfJHhO6TXfVVWQ8MBAAg@mail.gmail.com>
 <CAGHyZ6KT=yLv6KzJ9H1QrDDnh7Y-V87wFhRf4_KVkoJ-T=_kbA@mail.gmail.com>
 <563BA3AD.3010403@cloudera.com>
 <CAF-WD4QWo+Nuv5du6UCM0ZT0FYxaAezZ0e+TxRrdrR5LaoVyXg@mail.gmail.com>
 <5644CD9D.8090107@cloudera.com>
 <CADoiZqp9f3FeeWbeQ9ywrv0kN4q6iHv3600cooHK69_xsNLJvQ@mail.gmail.com>
 <985793206.1965493.1447690214503.JavaMail.yahoo@mail.yahoo.com>
From: Ryan Blue <blue@cloudera.com>
Message-ID: <564A3AEB.8050200@cloudera.com>
Date: Mon, 16 Nov 2015 12:22:03 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <985793206.1965493.1447690214503.JavaMail.yahoo@mail.yahoo.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

On 11/16/2015 08:10 AM, Sam Groth wrote:
> So I have developed for a similar scenario where we had APIs in 3 different languages that needed to be kept in sync. The releases were split, and we only had some high level tests to verify compatibility and therefore had to be very careful what changes we made to avoid failures across many different use cases of our APIs. There were a few people who were required to review any changes to this API. I feel that in a project with as many implementations as Avro, it will be difficult to have this same level of review. If it's possible to write some linked tests for most of the implementations like Niels suggested, it should be ok to split the releases. We would still have to pay close attention to changes in these test cases.
>
> Sam

I agree with the need to validate that files are to spec, but we 
currently have that problem regardless of whether we release components 
as a group or individually. Right now we don't do much cross-language 
testing at all so I think this should be a goal (an important one!), not 
a requirement for changing our releases.

>       On Saturday, November 14, 2015 7:54 AM, Niels Basjes <Niels@basjes.nl> wrote:
>
>
>   Hi,
>
> First of all a +1 from me to move to git.

Sounds like we have consensus around moving to git, so at least we can 
get that started.

> Regarding the "how many parts and the release cycle";
>
> In general I'm in the "release often" camp.
> Yet I understand the needless confusion if a certain language has not
> changed.
>
> How about this idea:
>
> We create a separate project (avro-spec) with the specification/testcases
> and such that guard the language specification. All language specific
> implementations must reference this (perhaps by using the git feature of a
> "linked submodule") and run the tests during the language specific deploy.
> The version of this module therefor the version of the Avro format
> specification.

This is similar to what we do in Parquet: we have a parquet-format 
project that has meaningful versions for file compatibility. The version 
for this dependency doesn't necessarily determine the version of a file 
because the API can choose what underlying version it uses.

> I think it would be good if these modules use a version that essentially is
> <avro-spec version>-<library version>
>
> This way the libraries can have a separate release cycle and still clearly
> indicate the supported Avro specifcation.

I think this makes sense. Right now, while we have only one version of 
the file format, do we need to include the format version in all version 
numbers? We could put off that step when we have more than one format 
version to worry about.

rb


-- 
Ryan Blue
Software Engineer
Cloudera, Inc.