couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Shorin <kxe...@gmail.com>
Subject Re: On dependency management and CI issues associated with it
Date Wed, 13 Apr 2016 06:35:05 GMT
Hi Robert,

Point about flattening to a single repository is valid: in the end, we
have our apps repos in broken state all the time as they are not
declare their decencies. So noone can pick fabric@master and run it -
he'll spend quite a lot of time to figure the deps of the right
versions. But the idea to solve the problem by reducing set of
repositories we have to test is good.


Hi Iliya,

I have alternative solution for you:

- Turn-off Travis CI everywhere where we cannot be sure about testing
without depended PRs (all except third-party modules, fauxton, docs,
and few more independent projects like couch-epi);
- Require everyone to submit additional PR to apache/couchdb repo with
commit hashes update;
- On this apache/couchdb PR we'll run CI testing;
- If you rebase/update any of your subcomponent PRs you must update
commit hash on apache/couchdb one;

Pros:
- We won't forget to update rebar.config when new changes lands;
- We will always run complete integration testing with all the right
deps states;
- We won't have to invent any complicated integration solutions to
deal with sub-repos testing;
- No new new steps/files/work introduced, so there is no need to care
about learning curve;

Cons:
- Need to be a bit tricky on Travis builder to realize on which remote
(fork) new rebar.config hashes are to correctly checkout them, though
that is not a rocket science since we have access to git information
there.

The Jenkins CI role here is to ensure that we have master build right
and releases build right, on the various OSes.

Sounds simpler and better for me, how it does for you?

--
,,,^..^,,,


On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson
<rnewson@apache.org> wrote:
> I'd like us to instead consider flattening to a single repository. I've found no value
and only pain from the multiple repositories approach (43 in total!).
>
> The contention is that multiple repositories enforces application boundaries (low coupling
/ high cohesion) but I've not felt that in reality. We don't, and couldn't meaningfully, release
any of our components separately, and, as Ilya makes clear, many enhancements require changes
to multiple repositories, and we break this into multiple commits, losing the ability to look
at an enhancement in toto.
>
> If what Ilya is proposing is the solution, I think it's the solution to a problem we
should not have.
>
> B.
>
>> On 12 Apr 2016, at 16:22, Ilya Khlopotov <iilyak@ca.ibm.com> wrote:
>>
>>
>>
>> Dear community,
>>
>>
>> There is a problem with contributors workflow which renders our CI system
>> useless. As you might know couchdb project consists of multiple
>> repositories. Most of the time changes cross the repositories boundaries.
>> When this happens the push to any of the repositories causes CI failures.
>> CI fails since it uses the old version of dependencies from main repository
>> of the project. Here is what we can do about it.
>>
>> # Proposal
>>
>> Let's use multiple files for dependency management.
>>
>> - deps.json - serves the same purpose as dependencies list from current
>> rebar.config.script
>> - proposed.deps.json - here we specify list of PRs we want to commit
>> atomically
>> - override.deps.json - local file outside of version control which we
>> consult in order to include development tools specific to contributor (code
>> reloader, debugger, tracer, profiler, binpp, ...)
>>
>> Bellow is the example of a content of these files:
>>
>> ## deps.json
>> {
>>    "src/b64url": [
>>        "https://github.com/apache/couchdb-b64url",
>>        "6895652d80f95cdf04efb14625abed868998f174"
>>    ],
>>    "src/cassim": [
>>        "https://github.com/apache/couchdb-cassim",
>>        "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
>>    ],
>>    "src/chttpd": [
>>        "https://github.com/apache/couchdb-chttpd",
>>        "54e8f6147486d9afc5245e0143d15a4dd1185654"
>>    ],
>>    "src/meck": [
>>        "https://github.com/apache/couchdb-meck",
>>        "tree/0.8.2"
>>    ],
>> ....
>> }
>>
>> ## proposed.deps.json
>> {
>>    "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
>>    "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
>>    "src/couch_tests": [
>>        "https://github.com/apache/couchdb-erlang-tests",
>>        "tree/branch"
>>    ],
>> }
>>
>> # Interface
>>
>> I propose to write a simple CLI tool to work with this structure. Bellow is
>> a list of commands which we need to support (for minimal version)
>>
>> ## Adding new dependency
>>
>> git propose add https://github.com/apache/couchdb-foo
>> a2d5ad2eedc960248b806f61df0a1009462bdb46
>> git propose add https://github.com/apache/couchdb-bar tree/branch_name
>>
>> ## Adding new PR to the change set
>>
>> git propose add https://github.com/apache/couchdb-config/pull/4
>>
>> ## Checking out right dependencies
>>
>> git propose checkout
>>
>> ## Checking out release
>>
>> git propose checkout --release # this would ignore proposed.deps.json if it
>> exists
>>
>> ## Merge the change
>>
>> This command would do the following:
>> - Parse proposed.deps.json
>> - Retrieve merge commit sha for every PR (exit if dependency is not merged
>> yet)
>> - Update dependencies in deps.json with correct merge commit sha
>> - remove proposed.deps.json
>>
>> # Workflow
>>
>> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
>> git checkout -b feature-ZZZ
>> cd src/X && hack dependency X
>> cd ../..
>> cd src/Y && hack dependency Y
>> issue PRs for X and Y
>> cd ../..
>> git propose add https://github.com/apache/couchdb-X/pull/4
>> git propose add https://github.com/apache/couchdb-Y/pull/49
>> git add proposed.deps.json
>> git commit -m "Commit feature {something} which does {a thing} and can be
>> tested as {procedure}"
>> git push origin  feature-ZZZ
>> ^ this would trigger our CI
>> CI would do
>> git propose checkout && ./configure && make check
>>
>> # Pros and Cons
>>
>> ## Pros
>>
>> - Changes are merged atomically
>> - CI runs against expected versions of deps
>> - Enables git bisect
>> - Reduce tasks that needs to be done by ASF committer (no need to update
>> rebar.config.script manually)
>> - Simplifies testing of PRs by reviewers
>> - Simplifies rebar.config since rebar is not used for managing deps
>>
>> ## Cons
>>
>> - some github.com specifics (concept of PRs and access to github API to get
>> info about PR)
>> - we need to have github.com as one of the remotes
>> - we trigger CI only on push to main repository
>>
>> # Implementation
>>
>> We write a git-propose script in python and place it in ./bin. We add ./bin
>> into either GIT_EXEC_PATH or PATH. You always can call the script directly
>> (as ./bin/git-propose) if you don't like amending
>> your environment.
>>
>> # Later improvements
>>
>> - We can issue PRs from the tool itself
>> - We can merge from the tool itself
>> - We can implement support for multiple remotes (asf, github, private)
>> - We can implement support for multiple git transports (for the first
>> version all repositories in *.deps.json files would use https://)
>>
>> Sincerely,
>> ILYA KHLOPOTOV
>

Mime
View raw message