couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilya Khlopotov" <iil...@ca.ibm.com>
Subject On dependency management and CI issues associated with it
Date Tue, 12 Apr 2016 15:22:14 GMT


Dear community,


There is a problem with contributors workflow which renders our CI system
useless. As you might know couchdb project consists of multiple
repositories. Most of the time changes cross the repositories boundaries.
When this happens the push to any of the repositories causes CI failures.
CI fails since it uses the old version of dependencies from main repository
of the project. Here is what we can do about it.

# Proposal

Let's use multiple files for dependency management.

 - deps.json - serves the same purpose as dependencies list from current
rebar.config.script
 - proposed.deps.json - here we specify list of PRs we want to commit
atomically
 - override.deps.json - local file outside of version control which we
consult in order to include development tools specific to contributor (code
reloader, debugger, tracer, profiler, binpp, ...)

Bellow is the example of a content of these files:

## deps.json
{
    "src/b64url": [
        "https://github.com/apache/couchdb-b64url",
        "6895652d80f95cdf04efb14625abed868998f174"
    ],
    "src/cassim": [
        "https://github.com/apache/couchdb-cassim",
        "9bbfe82125284fa7cb3317079e8bc1dc876a07bf"
    ],
    "src/chttpd": [
        "https://github.com/apache/couchdb-chttpd",
        "54e8f6147486d9afc5245e0143d15a4dd1185654"
    ],
    "src/meck": [
        "https://github.com/apache/couchdb-meck",
        "tree/0.8.2"
    ],
....
}

## proposed.deps.json
{
    "src/couch": "https://github.com/apache/couchdb-couch/pull/124",
    "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"
    "src/couch_tests": [
        "https://github.com/apache/couchdb-erlang-tests",
        "tree/branch"
    ],
}

# Interface

I propose to write a simple CLI tool to work with this structure. Bellow is
a list of commands which we need to support (for minimal version)

## Adding new dependency

git propose add https://github.com/apache/couchdb-foo
a2d5ad2eedc960248b806f61df0a1009462bdb46
git propose add https://github.com/apache/couchdb-bar tree/branch_name

## Adding new PR to the change set

git propose add https://github.com/apache/couchdb-config/pull/4

## Checking out right dependencies

git propose checkout

## Checking out release

git propose checkout --release # this would ignore proposed.deps.json if it
exists

## Merge the change

This command would do the following:
- Parse proposed.deps.json
- Retrieve merge commit sha for every PR (exit if dependency is not merged
yet)
- Update dependencies in deps.json with correct merge commit sha
- remove proposed.deps.json

# Workflow

export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv`
git checkout -b feature-ZZZ
cd src/X && hack dependency X
cd ../..
cd src/Y && hack dependency Y
issue PRs for X and Y
cd ../..
git propose add https://github.com/apache/couchdb-X/pull/4
git propose add https://github.com/apache/couchdb-Y/pull/49
git add proposed.deps.json
git commit -m "Commit feature {something} which does {a thing} and can be
tested as {procedure}"
git push origin  feature-ZZZ
^ this would trigger our CI
CI would do
git propose checkout && ./configure && make check

# Pros and Cons

## Pros

- Changes are merged atomically
- CI runs against expected versions of deps
- Enables git bisect
- Reduce tasks that needs to be done by ASF committer (no need to update
rebar.config.script manually)
- Simplifies testing of PRs by reviewers
- Simplifies rebar.config since rebar is not used for managing deps

## Cons

- some github.com specifics (concept of PRs and access to github API to get
info about PR)
- we need to have github.com as one of the remotes
- we trigger CI only on push to main repository

# Implementation

We write a git-propose script in python and place it in ./bin. We add ./bin
into either GIT_EXEC_PATH or PATH. You always can call the script directly
(as ./bin/git-propose) if you don't like amending
your environment.

# Later improvements

- We can issue PRs from the tool itself
- We can merge from the tool itself
- We can implement support for multiple remotes (asf, github, private)
- We can implement support for multiple git transports (for the first
version all repositories in *.deps.json files would use https://)

Sincerely,
ILYA KHLOPOTOV

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message