Mailing-List: contact users-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@qpid.apache.org
MIME-Version: 1.0
Date: Tue, 23 Feb 2016 20:06:42 -0800
Message-ID: 
 <CANVjZ_TW_=BMqzGa1u7UQKDKSFZaU1qWLvqA666hGA5T+x7vJg@mail.gmail.com>
Subject: Qpid Subversion reorganization proposal
From: Justin Ross <justin.ross@gmail.com>
To: "users@qpid.apache.org" <users@qpid.apache.org>
Content-Type: multipart/alternative; boundary=089e01227e849395a2052c7c316c

--089e01227e849395a2052c7c316c
Content-Type: text/plain; charset=UTF-8

Hi, everyone.  I've been preparing a proof of concept for the reorg, and
it's reached a stage where it's ready for comment.

    https://github.com/ssorj/qpid-svn-reorg

Look here for an idea of how the Qpid subversion top level would appear
after these changes:

    https://github.com/ssorj/qpid-svn-reorg/tree/trunk/qpid

I have remaining work to get the C++ tests in shape, but the overall form
of things is what I wish to propose.  I would appreciate your comments.
This is a functioning alternate source organization, so you can download it
and test it directly.

So you can make comments inline, I've appended the markdown README below.
I will produce revised proposals after incorporating your input.

Thanks,
Justin

---

# Qpid Subversion reorganization

## Overview

This proof of concept represents an effort to achieve the source tree
layout proposed [here][1].  It allows the Qpid project to produce
independent releases of Qpid C++ and Python as well as other modules
that have heretofore been bundled into one large Qpid release.

[1]:
https://cwiki.apache.org/confluence/display/qpid/Source+tree+layout+proposal

### End-goal top-level codebases and source artifacts

    New at the top level

    qpid-cpp        qpid/cpp/trunk            qpid-cpp-VERSION.tar.gz
    qpid-java-qmf   qpid/java-qmf/trunk       qpid-java-qmf-VERSION.tar.gz
    qpid-python     qpid/python/trunk         qpid-python-VERSION.tar.gz
    qpid-wcf        qpid/wcf/trunk            qpid-wcf-VERSION.tar.gz
    saslwrapper     qpid/saslwrapper/trunk    saslwrapper-VERSION.tar.gz

    Existing

    qpid-java       qpid/java/trunk           qpid-java-VERSION.tar.gz

    Existing and migrated to Git

    qpid-dispatch   qpid-dispatch.git         qpid-dispatch-VERSION.tar.gz
    qpid-jms        qpid-jms.git              apache-qpid-jms-VERSION.tar.gz
    qpid-proton     qpid-proton.git           qpid-proton-VERSION.tar.gz

    No source releases

    qpid-maven      qpid/maven/trunk          [NA]
    qpid-site       qpid/site                 [NA]
    qpid-specs      qpid/specs                [NA]

In the current proposal, qpid-java-qmf stands apart from qpid-java,
but it may make more sense for java-qmf to take a place inside of
qpid-java.

## Dependencies

    qpid-cpp        depends on              qpid-proton for amqp 1.0 support
    qpid-cpp        depends on              qpid-python for its tests
    qpid-dispatch   depends on              qpid-proton
    qpid-java       depends on              qpid-python for its tests
    qpid-java-qmf   depends on              qpid-java
    qpid-jms        depends on              qpid-proton
    qpid-python     optionally depends on   saslwrapper
    qpid-wcf        depends on              qpid-cpp

## Changes in the Subversion tree

Note that the proof of concept does not in fact move the new top-level
codebases into the standard Subversion structure with trunk, branches,
and tags.  In order to minimize diffs, they instead remain in their
current locations, but updated to remove cross-tree source
dependencies.

### Relocations

Migrate qpid/trunk/qpid/tools python and ruby content to
qpid/cpp/management:

    qpid/trunk/tools/setup.py                 ->
qpid/cpp/trunk/management/setup.py
    qpid/trunk/tools/MANIFEST.in              ->
qpid/cpp/trunk/management/MANIFEST.in
    qpid/trunk/tools/*.txt                    ->
qpid/cpp/trunk/management/python/*.txt
    qpid/trunk/tools/src/py/$libs             ->
qpid/cpp/trunk/management/python/lib/$libs
    qpid/trunk/extras/qmf/src/py/qmf          ->
qpid/cpp/trunk/management/python/lib/qmf
    qpid/trunk/tools/src/py/$tools            ->
qpid/cpp/trunk/management/python/bin/$tools
    qpid/trunk/tools/src/ruby/qpid_management ->
qpid/cpp/trunk/management/ruby/qpid_management

Migrate qpid/trunk/qpid/tools java content to qpid/java-qmf:

    qpid/trunk/tools/src/java                 -> qpid/java-qmf/trunk

Migrate qpid/trunk/qpid/tests to qpid/python:

    qpid/trunk/tests/src/py/qpid_tests        ->
qpid/python/trunk/qpid_tests

Migrate Windows packaging scripts to qpid/cpp:

    qpid/trunk/qpid/packaging/windows         ->
qpid/cpp/trunk/packaging/windows

Migrate other self-contained components to the top level:

    qpid/trunk/qpid/extras/sasl               -> qpid/saslwrapper/trunk
    qpid/trunk/qpid/wcf                       -> qpid/wcf/trunk

### Deletions

Obsolete:

    qpid/trunk/qpid/QPID_VERSION.txt          # No longer makes sense
    qpid/trunk/qpid/LICENSE                   # No longer makes sense
    qpid/trunk/qpid/NOTICE                    # No longer makes sense
    qpid/trunk/qpid/extras/dispatch           # Obsolete stub
    qpid/trunk/qpid/doc/website               # Obsolete stub

After migration:

    qpid/trunk/qpid/doc/book                  # Moved under
qpid/cpp/trunk/docs
    qpid/trunk/qpid/extras/qmf                # Moved under
qpid/cpp/trunk/management/python

Apparently no longer in use.  I'm judging "in use" by whether there
have been any changes--not an excellent metric.  If you are using any
of these, please inform me so that I preserve them.

    qpid/trunk/qpid/review                    # Not touched since 2009
    qpid/trunk/qpid/sandbox                   # Not touched since 2010
    qpid/trunk/qpid/bin                       # Not touched since 2012
    qpid/trunk/qpid/buildtools                # Not touched since 2010
    qpid/trunk/qpid/doc/dev-readme            # Not touched since 2010
    qpid/trunk/qpid/etc                       # Not touched since 2008

## C++ subtree changes

Relocate some docs:

    cpp/design_docs           -> cpp/docs/design
    cpp/docs/src/*            -> cpp/docs/design
    cpp/DESIGN                -> cpp/docs/design/overview.txt

    cpp/AMQP_1.0              -> cpp/docs/amqp-1.0.txt
    cpp/SSL                   -> cpp/docs/ssl.txt
    cpp/README-HA.txt         -> cpp/docs/ha.txt

Collect winsdk stuff:

    cpp/bld-winsdk.ps1        -> cpp/packaging/winsdk/bld-winsdk.ps1
    cpp/README-winsdk.ps1     -> cpp/packaging/winsdk/README.txt

Clean up names and extensions:

    cpp/QPID_VERSION.txt      -> cpp/VERSION.txt  # Matching what Dispatch
has
    cpp/NOTICE                -> cpp/NOTICE.txt
    cpp/LICENSE               -> cpp/LICENSE.txt  # Using .txt extension
for these
    cpp/INSTALL               -> cpp/INSTALL.txt  # is widespread practice
now
    cpp/INSTALL-WINDOWS       -> cpp/INSTALL-WINDOWS.txt

## Possibly impacted parties

 - Steve Huston - Windows packaging code (qpid/packaging/windows)
 - Java broker folks - Relocation of Python qpid_tests may affect Java
   broker testing

## How to setup the Qpid Python dependency

    cd qpid/python/trunk
    ./setup.py install --user
    export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages
    export PATH=$HOME/.local/bin:$PATH

## C++ tests

Changes:

 - Run scripts ("run_" scripts) organized by feature area
 - Run scripts are uniformly named
 - Run scripts produce temporary work dirs of the form run_script\_name_XXXX
   - They are removed on test success
 - All run scripts are runnable directly from src/tests build dir
 - Python test env and test broker tools - should be cross platform
   - Permits removal of many scripts that have both bash and
     powershell variants
   - Broker creation is more consistent
   - Broker termination and checking too
 - Modernized bash scripts

Test results after initial relocation:

    The following tests FAILED:
     1 - api_check_qpidtypes (Failed)
     2 - api_check_qpidmessaging (Failed)
    20 - interop_tests (Failed)
    21 - ha_tests (Failed)
    22 - qpidd_qmfv2_tests (Failed)
    23 - interlink_tests (Failed)
    24 - idle_timeout_tests (Failed)
    Errors while running CTest
    Makefile:149: recipe for target 'test' failed
    make: *** [test] Error 8

It was actually worse than this suggests.  Lots of the tests silently
pass (!) if the python testing tools are not available.

Current test results on Fedora 23:

    Total Test time (real) = 2053.14 sec

    The following tests FAILED:
     1 - api_check_qpidtypes (Failed)       [C++11 abi changes]
     2 - api_check_qpidmessaging (Failed)   [C++11 abi changes]
     4 - acl_tests (Failed)                 [valgrind error]
    12 - python_tests (Failed)              ["amqp:precondition-failed: No
target specified!"]
    13 - queue_redirect_tests (Failed)      [valgrind error]
    26 - linearstore_python_tests (Failed)  [valgrind error]
    Errors while running CTest

## Notes

 - Gah! Circular dependency.  Qpid Python (via qpid.testlib) depends
   on qpidtoollibs.  This should be removed: Qpid C++ depends on Qpid
   Python and not the reverse in any way.

 - The new test regime more uniformly runs brokers with valgrind, and
   this exposes new valgrind errors.

 - I removed the following test install utils from src/tests:
   qpid-build-rinstall, install_env.sh, etc.  If these are still
   important, I'd like to bring them back in a subdirectory or a
   different project altogether.  They are not central to the C++
   tests.

## Todo

 - Rewrite the index in qpid/README.txt - Update all the top-level
   info
 - Lots of tedious, infuriating Windows testing
   - Current results at
https://ci.appveyor.com/project/ssorj/qpid-svn-reorg/

--089e01227e849395a2052c7c316c--