Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CA+Y+4444bFFe6DHPUtwLnUJs9B-qrj0w9+6EG=FVPK5L-6_eHg@mail.gmail.com>
References: 
 <CA+96GG4mmZcFfAthaO7zCShwpAU7Wn3+RDC9JCKKP7JzS2DULA@mail.gmail.com>
	<CA+Y+4444bFFe6DHPUtwLnUJs9B-qrj0w9+6EG=FVPK5L-6_eHg@mail.gmail.com>
Date: Fri, 12 Oct 2012 10:25:43 -0400
Message-ID: 
 <CA+96GG7PJPyJ3URXCetPFG8Q0gr_TUs40ZS=i=sM=AMTwv77=w@mail.gmail.com>
Subject: Re: [VOTE] Apache Cloudstack 4.0.0-incubating Release, first round
From: Chip Childers <chip.childers@sungard.com>
To: cloudstack-dev@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Noah,

The level of detail and attention that you gave this is wonderful.  I
really want to thank you for this.

Comments inline:

On Fri, Oct 12, 2012 at 4:46 AM, Noah Slater <nslater@tumbolia.org> wrote:
> Okay, stepping through the test procedure.
> *
> *
> *> Download DevCloud image from
> http://download.cloud.com/templates/devcloud/DevCloud.ova into your machine.
> *
>
> We should add a warning that this is a 1.5GB image!

Done!

>
> Who maintains this? Who WILL maintain this?

The current image we are pointing people to was created by Edison
(IIRC, manually).

We are working on tools to automatically generate the image in
tools/devcloud/*.  This includes a Veewee install config, Vagrant
configs and puppet configs.  The community (specifically folks from
@basho.com) are working to make this all cleaner right now, by
building off the work that Edison, myself and others (Adam /
Sebastian) started.

That being said, it's still valid for us to use this as a test target
for ease of use.

> *> Open VirtualBox, File->Import Appliance->choose DevCloud.ova*
>
> The license screen lists the license terms as:
>
> "*Apache License 2.0*"

Actually, it shouldn't include that line at all.  We aren't able to
license the whole thing, since it's a full OS!  However, this isn't a
release artifact, so it can be ignored for the purpose of evaluating
the release.

> That's it. Should this include the license text instead?
>
> *> Install your required development tools (if not already available)*
>
> Where? On my local machine? Or into the running DevCloud image?
>
> I am assuming my local machine for now.
>
> *> sudo aptitude install openjdk-6-jdk ant*
>
> I've Googled around, and I can't find any obvious way to install this on OS
> X 10.8. I found a link to the official Java SE 6 JDK, but I don't know
> enough about Java to know whether this would be fine, or whether I need the
> OpenJDK.

For Mac, you can skip that.

> Looking at the rest of the instructions, I think I'm going to have a hard
> time installing all this without some guidance.
>
> Is there anyone on the ML who has this set up on OS X 10.8? Can you add
> instructions for getting set up?

I use OSX - I'll try to edit the instructions with the appropriate Mac
Ports packages that are actually needed.  I'm going to assume that the
user already knows how to use mac ports (there's only so far I'm
willing to go with the instructions).  Alternatively, I considered
just changing that whole section to a list of the packages that the
user has to have installed, and assume that they figure it out for
their own platform...  but I thought better of that.  A real example
is more useful, and I'm happy to try to remember the steps for Mac for
you.

> Skipping this section because I cannot complete it without further
> instruction.
>
> *> mkdir -p ~/cloudstack; cd ~/cloudstack*
>
> I would use something under /tmp for safety.

Done!

> I would also include a step to blow away any previous dir. (You'd be
> surprised how often people will just try to do a second round of voting
> from the same dir, and files get mixed up and confused.)

Done!

> *> Download the artifacts:*
>
> We are exporting a `zip` file and a `tar.gz` file. Which means we have have
> two source artefacts to vote on. I've not encountered this before. I think
> this means that I have to test both of them separately, which is a burden.
> Can we just pick one archive format and vote on that? Realistically, who is
> going to be installing from source anyway? Do we really need the `zip`
> format here?

Perhaps we don't need the zip format here.

I was following examples like:
http://maven.apache.org/download.html
http://ant.apache.org/srcdownload.cgi

However, I see that Hadoop only does tar.gz (well, that, plus RPM and
DEB packages, which we're not doing officially via ASF infra).

I'll start a thread specifically about this.

> The `zip` file is 16M, compared to 10M for the `tar.gz` file. Seems odd.

It's not odd.  They are different compression ratios.

> *> Import keys: *
>
> We should mention that the user import from the KEYS file first, and
> suggest importing from a public key server second.

Done.

> *>  gpg --verify apache-cloudstack-4.0.0-incubating-src.tar.gz.asc*
>
> Good sig.
>
> (We perhaps need to indicate what output people are looking for here. This
> is a crucial step.)
>
> *> gpg --verify apache-cloudstack-4.0.0-incubating-src.zip.asc*
>
> Good sig.
>
> (We perhaps need to indicate what output people are looking for here. This
> is a crucial step.)
>
> *> gpg --print-md MD5 apache-cloudstack-4.0.0-incubating-src.tar.gz | diff
> - apache-cloudstack-4.0.0-incubating-src.tar.gz.md5*
> *> gpg --print-md MD5 apache-cloudstack-4.0.0-incubating-src.zip | diff -
> apache-cloudstack-4.0.0-incubating-src.zip.md5*
> *> gpg --print-md SHA512 apache-cloudstack-4.0.0-incubating-src.tar.gz |
> diff - apache-cloudstack-4.0.0-incubating-src.tar.gz.sha*
> *> gpg --print-md SHA512 apache-cloudstack-4.0.0-incubating-src.zip | diff
> - apache-cloudstack-4.0.0-incubating-src.zip.sha*
>
> No output. With my user hat on, I don't know whether this was successful or
> not.
>
> `md5sum` and `sha1sum` might be simpler ways of verifying the hashes, but
> this would require the hashes to be generated with these tools as well.
>
> Consider this example:
>
> $ md5sum apache-cloudstack-4.0.0-incubating-src.tar.gz >
> apache-cloudstack-4.0.0-incubating-src.tar.gz.md5.example
> $ md5sum --check apache-cloudstack-4.0.0-incubating-src.tar.gz.md5.example
> apache-cloudstack-4.0.0-incubating-src.tar.gz: OK

I added instructions explaining that you are looking for no output
from the diffs.  I'm going to leave the build process as-is for now.

> *> Extract the source code and verify the contents: [unzip OR tar command]*
>
> This is a problem.
>
> People testing are instructed to only test on artefact. Either we reduce
> our artefacts to one instead of two, or people must unpack both and test
> both. Or we vote on them both separately. I favour having one artefact.

People don't HAVE to do anything really.  That being said, I
absolutely get the point!

We'll probably end up releasing just the tar.gz archive version, but
(as I said above) I'll start a different thread to get consensus on
this.

> At this point I made two directories, and ran my tests in parallel on both
> artefacts.
>
> I also ran this:
>
> $ diff -r tar/apache-cloudstack-4.0.0-incubating-src
> zip/apache-cloudstack-4.0.0-incubating-src
>
> No output, so we're good there. That 6M difference must be compression
> efficiency(!)
>
> I also ran these commands:
>
> $ mkdir git
> $ mkdir tree
> $ git clone https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.gitgit
> $ cd git
> $ git archive --prefix=../tree/ 5ee613fbf2da5f6212f2f166c980506e0c6cb5cb |
> tar Pxf -
> $ diff -r tar/apache-cloudstack-4.0.0-incubating-src tree/
> $ diff -r zip/apache-cloudstack-4.0.0-incubating-src tree/
>
> No output.
>
> Which means that the artefacts do not differ from the tree-ish provided in
> the vote email.

Good!

> On the one hand this is great! But I find it surprising. I was expecting
> there to be a build system that generated files as it prepared the
> artefacts. But I guess I am coming from an Autoconf world, where much of it
> is bootstrapped by the RM at release time.

That's not the way we're doing it.

> Either way, this check should be part of the test procedure. It verifies
> the artefact came from the tree-ish in the email.

Great addition!  I skipped this when reviewing the couchdb example,
specifically due to the autotools stuff.  But I get the point of that
step now.  I've added it to the instructions.  Done!

> *> mvn --projects='org.apache.cloudstack:cloudstack'
> org.apache.rat:apache-rat-plugin:0.8:check*
>
> This passes.
>
> But it downloads a LOT of jars. So, of course, I checked every license.
> *grin*
>
> http://repo1.maven.org/maven2/org/apache/rat/apache-rat-core/0.8/apache-rat-core-0.8.jar(Apache)
> http://repo1.maven.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar(Apache)
> http://repo1.maven.org/maven2/commons-lang/commons-lang/2.1/commons-lang-2.1.jar(Apache)
> http://repo1.maven.org/maven2/commons-io/commons-io/1.4/commons-io-1.4.jar(Apache)
> http://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.0/commons-compress-1.0.jar(Apache)
> http://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.jar(Apache)
> http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/1.4/plexus-utils-1.4.jar(Apache)
> http://repo1.maven.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.0-alpha-8/doxia-site-renderer-1.0-alpha-8.jar(Apache)
> http://repo1.maven.org/maven2/org/apache/maven/doxia/doxia-core/1.0-alpha-8/doxia-core-1.0-alpha-8.jar(Apache)
> http://repo1.maven.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-8/doxia-sink-api-1.0-alpha-8.jar(Apache)
> http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-i18n/1.0-beta-6/plexus-i18n-1.0-beta-6.jar(Apache)
> http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-velocity/1.1.3/plexus-velocity-1.1.3.jar(Apache)
> http://repo1.maven.org/maven2/velocity/velocity/1.4/velocity-1.4.jar(Apache)
> http://repo1.maven.org/maven2/velocity/velocity-dep/1.4/velocity-dep-1.4.jar(Apache)
> http://repo1.maven.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.0-alpha-8/doxia-decoration-model-1.0-alpha-8.jar(Apache)
> http://repo1.maven.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.0.4/maven-reporting-impl-2.0.4.jar(Apache)
> http://repo1.maven.org/maven2/commons-validator/commons-validator/1.2.0/commons-validator-1.2.0.jar(Apache)
> http://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar(Apache)
> http://repo1.maven.org/maven2/commons-digester/commons-digester/1.6/commons-digester-1.6.jar(Apache)
> http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar(Apache)
> http://repo1.maven.org/maven2/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar(Apache)
> http://repo1.maven.org/maven2/oro/oro/2.0.7/oro-2.0.7.jar (Apache)
> http://repo1.maven.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.4/maven-reporting-api-2.0.4.jar(Apache)
> http://repo1.maven.org/maven2/org/apache/maven/shared/maven-plugin-testing-harness/1.1/maven-plugin-testing-harness-1.1.jar(Apache)
> http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interactivity-api/1.0-alpha-4/plexus-interactivity-api-1.0-alpha-4.jar
> (Apache)
> http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-archiver/1.0-alpha-7/plexus-archiver-1.0-alpha-7.jar(Apache)
>
> All of them, as far as I can tell, are available under the Apache License.
>
> I then ran:
>
> $ cat pom.xml | grep '<exclude>' | sed 's,.*<exclude>,^,' | sed
> 's,</exclude>,,' | sed 's,\*\*/\*,.*,' | sed 's,\*\*,.*,g' | sed
> 's,/\*,/.*,' > ../license.skip
>
> $ grep -rL 'http://www.apache.org/licenses/LICENSE-2.0' * | grep -vEf
> ../license.skip
>
> This produced a lot of files, so I filtered them by removing images.
>
> I got the following list, with comments inline:
>
>> DISCLAIMER
>
> Why is this not picked up by RAT?

I don't understand the question.  It is.  This file is per-ASF
guidelines, and RAT knows that a DISCLAIMER file is a NOTICE, and
reports it as such.  See the output file:  target/rat.txt

>> KEYS
>
> This should not be in the repository.

Interesting.  OK.  Where does it normally live?

>> awsapi/NOTICE.txt
>> awsapi/README.txt
>
> These files should probably not exist.
>
> This information should be in the top level NOTICE and LICENSE files.

It's already there, but the files should have been purged.  Thanks!

CLOUDSTACK-326

>> debian/changelog
>> debian/README
>
> I am guessing RAT recognises these as files not needing a license.

You can check that by reviewing target/rat.txt, but yes...  it
understands that they are notice files.

>> deps/XenServerJava/LICENSE.txt
>> deps/XenServerJava/README.txt
>
> These files should probably not exist.

CLOUDSTACK-326

> This information should be in the top level NOTICE and LICENSE files.

Already there.

>> docs/publican-cloudstack/NOTICE
>
> This file should probably not exist.
>
> This information should be in the top level NOTICE and LICENSE files.

CLOUDSTACK-326

Plus, I found a stray LICENSE in that same folder.

>> patches/systemvm/debian/config/etc/logrotate.conf
>> patches/systemvm/debian/config/etc/logrotate.d/apache2
>> patches/systemvm/debian/config/etc/logrotate.d/dnsmasq
>> patches/systemvm/debian/config/etc/logrotate.d/haproxy
>> patches/systemvm/debian/config/etc/logrotate.d/ppp
>> patches/systemvm/debian/config/etc/logrotate.d/rsyslog
>> patches/systemvm/debian/config/etc/sysctl.conf
>> patches/systemvm/debian/config/opt/cloud/bin/vpc_passwd_server
>> patches/systemvm/debian/config.dat
>
> Why are these files not picked up by RAT?

Great catch.  That's because some pom.xml fixes didn't get
cherry-picked over.  I'm looking at that now.  The 4.0 branch has
"exclude subprojects" set to true, which it should not.  It's also
missing some exclude directives, that are required once that flag is
flipped.  I'll get it fixed up.

>> patches/systemvm/debian/README
>
> What is this doing here?

Low level instructions for using the code in that folder.  Why is this an issue?

>
>> patches/systemvm/debian/vpn/etc/ipsec.conf
>> patches/systemvm/debian/vpn/etc/ipsec.secrets
>> patches/systemvm/debian/vpn/etc/ppp/options.xl2tpd
>> patches/systemvm/debian/vpn/etc/xl2tpd/xl2tpd.conf
>
> Why are these files not picked up by RAT?

Same issue as above (the pom.xml misconfiguration in 4.0).

>
>> test/conf/README
>> test/integration/component/README
>> test/integration/README
>> test/integration/smoke/README
>> tools/devcloud/veewee/README
>
> Should we have this many README files?

Prasanna covered this in another email.

> Moving on to a more manual check of the files...
>
> `waf` seems to be some sort of composite binary file. Can we ship this? My
> guess is no.

It's BSD licensed.

http://code.google.com/p/waf/

> LICENSE is ISO-8859 encoded. This should be changed to UTF-8 to match our
> other text files.

LICENSE is generated by Apache Whisker (using
tools/whisker/descriptor.xml as the source).  That's the format that
it generates the file in apparently.

Do you happen to know how to convert it?  I can do that after
regenerating the files next time.

> I then ran this file (based on experimentation with `file` over the source
> tree):
>
> cat > ../okay_file_types.txt << EOF
> ASCII C\+\+ program text
> a python script text executable
> Bourne-Again shell script text executable
> a python script text executable
> ASCII English text
> exported SGML document text
> XML  document text
> a bash script text executable
> PNG image data
> ASCII c program text
> SVG Scalable Vector Graphics image
> HTML document text
> POSIX shell script text executable
> PNG image data
> GIF image data
> Lotus 1-2-3
> RFC1421 Security Certificate Signing Request text
> a /usr/bin/make -f script text executable
> ASCII text
> JPEG image data
> UTF-8 Unicode
> Netpbm PBM image text
> a /usr/bin/python script text executable
> ISO-8859 English text
> a /usr/bin/python -u script text executable
> DOS batch file text
> a /usr/bin/python -W ignore::Depr script text executable
> EOF
>
> And then this:
>
> $ find . -type f | xargs file | grep -vEf ../okay_file_types.txt | sed
> 's,:.*,,'
>
> Which gave me this output:
>
> ./awsapi/resource/AmazonEC2/xes.keystore
> ./client/tomcatconf/cloudmanagementserver.keystore
> ./console-proxy/certs/realhostip.keystore
> ./utils/certs/cloud.keystore
>
> These files are binary. Do we need to ship the source?

Yes.  They are in our RAT exclude list, and are needed for the system
to function on initial install.  The software provides options for
replacing them during a production deployment, but they are needed.

> Then I ran:
>
> $ grep -rsi "copyright" * | grep -v "regarding copyright ownership" | sed
> 's,:, ,' | awk '{print $1}' | sort | uniq > ../files_to_check.txt
>
> I manually reconciled this list with the pom.xml file.
>
>> awsapi/src/com/cloud/bridge/io/DimeDelimitedInputStream.java
>
> This file has two license headers in. One of them is incorrect.

CLOUDSTACK-327 - good find

>> deps/XenServerJava/BSD
>> deps/XenServerJava/LICENSE.Apache-2.0.txt
>> deps/XenServerJava/LICENSE.txt
>
> These should not be here. They need to be moved to the top level LICENSE.
>
> (And the exclusion in pom.xml needs to be removed.)

Added to CLOUDSTACK-326

>
>> docs/publican-cloudstack/LICENSE
>> docs/publican-cloudstack/NOTICE
>
> These files need to be removed, and merged into the top level LICENSE and
> NOTICE.

You already covered these files.

>> plugins/network-elements/nicira-nvp/README.NiciraIntegration
>
> Seems like an odd file.
>
> It has our standard header, but also a separate copyright statement. Which
> is it?
>
> The file is not mentioned in LICENSE.

It was developed as ASF.  We need to simply clean it up.  I think
there was just confusion about what was appropriate way back then...

Added to CLOUDSTACK-326


>> scripts/vm/hypervisor/xenserver/xcposs/NFSSR.py
>> scripts/vm/hypervisor/xenserver/xcpserver/NFSSR.py
>> scripts/vm/hypervisor/xenserver/xenserver56/NFSSR.py
>> scripts/vm/hypervisor/xenserver/xenserver56fp1/NFSSR.py
>> scripts/vm/hypervisor/xenserver/xenserver60/NFSSR.py
>
> These files have our standard boilerplate, but claim they are "(C) 2008
> Citrix Systems Inc".

Added to CLOUDSTACK-327

>> test/integration/component/test_allocation_states.py
>
> This file claims "Copyright 2012 Citrix Systems, Inc." No mention in
> LICENSE.

Already resolved (will be in the next RC).

>> tools/whisker/descriptor.xml
>
> This file has our standard boiler plate, but is crammed full of other
> licenses. Is this okay?

Yup - that's the metadata that generates our LICENSE and NOTICE files
in the first place!

>> ui/lib/excanvas.js
>
> This file claims it is "Copyright 2006 Google Inc.". No mention in LICENSE.

Great find.  CLOUDSTACK-327

> Okay, back to the test procedure instructions...
>
> *> Compile*
>
> Section skipped because I cannot able to set up my build environment
> without further instruction.
>
> I am unable to complete the rest of the testing procedure for the same
> reason.
>
> One concern is that even though I got the image to boot, I could not access
> it via localhost:8080.

That's because you didn't build / deploy / start the server!  ;-)

> That's it for now! :-)
>
> --
> NS