Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Date: Wed, 13 Jul 2016 04:23:24 -0400 (EDT)
From: Joan Touzet <wohali@apache.org>
Reply-To: Joan Touzet <wohali@apache.org>
To: dev@couchdb.apache.org
Message-ID: <681855389.42.1468398228411.JavaMail.Joan@Brain>
In-Reply-To: <1927053050.278.1468360965287.JavaMail.Joan@Brain>
References: <487167227.99.1466196508652.JavaMail.Joan@Brain> <CAGHTdUUGtPP+1+Y=_tPi8XQjWECrMy5dwmnOqBH--46KWDi1oA@mail.gmail.com> <CAGHTdUWDE1m1TS0X7gqNA3ZXPj0uVnHYcRNchvSWq09Oz6-4yA@mail.gmail.com> <CA+pq5frA8UK1bEyNcvQgQqXOTS_AwHpMbTMu9zpgi7cDGQVeiA@mail.gmail.com> <CAGHTdUX9uh2Z-bhrhebqA+0koO2K3d6u72tjfwMbzJ2fz3atMg@mail.gmail.com> <CA+pq5fqvpbdegFfWYONm3C_51KGGjj58-_EW6ddYUqpwU0H2QQ@mail.gmail.com> <CAJ_m3YD-_=0W=pji_oWaf5LfjRH-TnFeN7JEDdtCVLq0rdS25Q@mail.gmail.com> <1927053050.278.1468360965287.JavaMail.Joan@Brain>
Subject: Re: 2.0 & Windows: status update
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Thread-Topic: 2.0 & Windows: status update
Thread-Index: dqJu+iu2f0rdE1d/x2LFzCjx3IHrZ9gwSdJQ
archived-at: Wed, 13 Jul 2016 08:24:15 -0000

I took a look at the eunit failures and found that the entire
couchdb_os_daemon_test module fails due to issues with how it expects
to launch daemons.

The main issue is that the primary test harness is a .escript file, which
on *nix is magically parsed via the #!/usr/bin/escript header. On Windows
we are just trying to directly launch the .escript file which fails (since
Windows has no idea how to execute *.escript files). There are also .sh
scripts that are part of the test harness that will not run correctly.

I've submitted a PR to simply bypass this entire test module for now. It
sure would be swell to make it work but it'll be a fair bit of fiddling,
especially in a way that makes eunit happy, to get it to work.

Besides, there are other tests that drive the mrview os daemon; we will
see massive failures in the JS test suite if the entire os daemon launching
process fails.

https://github.com/apache/couchdb-couch/pull/184 hopefully will land soon
allowing the main couch eunit tests to pass.

More worryingly we have a failure in jiffy on Windows:

https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gis=
tfile1-txt-L381-L389

Anyone who can help look at this one? (Paul?)

-Joan

----- Original Message -----
From: "Joan Touzet" <wohali@apache.org>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 6:02:12 PM
Subject: Re: 2.0 & Windows: status update

Thanks, Paul. I'm starting to look at this today.

In better news, current Windows JS tests now match the *nix JS test results=
.
Only one test, replication.js, is failing (ignoring the ignored tests).

Results: https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829

Mr. Newson is looking at this failure right now and says we may be able
to improve upon the testing methodology.

-Joan

----- Original Message -----
From: "Paul Davis" <paul.joseph.davis@gmail.com>
To: dev@couchdb.apache.org
Sent: Tuesday, July 12, 2016 4:37:15 PM
Subject: Re: 2.0 & Windows: status update

The logs posted at [1] show that we're seeing OS processes die with an
exit code of 4. The most likely place I can find that that comes from
is couchspawnkillable_win.c [2] which is nicely Windows specific so
would do a lot to explain why we don't see it on *nix systems.
Unfortunately other than pointing out that the subprocess creation
seems to fail I don't have any idea or suggestion on how to debug
further.

[1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
[2] https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/=
couchspawnkillable_win.c#L106

On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
<sebastianrothbucher@googlemail.com> wrote:
> Hi all,
>
> just to follow up on that: there is another PR coming up (
> https://github.com/apache/couchdb/pull/427) that tests for some more fixe=
s
> and brings even more stability. In the meantime, deleting dev/lib is inde=
ed
> the best way to produce reliable results. So is switching between
> auth-tests-wip and master. But there's progress =3D> it might all end up
> nicely on master.
>
> Good luck, thanks & best
>     Sebastian
>
>
> On Mon, Jun 20, 2016 at 5:26 PM, Nick North <north.n@gmail.com> wrote:
>
>> Thanks Sebastian. I'm looking at eunit at the moment, but hope to come b=
ack
>> to these.
>>
>> Nick
>>
>> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>
>> > Hi Joan, Nick,
>> >
>> > the following gist provides a current run of the test against the late=
st
>> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> branch):
>> >
>> >
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7=
e0de
>> > Maybe you can use them
>> >
>> > Here's what I did to get both latest tests and latest code:
>> > git checkout master
>> > ./configure -c --disable-docs --disable-fauxton
>> > make clean
>> > make
>> > git checkout auth-tests-wip
>> >
>> > Currently, make javascript seems not optimal as one tests (needs
>> > investigation) seems to mess up the setup for all that's following.
>> Hence,
>> > I took this drastic measure to produce the logs:
>> >
>> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run -n 1 -=
q
>> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a
>> jstest2.log;
>> > done
>> >
>> > Maybe it makes sense for you to start w/ something similar to produce
>> some
>> > meaningful results.
>> >
>> > Best
>> >    Sebastian
>> >
>> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <north.n@gmail.com> wrote:
>> >
>> > > Sorry - I meant a single node cluster in that last message. And I me=
ant
>> > to
>> > > sign my name correctly.
>> > >
>> > > Nick
>> > >
>> > > On Sun, 19 Jun 2016 at 16:56 Nick North <north.n@gmail.com> wrote:
>> > >
>> > > > I'm trying these tests now, and find that there are still a lot of=
 JS
>> > > > failures with a single cluster. Many of them look suspiciously
>> similar
>> > at
>> > > > an initial glance, but I hope to look in more detail tomorrow.
>> > > >
>> > > > Nicj
>> > > >
>> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <jan@apache.org> wrote:
>> > > >
>> > > >>
>> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wohali@apache.org> wrote=
:
>> > > >> >
>> > > >> > Hello everyone,
>> > > >> >
>> > > >> > I'd like to update the community on the status of the 2.0 port =
to
>> > > >> Microsoft Windows. There are three parts to this email: the build
>> > > >> tools/chain themselves, support in CouchDB for the Windows build
>> > > process,
>> > > >> and testing results. I'll cover them in that order.
>> > > >> >
>> > > >> > -Joan
>> > > >> >
>> > > >> > Build Tools/Chain
>> > > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > > >> > ** TL;DR: New glazier repo to join couchdb, contains scripts an=
d
>> > > README
>> > > >> to build CouchDB 2.0 on Windows.
>> > > >> >
>> > > >> > Our work to date has been going on in Dave Cottlehuber's glazie=
r
>> > > >> repository at
>> > > >> >
>> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> > > >> >
>> > > >> > The reason for the extra repository is that the Windows build
>> > process
>> > > >> is *very* ugly, involving 3 distinct build chains (Visual Studio,
>> > Cygwin
>> > > >> and the Mozilla Build system) to build all of the necessary
>> > > prerequisites.
>> > > >> The repository includes a number of support scripts to set up tha=
t
>> > > >> environment, a README with a detailed walkthrough, and some patch=
es
>> > > >> necessary to the prerequisites to get them to build under the mod=
ern
>> > > >> Windows b uild system.
>> > > >> >
>> > > >> > Parenthetically, it _is_ possible to use binary installs for th=
e
>> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, Nic=
k
>> > North
>> > > >> and I have evolved the glazier system over a number of years and
>> it's
>> > > >> proven quite effective. Plus, we don't have to worry about the
>> > > provenance
>> > > >> of any of the binaries since we build everything from source
>> directly,
>> > > and
>> > > >> that's important when we put up an unofficial Windows build for
>> > > download at
>> > > >> https://couchdb.apache.org/ .
>> > > >> >
>> > > >> > Good news: as of today I've requested and Infra has created a n=
ew
>> > > >> apache couchdb-glazier repo, and it's my intent to mirror
>> dch/glazier
>> > > over
>> > > >> into the ASF's repo once things have stabilized a bit more (PR an=
d
>> > > merge of
>> > > >> the release/couchdb_2.0 branch, and pending progress on steps 2 a=
nd
>> 3
>> > > >> below). Dave and I did an audit of the repository as it stands, a=
nd
>> > > since
>> > > >> all checkins come from CouchDB contributors already, we are good =
to
>> go
>> > > from
>> > > >> a licensing perspective.
>> > > >> >
>> > > >> >
>> > > >> > Overall CouchDB Windows support
>> > > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of
>> top-level
>> > > >> Makefile in progress.
>> > > >> >
>> > > >> > There are two aspects to native CouchDB Windows support. The fi=
rst
>> > is
>> > > >> anything within the CouchDB code itself that assumes a Unix-like
>> > > >> environment. Fortunately, most of these problems have been worked
>> out
>> > in
>> > > >> prior releases. I'm not aware of any outstanding issues here (exc=
ept
>> > one
>> > > >> point below under test results).
>> > > >> >
>> > > >> > The other aspect is the build setup within the couchdb repo
>> itself.
>> > > >> I've already converted the bash configure script into a PowerShel=
l
>> > > >> configure script that works fine. However, the Makefile has bashi=
sms
>> > in
>> > > it
>> > > >> and assumes GNU Make. I've started a conversion of this into Wind=
ows
>> > > NMake
>> > > >> format and will submit a PR for a Makefile.win in due course.
>> > > >> >
>> > > >> > I want to answer two frequent questions we get here before they
>> get
>> > > >> re-asked:
>> > > >> >
>> > > >> >  1) Why not use a cygwin environment to retain compatibility wi=
th
>> > the
>> > > >> Unix build process? The answer is that performance suffers, the
>> build
>> > > chain
>> > > >> is onerous, there are link-time problems when trying to link agai=
nst
>> > > things
>> > > >> built using Visual Studio, and there are still assumptions on pat=
hs
>> > that
>> > > >> don't work out. We can't get away from making Windows-specific
>> > > >> customizations to the build process anyway, so we might as well t=
ake
>> > the
>> > > >> extra step and support the build process properly. It's not THAT
>> much
>> > > work
>> > > >> to convert the Makefile and configure script, and our top-level
>> > Makefile
>> > > >> really isn't much more than a shell script anyway (every target i=
s a
>> > > .PHONY
>> > > >> target!). In fact, a TODO for an enterprising developer might be =
to
>> > > rewrite
>> > > >> our top-level Makefile/Makefile.win as a Python script that "does
>> the
>> > > right
>> > > >> thing" on both platforms, the same way our dev/run script works
>> today.
>> > > >> >
>> > > >> >  2) Why not use the new "Bash and Ubuntu on Windows" functional=
ity
>> > > >> Microsoft has announced for Windows 10? There are two distinct
>> > problems
>> > > >> here. The first is that there is a very large install base still =
of
>> > > Windows
>> > > >> 7 and 8 (and Windows Server) machines that cannot run this
>> subsystem.
>> > > The
>> > > >> second is that Microsoft themselves say this about the
>> functionality:
>> > > >> >
>> > > >> >     "Second, while you=E2=80=99ll be able to run native Bash an=
d many
>> Linux
>> > > >> command-line tools on Windows, it=E2=80=99s important to note tha=
t this is a
>> > > >> developer toolset to help you write and build all your code for a=
ll
>> > your
>> > > >> scenarios and platforms. This is not a server platform upon which
>> you
>> > > will
>> > > >> host websites, run server infrastructure, etc."
>> > > >> >
>> > > >> > Given this strong warning from Microsoft themselves (which hint=
s
>> at
>> > > >> performance consideratings), and the fact that download statistic=
s
>> > show
>> > > an
>> > > >> equal number of downloads of the CouchDB .tar source and the Wind=
ows
>> > > .zip
>> > > >> installer from our couchdb.apache.org website, we need to conside=
r
>> > that
>> > > >> people are running CouchDB on Windows not just as a developer too=
l
>> but
>> > > as a
>> > > >> fully-fledged server. As such it behooves us to build it "properl=
y"
>> > as a
>> > > >> normal Windows binary/service.
>> > > >> >
>> > > >>
>> > > >> Great progress Joan! Thank you! :)
>> > > >>
>> > > >> > Test Results
>> > > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > > >> > ** TL;DR: Lots of things are failing. Joan needs help interpret=
ing
>> > the
>> > > >> results or she will go around the bend.
>> > > >> >
>> > > >> > Here are the current test results in gist form.
>> > > >> >
>> > > >> > EUnit:
>> > > >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff73172=
2
>> > > >> >
>> > > >> > JS tests:
>> > > >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219b=
f
>> > > >> >
>> > > >> > For the EUnit tests, everything other than os_process stuff see=
ms
>> to
>> > > be
>> > > >> working. Honestly, I think we can release without os_process supp=
ort
>> > on
>> > > >> Windows, though I should file a bug to track this. I am actually
>> > > inclined
>> > > >> to disable os_process support on Windows and the related eunit te=
sts
>> > > rather
>> > > >> than fix this rarely-needed functionality, unless someone on this
>> list
>> > > >> objects.
>> > > >>
>> > > >> You are probably thinking about CouchDB Externals, which definite=
ly
>> > use
>> > > >> os_process functionality and which I=E2=80=99d also be fine with =
dropping
>> > > support
>> > > >> for Windows for now, but os_process is also used by the view serv=
er,
>> > so
>> > > we
>> > > >> should definitely get them passing.
>> > > >>
>> > > >>
>> > > >> > For the JS tests, a *lot* is failing. I need to know how much t=
his
>> > > >> differs from a Linux/OSX baseline today, can anyone help me follo=
w
>> up
>> > > here?
>> > > >>
>> > > >> Can you try running these against a -n 1 cluster? We are not set =
up
>> to
>> > > >> run JS tests against more nodes at this point.
>> > > >>
>> > > >> On master/unix most if not all JS tests should either pass or
>> skipped
>> > > >> with a TODO message.
>> > > >>
>> > > >>
>> > > >> Best
>> > > >> Jan
>> > > >> --
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>>