couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick North <nort...@gmail.com>
Subject Re: 2.0 & Windows: status update
Date Wed, 13 Jul 2016 18:06:41 GMT
Ah - my mistake, sorry: I didn't merge it after all. The PR is outstanding
at https://github.com/apache/couchdb-jiffy/pull/3, with some discussion
between @kxepal and myself. There's a note to Paul there asking wiser heads
than mine to make a decision on whether to add it upstream.

Nick

On Wed, 13 Jul 2016 at 18:00 Nick North <north.n@gmail.com> wrote:

> Maybe I did something stupid with the PR. If I can get to a PC this
> evening I'll take a look.
>
> Nick
> On Wed, 13 Jul 2016 at 17:41, Paul Davis <paul.joseph.davis@gmail.com>
> wrote:
>
>> That Jiffy error does seem like a conversion issue either from integer
>> widths or endianess. I don't remember seeing a patch for that on
>> Windows and looking back I'm not seeing anything from after 2013 that
>> looks related (in davisp/jiffy). Looking at the apache/jiffy repo we
>> use merges so I'd think it'd be unlikely to have been overwritten but
>> I guess its always possible.
>>
>> Paul
>>
>> On Wed, Jul 13, 2016 at 3:31 AM, Nick North <north.n@gmail.com> wrote:
>> > I think I submitted a PR for that Jiffy problem a while back: it's some
>> > 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm
>> away
>> > from any useful tools for the next few days so can't check myself, but
>> take
>> > a look at the activity on the couchdb-jiffy repository. It's possible
>> that
>> > there has been a pull from upstream that overwrote my change.
>> >
>> > Nick
>> >
>> > On Wed, 13 Jul 2016 at 09:24 Joan Touzet <wohali@apache.org> wrote:
>> >
>> >> I took a look at the eunit failures and found that the entire
>> >> couchdb_os_daemon_test module fails due to issues with how it expects
>> >> to launch daemons.
>> >>
>> >> The main issue is that the primary test harness is a .escript file,
>> which
>> >> on *nix is magically parsed via the #!/usr/bin/escript header. On
>> Windows
>> >> we are just trying to directly launch the .escript file which fails
>> (since
>> >> Windows has no idea how to execute *.escript files). There are also .sh
>> >> scripts that are part of the test harness that will not run correctly.
>> >>
>> >> I've submitted a PR to simply bypass this entire test module for now.
>> It
>> >> sure would be swell to make it work but it'll be a fair bit of
>> fiddling,
>> >> especially in a way that makes eunit happy, to get it to work.
>> >>
>> >> Besides, there are other tests that drive the mrview os daemon; we will
>> >> see massive failures in the JS test suite if the entire os daemon
>> launching
>> >> process fails.
>> >>
>> >> https://github.com/apache/couchdb-couch/pull/184 hopefully will land
>> soon
>> >> allowing the main couch eunit tests to pass.
>> >>
>> >> More worryingly we have a failure in jiffy on Windows:
>> >>
>> >>
>> >>
>> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389
>> >>
>> >> Anyone who can help look at this one? (Paul?)
>> >>
>> >> -Joan
>> >>
>> >> ----- Original Message -----
>> >> From: "Joan Touzet" <wohali@apache.org>
>> >> To: dev@couchdb.apache.org
>> >> Sent: Tuesday, July 12, 2016 6:02:12 PM
>> >> Subject: Re: 2.0 & Windows: status update
>> >>
>> >> Thanks, Paul. I'm starting to look at this today.
>> >>
>> >> In better news, current Windows JS tests now match the *nix JS test
>> >> results.
>> >> Only one test, replication.js, is failing (ignoring the ignored tests).
>> >>
>> >> Results:
>> >> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829
>> >>
>> >> Mr. Newson is looking at this failure right now and says we may be able
>> >> to improve upon the testing methodology.
>> >>
>> >> -Joan
>> >>
>> >> ----- Original Message -----
>> >> From: "Paul Davis" <paul.joseph.davis@gmail.com>
>> >> To: dev@couchdb.apache.org
>> >> Sent: Tuesday, July 12, 2016 4:37:15 PM
>> >> Subject: Re: 2.0 & Windows: status update
>> >>
>> >> The logs posted at [1] show that we're seeing OS processes die with an
>> >> exit code of 4. The most likely place I can find that that comes from
>> >> is couchspawnkillable_win.c [2] which is nicely Windows specific so
>> >> would do a lot to explain why we don't see it on *nix systems.
>> >> Unfortunately other than pointing out that the subprocess creation
>> >> seems to fail I don't have any idea or suggestion on how to debug
>> >> further.
>> >>
>> >> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1
>> >> [2]
>> >>
>> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106
>> >>
>> >> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher
>> >> <sebastianrothbucher@googlemail.com> wrote:
>> >> > Hi all,
>> >> >
>> >> > just to follow up on that: there is another PR coming up (
>> >> > https://github.com/apache/couchdb/pull/427) that tests for some more
>> >> fixes
>> >> > and brings even more stability. In the meantime, deleting dev/lib is
>> >> indeed
>> >> > the best way to produce reliable results. So is switching between
>> >> > auth-tests-wip and master. But there's progress => it might all
end
>> up
>> >> > nicely on master.
>> >> >
>> >> > Good luck, thanks & best
>> >> >     Sebastian
>> >> >
>> >> >
>> >> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <north.n@gmail.com>
>> wrote:
>> >> >
>> >> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope
to
>> come
>> >> back
>> >> >> to these.
>> >> >>
>> >> >> Nick
>> >> >>
>> >> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher <
>> >> >> sebastianrothbucher@googlemail.com> wrote:
>> >> >>
>> >> >> > Hi Joan, Nick,
>> >> >> >
>> >> >> > the following gist provides a current run of the test against
the
>> >> latest
>> >> >> > master of CouchDB - and the latest tests (from the auth-tests-wip
>> >> >> branch):
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de
>> >> >> > Maybe you can use them
>> >> >> >
>> >> >> > Here's what I did to get both latest tests and latest code:
>> >> >> > git checkout master
>> >> >> > ./configure -c --disable-docs --disable-fauxton
>> >> >> > make clean
>> >> >> > make
>> >> >> > git checkout auth-tests-wip
>> >> >> >
>> >> >> > Currently, make javascript seems not optimal as one tests
(needs
>> >> >> > investigation) seems to mess up the setup for all that's
>> following.
>> >> >> Hence,
>> >> >> > I took this drastic measure to produce the logs:
>> >> >> >
>> >> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run
>> -n 1
>> >> -q
>> >> >> > --with-admin-party-please test/javascript/run $t 2>&1
| tee -a
>> >> >> jstest2.log;
>> >> >> > done
>> >> >> >
>> >> >> > Maybe it makes sense for you to start w/ something similar
to
>> produce
>> >> >> some
>> >> >> > meaningful results.
>> >> >> >
>> >> >> > Best
>> >> >> >    Sebastian
>> >> >> >
>> >> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <north.n@gmail.com>
>> >> wrote:
>> >> >> >
>> >> >> > > Sorry - I meant a single node cluster in that last message.
And
>> I
>> >> meant
>> >> >> > to
>> >> >> > > sign my name correctly.
>> >> >> > >
>> >> >> > > Nick
>> >> >> > >
>> >> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <north.n@gmail.com>
>> wrote:
>> >> >> > >
>> >> >> > > > I'm trying these tests now, and find that there
are still a
>> lot
>> >> of JS
>> >> >> > > > failures with a single cluster. Many of them look
suspiciously
>> >> >> similar
>> >> >> > at
>> >> >> > > > an initial glance, but I hope to look in more detail
tomorrow.
>> >> >> > > >
>> >> >> > > > Nicj
>> >> >> > > >
>> >> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <jan@apache.org>
>> wrote:
>> >> >> > > >
>> >> >> > > >>
>> >> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <wohali@apache.org>
>> >> wrote:
>> >> >> > > >> >
>> >> >> > > >> > Hello everyone,
>> >> >> > > >> >
>> >> >> > > >> > I'd like to update the community on the
status of the 2.0
>> port
>> >> to
>> >> >> > > >> Microsoft Windows. There are three parts to
this email: the
>> build
>> >> >> > > >> tools/chain themselves, support in CouchDB for
the Windows
>> build
>> >> >> > > process,
>> >> >> > > >> and testing results. I'll cover them in that
order.
>> >> >> > > >> >
>> >> >> > > >> > -Joan
>> >> >> > > >> >
>> >> >> > > >> > Build Tools/Chain
>> >> >> > > >> > =================
>> >> >> > > >> > ** TL;DR: New glazier repo to join couchdb,
contains
>> scripts
>> >> and
>> >> >> > > README
>> >> >> > > >> to build CouchDB 2.0 on Windows.
>> >> >> > > >> >
>> >> >> > > >> > Our work to date has been going on in Dave
Cottlehuber's
>> >> glazier
>> >> >> > > >> repository at
>> >> >> > > >> >
>> >> >> > > >> >  https://github.com/dch/glazier/tree/release/couchdb_2.0
>> >> >> > > >> >
>> >> >> > > >> > The reason for the extra repository is
that the Windows
>> build
>> >> >> > process
>> >> >> > > >> is *very* ugly, involving 3 distinct build chains
(Visual
>> Studio,
>> >> >> > Cygwin
>> >> >> > > >> and the Mozilla Build system) to build all of
the necessary
>> >> >> > > prerequisites.
>> >> >> > > >> The repository includes a number of support
scripts to set up
>> >> that
>> >> >> > > >> environment, a README with a detailed walkthrough,
and some
>> >> patches
>> >> >> > > >> necessary to the prerequisites to get them to
build under the
>> >> modern
>> >> >> > > >> Windows b uild system.
>> >> >> > > >> >
>> >> >> > > >> > Parenthetically, it _is_ possible to use
binary installs
>> for
>> >> the
>> >> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM
1.8.5), but Dave,
>> >> Nick
>> >> >> > North
>> >> >> > > >> and I have evolved the glazier system over a
number of years
>> and
>> >> >> it's
>> >> >> > > >> proven quite effective. Plus, we don't have
to worry about
>> the
>> >> >> > > provenance
>> >> >> > > >> of any of the binaries since we build everything
from source
>> >> >> directly,
>> >> >> > > and
>> >> >> > > >> that's important when we put up an unofficial
Windows build
>> for
>> >> >> > > download at
>> >> >> > > >> https://couchdb.apache.org/ .
>> >> >> > > >> >
>> >> >> > > >> > Good news: as of today I've requested and
Infra has
>> created a
>> >> new
>> >> >> > > >> apache couchdb-glazier repo, and it's my intent
to mirror
>> >> >> dch/glazier
>> >> >> > > over
>> >> >> > > >> into the ASF's repo once things have stabilized
a bit more
>> (PR
>> >> and
>> >> >> > > merge of
>> >> >> > > >> the release/couchdb_2.0 branch, and pending
progress on
>> steps 2
>> >> and
>> >> >> 3
>> >> >> > > >> below). Dave and I did an audit of the repository
as it
>> stands,
>> >> and
>> >> >> > > since
>> >> >> > > >> all checkins come from CouchDB contributors
already, we are
>> good
>> >> to
>> >> >> go
>> >> >> > > from
>> >> >> > > >> a licensing perspective.
>> >> >> > > >> >
>> >> >> > > >> >
>> >> >> > > >> > Overall CouchDB Windows support
>> >> >> > > >> > ===============================
>> >> >> > > >> > ** TL;DR: Windows support in 2.0 a priority,
conversion of
>> >> >> top-level
>> >> >> > > >> Makefile in progress.
>> >> >> > > >> >
>> >> >> > > >> > There are two aspects to native CouchDB
Windows support.
>> The
>> >> first
>> >> >> > is
>> >> >> > > >> anything within the CouchDB code itself that
assumes a
>> Unix-like
>> >> >> > > >> environment. Fortunately, most of these problems
have been
>> worked
>> >> >> out
>> >> >> > in
>> >> >> > > >> prior releases. I'm not aware of any outstanding
issues here
>> >> (except
>> >> >> > one
>> >> >> > > >> point below under test results).
>> >> >> > > >> >
>> >> >> > > >> > The other aspect is the build setup within
the couchdb repo
>> >> >> itself.
>> >> >> > > >> I've already converted the bash configure script
into a
>> >> PowerShell
>> >> >> > > >> configure script that works fine. However, the
Makefile has
>> >> bashisms
>> >> >> > in
>> >> >> > > it
>> >> >> > > >> and assumes GNU Make. I've started a conversion
of this into
>> >> Windows
>> >> >> > > NMake
>> >> >> > > >> format and will submit a PR for a Makefile.win
in due course.
>> >> >> > > >> >
>> >> >> > > >> > I want to answer two frequent questions
we get here before
>> they
>> >> >> get
>> >> >> > > >> re-asked:
>> >> >> > > >> >
>> >> >> > > >> >  1) Why not use a cygwin environment to
retain
>> compatibility
>> >> with
>> >> >> > the
>> >> >> > > >> Unix build process? The answer is that performance
suffers,
>> the
>> >> >> build
>> >> >> > > chain
>> >> >> > > >> is onerous, there are link-time problems when
trying to link
>> >> against
>> >> >> > > things
>> >> >> > > >> built using Visual Studio, and there are still
assumptions on
>> >> paths
>> >> >> > that
>> >> >> > > >> don't work out. We can't get away from making
>> Windows-specific
>> >> >> > > >> customizations to the build process anyway,
so we might as
>> well
>> >> take
>> >> >> > the
>> >> >> > > >> extra step and support the build process properly.
It's not
>> THAT
>> >> >> much
>> >> >> > > work
>> >> >> > > >> to convert the Makefile and configure script,
and our
>> top-level
>> >> >> > Makefile
>> >> >> > > >> really isn't much more than a shell script anyway
(every
>> target
>> >> is a
>> >> >> > > .PHONY
>> >> >> > > >> target!). In fact, a TODO for an enterprising
developer
>> might be
>> >> to
>> >> >> > > rewrite
>> >> >> > > >> our top-level Makefile/Makefile.win as a Python
script that
>> "does
>> >> >> the
>> >> >> > > right
>> >> >> > > >> thing" on both platforms, the same way our dev/run
script
>> works
>> >> >> today.
>> >> >> > > >> >
>> >> >> > > >> >  2) Why not use the new "Bash and Ubuntu
on Windows"
>> >> functionality
>> >> >> > > >> Microsoft has announced for Windows 10? There
are two
>> distinct
>> >> >> > problems
>> >> >> > > >> here. The first is that there is a very large
install base
>> still
>> >> of
>> >> >> > > Windows
>> >> >> > > >> 7 and 8 (and Windows Server) machines that cannot
run this
>> >> >> subsystem.
>> >> >> > > The
>> >> >> > > >> second is that Microsoft themselves say this
about the
>> >> >> functionality:
>> >> >> > > >> >
>> >> >> > > >> >     "Second, while you’ll be able to
run native Bash and
>> many
>> >> >> Linux
>> >> >> > > >> command-line tools on Windows, it’s important
to note that
>> this
>> >> is a
>> >> >> > > >> developer toolset to help you write and build
all your code
>> for
>> >> all
>> >> >> > your
>> >> >> > > >> scenarios and platforms. This is not a server
platform upon
>> which
>> >> >> you
>> >> >> > > will
>> >> >> > > >> host websites, run server infrastructure, etc."
>> >> >> > > >> >
>> >> >> > > >> > Given this strong warning from Microsoft
themselves (which
>> >> hints
>> >> >> at
>> >> >> > > >> performance consideratings), and the fact that
download
>> >> statistics
>> >> >> > show
>> >> >> > > an
>> >> >> > > >> equal number of downloads of the CouchDB .tar
source and the
>> >> Windows
>> >> >> > > .zip
>> >> >> > > >> installer from our couchdb.apache.org website,
we need to
>> >> consider
>> >> >> > that
>> >> >> > > >> people are running CouchDB on Windows not just
as a developer
>> >> tool
>> >> >> but
>> >> >> > > as a
>> >> >> > > >> fully-fledged server. As such it behooves us
to build it
>> >> "properly"
>> >> >> > as a
>> >> >> > > >> normal Windows binary/service.
>> >> >> > > >> >
>> >> >> > > >>
>> >> >> > > >> Great progress Joan! Thank you! :)
>> >> >> > > >>
>> >> >> > > >> > Test Results
>> >> >> > > >> > ============
>> >> >> > > >> > ** TL;DR: Lots of things are failing. Joan
needs help
>> >> interpreting
>> >> >> > the
>> >> >> > > >> results or she will go around the bend.
>> >> >> > > >> >
>> >> >> > > >> > Here are the current test results in gist
form.
>> >> >> > > >> >
>> >> >> > > >> > EUnit:
>> >> >> > > >>
>> >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722
>> >> >> > > >> >
>> >> >> > > >> > JS tests:
>> >> >> > > >>
>> >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf
>> >> >> > > >> >
>> >> >> > > >> > For the EUnit tests, everything other than
os_process stuff
>> >> seems
>> >> >> to
>> >> >> > > be
>> >> >> > > >> working. Honestly, I think we can release without
os_process
>> >> support
>> >> >> > on
>> >> >> > > >> Windows, though I should file a bug to track
this. I am
>> actually
>> >> >> > > inclined
>> >> >> > > >> to disable os_process support on Windows and
the related
>> eunit
>> >> tests
>> >> >> > > rather
>> >> >> > > >> than fix this rarely-needed functionality, unless
someone on
>> this
>> >> >> list
>> >> >> > > >> objects.
>> >> >> > > >>
>> >> >> > > >> You are probably thinking about CouchDB Externals,
which
>> >> definitely
>> >> >> > use
>> >> >> > > >> os_process functionality and which I’d also
be fine with
>> dropping
>> >> >> > > support
>> >> >> > > >> for Windows for now, but os_process is also
used by the view
>> >> server,
>> >> >> > so
>> >> >> > > we
>> >> >> > > >> should definitely get them passing.
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> > For the JS tests, a *lot* is failing. I
need to know how
>> much
>> >> this
>> >> >> > > >> differs from a Linux/OSX baseline today, can
anyone help me
>> >> follow
>> >> >> up
>> >> >> > > here?
>> >> >> > > >>
>> >> >> > > >> Can you try running these against a -n 1 cluster?
We are not
>> set
>> >> up
>> >> >> to
>> >> >> > > >> run JS tests against more nodes at this point.
>> >> >> > > >>
>> >> >> > > >> On master/unix most if not all JS tests should
either pass or
>> >> >> skipped
>> >> >> > > >> with a TODO message.
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >> Best
>> >> >> > > >> Jan
>> >> >> > > >> --
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > > >>
>> >> >> > >
>> >> >> >
>> >> >>
>> >>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message