mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Olivier <cjolivie...@gmail.com>
Subject Re: [DISCUSS] Build OSX builds in CI (possibly with TravisCI).
Date Tue, 12 Dec 2017 18:11:54 GMT
I am on vacation starting Thursday.

On Tue, Dec 12, 2017 at 9:49 AM kellen sunderland <
kellen.sunderland@gmail.com> wrote:

> Absolutely, let's do an investigation and see if it's possible to
> virtualize.  Would you have time to look into it a bit further?
>
> On Tue, Dec 12, 2017 at 6:47 PM, Chris Olivier <cjolivier01@gmail.com>
> wrote:
>
> > Don’t get me wrong, I’m not saying this Mac OS Jenkins solution is doable
> > but I feel like we should investigate because the payoff would be large.
> >
> >
> > On Tue, Dec 12, 2017 at 9:38 AM Chris Olivier <cjolivier01@gmail.com>
> > wrote:
> >
> > > Apple’s Darwin OS Is recently open-sourced.
> > > https://github.com/PureDarwin/PureDarwin
> > >
> > > How to convert this into a non-GUI VM I am not sure but I am willing to
> > > bet that people have done it already.
> > >
> > > On Tue, Dec 12, 2017 at 9:16 AM kellen sunderland <
> > > kellen.sunderland@gmail.com> wrote:
> > >
> > >> It might be technically possible, but I think it would violate the
> MacOS
> > >> license: http://store.apple.com/Catalog/US/Images/MacOSX.htm
> > >>
> > >> "2. Permitted License Uses and Restrictions.
> > >> A. This License allows you to install and use one copy of the Apple
> > >> Software on a single Apple-labeled computer at a time. This License
> does
> > >> not allow the Apple Software to exist on more than one computer at a
> > >> time,and you may not make the Apple Software available over a network
> > >> where
> > >> it could be used by multiple computers at the same time. You may make
> > one
> > >> copy of the Apple Software (excluding the Boot ROM code) in
> > >> machine-readable form for backup purposes only; provided that the
> backup
> > >> copy must include all copyright or other proprietary notices contained
> > on
> > >> the original. "
> > >>
> > >> I could be wrong though, does anyone know the details of MacOS
> > licensing /
> > >> virtualization?
> > >>
> > >> On Tue, Dec 12, 2017 at 6:10 PM, Chris Olivier <cjolivier01@gmail.com
> >
> > >> wrote:
> > >>
> > >> > googling seems to be full of running OSX (and even open-sourced
> > >> PureDarwin)
> > >> > in VMs. One could conceivably run a VM on an EC2 instance, right?
> > >> >
> > >> > On Tue, Dec 12, 2017 at 9:01 AM kellen sunderland <
> > >> > kellen.sunderland@gmail.com> wrote:
> > >> >
> > >> > > It would be ideal if we could cover OSX in Jenkins, but the only
> > >> solution
> > >> > > that I'm aware of would require physical machines to be the
> workers.
> > >> I
> > >> > > would be weakly opposed to having physical servers running on
PRs.
> > >> The
> > >> > > downsides that I see in order of importance:
> > >> > >
> > >> > > -  We can't autoscale physical hardware.   If we find that the
> load
> > is
> > >> > too
> > >> > > high we have to buy more machines.
> > >> > > -  Security would be tricky, as they'd have to be connected to
the
> > >> > internet
> > >> > > and then to our Jekins master instance.  Connecting via a wired
> > >> network
> > >> > > would probably not be possible on most corporate networks as
these
> > >> > machines
> > >> > > are by definition running arbitrary code from the internet. 
Many
> > >> > corporate
> > >> > > sites have public wifi that this machine could potentially connect
> > to,
> > >> > but
> > >> > > then our PRs start failing if the wifi disconnects temporarily.
> To
> > >> > connect
> > >> > > to the master we would need to setup a vpn solution with endpoints
> > in
> > >> our
> > >> > > vpc on AWS.  This is possible but would probably require a lot
of
> > >> > security
> > >> > > work.
> > >> > > -  We can't just create a simple startup script or yaml file
that
> is
> > >> > > checked into GitHub to manage the machine.  Someone will actually
> > >> have to
> > >> > > physically administer the machine, apply updates, etc. which
will
> > make
> > >> > > community ownership difficult.
> > >> > >
> > >> > > Specific to an OSX build:
> > >> > > -  We can't virtualize OSX which means we'd only be able to cover
> > one
> > >> OSX
> > >> > > build environment per physical device.  We couldn't target a
> matrix
> > of
> > >> > OSX
> > >> > > and Xcode versions as in Travis.
> > >> > >
> > >> > > -Kellen
> > >> > >
> > >> > > On Tue, Dec 12, 2017 at 5:46 PM, Chris Olivier <
> > cjolivier01@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > So why Travis when we could possibly use Jenkins?
> > >> > > >
> > >> > > > On Tue, Dec 12, 2017 at 7:59 AM Marco de Abreu <
> > >> > > > marco.g.abreu@googlemail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Yes that's correct, Chris.
> > >> > > > >
> > >> > > > > Am 12.12.2017 4:46 nachm. schrieb "Chris Olivier" <
> > >> > > cjolivier01@gmail.com
> > >> > > > >:
> > >> > > > >
> > >> > > > > > A quick google search seems to indicate that Mac
can be used
> > as
> > >> a
> > >> > > > Jenkins
> > >> > > > > > slave. Is this correct?
> > >> > > > > >
> > >> > > > > > On Tue, Dec 12, 2017 at 7:42 AM Steffen Rochel
<
> > >> > > > steffenrochel@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > +1 for #1 and #2
> > >> > > > > > >
> > >> > > > > > > I’m working on getting a MacPro to add
to CI system.
> > >> > > > > > > On Tue, Dec 12, 2017 at 1:43 AM kellen sunderland
<
> > >> > > > > > > kellen.sunderland@gmail.com> wrote:
> > >> > > > > > >
> > >> > > > > > > > Background:  TravisCI is a startup providing
managed
> > >> continuous
> > >> > > > > > > > integration services with GitHub integration
and YAML
> > based
> > >> > > > > > > configuration.
> > >> > > > > > > > TravisCI is one of the few CI providers
that will build
> a
> > >> > variety
> > >> > > > of
> > >> > > > > > > > OSX/MacOS builds for software projects.
 Their pricing
> > >> ranges
> > >> > > from
> > >> > > > > Free
> > >> > > > > > > > (for open source, 1 concurrent job,
to $489 monthly for
> 10
> > >> > > > concurrent
> > >> > > > > > > jobs).
> > >> > > > > > > >
> > >> > > > > > > > Problem: We’ve had a few OSX build
issues slip into
> MXNet
> > >> > master
> > >> > > in
> > >> > > > > the
> > >> > > > > > > > past few weeks.  We’ve previously
had a Travis CI based
> > >> testing
> > >> > > > > system
> > >> > > > > > > that
> > >> > > > > > > > would have caught these issues.
> > >> > > > > > > >
> > >> > > > > > > > Proposals so far:
> > >> > > > > > > >
> > >> > > > > > > > 1) Use TravisCI in it’s free mode
for a very minimal
> > sanity
> > >> > check
> > >> > > > on
> > >> > > > > > OSX.
> > >> > > > > > > > If we compile the program, and for example
run C++ unit
> > >> tests
> > >> > > we’re
> > >> > > > > > > > unlikely to run into problems with queued
builds.  The
> > total
> > >> > > build
> > >> > > > > time
> > >> > > > > > > > here should be less than 15 minutes.
 Configuration
> should
> > >> be
> > >> > > quite
> > >> > > > > > > simple
> > >> > > > > > > > and easy to maintain.  Error messages
should also be
> > >> obvious to
> > >> > > > > > > > contributors.
> > >> > > > > > > > 2) Run clang in Linux with our current
CI.  Building
> with
> > >> clang
> > >> > > > > should
> > >> > > > > > > > take less than 10 minutes, should flush
out a large
> subset
> > >> of
> > >> > the
> > >> > > > > > issues
> > >> > > > > > > > we’ve seen with OSX, and be quite
easy to maintain.
> > >> > > > > > > > 3) Run full test-suites in TravisCI,
equaling the level
> of
> > >> > > coverage
> > >> > > > > we
> > >> > > > > > > > provide to Linux in Jenkins.  This could
require us to
> > >> > subscribe
> > >> > > > to a
> > >> > > > > > > > monthly package with Travis to ensure
our build queue
> > >> doesn’t
> > >> > > grow
> > >> > > > to
> > >> > > > > > an
> > >> > > > > > > > unacceptable length.  It may also require
a volunteer to
> > >> setup
> > >> > > and
> > >> > > > > > > maintain
> > >> > > > > > > > long-term.
> > >> > > > > > > >
> > >> > > > > > > > I’d +1 #1 and #2 as I think those
should be low-cost,
> > >> > > low-maintence
> > >> > > > > > > > solutions that should catch the majority
of the problems
> > >> we’ve
> > >> > > seen
> > >> > > > > > thus
> > >> > > > > > > > far.
> > >> > > > > > > >
> > >> > > > > > > > -Kellen
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message