mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Olivier <cjolivie...@gmail.com>
Subject Re: [DISCUSS] Build OSX builds in CI (possibly with TravisCI).
Date Tue, 12 Dec 2017 17:47:52 GMT
Don’t get me wrong, I’m not saying this Mac OS Jenkins solution is doable
but I feel like we should investigate because the payoff would be large.


On Tue, Dec 12, 2017 at 9:38 AM Chris Olivier <cjolivier01@gmail.com> wrote:

> Apple’s Darwin OS Is recently open-sourced.
> https://github.com/PureDarwin/PureDarwin
>
> How to convert this into a non-GUI VM I am not sure but I am willing to
> bet that people have done it already.
>
> On Tue, Dec 12, 2017 at 9:16 AM kellen sunderland <
> kellen.sunderland@gmail.com> wrote:
>
>> It might be technically possible, but I think it would violate the MacOS
>> license: http://store.apple.com/Catalog/US/Images/MacOSX.htm
>>
>> "2. Permitted License Uses and Restrictions.
>> A. This License allows you to install and use one copy of the Apple
>> Software on a single Apple-labeled computer at a time. This License does
>> not allow the Apple Software to exist on more than one computer at a
>> time,and you may not make the Apple Software available over a network
>> where
>> it could be used by multiple computers at the same time. You may make one
>> copy of the Apple Software (excluding the Boot ROM code) in
>> machine-readable form for backup purposes only; provided that the backup
>> copy must include all copyright or other proprietary notices contained on
>> the original. "
>>
>> I could be wrong though, does anyone know the details of MacOS licensing /
>> virtualization?
>>
>> On Tue, Dec 12, 2017 at 6:10 PM, Chris Olivier <cjolivier01@gmail.com>
>> wrote:
>>
>> > googling seems to be full of running OSX (and even open-sourced
>> PureDarwin)
>> > in VMs. One could conceivably run a VM on an EC2 instance, right?
>> >
>> > On Tue, Dec 12, 2017 at 9:01 AM kellen sunderland <
>> > kellen.sunderland@gmail.com> wrote:
>> >
>> > > It would be ideal if we could cover OSX in Jenkins, but the only
>> solution
>> > > that I'm aware of would require physical machines to be the workers.
>> I
>> > > would be weakly opposed to having physical servers running on PRs.
>> The
>> > > downsides that I see in order of importance:
>> > >
>> > > -  We can't autoscale physical hardware.   If we find that the load is
>> > too
>> > > high we have to buy more machines.
>> > > -  Security would be tricky, as they'd have to be connected to the
>> > internet
>> > > and then to our Jekins master instance.  Connecting via a wired
>> network
>> > > would probably not be possible on most corporate networks as these
>> > machines
>> > > are by definition running arbitrary code from the internet.  Many
>> > corporate
>> > > sites have public wifi that this machine could potentially connect to,
>> > but
>> > > then our PRs start failing if the wifi disconnects temporarily.  To
>> > connect
>> > > to the master we would need to setup a vpn solution with endpoints in
>> our
>> > > vpc on AWS.  This is possible but would probably require a lot of
>> > security
>> > > work.
>> > > -  We can't just create a simple startup script or yaml file that is
>> > > checked into GitHub to manage the machine.  Someone will actually
>> have to
>> > > physically administer the machine, apply updates, etc. which will make
>> > > community ownership difficult.
>> > >
>> > > Specific to an OSX build:
>> > > -  We can't virtualize OSX which means we'd only be able to cover one
>> OSX
>> > > build environment per physical device.  We couldn't target a matrix of
>> > OSX
>> > > and Xcode versions as in Travis.
>> > >
>> > > -Kellen
>> > >
>> > > On Tue, Dec 12, 2017 at 5:46 PM, Chris Olivier <cjolivier01@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > So why Travis when we could possibly use Jenkins?
>> > > >
>> > > > On Tue, Dec 12, 2017 at 7:59 AM Marco de Abreu <
>> > > > marco.g.abreu@googlemail.com>
>> > > > wrote:
>> > > >
>> > > > > Yes that's correct, Chris.
>> > > > >
>> > > > > Am 12.12.2017 4:46 nachm. schrieb "Chris Olivier" <
>> > > cjolivier01@gmail.com
>> > > > >:
>> > > > >
>> > > > > > A quick google search seems to indicate that Mac can be
used as
>> a
>> > > > Jenkins
>> > > > > > slave. Is this correct?
>> > > > > >
>> > > > > > On Tue, Dec 12, 2017 at 7:42 AM Steffen Rochel <
>> > > > steffenrochel@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > +1 for #1 and #2
>> > > > > > >
>> > > > > > > I’m working on getting a MacPro to add to CI system.
>> > > > > > > On Tue, Dec 12, 2017 at 1:43 AM kellen sunderland <
>> > > > > > > kellen.sunderland@gmail.com> wrote:
>> > > > > > >
>> > > > > > > > Background:  TravisCI is a startup providing managed
>> continuous
>> > > > > > > > integration services with GitHub integration and
YAML based
>> > > > > > > configuration.
>> > > > > > > > TravisCI is one of the few CI providers that will
build a
>> > variety
>> > > > of
>> > > > > > > > OSX/MacOS builds for software projects.  Their
pricing
>> ranges
>> > > from
>> > > > > Free
>> > > > > > > > (for open source, 1 concurrent job, to $489 monthly
for 10
>> > > > concurrent
>> > > > > > > jobs).
>> > > > > > > >
>> > > > > > > > Problem: We’ve had a few OSX build issues slip
into MXNet
>> > master
>> > > in
>> > > > > the
>> > > > > > > > past few weeks.  We’ve previously had a Travis
CI based
>> testing
>> > > > > system
>> > > > > > > that
>> > > > > > > > would have caught these issues.
>> > > > > > > >
>> > > > > > > > Proposals so far:
>> > > > > > > >
>> > > > > > > > 1) Use TravisCI in it’s free mode for a very
minimal sanity
>> > check
>> > > > on
>> > > > > > OSX.
>> > > > > > > > If we compile the program, and for example run
C++ unit
>> tests
>> > > we’re
>> > > > > > > > unlikely to run into problems with queued builds.
 The total
>> > > build
>> > > > > time
>> > > > > > > > here should be less than 15 minutes.  Configuration
should
>> be
>> > > quite
>> > > > > > > simple
>> > > > > > > > and easy to maintain.  Error messages should also
be
>> obvious to
>> > > > > > > > contributors.
>> > > > > > > > 2) Run clang in Linux with our current CI.  Building
with
>> clang
>> > > > > should
>> > > > > > > > take less than 10 minutes, should flush out a
large subset
>> of
>> > the
>> > > > > > issues
>> > > > > > > > we’ve seen with OSX, and be quite easy to maintain.
>> > > > > > > > 3) Run full test-suites in TravisCI, equaling
the level of
>> > > coverage
>> > > > > we
>> > > > > > > > provide to Linux in Jenkins.  This could require
us to
>> > subscribe
>> > > > to a
>> > > > > > > > monthly package with Travis to ensure our build
queue
>> doesn’t
>> > > grow
>> > > > to
>> > > > > > an
>> > > > > > > > unacceptable length.  It may also require a volunteer
to
>> setup
>> > > and
>> > > > > > > maintain
>> > > > > > > > long-term.
>> > > > > > > >
>> > > > > > > > I’d +1 #1 and #2 as I think those should be
low-cost,
>> > > low-maintence
>> > > > > > > > solutions that should catch the majority of the
problems
>> we’ve
>> > > seen
>> > > > > > thus
>> > > > > > > > far.
>> > > > > > > >
>> > > > > > > > -Kellen
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message