aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "benley@gmail.com" <ben...@gmail.com>
Subject Re: Nix and Aurora
Date Thu, 13 Aug 2015 20:21:32 GMT
I've also had tremendous success with using Nix and Aurora together, but
haven't had such nice integration of the job definitions with Nix itself -
thank you for sharing this!

In case anyone else takes interest in this stuff: I took a different
approach, writing the job descriptions with Pystachio, while still using
Nix to package and distribute all dependencies.  A binding helper
translates references to packages (i.e. Nix attributes) from their name to
the store path where they can be accessed, and Nix takes care of retrieving
anything not already cached from a networked binary cache.  On the client
side, a simple script bundles the source derivation and/or the binary
outputs and publishes them to that network location prior to submitting a
job to the Aurora scheduler.

One nice thing about my method is that it doesn't use a nix profile for
each task, so there is also no need for post-task cleanup other than the
sandbox garbage collection that aurora and mesos already handle.

I like Rafik's implementation a lot, and will likely be using it at some
point, once some of these concepts are bridged together.  I'll share my
hacky version for reference or inspiration:

Binding helper and example jobs:
https://gist.github.com/benley/17f94a19c9b57b464a06

Hacky build/publish/launch script:
https://gist.github.com/benley/f1ce058a2f913674c408


On Thu, Aug 13, 2015 at 12:48 PM Rafik Salama <rafik@oysterbooks.com> wrote:

> I don't know how many people here would be interested in this, but I'm
> hoping to draw some interest from the list (besides benley who inspired the
> work).
>
> We've been using Nix <http://nixos.org/nix/> to build and distribute the
> programs we run on Aurora for a few months. It's been a tremendous success.
> We first came across Nix when looking for an alternative to Docker for
> isolating package dependencies and avoiding running jobs as root. Mesos
> gives us the isolation we need, so Docker seemed an unjustifiable overhead
> simply to isolate dependences.
>
> In our current setup, we create a Nix profile for each task, install the
> dependencies, run the processes, then clean up the Nix profile. All the
> jobs are configured using the standard Pystachio configuration and we've
> written some Python helpers to wrap the tasks with the proper Nix init and
> cleanup processes. Basically our jobs consist of two files: one which
> defines the dependencies using Nix expressions, and another which specifies
> the Aurora processes and tasks. The Aurora processes depend on the
> dependencies defined in Nix, but the link is implicit.
>
> You can see where I'm going with this.
>
> So, to form a more perfect union, I've been experimenting with replacing
> pystachio configurations with Nix expressions. I have a proof of concept of
> the work here: https://github.com/rafikk/nix-aurora
>
> The repo contains the hello_world.py example from the Aurora tutorial. The
> main thing to note is what happens to the fetch_package process. Since Nix
> produces a cryptographic hash of its inputs, we can avoid the checksum
> "trick". And since the package is really a dependency of the "hello_world"
> process, we can model this explicitly by specifying the package as a build
> input to the process and remove the "fetch_package" process altogether. Nix
> takes care of building all the package dependencies and provides the store
> paths. (this means we can even avoid copying the code into the sandbox in
> the first place. just reference it from /nix/store).
>
> As it's written, the example works with the Vagrant VM. A bit more work is
> required to distribute the sandbox and dependencies to the executors.
>
> I hope the benefits are apparent. Please let me know if you're interested
> in getting this to production-grade or have any feedback!
>
> Thanks,
> Rafik
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message