aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Smith <>
Subject Re: One more try at defaults and aliases
Date Fri, 13 Jun 2014 18:27:35 GMT
"to set up a configuration file”

This line to me just says “we’re reinventing wrapper scripts” where someone could create
an “alias” like ./ and ./ to do deploys to various clusters. In that sense,
these aliases etc are also “hiding” bad interfaces, just in a different way- and one which
isn’t as well-tested/understood as shell scripts. 

It seems like we’re set on putting time into this, so if we are, then ignore me and move
onward :). IF we were going to go with an approach, this seems reasonable based on ways that
I’ve seen people wrap their configs/invocations of the client.

On Jun 12, 2014, at 7:50 AM, Mark Chu-Carroll <> wrote:

> I'm nothing if not persistent! After feedback from my second try at
> proposing a way of doing defaults and aliases, I've got a third draft.
> Feedback please!
> # Defaults and Shorthands in the Aurora Client
> ## Motivation
> Most of the time, users are doing the same thing, over and over again.
> They're
> working mainly on one particular service, in one particular workspace. But
> they
> need to repeat the same parameters, over and over again, and they need to
> remember what those parameters are, what order they occur in, what format
> they
> use, and other details that can be difficult to remember.
> In order to avoid these problems, users set up custom scripts that make it
> easy
> for them to run commands like "`myservice start`", instead of "`aurora job
>  create west/markcc/prod/myserver
> src/main/aurora/myservice/myservice.aurora`"
> Scripts aren't necessarily a bad thing. We've designed the aurora client to
> be script friendly: every command has at least an option supporting
> easy-to-parse
> output. But scripts also often cover for problems that really shouldn't
> exist.
> Scripts get written for a variety of reasons. Among the many reasons that
> people implement
> scripts, there are scripts that add custom functionality for a specific
> group of users;
> scripts that can cover up for inadequate or missing core functionality; and
> there are
> scripts that cover up for poor user interfaces.
> In aurora, we've worked hard to eliminate cases where the core
> functionality of the
> command-line client is inadequate. But our user interface still needs a lot
> of work.
> In particular, commands in Aurora often require very long parameters, which
> users have
> a hard time remembering. Look at the example above - it's a pretty typical
> case. The
> user wants to create a job. They're going to be creating the same job, over
> and over again.
> But to run it, they need to type out 68 characters for the two parameters!
> This is a
> prime example of the kind of case where users will write scripts, not to
> provide new/special
> functionality, nor to cover for inadequate functionality, but just because
> it's so
> painful to remember and correctly type out an overly long string of
> parameters.
> To reduce this, we'd like to support a way for users to set up a
> configuration
> file that defines defaults and shorthands for their everyday work. With
> shorthands, a user that only works with a single service could say "aurora
> job
> create", instead of needing to spell out the full jobkey and configuration
> file
> location; a user working with multiple datacenters could say "`aurora job
> create @east`" or "`aurora job create @west`" to select the correct jobkey.
> ## Proposal: Aurora Init Files
> To allow users to customize shorthands, we'll provide a
> builtin capability to allow users to provide a configuration
> file, from which their customizations will be loaded.
> Many applications use a simple pattern to solve similar problems.
> Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> don't specifically tell the tool where to find a configuration, it looks
> in the current directory or one of its parents to find a file named
> "Vagrantfile".
> We'd like to follow a similar pattern, and create an "AuroraInit" file.
> The aurora init file is found by searching the following locations, in
> order:
>   * the contents of the "--init-file" parameter.
>   * if the "--init-file" parameter is unspecified, then look in the current
>     directory for a file named "AuroraInit".
>   * if no "AuroraInit" file exists in the current directory, then look in
> the users
>     home directory for an init file named ".aurora/init".
>> **Sidebar: Why fallback to the users home directory?**
>> Codebases are often shared between multiple projects, each of which lives
> in
>> a set of subdirectories within the codebase. For example, just look at
> the aurora
>> sourcecode, which includes the aurora scheduler, the aurora executor,
> thermos,
>> and the aurora client.
>> If multiple services live in a codebase, then users can't put an
> AuroraInit file in
>> the root directory of the codebase, because it would interfere with other
> users'
>> work.
>> The home directory fallback provides an easy way for users to set up a
> pointer
>> to the correct init file, which won't be removed by operations like
> switching branches
>> in a source repository. Users write shared init files, which are located
> in their projects
>> in the codebase, and create a symbolic link from their project init file
> to their home
>> directory.
> ### What goes into an init file?
> We should support the following kinds of things:
>   1. Universal defaults - user-defined default settings that will be
> applied to
>     all commands. For example, if there is a default config file that
> should always
>     be used if the user doesn't specify one, that would be a universal
> default.
>   2. Command specific defaults - users should be able to specify that they
> always want
>     to use certain parameter settings for a specific command. For example,
> if they
>     want to always use a default batch size of 10 for updates, but don't
> want to affect
>     batch sizes for other commands like kill, they could use a command
> specific default.
>    3. Aliases - shorthand names for longer parameters. A user could
> specify shorthands
>      "east" and "west" for full jobkeys in two different datacenters.
> ### Defaults
> A default specifies a set of _bindings_. If a parameter is omitted
> from a command, and there's a binding for that parameter, it will be
> automatically inserted into the command as if the user had typed it.
> The binding is specified in the configuration file using a Python
> dictionary.
> For example, if the defaults included  `{'jobspec':
> 'devcluster/me/prod/service'}`,
> then if you ran `aurora job create` without specifying any parameters, the
> command-line
> would automatically substitute `devcluster/me/prod/service` for the
> `jobspec` parameter.
> Defaults can be declared either globally (in which case they'll be inserted
> as parameters
> for all commands), or for specific commands (in which case they'll only be
> inserted for a
> single command).
> ### Aliases
> An alias is a short equivalent for a parameter. When a command line is
> provided by a user, aliases will be expanded inline.  A user can
> specifically
> mark an alias for expansion by prefixing it with "@"; if an alias
> appears on the command-line surrounded by whitespace, it will be
> replaced even if it isn't marked with an "@".
>> **Sidebar: why not just use shell substitutions?**
>> The @-substitution model proposed here is very similar to unix variable
>> substitution. So why implement the same thing all over again? Why not
> just tell
>> users to use their shell?
>> There are several reasons.
>> 1. All of the options and aliases that affect aurora command invocations
> can be specified
>> in one place: the AuroraInit file.
>> 2. Command shell substitution is complex - the ways in which the command
> shell does
>>  expansion, tokenization, substitution, and parsing can be very hard to
> follow. Adding
>>  the aurora aliases to that process just adds another layer of confusion
> to the process.
>>  With internal alias substitution, we can avoid the confusions of
> expansions and
>>  tokenization for aurora aliases.
>> 3. We can provide better debugging and error messages with our own alias
> substitution.
>>  For example, if a user specifies a job using aliases like "`aurora job
> create @c/@me/test/myservice`",
>>  we can create an error message that shows exactly what they typed, and
> how it expanded:
>> <pre>
>> Job "west/markcc/test/myservice" not found in configuration file
> config.aurora.
>> Jobkey was generated by alias expansion: original key was
> "@c/@me/test/myservice", where:
>>   - "@c" was expanded to "west"
>>   - "@me" was expanded to "markcc"
>>  config_file parameter "config.aurora" was specified from defaults.
>> </pre>
> ### Defining Shorthands and Defaults: Syntax and Examples
> The init file is, like aurora job configurations, a python source file
> using a Pystachio
> schema. The schema is loaded, and an `Init` object is retrieved from the
> top-level
> variable `init` in that file.
> The pystachio schema for init files is:
>    class CommandDefaults(Struct)
>      command = Required(String)
>      defaults=Map(String, String)
>    class Init(Struct):
>      defaults=Map(String, String)
>      command_defaults = Optional(CommandDefaults)
>      aliases=Map(String, String)
> For example, an init file could contain the following:
>    init=Init(
>      defaults={
>        "jobspec": "west/markcc/test/myservice",
>        "config_file": "./src/aurora/myservice.aurora"
>      },
>      aliases={
>        "east": "east/markcc/test/myservice",
>        "c": "east",
>        "me": "markchucarroll"
>      },
>      command_defaults(command="job update",
>        defaults={"--batch-size": 10}
>      ))
> With this configuration file, if the user ran "`aurora job create`" without
> any parameters,
> it would automatically be expanded to "`aurora job create
> west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> If a user ran "`aurora job create east`", the alias `east` would be
> expanded to "east/markcc/test/myservice", and the missing config file
> parameter would
> be instantiated using the default, to create a command-line:
> "`aurora job create east/markcc/test/myservice
> ./src/aurora/myservice.aurora`".
> If a user ran "`aurora job create @c/@me/test/myservice`", the two aliases
> would
> be expanded, and the omitted configuration parameter would be added:
> "`aurora job create east/markchucarroll/test/myservice
> ./src/aurora/myservice.aurora`".
> If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> parameters would
> get inserted from the global defaults, and the "--batch-size=10" would be
> inserted from
> the command defaults for "aurora job update".

View raw message