aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Chu-Carroll <mchucarr...@apache.org>
Subject Re: Updated proposal: Aurora Shorthands
Date Tue, 27 May 2014 18:28:51 GMT
I think that the answer there is that we want to *reduce* scripting, not
*eliminate* it.

Scripting happens for a bunch of different reasons:
(1) Filling functional gaps, where the original application is missing some
key functionality that users really need. For example, in my last job, we
had a script to submit code for code review.  The application we used for
code review used a very specific model of branching for submitting reviews,
which we thought was awkward and contrived. So we wrote scripts, so that we
didn't need to do it.
(2) Customizing an application for specific workflows. For example,
specific aurora users will have deploy processes that run specific tests,
create canaries in a particular way, etc.
(3) Patching around obscurity/complexity. Users don't like to type the same
long-winded things over and over again, or remember complex or obscure
parameters. For example, I've got shell scripts for running rbt in the
apache aurora codebase. I can't remember that the obscure  "-o -g" means
"create a new review", but "-o -r" means update an existing review. So I
made some helpers so that I don't need to remember them.

Type (1) scripts represent a clear lack in the application. If people need
to write type(1) scripts for aurora, that means that we've failed to
deliver an adequate tool for what they need.

Type (2) scripts are inevitable, and we shouldn't worry about them at all.

Type (3) scripts are also a sign of failure in the application.

I don't typically wrap git in a bunch of clumsy aliases, because, while I'm
not crazy about the way it works, for most routine tasks, the parameters
are clear, and I don't have trouble remembering them.

On the other hand, for pants, I set up command aliases - not because it's
hard to remember things, but because it's damned awkward to have to type
out "pants test/python/apache/aurora/client/cli:all" instead of just
"runtests".

For aurora, we've got a system which, I think, does a decent job of
providing users with the functionality that they need.  (Obviously it could
do better, and it's steadily moving in the direction of getting better at
it.) But what we don't do is make it *easy* to use. In the category 3
arena, I think we've got a massive failing. A typical aurora command
requires an easy to remember noun/verb pair - and then a 4-part jobkey
(which users have trouble remember the order of the key parts), and a long
path to a configuration file. For a typical invocation of the aurora
client, the parameters are about 70 characters long. Those 70 characters
are the same, over and over again, with minimal variation.

That's a problem that creates a need for users to write scripts, because
we're not doing a good enough job of covering their common usecases -
instead, we confront them with a need for verbose obscure parameters that
they need to remember.

The defaults/init file proposal is an attempt to address that case - making
the system easier to use for the most common use-cases, for the typical
users. They won't need to write wrapper scripts if we provide them with an
easy way of doing what they need to do.

   -Mark






On Sat, May 24, 2014 at 1:52 AM, Bill Farner <wfarner@apache.org> wrote:

> >
> > In order to avoid these problems, users set up custom scripts
> > that make it easy for them to run commands like "`myservice start`",
> > instead of "`aurora job create west/markcc/prod/myserver
> > src/main/aurora/myservice/myservice.aurora`"
> > To reduce this...
>
>
> The missing detail seems to be why the custom scripts need to be reduced.
>  In my experience, people do this to automate a deployment workflow, which
> aurora happens to be a part of.  Even with the shorthands, i'm pretty
> confident people will still have a script to roll up the "build/test", and
> "check that the service works" steps.  If that holds true, it seems that
> the goal of reducing scripting is not met.
>
>
>
> -=Bill
>
>
> On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> <mchucarroll@apache.org>wrote:
>
> > The bulk of the feedback from the first round of comments on this could
> be
> > summed up, roughly "get rid of the stupid command shortcuts you dope,
> > they're a menace!". Message received: they're gone.
> >
> > But I still think that the configurable defaults are valuable. Based on
> > some feedback, I've revised the proposal a bit. This includes trying to
> > strengthen the motivation/problem statement, and adding ways to provide
> > default parameters only for specific commands, plus many small changes.
> >
> > The updated proposal is attached below.
> >
> >
> > # Defaults and Shorthands in the Aurora Client
> >
> > ## Motivation
> >
> > Most of the time, users are doing the same thing, over and over again.
> > They're working mainly on one particular service, in one particular
> > workspace. But they need to repeat the same parameters, over and over
> > again, and they need to remember what those parameters are, what order
> > they occur in, what format they use, and other details that can be
> > difficult to remember.
> >
> > In order to avoid these problems, users set up custom scripts
> > that make it easy for them to run commands like "`myservice start`",
> > instead of "`aurora job create west/markcc/prod/myserver
> > src/main/aurora/myservice/myservice.aurora`"
> >
> > To reduce this, we'd like to support a way for users to set up a
> > configuration file that defines defaults and shorthands for their
> > everyday work. With shorthands, a user that only works with a single
> > service could say "aurora job create", instead of needing to spell
> > out the full jobkey and configuration file location; a user working with
> > multiple datacenters could say "`aurora job create @east`" or
> > "`aurora job create @west`" to select the correct jobkey.
> >
> > ## Proposal: Aurora Init Files
> >
> > To allow users to customize shorthands, we'll provide a
> > builtin capability to allow users to provide a configuration
> > file, from which their customizations will be loaded.
> > Many applications use a simple pattern to solve similar problems.
> > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > don't specifically tell the tool where to find a configuration, it looks
> > in the current directory or one of its parents to find a file named
> > "Vagrantfile".
> >
> > We'd like to follow a similar pattern, and create an "AuroraInit" file.
> > The aurora init file is found by searching the following locations, in
> > order:
> >
> >    * the contents of the "--init-file" parameter.
> >    * if the "--init-file" parameter is unspecified, then look in the
> > current
> >      directory for a file named "AuroraInit".
> >    * if no "AuroraInit" file exists in the current directory, then look
> in
> > the users
> >      home directory for an init file named ".AuroraInit".
> >
> >
> > ### What goes into an init file?
> >
> > We should support the following kinds of things:
> >
> >    1. Universal defaults - user-defined default settings that will be
> > applied to
> >      all commands. For example, if there is a default config file that
> > should always
> >      be used if the user doesn't specify one, that would be a universal
> > default.
> >    2. Command specific defaults - users should be able to specify that
> they
> > always want
> >      to use certain parameter settings for a specific command. For
> example,
> > if they
> >      want to always use a default batch size of 10 for updates, but don't
> > want to affect
> >      batch sizes for other commands like kill, they could use a command
> > specific default.
> >     3. Aliases - shorthand names for longer parameters. A user could
> > specify shorthands
> >       "east" and "west" for full jobkeys in two different datacenters.
> >
> > ### Using aliases and defaults
> >
> > A default specifies a set of _bindings_. If a parameter is omitted
> > from a command, and there's a binding for that parameter, it will be
> > automatically inserted into the command as if the user had typed it.
> >
> > An alias is a short equivalent for a parameter. When a command line is
> > provided by a user, aliases will be expanded inline.  A user can
> > specifically
> > mark an alias for expansion by prefixing it with "@"; if an alias
> > appears on the command-line surrounded by whitespace, it will be
> > replaced even if it isn't marked with an "@".
> >
> >
> > ### Defining Shorthands and Defaults
> >
> > The init file is, like aurora job configurations, a python source file
> > using a Pystachio
> > schema. The schema is loaded, and an `Init` object is retrieved from the
> > top-level
> > variable `init` in that file.
> >
> > The pystachio schema for init files is:
> >
> >     class CommandDefaults(Struct)
> >       command = Required(String)
> >       defaults=Map(String, String)
> >
> >     class Init(Struct):
> >       defaults=Map(String, String)
> >       command_defaults = Optional(CommandDefaults)
> >       aliases=Map(String, String)
> >
> >
> > For example, an init file could contain the following:
> >
> >     init=Init(
> >       defaults={
> >         "jobspec": "west/markcc/test/myservice",
> >         "config_file": "./src/aurora/myservice.aurora"
> >       },
> >       aliases={
> >         "east": "east/markcc/test/myservice",
> >         "c": "east",
> >         "me": "markchucarroll"
> >       },
> >       command_defaults(command="job update",
> >         defaults={"--batch-size": 10}
> >       ))
> >
> >
> > With this configuration file, if the user ran "`aurora job create`"
> without
> > any parameters,
> > it would automatically be expanded to "`aurora job create
> > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create east`", the alias `east` would be
> > expanded to "east/markcc/test/myservice", and the missing config file
> > parameter would
> > be instantiated using the default, to create a command-line:
> > "`aurora job create east/markcc/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> aliases
> > would
> > be expanded, and the omitted configuration parameter would be added:
> > "`aurora job create east/markchucarroll/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> > parameters would
> > get inserted from the global defaults, and the "--batch-size=10" would be
> > inserted from
> > the command defaults for "aurora job update".
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message