aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Chu-Carroll <mchucarr...@apache.org>
Subject Re: Updated proposal: Aurora Shorthands
Date Thu, 29 May 2014 17:24:58 GMT
WRT to the home directory fallback: I'm not hugely committed to it, but
I've discussed this with a couple of people, and all have liked it, for one
big reason: they're using git in the twitter codebase - so the working
directory that they use contains a ton of different projects. They can't
just put an AuroraInit file in the root directory of the workspace, because
it won't work. So they want to be able to write one, put it in git in some
directory, and then put a link to it into their homedir. Then they have
defaults in place.

WRT to separate init file: I think it's very important that the two be
different. The config file is a shared entity used by multiple people, and
possibly used in different ways by different people. For example, one
member of a team may typically only work with one of a team's services,
while all are defined in the same config file.  The user would to have
defaults to work with the one service that they actually work on - but
making it a default for all users of the config file would not work.

WRT @symbols: yes, we could do substitution via the shell or something
similar. But I think that the argument you made above about putting things
into the config file applies here: having all of the shorthands and
defaults that the user works with be stored in one place is better that
scattering among multiple files.  I also think that you're over-estimating
the degree of memory required to use it successfully; I've certainly seen
many cases of people using stuff like that. (Think about how people use
git; they know the names of their branches, and they know the order of
things for the commands they use most. For normal operation, that's fine.
Back in my google days, we had a crazy git->perforce bridge, and it used
aliases in exactly this way. It was hugely successful among engineers. I
think that this is similar.)

Finally, about the @ symbol: I'm not in love with it, but given my argument
above, if we want that functionality, we have to use something. My first
choice would have been to use pystachio-style stashes - but the braces are
already used by bash; in fact, all of the bracket characters already are
used. For single-character prefixes, there aren't many characters that
aren't already used by the shell - we can't use !, #, $, ^, &, *, ~, or ?.
For easy to type stuff, that left very little other than @. And @ is used
as a variable prefix in perl and ruby, so it's not without precedent or
familiarity to many people.

    -Mark


On Tue, May 27, 2014 at 7:19 PM, David McLaughlin <david@dmclaughlin.com>
wrote:

> I am a huge +1 on the idea of storing my run configurations in config files
> so that I don't need to type them out every time. I also really like the
> idea of command defaults.
>
> I do however have a couple of comments about the details of the proposal.
>
> I'm a little concerned about the fallback to the user's home directory. I
> feel like these config files will be per-project and checked in next to the
> aurora config file for all to share. Given that things like update batch
> sizes probably *should* be defined per-project and not per-user, I at least
> think we should encourage this. It's also not uncommon to run commands in a
> different cwd than you intended, and having such broad fallbacks seems to
> be just asking for trouble. I think it's perfectly fine to expect
> AuroraInit to either be in the correct working directory or for the user to
> supply an explicit path to the file.
>
> I also think once you get to that stage, you can make the observation that
> most production aurora configs will have an AuroraInit file next to them.
> So I wonder if we can just simplify here and let the user roll these into
> one file? It'd be nice to have a naming convention for aurora files so they
> could be dropped from the parameters too.
>
>
> The other concern I have is the syntax for injecting variables into
> commands with the @symbol. It seems like there are already tools for
> solving this - bash, python, etc. that should we just concede defeat to. If
> I'm typing my commands manually and relying on aliases I'd make a bet that
> I'm probably not going to be remembering what I alias'd different parts of
> my jobpath to and again, what the order of the jobpath bits are.
>
>
> Thanks,
> David
>
>
>
> On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> <mchucarroll@apache.org>wrote:
>
> > The bulk of the feedback from the first round of comments on this could
> be
> > summed up, roughly "get rid of the stupid command shortcuts you dope,
> > they're a menace!". Message received: they're gone.
> >
> > But I still think that the configurable defaults are valuable. Based on
> > some feedback, I've revised the proposal a bit. This includes trying to
> > strengthen the motivation/problem statement, and adding ways to provide
> > default parameters only for specific commands, plus many small changes.
> >
> > The updated proposal is attached below.
> >
> >
> > # Defaults and Shorthands in the Aurora Client
> >
> > ## Motivation
> >
> > Most of the time, users are doing the same thing, over and over again.
> > They're working mainly on one particular service, in one particular
> > workspace. But they need to repeat the same parameters, over and over
> > again, and they need to remember what those parameters are, what order
> > they occur in, what format they use, and other details that can be
> > difficult to remember.
> >
> > In order to avoid these problems, users set up custom scripts
> > that make it easy for them to run commands like "`myservice start`",
> > instead of "`aurora job create west/markcc/prod/myserver
> > src/main/aurora/myservice/myservice.aurora`"
> >
> > To reduce this, we'd like to support a way for users to set up a
> > configuration file that defines defaults and shorthands for their
> > everyday work. With shorthands, a user that only works with a single
> > service could say "aurora job create", instead of needing to spell
> > out the full jobkey and configuration file location; a user working with
> > multiple datacenters could say "`aurora job create @east`" or
> > "`aurora job create @west`" to select the correct jobkey.
> >
> > ## Proposal: Aurora Init Files
> >
> > To allow users to customize shorthands, we'll provide a
> > builtin capability to allow users to provide a configuration
> > file, from which their customizations will be loaded.
> > Many applications use a simple pattern to solve similar problems.
> > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > don't specifically tell the tool where to find a configuration, it looks
> > in the current directory or one of its parents to find a file named
> > "Vagrantfile".
> >
> > We'd like to follow a similar pattern, and create an "AuroraInit" file.
> > The aurora init file is found by searching the following locations, in
> > order:
> >
> >    * the contents of the "--init-file" parameter.
> >    * if the "--init-file" parameter is unspecified, then look in the
> > current
> >      directory for a file named "AuroraInit".
> >    * if no "AuroraInit" file exists in the current directory, then look
> in
> > the users
> >      home directory for an init file named ".AuroraInit".
> >
> >
> > ### What goes into an init file?
> >
> > We should support the following kinds of things:
> >
> >    1. Universal defaults - user-defined default settings that will be
> > applied to
> >      all commands. For example, if there is a default config file that
> > should always
> >      be used if the user doesn't specify one, that would be a universal
> > default.
> >    2. Command specific defaults - users should be able to specify that
> they
> > always want
> >      to use certain parameter settings for a specific command. For
> example,
> > if they
> >      want to always use a default batch size of 10 for updates, but don't
> > want to affect
> >      batch sizes for other commands like kill, they could use a command
> > specific default.
> >     3. Aliases - shorthand names for longer parameters. A user could
> > specify shorthands
> >       "east" and "west" for full jobkeys in two different datacenters.
> >
> > ### Using aliases and defaults
> >
> > A default specifies a set of _bindings_. If a parameter is omitted
> > from a command, and there's a binding for that parameter, it will be
> > automatically inserted into the command as if the user had typed it.
> >
> > An alias is a short equivalent for a parameter. When a command line is
> > provided by a user, aliases will be expanded inline.  A user can
> > specifically
> > mark an alias for expansion by prefixing it with "@"; if an alias
> > appears on the command-line surrounded by whitespace, it will be
> > replaced even if it isn't marked with an "@".
> >
> >
> > ### Defining Shorthands and Defaults
> >
> > The init file is, like aurora job configurations, a python source file
> > using a Pystachio
> > schema. The schema is loaded, and an `Init` object is retrieved from the
> > top-level
> > variable `init` in that file.
> >
> > The pystachio schema for init files is:
> >
> >     class CommandDefaults(Struct)
> >       command = Required(String)
> >       defaults=Map(String, String)
> >
> >     class Init(Struct):
> >       defaults=Map(String, String)
> >       command_defaults = Optional(CommandDefaults)
> >       aliases=Map(String, String)
> >
> >
> > For example, an init file could contain the following:
> >
> >     init=Init(
> >       defaults={
> >         "jobspec": "west/markcc/test/myservice",
> >         "config_file": "./src/aurora/myservice.aurora"
> >       },
> >       aliases={
> >         "east": "east/markcc/test/myservice",
> >         "c": "east",
> >         "me": "markchucarroll"
> >       },
> >       command_defaults(command="job update",
> >         defaults={"--batch-size": 10}
> >       ))
> >
> >
> > With this configuration file, if the user ran "`aurora job create`"
> without
> > any parameters,
> > it would automatically be expanded to "`aurora job create
> > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create east`", the alias `east` would be
> > expanded to "east/markcc/test/myservice", and the missing config file
> > parameter would
> > be instantiated using the default, to create a command-line:
> > "`aurora job create east/markcc/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> aliases
> > would
> > be expanded, and the omitted configuration parameter would be added:
> > "`aurora job create east/markchucarroll/test/myservice
> > ./src/aurora/myservice.aurora`".
> >
> > If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> > parameters would
> > get inserted from the global defaults, and the "--batch-size=10" would be
> > inserted from
> > the command defaults for "aurora job update".
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message