Mailing-List: contact dev-help@aurora.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAOOJoEy2BK8rm-be-9BXFM0dfbDTBV9x6aN0SuJFTX0DJKr4fg@mail.gmail.com>
References: 
 <CAFGkSCnm+x2n4a3EdDs2xuxyjLz3U+Y9x_2uV_DQheu2hP23HQ@mail.gmail.com>
	<CAOOJoEwR+20ToFp7CJGtmN7AHqHK8FKdDtXkLv9DOJKpDrgc2g@mail.gmail.com>
	<CAFGkSCnG6CLe8w9tjxWQFxO0tysU8BFrSET-T=J=+_gG85ij=g@mail.gmail.com>
	<CAOOJoEy2BK8rm-be-9BXFM0dfbDTBV9x6aN0SuJFTX0DJKr4fg@mail.gmail.com>
Date: Thu, 29 May 2014 16:23:12 -0400
Message-ID: 
 <CAFGkSCkssVX6S8Ei71Ae6A=UoR80eRoxAZFpyAdW2uOz3LP2hw@mail.gmail.com>
Subject: Re: Updated proposal: Aurora Shorthands
From: Mark Chu-Carroll <mchucarroll@apache.org>
To: dev@aurora.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113455f4b8493b04fa8fb253

--001a113455f4b8493b04fa8fb253
Content-Type: text/plain; charset=UTF-8

The reason that I use an alias for rbt isn't because I'm trying to save
keystrokes. It's because "-o", "-g" and "-r" mean nothing to me.  More
keystrokes is fine, if it comes with clarity - that's why I went along with
dropping the automatic shorthands.

The stuff that you can do with Brian's aliases proposal does work equally
well in mine; any alias is automatically substituted, with or without "@",
if it's used in bare form (that is, not as a part of another string in the
command line.)  Substituting inline, though, I still believe is useful. If
we implement it, and find that no one uses it, I'll happily remove it.

    -Mark


On Thu, May 29, 2014 at 4:10 PM, David McLaughlin <david@dmclaughlin.com>
wrote:

> I see the point about having global client defaults, but I still think we
> should avoid inferring the jobkey from the home directory. I'd suggest only
> inferring command_defaults, or even storing them separately - but this also
> adds complexity. Interested in what other people think here.
>
> With regards to symbols and the general theme of keystrokes, specifically
> this point:
>
> "I also think that you're over-estimating the degree of memory required to
> > use it successfully; I've certainly seen
> > many cases of people using stuff like that."
>
>
> Look at rbt, which you mentioned you wrote a wrapper around.
>
> This posts a new review:
>
> ./rbt post -o -g
>
> This updates an existing review:
>
> ./rbt post -o -r <id>
>
> This has very few key strokes. It has only one parameter change between the
> two commands that you need to remember. It is very simple when you look at
> it in an isolated fashion. Yet people will still write wrappers around it
> (or just hope it's in my bash history like I do).
>
> In this case I can't imagine constructing partial job keys from variables
> would save you anything. I think if I'm running the commands often enough
> that I'd remember the aliases, I'd either most likely be able to remember
> my cluster and role names too - or more realistically, I'd just add the
> full jobkey path to my AuroraInit file and be done with it.
>
> I'm only just catching on to the fact that you can't define more than one
> job key in the AuroraInit file and aliases might be the way around that. To
> solve that, I actually liked Brian's suggestion in the other thread:
>
> > Perhaps a stop-gap would be to require explicit aliases, e.g.
> >
> >   [aliases]
> >   test-west = wickman/west/test/service
> >   test-east = wickman/east/test/service
> >   prod-west = wickman/west/test/service
> >   prod-east = wickman/east/test/service
> >
>
> Maybe we could ask users to define pystachio structs with all their
> options, then have a profile (or target) option like:
>
> ./aurora job update --profile=prod
> ./aurora job kill --profile=prod
>
> etc.
>
>
> On Thu, May 29, 2014 at 10:24 AM, Mark Chu-Carroll <mchucarroll@apache.org
> >
> wrote:
>
> > WRT to the home directory fallback: I'm not hugely committed to it, but
> > I've discussed this with a couple of people, and all have liked it, for
> one
> > big reason: they're using git in the twitter codebase - so the working
> > directory that they use contains a ton of different projects. They can't
> > just put an AuroraInit file in the root directory of the workspace,
> because
> > it won't work. So they want to be able to write one, put it in git in
> some
> > directory, and then put a link to it into their homedir. Then they have
> > defaults in place.
> >
> > WRT to separate init file: I think it's very important that the two be
> > different. The config file is a shared entity used by multiple people,
> and
> > possibly used in different ways by different people. For example, one
> > member of a team may typically only work with one of a team's services,
> > while all are defined in the same config file.  The user would to have
> > defaults to work with the one service that they actually work on - but
> > making it a default for all users of the config file would not work.
> >
> > WRT @symbols: yes, we could do substitution via the shell or something
> > similar. But I think that the argument you made above about putting
> things
> > into the config file applies here: having all of the shorthands and
> > defaults that the user works with be stored in one place is better that
> > scattering among multiple files.  I also think that you're
> over-estimating
> > the degree of memory required to use it successfully; I've certainly seen
> > many cases of people using stuff like that. (Think about how people use
> > git; they know the names of their branches, and they know the order of
> > things for the commands they use most. For normal operation, that's fine.
> > Back in my google days, we had a crazy git->perforce bridge, and it used
> > aliases in exactly this way. It was hugely successful among engineers. I
> > think that this is similar.)
> >
> > Finally, about the @ symbol: I'm not in love with it, but given my
> argument
> > above, if we want that functionality, we have to use something. My first
> > choice would have been to use pystachio-style stashes - but the braces
> are
> > already used by bash; in fact, all of the bracket characters already are
> > used. For single-character prefixes, there aren't many characters that
> > aren't already used by the shell - we can't use !, #, $, ^, &, *, ~, or
> ?.
> > For easy to type stuff, that left very little other than @. And @ is used
> > as a variable prefix in perl and ruby, so it's not without precedent or
> > familiarity to many people.
> >
> >     -Mark
> >
> >
> > On Tue, May 27, 2014 at 7:19 PM, David McLaughlin <david@dmclaughlin.com
> >
> > wrote:
> >
> > > I am a huge +1 on the idea of storing my run configurations in config
> > files
> > > so that I don't need to type them out every time. I also really like
> the
> > > idea of command defaults.
> > >
> > > I do however have a couple of comments about the details of the
> proposal.
> > >
> > > I'm a little concerned about the fallback to the user's home
> directory. I
> > > feel like these config files will be per-project and checked in next to
> > the
> > > aurora config file for all to share. Given that things like update
> batch
> > > sizes probably *should* be defined per-project and not per-user, I at
> > least
> > > think we should encourage this. It's also not uncommon to run commands
> > in a
> > > different cwd than you intended, and having such broad fallbacks seems
> to
> > > be just asking for trouble. I think it's perfectly fine to expect
> > > AuroraInit to either be in the correct working directory or for the
> user
> > to
> > > supply an explicit path to the file.
> > >
> > > I also think once you get to that stage, you can make the observation
> > that
> > > most production aurora configs will have an AuroraInit file next to
> them.
> > > So I wonder if we can just simplify here and let the user roll these
> into
> > > one file? It'd be nice to have a naming convention for aurora files so
> > they
> > > could be dropped from the parameters too.
> > >
> > >
> > > The other concern I have is the syntax for injecting variables into
> > > commands with the @symbol. It seems like there are already tools for
> > > solving this - bash, python, etc. that should we just concede defeat
> to.
> > If
> > > I'm typing my commands manually and relying on aliases I'd make a bet
> > that
> > > I'm probably not going to be remembering what I alias'd different parts
> > of
> > > my jobpath to and again, what the order of the jobpath bits are.
> > >
> > >
> > > Thanks,
> > > David
> > >
> > >
> > >
> > > On Thu, May 22, 2014 at 11:48 AM, Mark Chu-Carroll
> > > <mchucarroll@apache.org>wrote:
> > >
> > > > The bulk of the feedback from the first round of comments on this
> could
> > > be
> > > > summed up, roughly "get rid of the stupid command shortcuts you dope,
> > > > they're a menace!". Message received: they're gone.
> > > >
> > > > But I still think that the configurable defaults are valuable. Based
> on
> > > > some feedback, I've revised the proposal a bit. This includes trying
> to
> > > > strengthen the motivation/problem statement, and adding ways to
> provide
> > > > default parameters only for specific commands, plus many small
> changes.
> > > >
> > > > The updated proposal is attached below.
> > > >
> > > >
> > > > # Defaults and Shorthands in the Aurora Client
> > > >
> > > > ## Motivation
> > > >
> > > > Most of the time, users are doing the same thing, over and over
> again.
> > > > They're working mainly on one particular service, in one particular
> > > > workspace. But they need to repeat the same parameters, over and over
> > > > again, and they need to remember what those parameters are, what
> order
> > > > they occur in, what format they use, and other details that can be
> > > > difficult to remember.
> > > >
> > > > In order to avoid these problems, users set up custom scripts
> > > > that make it easy for them to run commands like "`myservice start`",
> > > > instead of "`aurora job create west/markcc/prod/myserver
> > > > src/main/aurora/myservice/myservice.aurora`"
> > > >
> > > > To reduce this, we'd like to support a way for users to set up a
> > > > configuration file that defines defaults and shorthands for their
> > > > everyday work. With shorthands, a user that only works with a single
> > > > service could say "aurora job create", instead of needing to spell
> > > > out the full jobkey and configuration file location; a user working
> > with
> > > > multiple datacenters could say "`aurora job create @east`" or
> > > > "`aurora job create @west`" to select the correct jobkey.
> > > >
> > > > ## Proposal: Aurora Init Files
> > > >
> > > > To allow users to customize shorthands, we'll provide a
> > > > builtin capability to allow users to provide a configuration
> > > > file, from which their customizations will be loaded.
> > > > Many applications use a simple pattern to solve similar problems.
> > > > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > > > don't specifically tell the tool where to find a configuration, it
> > looks
> > > > in the current directory or one of its parents to find a file named
> > > > "Vagrantfile".
> > > >
> > > > We'd like to follow a similar pattern, and create an "AuroraInit"
> file.
> > > > The aurora init file is found by searching the following locations,
> in
> > > > order:
> > > >
> > > >    * the contents of the "--init-file" parameter.
> > > >    * if the "--init-file" parameter is unspecified, then look in the
> > > > current
> > > >      directory for a file named "AuroraInit".
> > > >    * if no "AuroraInit" file exists in the current directory, then
> look
> > > in
> > > > the users
> > > >      home directory for an init file named ".AuroraInit".
> > > >
> > > >
> > > > ### What goes into an init file?
> > > >
> > > > We should support the following kinds of things:
> > > >
> > > >    1. Universal defaults - user-defined default settings that will be
> > > > applied to
> > > >      all commands. For example, if there is a default config file
> that
> > > > should always
> > > >      be used if the user doesn't specify one, that would be a
> universal
> > > > default.
> > > >    2. Command specific defaults - users should be able to specify
> that
> > > they
> > > > always want
> > > >      to use certain parameter settings for a specific command. For
> > > example,
> > > > if they
> > > >      want to always use a default batch size of 10 for updates, but
> > don't
> > > > want to affect
> > > >      batch sizes for other commands like kill, they could use a
> command
> > > > specific default.
> > > >     3. Aliases - shorthand names for longer parameters. A user could
> > > > specify shorthands
> > > >       "east" and "west" for full jobkeys in two different
> datacenters.
> > > >
> > > > ### Using aliases and defaults
> > > >
> > > > A default specifies a set of _bindings_. If a parameter is omitted
> > > > from a command, and there's a binding for that parameter, it will be
> > > > automatically inserted into the command as if the user had typed it.
> > > >
> > > > An alias is a short equivalent for a parameter. When a command line
> is
> > > > provided by a user, aliases will be expanded inline.  A user can
> > > > specifically
> > > > mark an alias for expansion by prefixing it with "@"; if an alias
> > > > appears on the command-line surrounded by whitespace, it will be
> > > > replaced even if it isn't marked with an "@".
> > > >
> > > >
> > > > ### Defining Shorthands and Defaults
> > > >
> > > > The init file is, like aurora job configurations, a python source
> file
> > > > using a Pystachio
> > > > schema. The schema is loaded, and an `Init` object is retrieved from
> > the
> > > > top-level
> > > > variable `init` in that file.
> > > >
> > > > The pystachio schema for init files is:
> > > >
> > > >     class CommandDefaults(Struct)
> > > >       command = Required(String)
> > > >       defaults=Map(String, String)
> > > >
> > > >     class Init(Struct):
> > > >       defaults=Map(String, String)
> > > >       command_defaults = Optional(CommandDefaults)
> > > >       aliases=Map(String, String)
> > > >
> > > >
> > > > For example, an init file could contain the following:
> > > >
> > > >     init=Init(
> > > >       defaults={
> > > >         "jobspec": "west/markcc/test/myservice",
> > > >         "config_file": "./src/aurora/myservice.aurora"
> > > >       },
> > > >       aliases={
> > > >         "east": "east/markcc/test/myservice",
> > > >         "c": "east",
> > > >         "me": "markchucarroll"
> > > >       },
> > > >       command_defaults(command="job update",
> > > >         defaults={"--batch-size": 10}
> > > >       ))
> > > >
> > > >
> > > > With this configuration file, if the user ran "`aurora job create`"
> > > without
> > > > any parameters,
> > > > it would automatically be expanded to "`aurora job create
> > > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job create east`", the alias `east` would be
> > > > expanded to "east/markcc/test/myservice", and the missing config file
> > > > parameter would
> > > > be instantiated using the default, to create a command-line:
> > > > "`aurora job create east/markcc/test/myservice
> > > > ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> > > aliases
> > > > would
> > > > be expanded, and the omitted configuration parameter would be added:
> > > > "`aurora job create east/markchucarroll/test/myservice
> > > > ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job update`", then the `jobspec` and
> > `config_file`
> > > > parameters would
> > > > get inserted from the global defaults, and the "--batch-size=10"
> would
> > be
> > > > inserted from
> > > > the command defaults for "aurora job update".
> > > >
> > >
> >
>

--001a113455f4b8493b04fa8fb253--