aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Chu-Carroll <mchucarr...@apache.org>
Subject Re: One more try at defaults and aliases
Date Thu, 19 Jun 2014 20:44:15 GMT
That's what I thought - but in pants/blaze/maven/..., I thought most people
normally left their command-line in the project root. I certainly do. If
you're doing that - and a lot of people do - then you can't easily
disambiguate between different services living in the same repos.

    -Mark


On Thu, Jun 19, 2014 at 4:40 PM, Bill Farner <wfarner@apache.org> wrote:

> On Thu, Jun 19, 2014 at 1:36 PM, Mark Chu-Carroll <mchucarroll@apache.org>
> wrote:
>
> > Maybe I'm misunderstanding you?
> >
> > Where else should you look for a default configuration file, where it can
> > automatically differentiate between different services stored in the same
> > repository?
> >
>
> Working directory would be most intuitive to me.
>
>
> >
> >   -Mark
> >
> >
> > On Thu, Jun 19, 2014 at 4:35 PM, Bill Farner <wfarner@apache.org> wrote:
> >
> > > > in the root of the project directory
> > >
> > > FWIW i don't see any particular reason why it needs to be in the root
> of
> > a
> > > repo/project dir.
> > >
> > > -=Bill
> > >
> > >
> > > On Thu, Jun 19, 2014 at 1:31 PM, Mark Chu-Carroll <
> > mchucarroll@apache.org>
> > > wrote:
> > >
> > > > I actually don't agree with standardizing the config file name.
> > > >
> > > > Here's the issue: if you're working with typical modern tools -
> gradle,
> > > > maven, pants, sbt, ..., you're doing your work in the root of a
> fairly
> > > deep
> > > > source hierarchy. Living in that hierarchy, there are likely multiple
> > > > aurora job definitions.
> > > >
> > > > There's no way of just picking a default config file name that's not
> > > going
> > > > to cause confusion and disruption, because the config file sitting in
> > the
> > > > root of the project directory is going to be wrong for many users.
> > > >
> > > > That was the point of the home-directory fallback thing; it provided
> > > users
> > > > with a way to define a default config file from outside of the
> > workspace,
> > > > so that different users could set it differently.
> > > >
> > > > In any case: I'm punting on this for now. It doesn't seem like
> there's
> > > > anything close to a consensus that this is worth doing, so I'm going
> to
> > > > shelve it, and we can come back to it if the consensus changes later.
> > > >
> > > >     -Mark
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jun 19, 2014 at 4:08 PM, James Oliver <jdo500@gmail.com>
> > wrote:
> > > >
> > > > > +1 for standardizing the config file name
> > > > > On Jun 19, 2014 1:01 PM, "Bill Farner" <wfarner@apache.org>
wrote:
> > > > >
> > > > > > I agree with Joe, and tend to disagree with the motivation (for
> > > > aliases,
> > > > > at
> > > > > > least).  The end-result seems to be obfuscation (i'd likely
need
> to
> > > > peek
> > > > > in
> > > > > > the init file to figure out what a command does, or what command
> to
> > > > run),
> > > > > > and i'm doubtful it will result in converting users away from
> > wrapper
> > > > > > scripts.
> > > > > >
> > > > > > I also feel like our configuration files are already very complex
> > > > (nature
> > > > > > of the beast), and aliases make them even more so.
> > > > > >
> > > > > > I am, however, more comfortable with the standard/assumed config
> > file
> > > > > > naming (i.e. like Vagrantfile).
> > > > > >
> > > > > >
> > > > > > -=Bill
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 13, 2014 at 11:27 AM, Joseph Smith <
> > yasumoto7@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > "to set up a configuration file”
> > > > > > >
> > > > > > > This line to me just says “we’re reinventing wrapper
scripts”
> > where
> > > > > > > someone could create an “alias” like ./east.sh and
./west.sh to
> > do
> > > > > > deploys
> > > > > > > to various clusters. In that sense, these aliases etc are
also
> > > > “hiding”
> > > > > > bad
> > > > > > > interfaces, just in a different way- and one which isn’t
as
> > > > > > > well-tested/understood as shell scripts.
> > > > > > >
> > > > > > > It seems like we’re set on putting time into this, so
if we
> are,
> > > then
> > > > > > > ignore me and move onward :). IF we were going to go with
an
> > > > approach,
> > > > > > this
> > > > > > > seems reasonable based on ways that I’ve seen people
wrap their
> > > > > > > configs/invocations of the client.
> > > > > > >
> > > > > > > On Jun 12, 2014, at 7:50 AM, Mark Chu-Carroll <
> > > > mchucarroll@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I'm nothing if not persistent! After feedback from
my second
> > try
> > > at
> > > > > > > > proposing a way of doing defaults and aliases, I've
got a
> third
> > > > > draft.
> > > > > > > >
> > > > > > > > Feedback please!
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > # Defaults and Shorthands in the Aurora Client
> > > > > > > >
> > > > > > > > ## Motivation
> > > > > > > >
> > > > > > > > Most of the time, users are doing the same thing,
over and
> over
> > > > > again.
> > > > > > > > They're
> > > > > > > > working mainly on one particular service, in one particular
> > > > > workspace.
> > > > > > > But
> > > > > > > > they
> > > > > > > > need to repeat the same parameters, over and over
again, and
> > they
> > > > > need
> > > > > > to
> > > > > > > > remember what those parameters are, what order they
occur in,
> > > what
> > > > > > format
> > > > > > > > they
> > > > > > > > use, and other details that can be difficult to remember.
> > > > > > > >
> > > > > > > > In order to avoid these problems, users set up custom
scripts
> > > that
> > > > > make
> > > > > > > it
> > > > > > > > easy
> > > > > > > > for them to run commands like "`myservice start`",
instead of
> > > > > "`aurora
> > > > > > > job
> > > > > > > >  create west/markcc/prod/myserver
> > > > > > > > src/main/aurora/myservice/myservice.aurora`"
> > > > > > > >
> > > > > > > > Scripts aren't necessarily a bad thing. We've designed
the
> > aurora
> > > > > > client
> > > > > > > to
> > > > > > > > be script friendly: every command has at least an
option
> > > supporting
> > > > > > > > easy-to-parse
> > > > > > > > output. But scripts also often cover for problems
that really
> > > > > shouldn't
> > > > > > > > exist.
> > > > > > > >
> > > > > > > > Scripts get written for a variety of reasons. Among
the many
> > > > reasons
> > > > > > that
> > > > > > > > people implement
> > > > > > > > scripts, there are scripts that add custom functionality
for
> a
> > > > > specific
> > > > > > > > group of users;
> > > > > > > > scripts that can cover up for inadequate or missing
core
> > > > > functionality;
> > > > > > > and
> > > > > > > > there are
> > > > > > > > scripts that cover up for poor user interfaces.
> > > > > > > >
> > > > > > > > In aurora, we've worked hard to eliminate cases where
the
> core
> > > > > > > > functionality of the
> > > > > > > > command-line client is inadequate. But our user interface
> still
> > > > > needs a
> > > > > > > lot
> > > > > > > > of work.
> > > > > > > > In particular, commands in Aurora often require very
long
> > > > parameters,
> > > > > > > which
> > > > > > > > users have
> > > > > > > > a hard time remembering. Look at the example above
- it's a
> > > pretty
> > > > > > > typical
> > > > > > > > case. The
> > > > > > > > user wants to create a job. They're going to be creating
the
> > same
> > > > > job,
> > > > > > > over
> > > > > > > > and over again.
> > > > > > > > But to run it, they need to type out 68 characters
for the
> two
> > > > > > > parameters!
> > > > > > > > This is a
> > > > > > > > prime example of the kind of case where users will
write
> > scripts,
> > > > not
> > > > > > to
> > > > > > > > provide new/special
> > > > > > > > functionality, nor to cover for inadequate functionality,
but
> > > just
> > > > > > > because
> > > > > > > > it's so
> > > > > > > > painful to remember and correctly type out an overly
long
> > string
> > > of
> > > > > > > > parameters.
> > > > > > > >
> > > > > > > >
> > > > > > > > To reduce this, we'd like to support a way for users
to set
> up
> > a
> > > > > > > > configuration
> > > > > > > > file that defines defaults and shorthands for their
everyday
> > > work.
> > > > > With
> > > > > > > > shorthands, a user that only works with a single service
> could
> > > say
> > > > > > > "aurora
> > > > > > > > job
> > > > > > > > create", instead of needing to spell out the full
jobkey and
> > > > > > > configuration
> > > > > > > > file
> > > > > > > > location; a user working with multiple datacenters
could say
> > > > "`aurora
> > > > > > job
> > > > > > > > create @east`" or "`aurora job create @west`" to select
the
> > > correct
> > > > > > > jobkey.
> > > > > > > >
> > > > > > > > ## Proposal: Aurora Init Files
> > > > > > > >
> > > > > > > > To allow users to customize shorthands, we'll provide
a
> > > > > > > > builtin capability to allow users to provide a configuration
> > > > > > > > file, from which their customizations will be loaded.
> > > > > > > > Many applications use a simple pattern to solve similar
> > problems.
> > > > > > > > Vagrant uses a file named "Vagrantfile"; when you
run
> vagrant,
> > if
> > > > you
> > > > > > > > don't specifically tell the tool where to find a
> configuration,
> > > it
> > > > > > looks
> > > > > > > > in the current directory or one of its parents to
find a file
> > > named
> > > > > > > > "Vagrantfile".
> > > > > > > >
> > > > > > > > We'd like to follow a similar pattern, and create
an
> > "AuroraInit"
> > > > > file.
> > > > > > > > The aurora init file is found by searching the following
> > > locations,
> > > > > in
> > > > > > > > order:
> > > > > > > >
> > > > > > > >   * the contents of the "--init-file" parameter.
> > > > > > > >   * if the "--init-file" parameter is unspecified,
then look
> in
> > > the
> > > > > > > current
> > > > > > > >     directory for a file named "AuroraInit".
> > > > > > > >   * if no "AuroraInit" file exists in the current
directory,
> > then
> > > > > look
> > > > > > in
> > > > > > > > the users
> > > > > > > >     home directory for an init file named ".aurora/init".
> > > > > > > >
> > > > > > > >> **Sidebar: Why fallback to the users home directory?**
> > > > > > > >>
> > > > > > > >> Codebases are often shared between multiple projects,
each
> of
> > > > which
> > > > > > > lives
> > > > > > > > in
> > > > > > > >> a set of subdirectories within the codebase. For
example,
> just
> > > > look
> > > > > at
> > > > > > > > the aurora
> > > > > > > >> sourcecode, which includes the aurora scheduler,
the aurora
> > > > > executor,
> > > > > > > > thermos,
> > > > > > > >> and the aurora client.
> > > > > > > >>
> > > > > > > >> If multiple services live in a codebase, then
users can't
> put
> > an
> > > > > > > > AuroraInit file in
> > > > > > > >> the root directory of the codebase, because it
would
> interfere
> > > > with
> > > > > > > other
> > > > > > > > users'
> > > > > > > >> work.
> > > > > > > >>
> > > > > > > >> The home directory fallback provides an easy way
for users
> to
> > > set
> > > > > up a
> > > > > > > > pointer
> > > > > > > >> to the correct init file, which won't be removed
by
> operations
> > > > like
> > > > > > > > switching branches
> > > > > > > >> in a source repository. Users write shared init
files, which
> > are
> > > > > > located
> > > > > > > > in their projects
> > > > > > > >> in the codebase, and create a symbolic link from
their
> project
> > > > init
> > > > > > file
> > > > > > > > to their home
> > > > > > > >> directory.
> > > > > > > >
> > > > > > > > ### What goes into an init file?
> > > > > > > >
> > > > > > > > We should support the following kinds of things:
> > > > > > > >
> > > > > > > >   1. Universal defaults - user-defined default settings
that
> > will
> > > > be
> > > > > > > > applied to
> > > > > > > >     all commands. For example, if there is a default
config
> > file
> > > > that
> > > > > > > > should always
> > > > > > > >     be used if the user doesn't specify one, that
would be a
> > > > > universal
> > > > > > > > default.
> > > > > > > >   2. Command specific defaults - users should be able
to
> > specify
> > > > that
> > > > > > > they
> > > > > > > > always want
> > > > > > > >     to use certain parameter settings for a specific
command.
> > For
> > > > > > > example,
> > > > > > > > if they
> > > > > > > >     want to always use a default batch size of 10
for
> updates,
> > > but
> > > > > > don't
> > > > > > > > want to affect
> > > > > > > >     batch sizes for other commands like kill, they
could use
> a
> > > > > command
> > > > > > > > specific default.
> > > > > > > >    3. Aliases - shorthand names for longer parameters.
A user
> > > could
> > > > > > > > specify shorthands
> > > > > > > >      "east" and "west" for full jobkeys in two different
> > > > datacenters.
> > > > > > > >
> > > > > > > >
> > > > > > > > ### Defaults
> > > > > > > >
> > > > > > > > A default specifies a set of _bindings_. If a parameter
is
> > > omitted
> > > > > > > > from a command, and there's a binding for that parameter,
it
> > will
> > > > be
> > > > > > > > automatically inserted into the command as if the
user had
> > typed
> > > > it.
> > > > > > > > The binding is specified in the configuration file
using a
> > Python
> > > > > > > > dictionary.
> > > > > > > >
> > > > > > > > For example, if the defaults included  `{'jobspec':
> > > > > > > > 'devcluster/me/prod/service'}`,
> > > > > > > > then if you ran `aurora job create` without specifying
any
> > > > > parameters,
> > > > > > > the
> > > > > > > > command-line
> > > > > > > > would automatically substitute `devcluster/me/prod/service`
> for
> > > the
> > > > > > > > `jobspec` parameter.
> > > > > > > >
> > > > > > > > Defaults can be declared either globally (in which
case
> they'll
> > > be
> > > > > > > inserted
> > > > > > > > as parameters
> > > > > > > > for all commands), or for specific commands (in which
case
> > > they'll
> > > > > only
> > > > > > > be
> > > > > > > > inserted for a
> > > > > > > > single command).
> > > > > > > >
> > > > > > > >
> > > > > > > > ### Aliases
> > > > > > > >
> > > > > > > > An alias is a short equivalent for a parameter. When
a
> command
> > > line
> > > > > is
> > > > > > > > provided by a user, aliases will be expanded inline.
 A user
> > can
> > > > > > > > specifically
> > > > > > > > mark an alias for expansion by prefixing it with "@";
if an
> > alias
> > > > > > > > appears on the command-line surrounded by whitespace,
it will
> > be
> > > > > > > > replaced even if it isn't marked with an "@".
> > > > > > > >
> > > > > > > >> **Sidebar: why not just use shell substitutions?**
> > > > > > > >>
> > > > > > > >> The @-substitution model proposed here is very
similar to
> unix
> > > > > > variable
> > > > > > > >> substitution. So why implement the same thing
all over
> again?
> > > Why
> > > > > not
> > > > > > > > just tell
> > > > > > > >> users to use their shell?
> > > > > > > >>
> > > > > > > >> There are several reasons.
> > > > > > > >>
> > > > > > > >> 1. All of the options and aliases that affect
aurora command
> > > > > > invocations
> > > > > > > > can be specified
> > > > > > > >> in one place: the AuroraInit file.
> > > > > > > >> 2. Command shell substitution is complex - the
ways in which
> > the
> > > > > > command
> > > > > > > > shell does
> > > > > > > >>  expansion, tokenization, substitution, and parsing
can be
> > very
> > > > hard
> > > > > > to
> > > > > > > > follow. Adding
> > > > > > > >>  the aurora aliases to that process just adds
another layer
> of
> > > > > > confusion
> > > > > > > > to the process.
> > > > > > > >>  With internal alias substitution, we can avoid
the
> confusions
> > > of
> > > > > > > > expansions and
> > > > > > > >>  tokenization for aurora aliases.
> > > > > > > >> 3. We can provide better debugging and error messages
with
> our
> > > own
> > > > > > alias
> > > > > > > > substitution.
> > > > > > > >>  For example, if a user specifies a job using
aliases like
> > > > "`aurora
> > > > > > job
> > > > > > > > create @c/@me/test/myservice`",
> > > > > > > >>  we can create an error message that shows exactly
what they
> > > > typed,
> > > > > > and
> > > > > > > > how it expanded:
> > > > > > > >> <pre>
> > > > > > > >> Job "west/markcc/test/myservice" not found in
configuration
> > file
> > > > > > > > config.aurora.
> > > > > > > >> Jobkey was generated by alias expansion: original
key was
> > > > > > > > "@c/@me/test/myservice", where:
> > > > > > > >>   - "@c" was expanded to "west"
> > > > > > > >>   - "@me" was expanded to "markcc"
> > > > > > > >>  config_file parameter "config.aurora" was specified
from
> > > > defaults.
> > > > > > > >> </pre>
> > > > > > > >
> > > > > > > >
> > > > > > > > ### Defining Shorthands and Defaults: Syntax and Examples
> > > > > > > >
> > > > > > > > The init file is, like aurora job configurations,
a python
> > source
> > > > > file
> > > > > > > > using a Pystachio
> > > > > > > > schema. The schema is loaded, and an `Init` object
is
> retrieved
> > > > from
> > > > > > the
> > > > > > > > top-level
> > > > > > > > variable `init` in that file.
> > > > > > > >
> > > > > > > > The pystachio schema for init files is:
> > > > > > > >
> > > > > > > >    class CommandDefaults(Struct)
> > > > > > > >      command = Required(String)
> > > > > > > >      defaults=Map(String, String)
> > > > > > > >
> > > > > > > >    class Init(Struct):
> > > > > > > >      defaults=Map(String, String)
> > > > > > > >      command_defaults = Optional(CommandDefaults)
> > > > > > > >      aliases=Map(String, String)
> > > > > > > >
> > > > > > > >
> > > > > > > > For example, an init file could contain the following:
> > > > > > > >
> > > > > > > >    init=Init(
> > > > > > > >      defaults={
> > > > > > > >        "jobspec": "west/markcc/test/myservice",
> > > > > > > >        "config_file": "./src/aurora/myservice.aurora"
> > > > > > > >      },
> > > > > > > >      aliases={
> > > > > > > >        "east": "east/markcc/test/myservice",
> > > > > > > >        "c": "east",
> > > > > > > >        "me": "markchucarroll"
> > > > > > > >      },
> > > > > > > >      command_defaults(command="job update",
> > > > > > > >        defaults={"--batch-size": 10}
> > > > > > > >      ))
> > > > > > > >
> > > > > > > >
> > > > > > > > With this configuration file, if the user ran "`aurora
job
> > > create`"
> > > > > > > without
> > > > > > > > any parameters,
> > > > > > > > it would automatically be expanded to "`aurora job
create
> > > > > > > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > > > > > > >
> > > > > > > > If a user ran "`aurora job create east`", the alias
`east`
> > would
> > > be
> > > > > > > > expanded to "east/markcc/test/myservice", and the
missing
> > config
> > > > file
> > > > > > > > parameter would
> > > > > > > > be instantiated using the default, to create a command-line:
> > > > > > > > "`aurora job create east/markcc/test/myservice
> > > > > > > > ./src/aurora/myservice.aurora`".
> > > > > > > >
> > > > > > > > If a user ran "`aurora job create @c/@me/test/myservice`",
> the
> > > two
> > > > > > > aliases
> > > > > > > > would
> > > > > > > > be expanded, and the omitted configuration parameter
would be
> > > > added:
> > > > > > > > "`aurora job create east/markchucarroll/test/myservice
> > > > > > > > ./src/aurora/myservice.aurora`".
> > > > > > > >
> > > > > > > > If a user ran "`aurora job update`", then the `jobspec`
and
> > > > > > `config_file`
> > > > > > > > parameters would
> > > > > > > > get inserted from the global defaults, and the
> > "--batch-size=10"
> > > > > would
> > > > > > be
> > > > > > > > inserted from
> > > > > > > > the command defaults for "aurora job update".
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message