aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Justin Venus (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AURORA-1782) Thermos Executor is not handling apostrophe gracefully
Date Wed, 05 Oct 2016 14:43:20 GMT

     [ https://issues.apache.org/jira/browse/AURORA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Justin Venus updated AURORA-1782:
---------------------------------
    Description: 
This is a regression from the behavior in 0.15.0.

This is enough to cause an execution to fail.
{code}
python -c 'import sys

if __name__ == "__main__":
  sys.exit(0)'
{code}

This is the error seen when running an inline script from a Process().
{code}
Failed to parse the flags: Failed to load flag 'command': Failed to load value '{"shell":true,"value":"/bin/bash
-c 'python -c import': syntax error at line 1 near: 
{code}


Ideally I could escape the apostrophe and have it work for Process's that run 'chrooted/pivoted'
by thermos or bare.  My work around has been to write a templating hack for aurora's dsl.

{code}
# Template.render(name='foo', template='SOME_STRING_TEMPLATE')
# Returns a Process().  I can't just base64 the entire template (that would be too easy),

# b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace apostrophe
# with a pattern that is unlikely to be legitimately used and render a 'cmdline' that will

# undo the hack.  Of course using the octal or "'" directly proved troublesome, so I had to

# base64 encode the simple sed statement and decode the script on the remote side.  Like
# I said this is a complete hack.

class Template(object):

  class T(Struct):
    payload = Required(String)

  @classmethod
  def render(cls, name=None, template=None, **kwargs):
    assert name is not None
    assert template is not None
    return cls(name, template, 'template', **kwargs).process

  def cmdline(self):
    return ''.join([
        '(mkdir .decoder && ',
        '(test -x ./.decoder/decoder || ',
        '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) && ',
        'chmod +x ./.decoder/decoder)) || ',
        "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 1; done)) &&
",
        '((echo "{{template_payload}}" | ./.decoder/decoder) > {{template_filename}})'
    ])

  def decoder_script(self):
    return r"""#!/bin/bash
    sed "s/__APOSTROPHE__/'/g"
    """

  def postInit(self):
    self.process.in_scope = self.in_scope  # need to override imethod

  def __init__(self, name, template, prefix, filename=None, auto_init=True, **kwargs):
    self.resolved = False

    self.process = Process(
        name="%s:%s" % (prefix, name),
        cmdline=self.cmdline(), **kwargs).bind(
            self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
            template_filename=name if filename is None else filename,
            template_decoder=base64.b64encode(self.decoder_script()))
    if auto_init:
      self.postInit()

  def in_scope(self, *args, **kwargs):
    """ensure name/commandline is resolved before proceeding"""
    if self.resolved:
      return self.process
    self.process = Process.in_scope(self.process, *args, **kwargs)
    if '{{' not in str(self.process.name()):
      scopes = self.process.scopes()
      for scope in scopes:
        if not hasattr(scope, 'bind'):
          continue
        scope = scope.bind(**kwargs)
        if scope.check() and isinstance(scope, self.__class__.T):
          self.resolved = True
          payload = str(scope.payload())
          print(" INFO] Memoizing {}".format(self.process.name()))
          self.process = self.process.bind(template_payload=payload)
{code}

  was:
When using mesos containerizer with an image, the code path taken appears to be stripping
new line characters making it impossible to render files with '/bin/echo' or execute inline
python scripts.  I suspect this is due to the introduction of shlex.split() in the method
wrapped_cmdline() in the file src/main/python/apache/thermos/core/process.py, but I haven't
investigated any further.

This is a regression from the behavior in 0.15.0.   


This is enough to cause an execution to fail.
{code}
python -c 'import sys

if __name__ == "__main__":
  sys.exit(0)'
{code}

This is the error seen when running an inline script from a Process().
{code}
Failed to parse the flags: Failed to load flag 'command': Failed to load value '{"shell":true,"value":"/bin/bash
-c 'python -c import': syntax error at line 1 near: 
{code}



> Thermos Executor is not handling apostrophe gracefully
> ------------------------------------------------------
>
>                 Key: AURORA-1782
>                 URL: https://issues.apache.org/jira/browse/AURORA-1782
>             Project: Aurora
>          Issue Type: Bug
>    Affects Versions: 0.16.0
>            Reporter: Justin Venus
>
> This is a regression from the behavior in 0.15.0.
> This is enough to cause an execution to fail.
> {code}
> python -c 'import sys
> if __name__ == "__main__":
>   sys.exit(0)'
> {code}
> This is the error seen when running an inline script from a Process().
> {code}
> Failed to parse the flags: Failed to load flag 'command': Failed to load value '{"shell":true,"value":"/bin/bash
-c 'python -c import': syntax error at line 1 near: 
> {code}
> Ideally I could escape the apostrophe and have it work for Process's that run 'chrooted/pivoted'
by thermos or bare.  My work around has been to write a templating hack for aurora's dsl.
> {code}
> # Template.render(name='foo', template='SOME_STRING_TEMPLATE')
> # Returns a Process().  I can't just base64 the entire template (that would be too easy),

> # b/c then I can't use {{thermos.ports[http]}}.  So I go through and replace apostrophe
> # with a pattern that is unlikely to be legitimately used and render a 'cmdline' that
will 
> # undo the hack.  Of course using the octal or "'" directly proved troublesome, so I
had to 
> # base64 encode the simple sed statement and decode the script on the remote side.  Like
> # I said this is a complete hack.
> class Template(object):
>   class T(Struct):
>     payload = Required(String)
>   @classmethod
>   def render(cls, name=None, template=None, **kwargs):
>     assert name is not None
>     assert template is not None
>     return cls(name, template, 'template', **kwargs).process
>   def cmdline(self):
>     return ''.join([
>         '(mkdir .decoder && ',
>         '(test -x ./.decoder/decoder || ',
>         '((echo {{template_decoder}} | base64 -d > ./.decoder/decoder) &&
',
>         'chmod +x ./.decoder/decoder)) || ',
>         "(while [ ! -x ./.decoder/decoder ]; do echo resolving-decoder; sleep 1; done))
&& ",
>         '((echo "{{template_payload}}" | ./.decoder/decoder) > {{template_filename}})'
>     ])
>   def decoder_script(self):
>     return r"""#!/bin/bash
>     sed "s/__APOSTROPHE__/'/g"
>     """
>   def postInit(self):
>     self.process.in_scope = self.in_scope  # need to override imethod
>   def __init__(self, name, template, prefix, filename=None, auto_init=True, **kwargs):
>     self.resolved = False
>     self.process = Process(
>         name="%s:%s" % (prefix, name),
>         cmdline=self.cmdline(), **kwargs).bind(
>             self.__class__.T(payload=template.replace("'", '__APOSTROPHE__')),
>             template_filename=name if filename is None else filename,
>             template_decoder=base64.b64encode(self.decoder_script()))
>     if auto_init:
>       self.postInit()
>   def in_scope(self, *args, **kwargs):
>     """ensure name/commandline is resolved before proceeding"""
>     if self.resolved:
>       return self.process
>     self.process = Process.in_scope(self.process, *args, **kwargs)
>     if '{{' not in str(self.process.name()):
>       scopes = self.process.scopes()
>       for scope in scopes:
>         if not hasattr(scope, 'bind'):
>           continue
>         scope = scope.bind(**kwargs)
>         if scope.check() and isinstance(scope, self.__class__.T):
>           self.resolved = True
>           payload = str(scope.payload())
>           print(" INFO] Memoizing {}".format(self.process.name()))
>           self.process = self.process.bind(template_payload=payload)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message