aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Cheynier ...@pierre-cheynier.net>
Subject Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
Date Wed, 20 Apr 2016 10:46:54 GMT


> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote:
> > Sorry for the delay on this. After you filed the pull request, I investigated a
bit what will be required once Mesos 0.30.0 lands: https://issues.apache.org/jira/browse/AURORA-1632.
I think the problem goes beyond the failure to find `sys.executable` when $PATH is not set.
As even after switching back to chmod+x on the runner, the task failed further down the stack.
> > 
> > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which should allow
`sys.executable` to continue working and will allow any tasks users have running which have
come to rely on Thermos setting it for them to behave as expected. The problem is, I haven't
had time to figure out what we should set $PATH to yet ;) (anyone have any thoughts?).
> > 
> > I know this is probably more info than you bargained for when you opened what seemed
like a simple pull request. I'm not opposed to accepting this patch (with a TODO to restore
`sys.executable` when we figure out what to do about setting $PATH) if it unblocks your use
case, but can you confirm that you're actually able to run the Mesos agent with `--executor_environment_variables='{}'`
and still launch tasks?
> 
> Pierre Cheynier wrote:
>     Hi,
>     
>     Sorry for my delay on opening this review on apache.org.
>     Actually, I started to use Aurora and faced this issue cause my setup use `--executor_environment_variables`.
>     I first investigated in the Python default setup on CentOS, the pants/pex build system
and then the differences between the vagrant box provided in the repo and mine before discovering
that this is the origin of the issue.
>     
>     I admit this is a simple fix, I just tried to understand the reason of using this
mechanism and found that it was changed 2 years ago.
>     In 0.27.2  it works well by using this patch (I'm now able to run something).
> 
> Joshua Cohen wrote:
>     So he's the behavior I see when I run the Mesos agent with `--executor_environment_variables='{}'`.
Without this patch, as you note, tasks fail to start with a permission denied error (due to
`sys.executable` being unset). After applying this patch, tasks start up, but processes fail
to fork: https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0.
>     
>     Are you by chance running Docker tasks?
> 
> Pierre Cheynier wrote:
>     I'm running tasks on the Mesos containerizer for now.
>     Your problem is not purely a side-effect of not passing any environment variables
? (your tasks themselves maybe need PATH or LD_LIBRARY_PATH ?)
> 
> Joshua Cohen wrote:
>     It's not the task that requires PATH to be set, it's the thermos runner itself: https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/core/process.py#L397-L402
>     
>     This is why I'm confused as to how this works for you unless PATH is somehow being
set elsewhere (and if PATH is set elsewhere, then `sys.executable` should work as well).

Using the Mesos Containerizer, it probably depends on the shell used and the need for the
tasks itself to get a specific environment.

But I'm really surprised, cause even when I'm using your vagrant setup (patch cherry-picked
on top of the master branch + vagrant up), I'm able to launch your two sample tasks (hello_world
+ cron_hello_world).
I can provide screenshots if needed.

The portion of code you pointed me to is supposed to enrich the environment of the subshell.
I presume that if the child absolutely need the PATH/LD_LIBRARY_PATH/whatever, the operator
will have to add it in --executor_environment_variable OR the user will add it by itself.
This is what it's supposed to happen when Mesos 0.30 will be released, right ?


- Pierre


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45104/#review124697
-----------------------------------------------------------


On March 22, 2016, 9:50 a.m., Pierre Cheynier wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45104/
> -----------------------------------------------------------
> 
> (Updated March 22, 2016, 9:50 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> When using `--executor_environment_variables` without explicitely
> passing LD_LIBRARY_PATH, `sys.executable` returns an empty string
> resulting in a '[Errno 13] Permission denied' error for every launched
> task.
> 
> Moreover, it seems that this feature is coming in 0.30: "Executors no
> longer inherit environment variables from the agent".
> 
> This patch partially revert back 07ce21d where chmod_x method was
> removed in favor of using sys.executable.
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/executor/thermos_task_runner.py 3896e3841562600379705dbf78a6f62728246348

> 
> Diff: https://reviews.apache.org/r/45104/diff/
> 
> 
> Testing
> -------
> 
> Make Aurora executor run on CentOS7, while running mesos agent with `--executor_environment_variables`
option and no excplicit $PATH set.
> 
> 
> Thanks,
> 
> Pierre Cheynier
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message