airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: CgroupTaskRunner Failure
Date Fri, 05 Apr 2019 18:47:23 GMT
Have you checked the obvious:

Do you have disk space left? 

If you do it's possible you've hit a limit on the number of cgroups - either cos there are
too many running at the same time, or because something isn't cleaning them up.

-ash

> On 5 Apr 2019, at 19:44, Junda Yang <youngyjd@gmail.com> wrote:
> 
> Hi there,
> 
> Does anyone have experience in running CgroupTaskRunner? I tested it on our
> staging cluster but kept getting errors:
> 
> Traceback (most recent call last):
> 
>  File
> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/jobs.py",
> line 2540, in _execute
> 
>    self.task_runner.start()
> 
>  File
> "/srv/venvs/service/trusty/service_venv_python3.6/src/apache-airflow/airflow/contrib/task_runner/cgroup_task_runner.py",
> line 154, in start
> 
>    mem_cgroup_node.controller.limit_in_bytes = self._mem_mb_limit * 1024 *
> 1024
> 
>  File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/interfaces.py",
> line 63, in __set__
> 
>    return instance.set_property(self.filename, value)
> 
>  File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cgroupspy/controllers.py",
> line 63, in set_property
> 
>    return f.write(str(value))
> 
> OSError: [Errno 28] No space left on device
> 
> 
> Actually it succeeded in creating memory group but failed later creating
> cpu group in
> https://github.com/apache/airflow/blob/c3939c8e721870d263997e7aeaebc28e678d544b/airflow/contrib/task_runner/cgroup_task_runner.py#L139-#L156.
> However, If I switch the order, it could create cpu group and then failed
> creating memory group. So it sounds to me that it can only succeed in
> creating the first cgroup configuration. I did some researches online but
> relevant information is little. Any help is appreciated!
> 
> FYI, this is the system version that our Airflow is running on:
> 
> NAME="Ubuntu"
> 
> VERSION="14.04.6 LTS, Trusty Tahr"
> 
> ID=ubuntu
> 
> ID_LIKE=debian
> 
> PRETTY_NAME="Ubuntu 14.04.6 LTS"
> 
> VERSION_ID="14.04"
> 
> HOME_URL="http://www.ubuntu.com/"
> 
> SUPPORT_URL="http://help.ubuntu.com/"
> 
> Thanks,
> Junda


Mime
View raw message