airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Kaushik (Jira)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-6124) tmp folder gets deleted, shell-init: error retrieving current directory
Date Fri, 29 Nov 2019 13:32:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravi Kaushik updated AIRFLOW-6124:
----------------------------------
    Description: 
My dag has three SparkSubmitOperators and throws this error occasionally after executing one
task. If I trigger the job manually by ignoring all dependencies, it works well. Most likely
/tmp folder is getting deleted and it cannot find it again

 

{{```[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,547] \{base_hook.py:84}
INFO - Using connection to: id: spark_default. Host: yarn, Port: None, Schema: None, Login:
None, Password: None, extra: \{'deploy-mode': 'cluster', 'spark-binary': '/usr/bin/spark2-submit/'}
 [2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,548] \{spark_submit_hook.py:302}
INFO - Spark-Submit cmd: ['spark-submit', '--master', 'yarn', '--conf', 'spark.yarn.maxAppAttempts=2',
'--files', '/home/centos/airflow/dags/conf/aws_keys.pickle', '--num-executors', '4', '--executor-cores',
'4', '--executor-memory', '20G', '--name', 'universal_cross_matrix_joins_AUS_27-11-2019',
'--deploy-mode', 'cluster', '/home/centos/airflow/scripts/near-airflow-repo/scripts/universal_cross_matrix/join_ifa_ncid.py',
'--input_date', '27-11-2019', '--country', 'AUS', '--tenant_id', '63ce7457']
 [2019-11-29 12:58:54,553] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,553] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,555] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,555] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,557] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,557] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,560] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,560] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,563] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,563] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,565] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,564] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,567] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,567] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,569] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,569] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,571] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,571] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,574] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,574] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
 [2019-11-29 12:58:54,576] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,576] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory```}}

  was:
My dag has three SparkSubmitOperators and throws this error 

 

{{```[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,547] \{base_hook.py:84}
INFO - Using connection to: id: spark_default. Host: yarn, Port: None, Schema: None, Login:
None, Password: None, extra: \{'deploy-mode': 'cluster', 'spark-binary': '/usr/bin/spark2-submit/'}
[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,548] \{spark_submit_hook.py:302}
INFO - Spark-Submit cmd: ['spark-submit', '--master', 'yarn', '--conf', 'spark.yarn.maxAppAttempts=2',
'--files', '/home/centos/airflow/dags/conf/aws_keys.pickle', '--num-executors', '4', '--executor-cores',
'4', '--executor-memory', '20G', '--name', 'universal_cross_matrix_joins_AUS_27-11-2019',
'--deploy-mode', 'cluster', '/home/centos/airflow/scripts/near-airflow-repo/scripts/universal_cross_matrix/join_ifa_ncid.py',
'--input_date', '27-11-2019', '--country', 'AUS', '--tenant_id', '63ce7457']
[2019-11-29 12:58:54,553] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,553] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,555] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,555] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,557] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,557] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,560] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,560] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,563] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,563] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,565] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,564] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,567] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,567] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,569] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,569] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,571] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,571] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,574] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,574] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
[2019-11-29 12:58:54,576] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,576] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory```}}


> tmp folder gets deleted, shell-init: error retrieving current directory
> -----------------------------------------------------------------------
>
>                 Key: AIRFLOW-6124
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6124
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DAG
>    Affects Versions: 1.10.6
>            Reporter: Ravi Kaushik
>            Priority: Critical
>         Attachments: Error.png
>
>
> My dag has three SparkSubmitOperators and throws this error occasionally after executing
one task. If I trigger the job manually by ignoring all dependencies, it works well. Most
likely /tmp folder is getting deleted and it cannot find it again
>  
> {{```[2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,547]
\{base_hook.py:84} INFO - Using connection to: id: spark_default. Host: yarn, Port: None,
Schema: None, Login: None, Password: None, extra: \{'deploy-mode': 'cluster', 'spark-binary':
'/usr/bin/spark2-submit/'}
>  [2019-11-29 12:58:54,548] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,548] \{spark_submit_hook.py:302}
INFO - Spark-Submit cmd: ['spark-submit', '--master', 'yarn', '--conf', 'spark.yarn.maxAppAttempts=2',
'--files', '/home/centos/airflow/dags/conf/aws_keys.pickle', '--num-executors', '4', '--executor-cores',
'4', '--executor-memory', '20G', '--name', 'universal_cross_matrix_joins_AUS_27-11-2019',
'--deploy-mode', 'cluster', '/home/centos/airflow/scripts/near-airflow-repo/scripts/universal_cross_matrix/join_ifa_ncid.py',
'--input_date', '27-11-2019', '--country', 'AUS', '--tenant_id', '63ce7457']
>  [2019-11-29 12:58:54,553] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,553] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,555] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,555] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,557] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,557] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,560] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,560] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,563] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,563] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,565] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,564] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,567] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,567] \{spark_submit_hook.py:433}
INFO - shell-init: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,569] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,569] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,571] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,571] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,574] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,574] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory
>  [2019-11-29 12:58:54,576] \{logging_mixin.py:112} INFO - [2019-11-29 12:58:54,576] \{spark_submit_hook.py:433}
INFO - chdir: error retrieving current directory: getcwd: cannot access parent directories:
No such file or directory```}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message