airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Forestlzj (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-3128) SparkSqlOperator prints blank info endlessly after SparkContext stopped
Date Fri, 28 Sep 2018 09:30:00 GMT
Forestlzj created AIRFLOW-3128:
----------------------------------

             Summary: SparkSqlOperator prints blank info endlessly after SparkContext stopped
                 Key: AIRFLOW-3128
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3128
             Project: Apache Airflow
          Issue Type: Bug
          Components: contrib
            Reporter: Forestlzj
             Fix For: 1.10.0


environment: hdp 2.6.2 with spark 2 

spark_sql_test = SparkSqlOperator(task_id='spark_sql_test',dag=dag,sql='select * from default.test
limit 2;',name='airflow_spark_sql_test',executor_cores=2,num_executors=2)

Log:

[2018-09-28 17:14:08,754] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!\n'
[2018-09-28 17:14:08,778] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO SparkContext:
{color:#205081}Successfully stopped SparkContext\n'{color}
[2018-09-28 17:14:08,780] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO ShutdownHookManager:
Shutdown hook called\n'
[2018-09-28 17:14:08,780] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO ShutdownHookManager:
Deleting directory /tmp/spark-ba3b2fef-ae75-4e65-afb2-1b029b2c5c23\n'
[2018-09-28 17:14:08,781] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08 INFO ShutdownHookManager:
Deleting directory /tmp/spark-ba397c83-4003-4006-8ce9-179cc2df2dd7\n'
{color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO - b''{color}
{color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO - b''{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message