airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Anand (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-139) Executing VACUUM with PostgresOperator
Date Sat, 08 Oct 2016 17:55:21 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558422#comment-15558422
] 

Siddharth Anand edited comment on AIRFLOW-139 at 10/8/16 5:55 PM:
------------------------------------------------------------------

FYI, I haven't had a chance to follow this very closely, but we won't be merging fixes that
don't work for a range of postgres versions [9.3, 9.6] (inclusive). We might in fact support
versions as far back as 9.2. Also, we don't use Redshift and I hear that has different versioning,
so we may just need to break that out into a different operator unless someone has already
done that. 

BTW, this means that our travis builds will need to test against all of those postgres versions
and it doens't do that currently. Also, a vacuum test case is sorely needed. If someone wants
to take the lead on this, I'm happy to pair (with reviews and merging). 
-s


was (Author: sanand):
FYI, I haven't had a chance to follow this very closely, but we won't be merging fixes that
don't work for a range of postgres versions [9.3, 9.6] (inclusive). We might in fact support
versions as far back as 9.2. Also, we don't use Redshift and I hear that has different versioning,
so we may just need to break that out into a different operator unless someone has already
done that. 

-s

> Executing VACUUM with PostgresOperator
> --------------------------------------
>
>                 Key: AIRFLOW-139
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-139
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: Airflow 1.7.0
>            Reporter: Rafael
>
> Dear Airflow Maintainers,
> h1. Environment
> * Airflow version: *v1.7.0*
> * Airflow components: *PostgresOperator*
> * Python Version: *Python 3.5.1*
> * Operating System: *15.4.0 Darwin*
> h1. Description of Issue
> I am trying to execute a `VACUUM` command as part of DAG with the `PostgresOperator`,
which fails with the following error:
> {quote}
> [2016-05-14 16:14:01,849] {__init__.py:36} INFO - Using executor SequentialExecutor
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 15, in <module>
>     args.func(args)
>   File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/bin/cli.py",
line 203, in run
>     pool=args.pool,
>   File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/models.py",
line 1067, in run
>     result = task_copy.execute(context=context)
>   File "/usr/local/lib/python3.5/site-packages/airflow/operators/postgres_operator.py",
line 39, in execute
>     self.hook.run(self.sql, self.autocommit, parameters=self.parameters)
>   File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/airflow/hooks/dbapi_hook.py",
line 109, in run
>     cur.execute(s)
> psycopg2.InternalError: VACUUM cannot run inside a transaction block
> {quote}
> I could create a small python script that performs the operation, as explained in [this
stackoverflow entry](http://stackoverflow.com/questions/1017463/postgresql-how-to-run-vacuum-from-code-outside-transaction-block).
However, I would like to know first if the `VACUUM` command should be supported by the `PostgresOperator`.
> h1. Reproducing the Issue
> The operator can be declared as follows:
> {quote}
> conn = ('postgres_default')
> t4 = PostgresOperator(
>     task_id='vacuum',
>     postgres_conn_id=conn,
>     sql=("VACUUM public.table"),
>     dag=dag
>     )
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message