airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-3783) Switch to HEADER arg to unload data from Redshift to S3
Date Mon, 14 Oct 2019 11:42:01 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950944#comment-16950944
] 

ASF subversion and git services commented on AIRFLOW-3783:
----------------------------------------------------------

Commit 5447b153665cc48b857f99545898172f76b734da in airflow's branch refs/heads/v1-10-test
from Felix Uellendall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5447b15 ]

[AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309)

replace the default unload_option 'PARALLEL OFF' by 'HEADER' as Redshift handles
header with parallel mode now

Co-authored-by: Shreya Chakraborty <shrechak@users.noreply.github.com>
(cherry picked from commit 3ed7b32775cf8328836cfb36af8e65cab01e221b)


> Switch to HEADER arg to unload data from Redshift to S3
> -------------------------------------------------------
>
>                 Key: AIRFLOW-3783
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3783
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: operators
>    Affects Versions: 1.10.5
>            Reporter: Shreya Chakraborty
>            Assignee: Felix Uellendall
>            Priority: Major
>             Fix For: 1.10.6
>
>
> Redshift has added a new *HEADER* option to the s3unload parameters.
> So we can remove the default PARALLEL OFF in the existing
> RedshiftToS3Transfer operator. The unload operation can now run parallely, hence faster,
along with headers, if required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message