airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (AIRFLOW-1583) PostgreSQL error when pushing a value bigger than 1GB
Date Sun, 02 Sep 2018 18:00:04 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned AIRFLOW-1583:
-------------------------------------

    Assignee: Rémy Frenoy  (was: Holden Karau's magical unicorn)

> PostgreSQL error when pushing a value bigger than 1GB
> -----------------------------------------------------
>
>                 Key: AIRFLOW-1583
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1583
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: xcom
>    Affects Versions: Airflow 1.8, Airflow 1.7.0
>            Reporter: Rémy Frenoy
>            Assignee: Rémy Frenoy
>            Priority: Minor
>   Original Estimate: 0.25h
>  Remaining Estimate: 0.25h
>
> When a task Operator or a PythonOperator returns a value, this value is automatically
pushed to XCom. This results in a postgresql error when the value is bigger than 1GB :
> ERROR:  invalid memory alloc request size 1073741824
> A size limit could be added on the value size in the xcom-push() function in model.py.
When value is higher than the limit size, a default value (maybe a string explaining that
the initial value was too high to be saved) could be pushed to XCom instead of the real one,
avoiding the postgresql error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message