airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rémy Frenoy (JIRA) <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1583) PostgreSQL error when pushing a value bigger than 1GB
Date Fri, 08 Sep 2017 14:47:00 GMT
Rémy Frenoy created AIRFLOW-1583:
------------------------------------

             Summary: PostgreSQL error when pushing a value bigger than 1GB
                 Key: AIRFLOW-1583
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1583
             Project: Apache Airflow
          Issue Type: Bug
          Components: xcom
    Affects Versions: Airflow 1.8, Airflow 1.7.0
            Reporter: Rémy Frenoy
            Assignee: Rémy Frenoy
            Priority: Minor


When a task Operator or a PythonOperator returns a value, this value is automatically pushed
to XCom. This results in a postgresql error when the value is bigger than 1GB :

ERROR:  invalid memory alloc request size 1073741824

A size limit could be added on the value size in the xcom-push() function in model.py. When
value is higher than the limit size, a default value (maybe a string explaining that the initial
value was too high to be saved) could be pushed to XCom instead of the real one, avoiding
the postgresql error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message