airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (AIRFLOW-1583) PostgreSQL error when pushing a value bigger than 1GB
Date Sun, 02 Sep 2018 18:00:04 GMT


Apache Spark reassigned AIRFLOW-1583:

    Assignee: Rémy Frenoy  (was: Holden Karau's magical unicorn)

> PostgreSQL error when pushing a value bigger than 1GB
> -----------------------------------------------------
>                 Key: AIRFLOW-1583
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: xcom
>    Affects Versions: Airflow 1.8, Airflow 1.7.0
>            Reporter: Rémy Frenoy
>            Assignee: Rémy Frenoy
>            Priority: Minor
>   Original Estimate: 0.25h
>  Remaining Estimate: 0.25h
> When a task Operator or a PythonOperator returns a value, this value is automatically
pushed to XCom. This results in a postgresql error when the value is bigger than 1GB :
> ERROR:  invalid memory alloc request size 1073741824
> A size limit could be added on the value size in the xcom-push() function in
When value is higher than the limit size, a default value (maybe a string explaining that
the initial value was too high to be saved) could be pushed to XCom instead of the real one,
avoiding the postgresql error.

This message was sent by Atlassian JIRA

View raw message