airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Chitwood (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1898) Large XComs are not supported and fail silently
Date Wed, 07 Feb 2018 19:08:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355900#comment-16355900
] 

Sean Chitwood commented on AIRFLOW-1898:
----------------------------------------

The size not specified in the migration, so SqlAlchemy uses it's default on mysql, BLOB.

As I see it, the options are:
 * Decide on an upper limit for xcom sizes, set the size in a migration and document it.
 * Document the size limits that are created by the DB backends.

> Large XComs are not supported and fail silently
> -----------------------------------------------
>
>                 Key: AIRFLOW-1898
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1898
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: xcom
>    Affects Versions: Airflow 1.8
>         Environment: MySQL
>            Reporter: Len Frodgers
>            Priority: Major
>
> I am using Airlfow backed by MySQL and having problems with large XComs (> 64 KB).
Xcom uses PickleType which is backed by BLOB on MySQL.
> Unfortunately, MySQL by default truncates anything longer than BLOB (64 KB) when saving,
so when unpickling such XComs, they are corrupt and an EOFError is raised.
> Two things we need:
> 1) Validation when saving the XCom that it is not too big
> 2) Use MEDIUMBLOB as the underlying data type for the XCOM column on MySQL so large XComs
can be stored (supports  up to 12 MB i think)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message