airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joy Gao (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-1613) Make MySqlToGoogleCloudStorageOperator compaitible with python3
Date Thu, 14 Sep 2017 20:16:00 GMT
Joy Gao created AIRFLOW-1613:

             Summary: Make MySqlToGoogleCloudStorageOperator compaitible with python3
                 Key: AIRFLOW-1613
             Project: Apache Airflow
          Issue Type: Bug
          Components: contrib
            Reporter: Joy Gao

In Python 3, map(...) returns an iterator, which can only be iterated over once. 
Therefore the current implementation will return an empty list after the first iteration of

schema = map(lambda schema_tuple: schema_tuple[0], cursor.description)
        file_no = 0
        tmp_file_handle = NamedTemporaryFile(delete=True)
        tmp_file_handles = {self.filename.format(file_no): tmp_file_handle}

        for row in cursor:
            # Convert datetime objects to utc seconds, and decimals to floats
            row = map(self.convert_types, row)
            row_dict = dict(zip(schema, row))

Moving it inside the loop for re-use.

This message was sent by Atlassian JIRA

View raw message