beam-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-11277) WriteToBigQuery with batch file loads does not respect schema update options when there are multiple load jobs
Date Mon, 01 Mar 2021 06:05:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-11277?focusedWorklogId=559202&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-559202
]

ASF GitHub Bot logged work on BEAM-11277:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Mar/21 06:04
            Start Date: 01/Mar/21 06:04
    Worklog Time Spent: 10m 
      Work Description: codecov[bot] edited a comment on pull request #14113:
URL: https://github.com/apache/beam/pull/14113#issuecomment-787047099


   # [Codecov](https://codecov.io/gh/apache/beam/pull/14113?src=pr&el=h1) Report
   > Merging [#14113](https://codecov.io/gh/apache/beam/pull/14113?src=pr&el=desc)
(6c028ba) into [master](https://codecov.io/gh/apache/beam/commit/07e1e02125082d9ec804428f139eb849d79a8ec8?el=desc)
(07e1e02) will **decrease** coverage by `0.00%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/14113/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/14113?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #14113      +/-   ##
   ==========================================
   - Coverage   83.01%   83.01%   -0.01%     
   ==========================================
     Files         469      469              
     Lines       58355    58527     +172     
   ==========================================
   + Hits        48443    48585     +142     
   - Misses       9912     9942      +30     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/14113?src=pr&el=tree) | Coverage
Δ | |
   |---|---|---|
   | [...on/apache\_beam/runners/direct/watermark\_manager.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3Qvd2F0ZXJtYXJrX21hbmFnZXIucHk=)
| | |
   | [.../examples/snippets/transforms/elementwise/pardo.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9wYXJkby5weQ==)
| | |
   | [...sdks/python/apache\_beam/runners/direct/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvX19pbml0X18ucHk=)
| | |
   | [...ild/srcs/sdks/python/apache\_beam/utils/sentinel.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvc2VudGluZWwucHk=)
| | |
   | [...python/apache\_beam/examples/wordcount\_debugging.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvd29yZGNvdW50X2RlYnVnZ2luZy5weQ==)
| | |
   | [...hon/apache\_beam/io/gcp/datastore/v1new/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy9fX2luaXRfXy5weQ==)
| | |
   | [.../python/apache\_beam/portability/api/metrics\_pb2.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL21ldHJpY3NfcGIyLnB5)
| | |
   | [...s/sdks/python/apache\_beam/transforms/ptransform.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9wdHJhbnNmb3JtLnB5)
| | |
   | [...\_beam/testing/benchmarks/nexmark/queries/query4.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL25leG1hcmsvcXVlcmllcy9xdWVyeTQucHk=)
| | |
   | [...y38/build/srcs/sdks/python/apache\_beam/pipeline.py](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcGlwZWxpbmUucHk=)
| | |
   | ... and [928 more](https://codecov.io/gh/apache/beam/pull/14113/diff?src=pr&el=tree-more)
| |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14113?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14113?src=pr&el=footer).
Last update [07e1e02...6c028ba](https://codecov.io/gh/apache/beam/pull/14113?src=pr&el=lastupdated).
Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 559202)
    Time Spent: 2h 20m  (was: 2h 10m)

> WriteToBigQuery with batch file loads does not respect schema update options when there
are multiple load jobs
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-11277
>                 URL: https://issues.apache.org/jira/browse/BEAM-11277
>             Project: Beam
>          Issue Type: Bug
>          Components: io-py-gcp, runner-dataflow
>    Affects Versions: 2.21.0, 2.24.0, 2.25.0
>            Reporter: Chun Yang
>            Assignee: Chun Yang
>            Priority: P2
>         Attachments: repro.py
>
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> When multiple load jobs are needed to write data to a destination table, e.g., when the
data is spread over more than [10,000|https://cloud.google.com/bigquery/quotas#load_jobs]
URIs, WriteToBigQuery in FILE_LOADS mode will write data into temporary tables and then copy
the temporary tables into the destination table.
> When WriteToBigQuery is used with {{write_disposition=BigQueryDisposition.WRITE_APPEND}}
and {{additional_bq_parameters=\{"schemaUpdateOptions": ["ALLOW_FIELD_ADDITION"]\}}}, the
schema update options are not respected by the jobs that copy data from temporary tables into
the destination table. The effect is that for small jobs (<10K source URIs), schema field
addition is allowed, however, if the job is scaled to >10K source URIs, then schema field
addition will fail with an error such as:
> {code:none}Provided Schema does not match Table project:dataset.table. Cannot add fields
(field: field_name){code}
> I've been able to reproduce this issue with Python 3.7 and DataflowRunner on Beam 2.21.0
and Beam 2.25.0. I could not reproduce the issue with DirectRunner. A minimal reproducible
example is attached.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message