beam-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenbing Bai (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (BEAM-8965) WriteToBigQuery failed in BundleBasedDirectRunner
Date Wed, 10 Jun 2020 19:30:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenbing Bai resolved BEAM-8965.
-------------------------------
    Fix Version/s: 2.20.0
         Assignee: Wenbing Bai
       Resolution: Fixed

> WriteToBigQuery failed in BundleBasedDirectRunner
> -------------------------------------------------
>
>                 Key: BEAM-8965
>                 URL: https://issues.apache.org/jira/browse/BEAM-8965
>             Project: Beam
>          Issue Type: Bug
>          Components: io-py-gcp
>    Affects Versions: 2.16.0, 2.17.0, 2.18.0, 2.19.0
>            Reporter: Wenbing Bai
>            Assignee: Wenbing Bai
>            Priority: P2
>             Fix For: 2.20.0
>
>          Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> *{{WriteToBigQuery}}* fails in *{{BundleBasedDirectRunner}}* with error {{PCollection
of size 2 with more than one element accessed as a singleton view.}}
> Here is the code
>  
> {code:python}
> with Pipeline() as p:
>     query_results = (
>         p 
>         | beam.io.Read(beam.io.BigQuerySource(
>             query='SELECT ... FROM ...')
>         )
>     query_results | beam.io.gcp.WriteToBigQuery(
>             table=<your_table_name>,
>             method=WriteToBigQuery.Method.FILE_LOADS,
>             schema={"fields": []}
>         )
> {code}
>  
> Here is the error
>  
> {code:none}
>   File "apache_beam/runners/common.py", line 778, in apache_beam.runners.common.DoFnRunner.process
>     def process(self, windowed_value):
>   File "apache_beam/runners/common.py", line 782, in apache_beam.runners.common.DoFnRunner.process
>     self._reraise_augmented(exn)
>   File "apache_beam/runners/common.py", line 849, in apache_beam.runners.common.DoFnRunner._reraise_augmented
>     raise_with_traceback(new_exn)
>   File "apache_beam/runners/common.py", line 780, in apache_beam.runners.common.DoFnRunner.process
>     return self.do_fn_invoker.invoke_process(windowed_value)
>   File "apache_beam/runners/common.py", line 587, in apache_beam.runners.common.PerWindowInvoker.invoke_process
>     self._invoke_process_per_window(
>   File "apache_beam/runners/common.py", line 610, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
>     [si[global_window] for si in self.side_inputs]))
>   File "/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
>     _FilteringIterable(self._iterable, target_window), self._view_options)
>   File "/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 443, in _from_runtime_iterable
>     len(head), str(head[0]), str(head[1])))
> ValueError: PCollection of size 2 with more than one element accessed as a singleton
view. First two elements encountered are "gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f",
"gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f". [while running 'WriteToBigQuery/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)']
> {code}
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message