beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2985) BigQuery IO write transform is broken for DirectRunner
Date Sat, 23 Sep 2017 01:16:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177415#comment-16177415
] 

ASF GitHub Bot commented on BEAM-2985:
--------------------------------------

GitHub user chamikaramj opened a pull request:

    https://github.com/apache/beam/pull/3892

    [BEAM-2985] Updates WriteToBigQuery PTransform to get project id from GoogleCloud…

    …Options when using DirectRunner.
    
    WriteToBigQuery PTransform behaves differently for DirectRunner and DataflowRunner when
it comes to determining the project that the output table belongs to. If a project is not
specified, DataflowRunner defauls to GoogleCloudOptions.project while DirectRunner does not.
This PR fixes this inconsistency by defaulting to GoogleCloudOptions.project for DirectRunner
as well.
    
    Follow this checklist to help us incorporate your contribution quickly and easily:
    
     - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/)
filed for the change (usually before you start working on it).  Trivial changes like typos
do not require a JIRA issue.  Your pull request should address just this issue, without pulling
in other changes.
     - [ ] Each commit in the pull request should have a meaningful subject line and body.
     - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue.
     - [ ] Write a pull request description that is detailed enough to understand what the
pull request does, how, and why.
     - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
     - [ ] If this contribution is large, please file an Apache [Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/chamikaramj/beam bq_direct_runner_write

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3892
    
----
commit f99db7932cab90dda2741d22b291e7f1eaad7336
Author: chamikara@google.com <chamikara@google.com>
Date:   2017-09-23T00:59:50Z

    Updates WriteToBigQuery PTransform to get project id from GoogleCloudOptions when using
DirectRunner.
    
    WriteToBigQuery PTransform behaves differently for DirectRunner and DataflowRunner when
it comes to determining the project that the output table belongs to. If a project is not
specified, DataflowRunner defauls to GoogleCloudOptions.project while DirectRunner does not.
This PR fixes this inconsistency by defaulting to GoogleCloudOptions.project for DirectRunner
as well.

----


> BigQuery IO write transform is broken for DirectRunner
> ------------------------------------------------------
>
>                 Key: BEAM-2985
>                 URL: https://issues.apache.org/jira/browse/BEAM-2985
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Chamikara Jayalath
>            Assignee: Chamikara Jayalath
>
> I get following error when trying to run BigQuery tornadoes using DirectRunner.
> DataflowRunner seems to be working fine.
> python -m apache_beam.examples.cookbook.bigquery_tornadoes --output <dataset>.<table>
--project <project>
>  Request missing required parameter projectId
>  Traceback for above exception (most recent call last):
>   File "apache_beam/utils/retry.py", line 175, in wrapper
>     return fun(*args, **kwargs)
>   File "apache_beam/io/gcp/bigquery.py", line 828, in _get_table
>     response = self.client.tables.Get(request)
>   File "apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 608,
in Get
>     config, request, global_params=global_params)
>   File "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 695, in _RunMethod
>     download)
>   File "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 676, in PrepareHttpRequest
>     method_config, request, relative_path=url_builder.relative_path)
>   File "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 580, in __ConstructRelativePath
>     relative_path=relative_path)
>   File "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/util.py",
line 124, in ExpandRelativePath
>     'Request missing required parameter %s' % param)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message