beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Łukasz Gajowy (JIRA) <>
Subject [jira] [Commented] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"
Date Thu, 04 Jan 2018 16:13:00 GMT


Łukasz Gajowy commented on BEAM-3406:

In short: to fix this issue right away I think we need access to apache-beam-testing project
to set schema in bigquery table.
While creating the dashboard for FileBased ioit we created the table in the bigquery underneath
but the schema was not set and it is needed there. According to this:  [documentation|]
we can set the schema while loading data or right after creating the table. Perfkit does not
set the schema in any way for now.

We could set the schema manually and fix the issue, but AFAIK we do not have access to "apache-beam-testing"
where bigquery seems to be located. Could you grant it to us [~chamikara]?

I also created [an issue|]
on PKBs github to add an --autodetect flag while loading json files to bigquery - as i checked
locally, the schema we use can be autodetected. If this issue gets resolved and merged in
perfkit we won't have to worry about the schema anymore as it will be set automatically.

> FIeBasedIO IT fails due to "BigQuery error"
> -------------------------------------------
>                 Key: BEAM-3406
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: testing
>            Reporter: Chamikara Jayalath
>            Assignee: Łukasz Gajowy
> For example, 
> beam_performance.filebasedioit_pkb_results /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json.
Got return code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/
UserWarning: You have requested explicit scopes to be used with a GCE service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?

This message was sent by Atlassian JIRA

View raw message