Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Tue, 12 Sep 2017 19:17:00 +0000 (UTC)
From: "Reuven Lax (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13093865.1502386652000.94439.1505243820330@Atlassian.JIRA>
In-Reply-To: <JIRA.13093865.1502386652000@Atlassian.JIRA>
References: <JIRA.13093865.1502386652000@Atlassian.JIRA> <JIRA.13093865.1502386652448@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-2761) Write to empty BigQuery partition
 fails with "No schema specified on job or table." despite having provided
 schema
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 12 Sep 2017 19:17:12 -0000


    [ https://issues.apache.org/jira/browse/BEAM-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163509#comment-16163509 ] 

Reuven Lax commented on BEAM-2761:
----------------------------------

Hi, I ran the precise job listed in the bug using the latest Beam snapshot, and could not reproduce the failure. BigQuery successfully created an empty table called pets and did not fail with the error. I am going to resolve this issue for now. Please reopen if you can reproduce this on the Beam 2.2.0 snapshots.

> Write to empty BigQuery partition fails with "No schema specified on job or table." despite having provided schema
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-2761
>                 URL: https://issues.apache.org/jira/browse/BEAM-2761
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>    Affects Versions: 2.1.0, 2.2.0
>            Reporter: Fallon
>            Assignee: Reuven Lax
>            Priority: Minor
>             Fix For: 2.2.0
>
>         Attachments: beam-2761-stacktrace.txt
>
>
> In 2.1.0-SNAPSHOT and 2.2.0-SNAPSHOT, jobs writing an empty PCollection to a BigQuery partition fail with "java.lang.RuntimeException: Failed to create load job with id prefix". This is associated with a message "No schema specified on job or table" even though a schema is provided. See attached stack trace for the more detail on the error.
> Command to run job:
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.EmptyPCollection \
>      -Dexec.args="--runner=DataflowRunner --project=<GCP project> \
>                   --gcpTempLocation=<tmp location>" \
>      -Pdataflow-runner
> {code}
> Code to reproduce the problem:
> {code:title=EmptyPCollection.java|borderStyle=solid}
> public class EmptyPCollection {
>   public static void main(String[] args) {
>     PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
>     options.setTempLocation("<your tmp location>");
>     Pipeline pipeline = Pipeline.create(options);
>     String schema = "{\"fields\": [{\"name\": \"pet\", \"type\": \"string\", \"mode\": \"required\"}]}";
>     String table = "mydataset.pets";
>     List<String> pets = Arrays.asList("Dog", "Cat", "Goldfish");
>     PCollection<String> inputText = pipeline.apply(Create.of(pets)).setCoder(StringUtf8Coder.of());
>     PCollection<TableRow> rows = inputText.apply(ParDo.of(new DoFn<String, TableRow>() {
>       @ProcessElement
>       public void processElement(ProcessContext c) {
>         String text = c.element();
>         if (text.startsWith("X")) {  // change to (D)og and works fine
>           TableRow row = new TableRow();
>           row.set("pet", text);
>           c.output(row);
>         }
>       }
>     }));
>     rows.apply(BigQueryIO.writeTableRows().to(table).withJsonSchema(schema)
>         .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
>         .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
>     pipeline.run().waitUntilFinish();
>   }
> }
> {code}


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)