beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Bookman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box
Date Tue, 06 Jun 2017 22:00:19 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039733#comment-16039733
] 

Colin Bookman commented on BEAM-2418:
-------------------------------------

Same issue. Tried with the following arguments.


java -jar build/libs/*.jar \
  --runner=DataflowRunner \
  --project=my-project \
  --stagingLocation=gs://my-project.appspot.com/staging/ \
  --tempLocation=gs://my-project.appspot.com/temp/


```
Jun 06, 2017 2:57:37 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath:
will stage 1 files. Enable logging at DEBUG level to see which files will be staged.
Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder
for IngestEntities/ParDo(GqlQueryTranslate)/ParMultiDo(GqlQueryTranslate).out0 [PCollection].
Correct one of the following root causes:
  No Coder has been manually specified;  you may do so using .setCoder().
  Inferring a Coder from the CoderRegistry failed: Unable to provide a Coder for com.google.datastore.v1.Query.
  Building a Coder using a registered CoderProvider failed.
  See suppressed exceptions for detailed failures.
  Using the default output Coder from the producing PTransform failed: Unable to provide a
Coder for com.google.datastore.v1.Query.
  Building a Coder using a registered CoderProvider failed.
  See suppressed exceptions for detailed failures.
	at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
	at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:250)
	at org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:104)
	at org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:147)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:481)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:422)
	at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:277)
	at org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read.expand(DatastoreV1.java:581)
	at org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read.expand(DatastoreV1.java:226)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:482)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:441)
	at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
	at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:179)
	at com.google.cloud.dataflow.teleport.DatastoreToGcs.main(DatastoreToGcs.java:50)
	at com.google.cloud.dataflow.teleport.Main.main(Main.java:50)
```

> Datastore IO does not work out of the box
> -----------------------------------------
>
>                 Key: BEAM-2418
>                 URL: https://issues.apache.org/jira/browse/BEAM-2418
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions, sdk-java-gcp
>    Affects Versions: 2.0.0
>            Reporter: Stephen Sisk
>            Assignee: Davor Bonaci
>            Priority: Blocker
>             Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the core SDK
(protobuf in this case). ProtoCoder is not registered by default, so a user would need explicitly
include 'beam-sdks-java-extensions-protobuf' in their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the next release
to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message