beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <>
Subject [jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box
Date Wed, 07 Jun 2017 08:06:18 GMT


Luke Cwik commented on BEAM-2418:

The way in which you are building your jar file is broken since it includes multiple copies
of the same CoderProviderRegistrar file. Java does not understand jar files which have multiple
copies of the same file within it. In this specific case the three *META-INF/services/org.apache.beam.sdk.coders.CoderProviderRegistrar*
should have been concatenated together. 

lcwik@lcwik0:~/beam2418$ jar tvf dataflow-teleport-1.0-Alpha.jar | grep META-INF/services/org.apache.beam.sdk.coders.CoderProviderRegistrar
    71 Fri May 12 17:03:24 PDT 2017 META-INF/services/org.apache.beam.sdk.coders.CoderProviderRegistrar
   150 Fri May 12 16:56:14 PDT 2017 META-INF/services/org.apache.beam.sdk.coders.CoderProviderRegistrar
   130 Fri May 12 17:03:38 PDT 2017 META-INF/services/org.apache.beam.sdk.coders.CoderProviderRegistrar

The culprit seems to be that the build file in your project is incorrectly assembling the
*uber* jar:
task uberjar(type: Jar) {
    from files(sourceSets.main.output.classesDir)
    from {configurations.compile.collect {zipTree(it)}} {
        exclude "META-INF/*.SF"
        exclude "META-INF/*.DSA"
        exclude "META-INF/*.RSA"
    manifest {
        attributes 'Main-Class': mainClassName

Please take a look at the shadow gradle plugin and this section of their documentation about
merging resources (specifically 2.7.1. Merging Service Descriptor Files):

> Datastore IO does not work out of the box
> -----------------------------------------
>                 Key: BEAM-2418
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions, sdk-java-gcp
>    Affects Versions: 2.0.0
>            Reporter: Stephen Sisk
>            Assignee: Vikas Kedigehalli
>            Priority: Blocker
>             Fix For: Not applicable
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the core SDK
(protobuf in this case). ProtoCoder is not registered by default, so a user would need explicitly
include 'beam-sdks-java-extensions-protobuf' in their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the next release
to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]

This message was sent by Atlassian JIRA

View raw message