beam-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomo Suzuki (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-11968) Bump com.google.cloud:libraries-bom to 19.0.0
Date Mon, 15 Mar 2021 15:49:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301718#comment-17301718
] 

Tomo Suzuki edited comment on BEAM-11968 at 3/15/21, 3:48 PM:
--------------------------------------------------------------

I agree that it's nice to upgrade the version of the Libraries BOM. Thank you for PR.

But I wonder why the NoSuchMethodError happened.

h1. Troubleshooting

The Libraries BOM version was 13.2.0 in 2.28.0 tree https://github.com/apache/beam/blob/release-2.28.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L535.
That version of libraries BOM had google-http-java-client version 1.36.0.

{{com.google.api.services.storage.Storage$Builder}} is {{com.google.apis:google-api-services-storage}}
artifact. At 2.28.0 release, it was "v1-rev20200611-1.30.10" (the Libraries BOM does not have
the artifact in it). I don't see the {{Storage$Builder}} class has a reference to the missing
method.

{noformat}
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar
'com.google.api.services.storage.Storage$Builder' |grep isMtls
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar
'com.google.api.services.storage.Storage$Builder' |grep chooseEndpoint
suztomo-macbookpro44% 
{noformat}


[~fokko] From the observation above, I'm assuming that you specify the version of {{com.google.apis:google-api-services-storage}}
by yourself or there are other dependencies that fetched the different version than "v1-rev20200611-1.30.10".
*Would you provide the dependency graph* ({{mvn dependency:tree}} or {{./gradlew dependencies}})
of the problematic project? The tree might reveal the unexpected consequence in Beam's dependency
management and I'd like to fix it if any.




was (Author: suztomo):
I agree that it's nice to upgrade the version of the Libraries BOM.

h1. Troubleshooting

The Libraries BOM version was 13.2.0 in 2.28.0 tree https://github.com/apache/beam/blob/release-2.28.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L535.
That version of libraries BOM had google-http-java-client version 1.36.0.

{{com.google.api.services.storage.Storage$Builder}} is {{com.google.apis:google-api-services-storage}}
artifact. At 2.28.0 release, it was "v1-rev20200611-1.30.10" (the Libraries BOM does not have
the artifact in it). I don't see the {{Storage$Builder}} class has a reference to the missing
method.

{noformat}
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar
'com.google.api.services.storage.Storage$Builder' |grep isMtls
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar
'com.google.api.services.storage.Storage$Builder' |grep chooseEndpoint
suztomo-macbookpro44% 
{noformat}


[~fokko] From the observation above, I'm assuming that you specify the version of {{com.google.apis:google-api-services-storage}}
by yourself or there are other dependencies that fetched the different version than "v1-rev20200611-1.30.10".
*Would you provide the dependency graph* ({{mvn dependency:tree}} or {{./gradlew dependencies}})
of the problematic project? The tree might reveal the unexpected consequence in Beam's dependency
management and I'd like to fix it if any.



> Bump com.google.cloud:libraries-bom to 19.0.0
> ---------------------------------------------
>
>                 Key: BEAM-11968
>                 URL: https://issues.apache.org/jira/browse/BEAM-11968
>             Project: Beam
>          Issue Type: Bug
>          Components: extensions-java-gcp
>    Affects Versions: 2.28.0
>            Reporter: Fokko Driesprong
>            Assignee: Fokko Driesprong
>            Priority: P2
>             Fix For: 2.29.0
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Missing the isMtls() in google-http-client. Looks like it has been added since 1.38.0:
> [https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpTransport.java#L136]
> Original stacktrace:
> java.lang.RuntimeException: Encountered checked exception when constructing an instance
from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
>  at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:233)
>  at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
>  at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
>  at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
> ...
>  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
>  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke (Method.java:564)
>  at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
>  at java.lang.Thread.run (Thread.java:832)
> Caused by: java.lang.NoSuchMethodError: 'boolean com.google.api.client.http.HttpTransport.isMtls()'
>  at com.google.api.services.storage.Storage$Builder.chooseEndpoint (Storage.java:11151)
>  at com.google.api.services.storage.Storage$Builder.<init> (Storage.java:11184)
>  at org.apache.beam.sdk.extensions.gcp.util.Transport.newStorageClient (Transport.java:97)
>  at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:104)
>  at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:93)
>  at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:605)
>  at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:546)
>  at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:171)
>  at com.sun.proxy.$Proxy41.getGcsUtil (Unknown Source)
>  at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible
(GcsPathValidator.java:83)
>  at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported
(GcsPathValidator.java:53)
>  at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:291)
>  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
>  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke (Method.java:564)
>  at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:217)
>  at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
>  at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
>  at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
> ...
>  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
>  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke (Method.java:564)
>  at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
>  at java.lang.Thread.run (Thread.java:832)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message