beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1187) GCP Transport not performing timed backoff after connection failure
Date Fri, 23 Jun 2017 16:41:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061198#comment-16061198
] 

ASF GitHub Bot commented on BEAM-1187:
--------------------------------------

GitHub user lukecwik opened a pull request:

    https://github.com/apache/beam/pull/3430

    [BEAM-1187] Improve logging to contain the number of retries done due to IOException and
unsuccessful response codes.

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [x] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [x] Make sure tests pass via `mvn clean verify`.
     - [x] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [x] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lukecwik/incubator-beam beam1187

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3430
    
----
commit f20a47c450f5a6b05cbb410506b233fee71dbb0c
Author: Luke Cwik <lcwik@google.com>
Date:   2017-06-23T16:32:49Z

    [BEAM-1187] Improve logging to contain the number of retries done due to IOException and
unsuccessful response codes.

----


> GCP Transport not performing timed backoff after connection failure
> -------------------------------------------------------------------
>
>                 Key: BEAM-1187
>                 URL: https://issues.apache.org/jira/browse/BEAM-1187
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow, sdk-java-core, sdk-java-gcp
>            Reporter: Luke Cwik
>            Priority: Minor
>
> The http request retries are failing and seemingly being immediately retried if there
is a connection exception. Note that below all the times are the same, and also that we are
logging too much. This seems to be related to the interaction by the chaining http request
initializer combining the Credential initializer followed by the RetryHttpRequestInitializer.
Also, note that we never log "Request failed with IOException, will NOT retry" which implies
that the retry logic never made it to the RetryHttpRequestInitializer.
> Action items are:
> 1) Ensure that the RetryHttpRequestInitializer is used
> 2) Ensure that calls do backoff
> 3) Reduce the logging to one terminal statement saying that we retried X times and final
failure was YYY.
> Dump of console output:
> Dec 20, 2016 9:12:20 AM com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner
fromOptions
> INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath:
will stage 1 files. Enable logging at DEBUG level to see which files will be staged.
> Dec 20, 2016 9:12:21 AM com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner
run
> INFO: Executing pipeline on the Dataflow Service, which will have billing implications
related to Google Compute Engine usage and other Google Cloud Services.
> Dec 20, 2016 9:12:21 AM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElements
> INFO: Uploading 1 files from PipelineOptions.filesToStage to staging location to prepare
for execution.
> Dec 20, 2016 9:12:21 AM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElements
> INFO: Uploading PipelineOptions.filesToStage complete: 1 files newly uploaded, 0 files
cached
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> 	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> 	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
> 	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
> 	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
> 	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
> 	at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> 	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> 	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
> 	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
> 	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
> 	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
> 	at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> 	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> 	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
> 	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
> 	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
> 	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
> 	at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> 	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> 	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
> 	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
> 	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
> 	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
> 	at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)
> Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute
> WARNING: exception thrown while executing request
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> 	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> 	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> 	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
> 	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
> 	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
> 	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
> 	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:632)
> 	at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:201)
> 	at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.numbersStreamingFromPubsub(NumbersStreaming.java:378)
> 	at com.google.cloud.dataflow.integration.NumbersStreaming.main(NumbersStreaming.java:831)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message