beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-4041) Performance tests fail due to kubernetes load balancer problems
Date Fri, 18 May 2018 14:41:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-4041?focusedWorklogId=103393&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103393
]

ASF GitHub Bot logged work on BEAM-4041:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 18/May/18 14:40
            Start Date: 18/May/18 14:40
    Worklog Time Spent: 10m 
      Work Description: DariuszAniszewski opened a new pull request #5425: [BEAM-4041] Increase
timeout for getting K8s LoadBalancer external IP
URL: https://github.com/apache/beam/pull/5425
 
 
   From some time we're observing that obtaining K8s LoadBalancer IP fails, even with 3 minutes
timeout. This came less often but still in this week (at least) two jobs suffered this problem:
   
   - `beam_PerformanceTests_MongoDBIO_IT` [build 168](https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_MongoDBIO_IT/168/console)
   - `beam_PerformanceTests_HadoopInputFormat` [build 256](https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/256/console)
and [build 257](https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_HadoopInputFormat/257/console)
   
   Since [PR 1641 to PerfkiBenchmarker](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1641)
it's now possible to configure retry-mechanism policy. 
   This PR increases overall timeout from 3 minutes to 6 minutes.
   
   ----------------
   
   Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically
link the pull request to the issue.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`)
to look at it.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 103393)
            Time Spent: 10m
    Remaining Estimate: 0h

> Performance tests fail due to kubernetes load balancer problems
> ---------------------------------------------------------------
>
>                 Key: BEAM-4041
>                 URL: https://issues.apache.org/jira/browse/BEAM-4041
>             Project: Beam
>          Issue Type: Bug
>          Components: testing
>            Reporter: Łukasz Gajowy
>            Assignee: Jason Kuster
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Recently, as we added more IOITs to be run on jenkins using kubernetes, some of them
started to fail randomly, because they couldn't retrieve LoadBalancer address. Normally obtaining
the address took about one minute. Perfkit waits for the address (actively checking for
it) for 3 minutes. This should be enough for getting the address, yet it recently started
to exceed the 3 minutes limit. I also noticed that this error didn't happen when there were
fewer tests.
> Example logs:
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/31/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message