aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David McLaughlin" <da...@dmclaughlin.com>
Subject Re: Review Request 27698: Move zookeeper connection off the main thread to prevent client deadlocks.
Date Thu, 06 Nov 2014 21:00:06 GMT


> On Nov. 6, 2014, 8:55 p.m., Aurora ReviewBot wrote:
> > Master (53c732d) is red with this patch.
> >   ./build-support/jenkins/build.sh
> > 
> >     
> >       client = make_mock_client(proxy_url=None)
> > >     assert client.url == '%s://%s:%d' % (scheme, host, port)
> > 
> > src/test/python/apache/aurora/client/api/test_scheduler_client.py:364: 
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

> > /tmp/tmpKUzNvG/apache/aurora/client/api/scheduler_client.py:167: in url
> >     return self.raw_url
> > /tmp/tmpKUzNvG/apache/aurora/client/api/scheduler_client.py:172: in raw_url
> >     self._resolve()
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

> > 
> > self = <apache.aurora.client.api.scheduler_client.ZookeeperSchedulerClient object
at 0x7f83d8ac7290>
> > 
> >     def _resolve(self):
> >       """Resolve the uri associated with this scheduler from zookeeper."""
> >       joined = threading.Event()
> >       def on_join(elements):
> >         joined.set()
> >     
> >       zk, serverset = self.get_scheduler_serverset(self._cluster, verbose=self._verbose,
> >         port=self._zkport, on_join=on_join)
> >     
> >       joined.wait(timeout=self.SERVERSET_TIMEOUT.as_(Time.SECONDS))
> >     
> >       try:
> >         serverset_endpoints = deadline(lambda: list(serverset),
> >           timeout=self.SERVERSET_TIMEOUT.as_(Time.SECONDS), daemon=True, propagate=True)
> >       except Timeout:
> >         raise self.CouldNotConnect("Failed to connect to Zookeeper within %d seconds."
%
> > >         self.SERVERSET_TIMEOUT.as_(Time.SECONDS))
> > E       CouldNotConnect: Failed to connect to Zookeeper within 0 seconds.
> > 
> > /tmp/tmpKUzNvG/apache/aurora/client/api/scheduler_client.py:143: CouldNotConnect
> >  generated xml file: /jenkins/workspace/AuroraBot/dist/test-results/src.test.python.apache.aurora.client.api.scheduler_client.xml

> > !!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
> > =============== 1 failed, 60 passed, 1 warnings in 6.15 seconds ================
> > Build operating on top level addresses: set([BuildFileAddress(/jenkins/workspace/AuroraBot/src/test/python/BUILD,
all)])
> > src.test.python.apache.aurora.admin.admin_util                                 
.....   SUCCESS
> > src.test.python.apache.aurora.admin.host_maintenance                           
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.api                                   
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.disambiguator                         
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.instance_watcher                      
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.job_monitor                           
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.mux                                   
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.quota_check                           
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.restarter                             
.....   SUCCESS
> > src.test.python.apache.aurora.client.api.scheduler_client                      
.....   FAILURE
> > src.test.python.apache.aurora.client.binding_helper                            
.....   SUCCESS
> > src.test.python.apache.aurora.client.config                                    
.....   SUCCESS
> 
> David McLaughlin wrote:
>     Cannot reproduce this locally?

nm, I can. It's just intermittent. The joys.


- David


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27698/#review60236
-----------------------------------------------------------


On Nov. 6, 2014, 8:45 p.m., David McLaughlin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27698/
> -----------------------------------------------------------
> 
> (Updated Nov. 6, 2014, 8:45 p.m.)
> 
> 
> Review request for Aurora, Brian Wickman and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> The underlying Kazoo code here has an uninterruptable wait on the main thread, which
means if there are any problems establishing a connection then you need to hard kill the client
process. Moving the code to a separate thread.  
> 
> 
> Diffs
> -----
> 
>   src/main/python/apache/aurora/client/api/scheduler_client.py 3a851cc9f53380b4bf2e9df47080e7c476c3e38e

> 
> Diff: https://reviews.apache.org/r/27698/diff/
> 
> 
> Testing
> -------
> 
> ./pants src/test/python/apache/aurora/client:all
> src.test.python.apache.aurora.client.api.api                                    .....
  SUCCESS
> src.test.python.apache.aurora.client.api.disambiguator                          .....
  SUCCESS
> src.test.python.apache.aurora.client.api.instance_watcher                       .....
  SUCCESS
> src.test.python.apache.aurora.client.api.job_monitor                            .....
  SUCCESS
> src.test.python.apache.aurora.client.api.mux                                    .....
  SUCCESS
> src.test.python.apache.aurora.client.api.quota_check                            .....
  SUCCESS
> src.test.python.apache.aurora.client.api.restarter                              .....
  SUCCESS
> src.test.python.apache.aurora.client.api.scheduler_client                       .....
  SUCCESS
> src.test.python.apache.aurora.client.api.sla                                    .....
  SUCCESS
> src.test.python.apache.aurora.client.api.updater                                .....
  SUCCESS
> src.test.python.apache.aurora.client.api.updater_util                           .....
  SUCCESS
> src.test.python.apache.aurora.client.binding_helper                             .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.api                                    .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.bridge                                 .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.command_hooks                          .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.config                                 .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.cron                                   .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.help                                   .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.inspect                                .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.job                                    .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.logging                                .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.plugins                                .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.quota                                  .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.sla                                    .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.supdate                                .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.task                                   .....
  SUCCESS
> src.test.python.apache.aurora.client.cli.update                                 .....
  SUCCESS
> src.test.python.apache.aurora.client.commands.admin                             .....
  SUCCESS
> src.test.python.apache.aurora.client.commands.core                              .....
  SUCCESS
> src.test.python.apache.aurora.client.commands.hooks                             .....
  SUCCESS
> src.test.python.apache.aurora.client.commands.maintenance                       .....
  SUCCESS
> src.test.python.apache.aurora.client.commands.run                               .....
  SUCCESS
> src.test.python.apache.aurora.client.commands.ssh                               .....
  SUCCESS
> src.test.python.apache.aurora.client.config                                     .....
  SUCCESS
> src.test.python.apache.aurora.client.hooks.hooked_api                           .....
  SUCCESS
> src.test.python.apache.aurora.client.hooks.non_hooked_api                       .....
  SUCCESS
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message