pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [pulsar] f-ld commented on issue #4718: failed to get Partitioned metadata : Policies not found for mytenant/mynamespace namespace
Date Tue, 22 Oct 2019 18:01:17 GMT
f-ld commented on issue #4718: failed to get Partitioned metadata : Policies not found for
mytenant/mynamespace namespace
URL: https://github.com/apache/pulsar/issues/4718#issuecomment-545082655
 
 
   Getting the issue again with all kind of topics :
   - non partitioned global topics
   - partitioned global topics
   - local (i.e. only one cluster) non partitioned topics 
   - local (i.e. only one cluster) partitioned topics 
   
   And for the latter, I checked that all 3 zk instances have the same content for `/namespace/<mytenant>/<mynamespace>`
   
   When I do a `pulsar-admin topics list tenant/namespace` I get a response `HTTP 500 Internal
Server Error`.
   
   Checking the logs I can see on one broker (the one handling the request):
   ```
   17:25:53.474 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.1.122
- - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/persistent/tenant/namespace HTTP/1.1" 200
675 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.516 [pulsar-web-30-1] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Namespace bundle is not owned by any broker tenant/namespace/0x15555555_0x2aaaaaaa
   17:25:53.516 [pulsar-web-30-4] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Namespace bundle is not owned by any broker tenant/namespace/0x55555554_0x6aaaaaa9
   17:25:53.516 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x15555555_0x2aaaaaaa
HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.516 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x55555554_0x6aaaaaa9
HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.516 [pulsar-web-30-7] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x95555553_0xaaaaaaa8
HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
   17:25:53.518 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xd5555552_0xeaaaaaa7
HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.519 [pulsar-web-30-6] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Namespace bundle is not owned by any broker tenant/namespace/0x3fffffff_0x55555554
   17:25:53.519 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x3fffffff_0x55555554
HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.519 [pulsar-web-30-4] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Namespace bundle is not owned by any broker tenant/namespace/0xbffffffd_0xd5555552
   17:25:53.520 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x00000000_0x15555555
HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
   17:25:53.520 [pulsar-web-30-8] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Namespace bundle is not owned by any broker tenant/namespace/0x2aaaaaaa_0x3fffffff
   17:25:53.520 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xbffffffd_0xd5555552
HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 3
   17:25:53.520 [pulsar-web-30-8] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x2aaaaaaa_0x3fffffff
HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.521 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x6aaaaaa9_0x7ffffffe
HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
   17:25:53.521 [pulsar-web-30-7] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Namespace bundle is not owned by any broker tenant/namespace/0x7ffffffe_0x95555553
   17:25:53.522 [pulsar-web-30-7] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x7ffffffe_0x95555553
HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 4
   17:25:53.522 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xeaaaaaa7_0xffffffff
HTTP/1.1" 200 57 "-" "Pulsar-Java-v2.4.1" 1
   17:25:53.524 [pulsar-web-30-6] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Namespace bundle is not owned by any broker tenant/namespace/0xaaaaaaa8_0xbffffffd
   17:25:53.524 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xaaaaaaa8_0xbffffffd
HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.529 [pulsar-web-30-5] ERROR org.apache.pulsar.broker.admin.v2.NonPersistentTopics
- [null] Failed to get list of topics under namespace tenant/namespace
   java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException$NotFoundException:
Tenant does not exist
   	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_212]
   	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_212]
   	at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getList(NonPersistentTopics.java:184)
~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
   	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
   	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
   	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
   	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
   	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
   	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
   	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
   	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
[org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53)
[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
   	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
[org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
[org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
[org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
[org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
[org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
[org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.Server.handle(Server.java:503) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
[org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
[org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
[org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
[org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
[org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
[org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
   	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
   Caused by: org.apache.pulsar.client.admin.PulsarAdminException$NotFoundException: Tenant
does not exist
   	at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:214)
~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.client.admin.internal.TopicsImpl$5.failed(TopicsImpl.java:159) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
   	at org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017)
~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178)
~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312)
~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243)
~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234)
~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
   	at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	... 2 more
   Caused by: javax.ws.rs.NotFoundException: HTTP 404 Not Found
   	at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1086)
~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:99) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017)
~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178)
~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312)
~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
   	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
   	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243)
~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
   	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234)
~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
   	at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
   	... 2 more
   17:25:53.532 [pulsar-web-30-5] INFO  org.eclipse.jetty.server.RequestLog - 10.11.1.122
- - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace HTTP/1.1"
500 5406 "-" "Pulsar-Java-v2.4.1" 21
   ```
   
   On other 2 brokers I get:
   ```
   17:25:53.522 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x95555553_0xaaaaaaa8?authoritative=true
HTTP/1.1" 404 34 "-" "Pulsar-Java-v2.4.1" 3
   17:25:53.525 [pulsar-web-30-5] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x6aaaaaa9_0x7ffffffe?authoritative=true
HTTP/1.1" 404 34 "-" "Pulsar-Java-v2.4.1" 2
   ```
   and
   ```
   17:25:53.521 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xd5555552_0xeaaaaaa7?authoritative=true
HTTP/1.1" 200 113 "-" "Pulsar-Java-v2.4.1" 2
   17:25:53.524 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 -
- [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x00000000_0x15555555?authoritative=true
HTTP/1.1" 200 170 "-" "Pulsar-Java-v2.4.1" 2
   ```
   
   On zookeeper, on all 3 nodes, if I check the content of `/namespace/tenant/namespace` I
get:
   ```
   [zk: localhost:2181(CONNECTED) 0] ls /namespace/tenant/namespace
   [0xd5555552_0xeaaaaaa7, 0x6aaaaaa9_0x7ffffffe, 0xeaaaaaa7_0xffffffff, 0x95555553_0xaaaaaaa8,
0x00000000_0x15555555]
   [zk: localhost:2181(CONNECTED) 1] get /namespace/tenant/namespace/0xd5555552_0xeaaaaaa7
   {"nativeUrl":"pulsar://10.11.0.111:6650","httpUrl":"http://10.11.0.111:8080","disabled":false}
   [zk: localhost:2181(CONNECTED) 2] get /namespace/tenant/namespace/0x6aaaaaa9_0x7ffffffe
   {"nativeUrl":"pulsar://10.11.3.88:6650","httpUrl":"http://10.11.3.88:8080","disabled":false}
   [zk: localhost:2181(CONNECTED) 3] get /namespace/tenant/namespace/0xeaaaaaa7_0xffffffff
   {"nativeUrl":"pulsar://10.11.0.50:6650","httpUrl":"http://10.11.0.50:8080","disabled":false}
   [zk: localhost:2181(CONNECTED) 4] get /namespace/tenant/namespace/0x95555553_0xaaaaaaa8
   {"nativeUrl":"pulsar://10.11.3.88:6650","httpUrl":"http://10.11.3.88:8080","disabled":false}
   [zk: localhost:2181(CONNECTED) 5] get /namespace/tenant/namespace/0x00000000_0x15555555
   {"nativeUrl":"pulsar://10.11.0.111:6650","httpUrl":"http://10.11.0.111:8080","disabled":false}
   [zk: localhost:2181(CONNECTED) 5] get /admin/local-policies/tenant/namespace
   {"bundles":{"boundaries":["0x00000000","0x15555555","0x2aaaaaaa","0x3fffffff","0x55555554","0x6aaaaaa9","0x7ffffffe","0x95555553","0xaaaaaaa8","0xbffffffd","0xd5555552","0xeaaaaaa7","0xffffffff"],"numBundles":12}}
   ```
   So I indeed have some bundles that have "disappeared" from brokers. But I got no broker
crash or zookeeper issue whatsoever. 
   
   But I do not see any error in zookeeper logs apart maybe things like:
   ```
   14:21:52.112 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor
- Got user-level KeeperException when processing sessionid:0x200d5ed81570008 type:delete cxid:0x3032c
zxid:0x100008631 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode
= Directory not empty for /ledgers/00/0000
   14:21:52.415 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor
- Got user-level KeeperException when processing sessionid:0x200d5ed81570007 type:delete cxid:0x244de
zxid:0x100008634 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode
= Directory not empty for /ledgers/00/0000
   14:22:52.112 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor
- Got user-level KeeperException when processing sessionid:0x200d5ed81570008 type:delete cxid:0x3034d
zxid:0x100008636 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode
= Directory not empty for /ledgers/00/0000
   ```
   but those are info logs.
   
   Still a mystery for me ... 
   But learning everytime, so now going to rebuild the cluster and check a few more things
before using it to ensure it is all ok

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message