atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayub Khan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ATLAS-1068) Under stress conditions, all Atlas threads run in to BLOCKED state and returns 500 server error for a valid search query.
Date Wed, 03 Aug 2016 10:46:20 GMT

     [ https://issues.apache.org/jira/browse/ATLAS-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ayub Khan updated ATLAS-1068:
-----------------------------
    Assignee:     (was: Madhan Neethiraj)

> Under stress conditions, all Atlas threads run in to BLOCKED state and returns 500 server
error for a valid search query.
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: ATLAS-1068
>                 URL: https://issues.apache.org/jira/browse/ATLAS-1068
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: 0.7-incubating
>            Reporter: Ayub Khan
>            Priority: Critical
>         Attachments: jstack_dump.txt
>
>
> During some random experiments to check Atlas behavior under stress, found that most
of the Atlas threads are in blocked state and Atlas server was throwing below error in the
logs.
> Steps to repro:
> * Stop Atlas server
> * Start a batch command to create some 1000 hive ctas queries
> * Start Atlas server after some message built up in the kafka message queue.
> * Parallely run fulltext queries( using threads ) from command line (query=*)
> * Once the query count reaches to 100. Now in the logs, below exception can be seen.
> * Also Jstack trace of all the threads shows that all threads are in blocked state.
> *Impact: Response from Atlas is very slow, most of the times request timesout or 500
server error is observed.*
> Atlas application log snapshot:
> {noformat}
> 2016-07-30 16:09:52,894 WARN  - [qtp511473681-13:] ~ Could not send response error 500:
java.lang.IllegalStateException: Committed (HttpChannel:481)
> 2016-07-30 16:09:52,896 INFO  - [qtp511473681-13 - 8ccee9fb-8efb-43b2-8eba-2920cad69e04:]
~ Audit: admin/172.22.114.250 performed request GET http://atlas-r6-1088-simple-2.openstacklocal:21000/api/atlas/discovery/search/fulltext?limit=25&query=*&FSBX=HQKDJ
(172.22.114.26) at time 2016-07-30T16:09Z (AuditFilter:91)
> 2016-07-30 16:09:52,896 INFO  - [qtp511473681-13 - 8ccee9fb-8efb-43b2-8eba-2920cad69e04:]
~ Audit: admin/172.22.114.250-172.22.114.250 performed request GET http://atlas-r6-1088-simple-2.openstacklocal:21000/api/atlas/discovery/search/fulltext?limit=25&query=*&FSBX=HQKDJ
(172.22.114.26) at time 2016-07-30T16:09Z (AUDIT:104)
> 2016-07-30 16:09:52,897 INFO  - [qtp511473681-13 - 8ccee9fb-8efb-43b2-8eba-2920cad69e04:]
~ Converted query string with 1 replacements: [v."entityText":(*)] => [pad_t:(*)] (IndexSerializer:648)
> 2016-07-30 16:09:53,070 ERROR - [qtp511473681-15 - 9ea08823-81dd-4fcf-8324-32c760e9950c:]
~ The response of the WebApplicationException cannot be utilized as the response is already
committed. Re-throwing to the HTTP container (WebApplicationImpl:1514)
> javax.ws.rs.WebApplicationException: java.lang.Exception: Error writing JSON object.
> 	at com.sun.jersey.json.impl.provider.entity.JSONObjectProvider.writeTo(JSONObjectProvider.java:113)
> 	at com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$App.writeTo(JSONObjectProvider.java:65)
> 	at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> 	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> 	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> 	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> 	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> 	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> 	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> 	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> 	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> 	at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:71)
> 	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> 	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> 	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> 	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> 	at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> 	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> 	at org.springframework..web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
> 	at org.apache.atlas.web.filters.AtlasAuthorizationFilter.doFilter(AtlasAuthorizationFilter.java:154)
> 	at org.springframework..web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework..web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
> 	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:232)
> 	at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
> 	at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:187)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:301)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
> 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> 	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
> 	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
> 	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
> 	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> 	at org.eclipse.jetty.server.Server.handle(Server.java:499)
> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Error writing JSON object.
> 	... 75 more
> Caused by: org.codehaus.jettison.json.JSONException
> 	at org.codehaus.jettison.json.JSONObject.write(JSONObject.java:1385)
> 	at org.codehaus.jettison.json.JSONArray.write(JSONArray.java:897)
> 	at org.codehaus.jettison.json.JSONObject.write(JSONObject.java:1372)
> 	at com.sun.jersey.json.impl.provider.entity.JSONObjectProvider.writeTo(JSONObjectProvider.java:110)
> 	... 74 more
> Caused by: org.eclipse.jetty.io.EofException
> 	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
> 	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
> 	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
> 	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129)
> 	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:690)
> 	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
> 	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
> 	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:480)
> 	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:768)
> 	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
> 	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
> 	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140)
> 	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:355)
> 	at org.springframework.security.web.context.SaveContextOnUpdateOrErrorResponseWrapper$SaveContextServletOutputStream.write(SaveContextOnUpdateOrErrorResponseWrapper.java:435)
> 	at com.sun.jersey.spi.container.servlet.WebComponent$Writer.write(WebComponent.java:300)
> 	at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.write(ContainerResponse.java:135)
> 	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> 	at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> 	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> 	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
> 	at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
> 	at java.io.Writer.write(Writer.java:157)
> 	at org.codehaus.jettison.json.JSONObject.write(JSONObject.java:1374)
> 	... 77 more
> Caused by: java.io.IOException: Connection reset by peer
> 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> 	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> 	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> 	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
> 	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
> 	... 99 more
> 2016-07-30 16:09:53,122 WARN  - [qtp511473681-15 - 9ea08823-81dd-4fcf-8324-32c760e9950c:]
~ Committed before 500 null (Response:565)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message