ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suraj Nayak M <snay...@gmail.com>
Subject Re: Ambari UI not recovering after Machine power failure
Date Fri, 11 Jul 2014 07:51:35 GMT
This worked! :)

Thanks a lot for your help Yusaku, Sid and Jeff!

Just to understand why this happened I have few questions, so we can 
address the issue if it is commonly occurring on machine failure thus 
reducing manual intervention. /(I will try to find answers for the below 
questions myself in the source, would be great if I can get help here). /

  * Why did Postgresql corrupted on machine crash and did not recover on
    restart ?
  * If we had configured MySQL instead of Postgresql, will the problem
    re-occur on similar situation ? (I know this may take a while to
    test and infer conclusion)
  * What is stored in key_value_store and which component uses it ?
  * Can we have this as a quick fix in UI when the Web UI pops up the
    error ? Thus allowing admin to perform step by step guidance to
    recover from the failure(manual or self healing process, need
    discussion on this).
  * What did the POST call to http://localhost:8080/api/v1/persist API do ?

--
Suraj Nayak

On Friday 11 July 2014 05:17 AM, Yusaku Sako wrote:
> Please try the following (worked for me on my test cluster).
>
> 1. Delete all rows from key_value_store
> 1. ambari-server restart
> 2. Wait until the server comes up
> 3. curl -i -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d
> '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}'
> http://localhost:8080/api/v1/persist
> 4. Log back in to ambari-server
>
> Hope this helps.
> Yusaku
>
> On Thu, Jul 10, 2014 at 11:31 AM, Suraj Nayak M <snayakm@gmail.com> wrote:
>> Thanks Sid,
>>
>> Found the username, password and database name used by ambari :)
>>
>> Yusaku,
>>      Is it safe to truncate the table key_value_store table ? The old cluster
>> state will be restored after following steps mentioned below (confirmation
>> needed!):
>>
>> Truncate table key_value_store from postgresql.
>> Call the API as mentioned below in the email chain.
>>
>> --
>> Suraj Nayak
>>
>>
>> On Thursday 10 July 2014 10:29 PM, Siddharth Wagle wrote:
>>
>> Hi Suraj,
>>
>> You should find all db info in /etc/ambari-server/conf/ambari.properties.
>> (grep for jdbc).
>>
>> I believe you can just truncate the table key_value_store and then try
>> making the API call.
>> Since UI utilizes this table Yusaku would be the right person to +1 this
>> advice.
>>
>> -Sid
>>
>>
>> On Thu, Jul 10, 2014 at 1:34 AM, Suraj Nayak M <snayakm@gmail.com> wrote:
>>> Yes Sid, Even I suspected the Postgres corruption.
>>>
>>> I tried the API provided by Yasaku. I got 500 Internal error message
>>> mentioned below :
>>>
>>> HTTP/1.1 500  Internal Exception: org.postgresql.util.PSQLException:
>>> ERROR: missing chunk number 0 for toast value 50294 in pg_toast_16548 Error
>>> Code: 0 Call: SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)
>>> bind => [1 parameter bound] Query: ReadObjectQuery(name="readObject"
>>> referenceClass=KeyValueEntity sql="SELECT "key", "value" FROM
>>> key_value_store WHERE ("key" = ?)")
>>> Set-Cookie: AMBARISESSIONID=39gi1jtp0amp1orp572oipb8k;Path=/
>>> Content-Type: text/plain;charset=ISO-8859-1
>>> Content-Length: 467
>>> Server: Jetty(7.6.7.v20120910)
>>>
>>> {
>>>    "status": 500,
>>>    "message": "\nInternal Exception: org.postgresql.util.PSQLException:
>>> ERROR: missing chunk number 0 for toast value 50294 in pg_toast_16548\nError
>>> Code: 0\nCall: SELECT \"key\", \"value\" FROM key_value_store WHERE (\"key\"
>>> \u003d ?)\n\tbind \u003d\u003e [1 parameter bound]\nQuery:
>>> ReadObjectQuery(name\u003d\"readObject\" referenceClass\u003dKeyValueEntity
>>> sql\u003d\"SELECT \"key\", \"value\" FROM key_value_store WHERE (\"key\"
>>> \u003d ?)\")"
>>>
>>> I can notice a specific key, which is highlighter in red. Does deleting
>>> this solve the problem ?
>>>
>>> I am not able to take a dump of existing databases as I don't know the
>>> username and password of postgre.
>>>
>>> What is the default possource files at /var/lib/ambari-server/resources...
>>> Server PID at: /var/run/ambari-server/ambari-server.pid
>>> Server out at: /var/log/ambari-server/ambari-server.out
>>> Server log at: /var/log/ambari-server/ambari-server.log
>>> Ambari Server 'start' completed successfully.
>>> [root@vm1 ~]#  curl -i -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d
>>> curl: option -d: requires parameter
>>> curl: try 'curl --help' or 'curl --manual' for more information
>>> [root@vm1 ~]# '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}'
>>> -bash: {"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}:
command not found
>>> [root@vm1 ~]# '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJotgres
username, password and database name that
>>> Ambari uses ?
>>> I suspect the user is posgres, which is the default user name. If I hard
>>> reset the postgres which is mentioned in the this link, will ambari face
>>> problem ? Which is the configuration file Ambari uses to store postgres
>>> username and password?
>>>
>>> --
>>> Suraj Nayak
>>>
>>>
>>> On Thursday 10 July 2014 09:31 AM, Yusaku Sako wrote:
>>>
>>> Suraj,
>>>
>>> Here's the API call that Sid mentioned for restoring key_value_store data:
>>>
>>> curl -i -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{
>>> "CLUSTER_CURRENT_STATUS": "{\"clusterState\":\"CLUSTER_STARTED_5\"}"
>>> }' http://localhost:8080/api/v1/persist
>>>
>>> Hopefully this helps.
>>>
>>> Yusaku
>>>
>>> On Wed, Jul 9, 2014 at 8:49 PM, Siddharth Wagle <swagle@hortonworks.com>
>>> wrote:
>>>
>>> Hi Suraj,
>>>
>>> Found a similar thread on postgres forum,
>>>
>>> http://postgresql.1045698.n5.nabble.com/select-table-indicate-missing-chunk-number-0-for-toast-value-96635-in-pg-toast-2619-td5682176.html
>>>
>>> The cleanup commands at the end of the topic seem worth a try.
>>>
>>> Note: If key_value_store has corrupt data it can be deleted without
>>> affection state of the system, there is a API call to do it, Yusaku can
>>> probably help you get the API call for it.
>>>
>>> 1. Backup the database as it is.
>>> 2. I would try the API call and see if everything recovers.
>>> 3. Run the analyze command and re index on pg_statistic.
>>>
>>> -Sid
>>>
>>>
>>> On Wed, Jul 9, 2014 at 8:32 PM, Yusaku Sako <yusaku@hortonworks.com>
>>> wrote:
>>>
>>> "Caused by: org.postgresql.util.PSQLException: ERROR: missing chunk
>>> number 0 for toast value 50294 in pg_toast_16548"
>>>
>>> I think this suggests that your Postgres DB is corrupt.
>>>
>>> Yusaku
>>>
>>>
>>> On Wed, Jul 9, 2014 at 10:58 AM, Suraj Nayak M <snayakm@gmail.com> wrote:
>>>
>>> Jeff,
>>>
>>> I tried restarting both ambari-server and postgresql. Still the error
>>> exist.
>>> I have attached a part of the log file(log after ambari-server restart)
>>> so
>>> it might help to pin point the issue.
>>>
>>> I could see following line before the Exception :
>>>
>>> 11:03:04,784 ERROR [qtp2062231465-43] HeartBeatHandler:127 -
>>> CurrentResponseId unknown for vm1.somedomain.com - send register command
>>> 11:03:09,016 ERROR [qtp2062231465-43] HeartBeatHandler:127 -
>>> CurrentResponseId unknown for vm3.somedomain.com - send register command
>>> 11:03:20,792 ERROR [qtp2062231465-44] HeartBeatHandler:127 -
>>> CurrentResponseId unknown for vm2.somedomain.com - send register command
>>>
>>> Below are the output of the commands(Restart and status) :
>>>
>>> -------------------------------
>>>
>>> Status :
>>>
>>> [root@vm1 ~]# ambari-server status
>>> Using python  /usr/bin/python2.6
>>> Ambari-server status
>>> Ambari Server running
>>> Found Ambari Server PID: '4895 at:
>>> /var/run/ambari-server/ambari-server.pid
>>> [root@vm1 ~]# ps -ef | grep postgres
>>> postgres  1947     1  0 04:16 ?        00:00:01 /usr/bin/postmaster -p
>>> 5432
>>> -D /var/lib/pgsql/data
>>> postgres  1949  1947  0 04:16 ?        00:00:00 postgres: logger process
>>> postgres  1951  1947  0 04:16 ?        00:00:03 postgres: writer process
>>> postgres  1952  1947  0 04:16 ?        00:00:02 postgres: wal writer
>>> process
>>> postgres  1953  1947  0 04:16 ?        00:00:00 postgres: autovacuum
>>> launcher process
>>> postgres  1954  1947  0 04:16 ?        00:00:01 postgres: stats
>>> collector
>>> process
>>> postgres  4915  1947  1 04:43 ?        00:05:05 postgres: ambari ambari
>>> 127.0.0.1(57245) idle
>>> postgres  5018  1947  0 04:44 ?        00:00:00 postgres: ambari ambari
>>> 127.0.0.1(57267) idle
>>> postgres  5173  1947  0 04:45 ?        00:00:00 postgres: ambari ambari
>>> 127.0.0.1(57282) idle
>>> postgres  5174  1947  0 04:45 ?        00:00:00 postgres: ambari ambari
>>> 127.0.0.1(57283) idle
>>> postgres  5175  1947  0 04:45 ?        00:00:00 postgres: ambari ambari
>>> 127.0.0.1(57284) idle
>>> root     30657 30493  0 10:59 pts/0    00:00:00 grep postgres
>>> [root@vm1 ~]# service postgresql status
>>> postmaster (pid  1947) is running...
>>>
>>> Restart :
>>> [root@vm1 ~]# service postgresql restart
>>> Stopping postgresql service:                               [  OK  ]
>>> Starting postgresql service:                               [  OK  ]
>>> [root@vm1 ~]# ambari-server restart
>>> Using python  /usr/bin/python2.6
>>> Restarting ambari-server
>>> Using python  /usr/bin/python2.6
>>> Stopping ambari-server
>>> Ambari Server stopped
>>> Using python  /usr/bin/python2.6
>>> Starting ambari-server
>>> Ambari Server running with 'root' privileges.
>>> Organizing resource files at /var/lib/ambari-server/resources...
>>> Server PID at: /var/run/ambari-server/ambari-server.pid
>>> Server out at: /var/log/ambari-server/ambari-server.out
>>> Server log at: /var/log/ambari-server/ambari-server.log
>>> Ambari Server 'start' completed successfully.
>>>
>>> Status :
>>> [root@vm1 ~]# service postgresql status
>>> postmaster (pid  31003) is running...
>>>
>>> [root@vm1 ~]# ps -ef | grep postgres
>>> postgres 31003     1  0 11:02 ?        00:00:00 /usr/bin/postmaster -p
>>> 5432
>>> -D /var/lib/pgsql/data
>>> postgres 31005 31003  0 11:02 ?        00:00:00 postgres: logger process
>>> postgres 31007 31003  0 11:02 ?        00:00:00 postgres: writer process
>>> postgres 31008 31003  0 11:02 ?        00:00:00 postgres: wal writer
>>> process
>>> postgres 31009 31003  0 11:02 ?        00:00:00 postgres: autovacuum
>>> launcher process
>>> postgres 31010 31003  0 11:02 ?        00:00:00 postgres: stats
>>> collector
>>> process
>>> postgres 31079 31003  1 11:02 ?        00:00:03 postgres: ambari ambari
>>> 127.0.0.1(59377) idle
>>> postgres 31350 31003  0 11:05 ?        00:00:00 postgres: ambari ambari
>>> 127.0.0.1(59417) idle
>>> postgres 31351 31003  0 11:05 ?        00:00:00 postgres: ambari ambari
>>> 127.0.0.1(59418) idle
>>> postgres 31352 31003  0 11:05 ?        00:00:00 postgres: ambari ambari
>>> 127.0.0.1(59419) idle
>>> root     31501 30493  0 11:06 pts/0    00:00:00 grep postgres
>>>
>>> [root@vm1 ~]# ambari-server status
>>> Using python  /usr/bin/python2.6
>>> Ambari-server status
>>> Ambari Server running
>>> Found Ambari Server PID: '31061 at:
>>> /var/run/ambari-server/ambari-server.pid
>>>
>>> ----------------------------
>>>
>>>
>>> --
>>> Suraj Nayak
>>>
>>>
>>> On Wednesday 09 July 2014 07:15 PM, Jeff Sposetti wrote:
>>>
>>> Can you confirm ambari server is running? And stop/start as necessary?
>>>
>>> ambari-server status
>>> Also, confirm postgres is running after you stop/start ambari-server?
>>>
>>> ps -ef | grep postgres
>>>
>>> service postgresql status
>>>
>>>
>>>
>>> On Wed, Jul 9, 2014 at 8:49 AM, Suraj Nayak <snayakm@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I have a 3 node cluster with HA enabled via Ambari. Accidentally the
>>> power
>>> supply to all the nodes was disrupted. After the power was restored,
>>> the 3
>>> servers were turned on. Now if I try to login to Ambari Web UI, an
>>> error
>>> pops up  Error in retrieving web client state from ambari server.(After
>>> entering username and password)
>>>
>>> It doesn't show the cluster dashboard which was showing before. Instead
>>> it
>>> shows "Cluster Install Wizard" to setup a new cluster. How can I
>>> recover
>>> from the error? I do not want to restart the installation process
>>>
>>> Below is the error I found in /var/log/ambari-server/ambari-server.log
>>> log
>>> file.
>>>
>>> I found an email archive at
>>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/ambari-user/201402.mbox/%3CCF1BC3ED.2E43F%25are9004@nyp.org%3E.
>>> But I need suggestions on recovering but not starting the installation
>>> from
>>> scratch.
>>>
>>> Error :
>>>
>>> Local Exception Stack:
>>> Exception [EclipseLink-4002] (Eclipse Persistence Services -
>>> 2.4.0.v20120608-r11652):
>>> org.eclipse.persistence.exceptions.DatabaseException
>>> Internal Exception: org.postgresql.util.PSQLException: ERROR: missing
>>> chunk number 0 for toast value 50294 in pg_toast_16548
>>> Error Code: 0
>>> Call: SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)
>>>      bind => [1 parameter bound]
>>> Query: ReadObjectQuery(name="readObject" referenceClass=KeyValueEntity
>>> sql="SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)")
>>>      at
>>>
>>>
>>> org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:646)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:537)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1800)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:668)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2744)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2697)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:452)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1149)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1108)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:420)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1196)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2875)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1602)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1584)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1535)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:838)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.jpa.EntityManagerImpl.findInternal(EntityManagerImpl.java:778)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:671)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:543)
>>>      at
>>>
>>>
>>> org.apache.ambari.server.orm.dao.KeyValueDAO.findByKey(KeyValueDAO.java:42)
>>>      at
>>>
>>>
>>> org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53)
>>>      at
>>>
>>>
>>> org.apache.ambari.server.api.services.PersistKeyValueImpl.getValue(PersistKeyValueImpl.java:50)
>>>      at
>>>
>>>
>>> org.apache.ambari.server.api.services.PersistKeyValueService.getKey(PersistKeyValueService.java:83)
>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>      at
>>>
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>      at
>>>
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>      at java.lang.reflect.Method.invoke(Method.java:606)
>>>      at
>>>
>>>
>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>>>      at
>>>
>>>
>>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>>>      at
>>>
>>>
>>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>>>      at
>>>
>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>>>      at
>>>
>>>
>>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>      at
>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
>>>      at
>>>
>>>
>>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
>>>      at
>>>
>>>
>>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.apache.ambari.server.security.authorization.internal.InternalTokenAuthenticationFilter.doFilter(InternalTokenAuthenticationFilter.java:53)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
>>>      at
>>>
>>>
>>> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
>>>      at
>>>
>>>
>>> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
>>>      at
>>>
>>>
>>> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
>>>      at
>>>
>>>
>>> org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:48)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
>>>      at
>>>
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)
>>>      at
>>>
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>>      at
>>>
>>> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>>      at org.eclipse.jetty.server.Server.handle(Server.java:363)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)
>>>      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
>>>      at
>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>      at
>>>
>>>
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>      at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.postgresql.util.PSQLException: ERROR: missing chunk
>>> number
>>> 0 for toast value 50294 in pg_toast_16548
>>>      at
>>>
>>>
>>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
>>>      at
>>>
>>>
>>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
>>>      at
>>>
>>>
>>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
>>>      at
>>>
>>>
>>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
>>>      at
>>>
>>>
>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
>>>      at
>>>
>>>
>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:938)
>>>      at
>>>
>>>
>>> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:609)
>>>      ... 98 more
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the
>>> reader of
>>> this message is not the intended recipient, you are hereby notified that
>>> any
>>> printing, copying, dissemination, distribution, disclosure or forwarding
>>> of
>>> this communication is strictly prohibited. If you have received this
>>> communication in error, please contact the sender immediately and delete
>>> it
>>> from your system. Thank You.
>>>
>>>
>>> --
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of
>>> this message is not the intended recipient, you are hereby notified that
>>> any
>>> printing, copying, dissemination, distribution, disclosure or forwarding
>>> of
>>> this communication is strictly prohibited. If you have received this
>>> communication in error, please contact the sender immediately and delete
>>> it
>>> from your system. Thank You.
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader of
>> this message is not the intended recipient, you are hereby notified that any
>> printing, copying, dissemination, distribution, disclosure or forwarding of
>> this communication is strictly prohibited. If you have received this
>> communication in error, please contact the sender immediately and delete it
>> from your system. Thank You.
>>
>>


Mime
View raw message