ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yusaku Sako <yus...@hortonworks.com>
Subject Re: Ambari UI not recovering after Machine power failure
Date Fri, 11 Jul 2014 14:48:17 GMT
Suraj,

See my comments inline:


On Fri, Jul 11, 2014 at 12:51 AM, Suraj Nayak M <snayakm@gmail.com> wrote:
> This worked! :)
>
> Thanks a lot for your help Yusaku, Sid and Jeff!
>
> Just to understand why this happened I have few questions, so we can address
> the issue if it is commonly occurring on machine failure thus reducing
> manual intervention. (I will try to find answers for the below questions
> myself in the source, would be great if I can get help here).
>
> Why did Postgresql corrupted on machine crash and did not recover on restart
> ?

You can read up on Postgres (or other databases) getting corrupt on
machine shutdown, etc.  For example, see
http://rhaas.blogspot.com/2012/03/why-is-my-database-corrupted.html

> If we had configured MySQL instead of Postgresql, will the problem re-occur
> on similar situation ? (I know this may take a while to test and infer
> conclusion)

I'm no expert at making comments on MySQL vs Postgres, but for systems
utilizing *any* RDBMS, it is recommended that backups are taken at an
interval or (near) realtime through replication.

> What is stored in key_value_store and which component uses it ?

key_value_store is used only by Ambari Web UI to persist UI state.
Ambari Server itself only exposes this feature but does not use it.
Of course, any custom UI/client built upon the Ambari API could use it
as well to persist client state.

> Can we have this as a quick fix in UI when the Web UI pops up the error ?
> Thus allowing admin to perform step by step guidance to recover from the
> failure(manual or self healing process, need discussion on this).
> What did the POST call to http://localhost:8080/api/v1/persist API do ?

That is an interesting idea.
The "magic" call to clear the key_store_value (/api/v1/persist)
actually solves a number of issues, so it might be a good idea to
expose this somehow.

Yusaku


>
> --
> Suraj Nayak
>
>
> On Friday 11 July 2014 05:17 AM, Yusaku Sako wrote:
>
> Please try the following (worked for me on my test cluster).
>
> 1. Delete all rows from key_value_store
> 1. ambari-server restart
> 2. Wait until the server comes up
> 3. curl -i -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d
> '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}'
> http://localhost:8080/api/v1/persist
> 4. Log back in to ambari-server
>
> Hope this helps.
> Yusaku
>
> On Thu, Jul 10, 2014 at 11:31 AM, Suraj Nayak M <snayakm@gmail.com> wrote:
>
> Thanks Sid,
>
> Found the username, password and database name used by ambari :)
>
> Yusaku,
>     Is it safe to truncate the table key_value_store table ? The old cluster
> state will be restored after following steps mentioned below (confirmation
> needed!):
>
> Truncate table key_value_store from postgresql.
> Call the API as mentioned below in the email chain.
>
> --
> Suraj Nayak
>
>
> On Thursday 10 July 2014 10:29 PM, Siddharth Wagle wrote:
>
> Hi Suraj,
>
> You should find all db info in /etc/ambari-server/conf/ambari.properties.
> (grep for jdbc).
>
> I believe you can just truncate the table key_value_store and then try
> making the API call.
> Since UI utilizes this table Yusaku would be the right person to +1 this
> advice.
>
> -Sid
>
>
> On Thu, Jul 10, 2014 at 1:34 AM, Suraj Nayak M <snayakm@gmail.com> wrote:
>
> Yes Sid, Even I suspected the Postgres corruption.
>
> I tried the API provided by Yasaku. I got 500 Internal error message
> mentioned below :
>
> HTTP/1.1 500  Internal Exception: org.postgresql.util.PSQLException:
> ERROR: missing chunk number 0 for toast value 50294 in pg_toast_16548 Error
> Code: 0 Call: SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)
> bind => [1 parameter bound] Query: ReadObjectQuery(name="readObject"
> referenceClass=KeyValueEntity sql="SELECT "key", "value" FROM
> key_value_store WHERE ("key" = ?)")
> Set-Cookie: AMBARISESSIONID=39gi1jtp0amp1orp572oipb8k;Path=/
> Content-Type: text/plain;charset=ISO-8859-1
> Content-Length: 467
> Server: Jetty(7.6.7.v20120910)
>
> {
>   "status": 500,
>   "message": "\nInternal Exception: org.postgresql.util.PSQLException:
> ERROR: missing chunk number 0 for toast value 50294 in pg_toast_16548\nError
> Code: 0\nCall: SELECT \"key\", \"value\" FROM key_value_store WHERE (\"key\"
> \u003d ?)\n\tbind \u003d\u003e [1 parameter bound]\nQuery:
> ReadObjectQuery(name\u003d\"readObject\" referenceClass\
> u003dKeyValueEntity
> sql\u003d\"SELECT \"key\", \"value\" FROM key_value_store WHERE (\"key\"
> \u003d ?)\")"
>
> I can notice a specific key, which is highlighter in red. Does deleting
> this solve the problem ?
>
> I am not able to take a dump of existing databases as I don't know the
> username and password of postgre.
>
> What is the default possource files at /var/lib/ambari-server/resources...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Ambari Server 'start' completed successfully.
> [root@vm1 ~]#  curl -i -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d
> curl: option -d: requires parameter
> curl: try 'curl --help' or 'curl --manual' for more information
> [root@vm1 ~]#
> '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}'
> -bash:
> {"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}:
> command not found
> [root@vm1 ~]#
> '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJotgres
> username, password and database name that
> Ambari uses ?
> I suspect the user is posgres, which is the default user name. If I hard
> reset the postgres which is mentioned in the this link, will ambari face
> problem ? Which is the configuration file Ambari uses to store postgres
> username and password?
>
> --
> Suraj Nayak
>
>
> On Thursday 10 July 2014 09:31 AM, Yusaku Sako wrote:
>
> Suraj,
>
> Here's the API call that Sid mentioned for restoring key_value_store data:
>
> curl -i -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{
> "CLUSTER_CURRENT_STATUS": "{\"clusterState\":\"CLUSTER_STARTED_5\"}"
> }' http://localhost:8080/api/v1/persist
>
> Hopefully this helps.
>
> Yusaku
>
> On Wed, Jul 9, 2014 at 8:49 PM, Siddharth Wagle <swagle@hortonworks.com>
> wrote:
>
> Hi Suraj,
>
> Found a similar thread on postgres forum,
>
> http://postgresql.1045698.n5.nabble.com/select-table-indicate-missing-chunk-number-0-for-toast-value-96635-in-pg-toast-2619-td5682176.html
>
> The cleanup commands at the end of the topic seem worth a try.
>
> Note: If key_value_store has corrupt data it can be deleted without
> affection state of the system, there is a API call to do it, Yusaku can
> probably help you get the API call for it.
>
> 1. Backup the database as it is.
> 2. I would try the API call and see if everything recovers.
> 3. Run the analyze command and re index on pg_statistic.
>
> -Sid
>
>
> On Wed, Jul 9, 2014 at 8:32 PM, Yusaku Sako <yusaku@hortonworks.com>
> wrote:
>
> "Caused by: org.postgresql.util.PSQLException: ERROR: missing chunk
> number 0 for toast value 50294 in pg_toast_16548"
>
> I think this suggests that your Postgres DB is corrupt.
>
> Yusaku
>
>
> On Wed, Jul 9, 2014 at 10:58 AM, Suraj Nayak M <snayakm@gmail.com> wrote:
>
> Jeff,
>
> I tried restarting both ambari-server and postgresql. Still the error
> exist.
> I have attached a part of the log file(log after ambari-server restart)
> so
> it might help to pin point the issue.
>
> I could see following line before the Exception :
>
> 11:03:04,784 ERROR [qtp2062231465-43] HeartBeatHandler:127 -
> CurrentResponseId unknown for vm1.somedomain.com - send register command
> 11:03:09,016 ERROR [qtp2062231465-43] HeartBeatHandler:127 -
> CurrentResponseId unknown for vm3.somedomain.com - send register command
> 11:03:20,792 ERROR [qtp2062231465-44] HeartBeatHandler:127 -
> CurrentResponseId unknown for vm2.somedomain.com - send register command
>
> Below are the output of the commands(Restart and status) :
>
> -------------------------------
>
> Status :
>
> [root@vm1 ~]# ambari-server status
> Using python  /usr/bin/python2.6
> Ambari-server status
> Ambari Server running
> Found Ambari Server PID: '4895 at:
> /var/run/ambari-server/ambari-server.pid
> [root@vm1 ~]# ps -ef | grep postgres
> postgres  1947     1  0 04:16 ?        00:00:01 /usr/bin/postmaster -p
> 5432
> -D /var/lib/pgsql/data
> postgres  1949  1947  0 04:16 ?        00:00:00 postgres: logger process
> postgres  1951  1947  0 04:16 ?        00:00:03 postgres: writer process
> postgres  1952  1947  0 04:16 ?        00:00:02 postgres: wal writer
> process
> postgres  1953  1947  0 04:16 ?        00:00:00 postgres: autovacuum
> launcher process
> postgres  1954  1947  0 04:16 ?        00:00:01 postgres: stats
> collector
> process
> postgres  4915  1947  1 04:43 ?        00:05:05 postgres: ambari ambari
> 127.0.0.1(57245) idle
> postgres  5018  1947  0 04:44 ?        00:00:00 postgres: ambari ambari
> 127.0.0.1(57267) idle
> postgres  5173  1947  0 04:45 ?        00:00:00 postgres: ambari ambari
> 127.0.0.1(57282) idle
> postgres  5174  1947  0 04:45 ?        00:00:00 postgres: ambari ambari
> 127.0.0.1(57283) idle
> postgres  5175  1947  0 04:45 ?        00:00:00 postgres: ambari ambari
> 127.0.0.1(57284) idle
> root     30657 30493  0 10:59 pts/0    00:00:00 grep postgres
> [root@vm1 ~]# service postgresql status
> postmaster (pid  1947) is running...
>
> Restart :
> [root@vm1 ~]# service postgresql restart
> Stopping postgresql service:                               [  OK  ]
> Starting postgresql service:                               [  OK  ]
> [root@vm1 ~]# ambari-server restart
> Using python  /usr/bin/python2.6
> Restarting ambari-server
> Using python  /usr/bin/python2.6
> Stopping ambari-server
> Ambari Server stopped
> Using python  /usr/bin/python2.6
> Starting ambari-server
> Ambari Server running with 'root' privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Ambari Server 'start' completed successfully.
>
> Status :
> [root@vm1 ~]# service postgresql status
> postmaster (pid  31003) is running...
>
> [root@vm1 ~]# ps -ef | grep postgres
> postgres 31003     1  0 11:02 ?        00:00:00 /usr/bin/postmaster -p
> 5432
> -D /var/lib/pgsql/data
> postgres 31005 31003  0 11:02 ?        00:00:00 postgres: logger process
> postgres 31007 31003  0 11:02 ?        00:00:00 postgres: writer process
> postgres 31008 31003  0 11:02 ?        00:00:00 postgres: wal writer
> process
> postgres 31009 31003  0 11:02 ?        00:00:00 postgres: autovacuum
> launcher process
> postgres 31010 31003  0 11:02 ?        00:00:00 postgres: stats
> collector
> process
> postgres 31079 31003  1 11:02 ?        00:00:03 postgres: ambari ambari
> 127.0.0.1(59377) idle
> postgres 31350 31003  0 11:05 ?        00:00:00 postgres: ambari ambari
> 127.0.0.1(59417) idle
> postgres 31351 31003  0 11:05 ?        00:00:00 postgres: ambari ambari
> 127.0.0.1(59418) idle
> postgres 31352 31003  0 11:05 ?        00:00:00 postgres: ambari ambari
> 127.0.0.1(59419) idle
> root     31501 30493  0 11:06 pts/0    00:00:00 grep postgres
>
> [root@vm1 ~]# ambari-server status
> Using python  /usr/bin/python2.6
> Ambari-server status
> Ambari Server running
> Found Ambari Server PID: '31061 at:
> /var/run/ambari-server/ambari-server.pid
>
> ----------------------------
>
>
> --
> Suraj Nayak
>
>
> On Wednesday 09 July 2014 07:15 PM, Jeff Sposetti wrote:
>
> Can you confirm ambari server is running? And stop/start as necessary?
>
> ambari-server status
> Also, confirm postgres is running after you stop/start ambari-server?
>
> ps -ef | grep postgres
>
> service postgresql status
>
>
>
> On Wed, Jul 9, 2014 at 8:49 AM, Suraj Nayak <snayakm@gmail.com> wrote:
>
> Hi,
>
> I have a 3 node cluster with HA enabled via Ambari. Accidentally the
> power
> supply to all the nodes was disrupted. After the power was restored,
> the 3
> servers were turned on. Now if I try to login to Ambari Web UI, an
> error
> pops up  Error in retrieving web client state from ambari server.(After
> entering username and password)
>
> It doesn't show the cluster dashboard which was showing before. Instead
> it
> shows "Cluster Install Wizard" to setup a new cluster. How can I
> recover
> from the error? I do not want to restart the installation process
>
> Below is the error I found in /var/log/ambari-server/ambari-server.log
> log
> file.
>
> I found an email archive at
>
>
> http://mail-archives.apache.org/mod_mbox/ambari-user/201402.mbox/%3CCF1BC3ED.2E43F%25are9004@nyp.org%3E.
> But I need suggestions on recovering but not starting the installation
> from
> scratch.
>
> Error :
>
> Local Exception Stack:
> Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.4.0.v20120608-r11652):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: org.postgresql.util.PSQLException: ERROR: missing
> chunk number 0 for toast value 50294 in pg_toast_16548
> Error Code: 0
> Call: SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)
>     bind => [1 parameter bound]
> Query: ReadObjectQuery(name="readObject" referenceClass=KeyValueEntity
> sql="SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)")
>     at
>
>
> org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333)
>     at
>
>
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:646)
>     at
>
>
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:537)
>     at
>
>
> org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1800)
>     at
>
>
> org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)
>     at
>
>
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
>     at
>
>
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
>     at
>
>
> org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:668)
>     at
>
>
> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2744)
>     at
>
>
> org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2697)
>     at
>
>
> org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:452)
>     at
>
>
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1149)
>     at
>
>
> org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852)
>     at
>
>
> org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1108)
>     at
>
>
> org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:420)
>     at
>
>
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1196)
>     at
>
>
> org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2875)
>     at
>
>
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1602)
>     at
>
>
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1584)
>     at
>
>
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1535)
>     at
>
>
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:838)
>     at
>
>
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.findInternal(EntityManagerImpl.java:778)
>     at
>
>
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:671)
>     at
>
>
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:543)
>     at
>
>
> org.apache.ambari.server.orm.dao.KeyValueDAO.findByKey(KeyValueDAO.java:42)
>     at
>
>
> org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53)
>     at
>
>
> org.apache.ambari.server.api.services.PersistKeyValueImpl.getValue(PersistKeyValueImpl.java:50)
>     at
>
>
> org.apache.ambari.server.api.services.PersistKeyValueService.getKey(PersistKeyValueService.java:83)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
>
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
>
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at
>
>
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>     at
>
>
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>     at
>
>
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>     at
>
>
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>     at
>
>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>     at
>
>
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>     at
>
>
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>     at
>
>
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>     at
>
>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>     at
>
>
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>     at
>
>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>     at
>
>
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>     at
>
>
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>     at
>
>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>     at
>
>
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
>     at
>
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
>     at
>
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
>     at
>
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.apache.ambari.server.security.authorization.internal.InternalTokenAuthenticationFilter.doFilter(InternalTokenAuthenticationFilter.java:53)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
>     at
>
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>     at
>
>
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
>     at
>
>
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
>     at
>
>
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
>     at
>
>
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
>     at
>
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
>     at
>
>
> org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:48)
>     at
>
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
>     at
>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
>     at
>
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>     at
>
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
>     at
>
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
>     at
>
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)
>     at
>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)
>     at
>
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
>     at
>
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)
>     at
>
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>     at
>
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
>     at
>
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>     at org.eclipse.jetty.server.Server.handle(Server.java:363)
>     at
>
>
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)
>     at
>
>
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)
>     at
>
>
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)
>     at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
>     at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>     at
>
>
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>     at
>
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
>     at
>
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
>     at
>
>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>     at
>
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>     at java.lang.Thread.run(Thread.java:745)
> Caused by: org.postgresql.util.PSQLException: ERROR: missing chunk
> number
> 0 for toast value 50294 in pg_toast_16548
>     at
>
>
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
>     at
>
>
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
>     at
>
>
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
>     at
>
>
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
>     at
>
>
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
>     at
>
>
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
>     at
>
>
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:938)
>     at
>
>
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:609)
>     ... 98 more
>
>
> --
> Thanks
> Suraj Nayak
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the
> reader of
> this message is not the intended recipient, you are hereby notified that
> any
> printing, copying, dissemination, distribution, disclosure or forwarding
> of
> this communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and delete
> it
> from your system. Thank You.
>
>
> --
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified
> that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender
> immediately
> and delete it from your system. Thank You.
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of
> this message is not the intended recipient, you are hereby notified that
> any
> printing, copying, dissemination, distribution, disclosure or forwarding
> of
> this communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and delete
> it
> from your system. Thank You.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader of
> this message is not the intended recipient, you are hereby notified that any
> printing, copying, dissemination, distribution, disclosure or forwarding of
> this communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and delete it
> from your system. Thank You.
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message