atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayub Khan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (ATLAS-1321) Fix failure of Hive ITs
Date Thu, 24 Nov 2016 04:30:58 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692116#comment-15692116
] 

Ayub Khan edited comment on ATLAS-1321 at 11/24/16 4:30 AM:
------------------------------------------------------------

[~svimal2106]
The below change would work for local builds, but when it is packaged and run from the deployed
path(i.e; on deployed cluster), this might fail. Because, if atlas.home is not set, then the
user.dir property would point to(/usr/hdp/current/atlas-server/) and this does not have 'addons/models'
dir. This would result to model registration failure in the actual cluster.

{noformat}
         AtlasTypeDefStoreInitializer storeInitializer = new AtlasTypeDefStoreInitializer();
 
         String atlasHomeDir = System.getProperty("atlas.home");
-        String typesDirName = (StringUtils.isEmpty(atlasHomeDir) ? "." : atlasHomeDir) +
File.separator + "models";
-
+        String typesDirName = (StringUtils.isEmpty(atlasHomeDir) ? System.getProperty("user.dir")
+ File.separator + "addons" : atlasHomeDir) + File.separator + "models";
         storeInitializer.initializeStore(this, typeRegistry, typesDirName);
     }
{noformat}


Below snippet confirms absence of addons dir on deployed cluster: 
{noformat}
atlas-testing-ui-1:~ # ls -lrt /usr/hdp/current/atlas-server/
total 60
-rwxr-xr-x 1 root  root     900 Nov 19 05:19 NOTICE
-rwxr-xr-x 1 root  root   13430 Nov 19 05:19 LICENSE
-rwxr-xr-x 1 root  root     698 Nov 19 05:19 DISCLAIMER.txt
drwxr-xr-x 3 root  root    4096 Nov 21 10:15 libext
drwxr-xr-x 3 root  root    4096 Nov 21 10:16 server
drwxr-xr-x 2 root  root    4096 Nov 21 10:16 models
drwxr-xr-x 2 root  root    4096 Nov 21 10:16 examples
drwxr-xr-x 6 root  root    4096 Nov 21 10:16 docs
drwxr-xr-x 2 root  root    4096 Nov 21 10:16 bin
lrwxrwxrwx 1 root  root      23 Nov 21 10:17 conf -> /etc/atlas/2.5.3.0-27/0
drwxr-xr-x 2 root  root    4096 Nov 21 10:18 hook-bin
drwxr-xr-x 7 root  root    4096 Nov 21 10:18 hook
drwxr-xr-x 2 atlas hadoop  4096 Nov 21 10:39 data
atlas-testing-ui-1:~ #
{noformat}



was (Author: ayubkhan):
[~svimal2106]
I am not seeing the model registration failures with clean git pull. Am i missing something
here?

Also, the below change would work for local builds, but when it is packaged and run from the
deployed path(i.e; on deployed cluster), this might fail. Because, if atlas.home is not set,
then the user.dir property would point to(/usr/hdp/current/atlas-server/) and this does not
have 'addons/models' dir. This would result to model registration failure in the actual cluster.

{noformat}
         AtlasTypeDefStoreInitializer storeInitializer = new AtlasTypeDefStoreInitializer();
 
         String atlasHomeDir = System.getProperty("atlas.home");
-        String typesDirName = (StringUtils.isEmpty(atlasHomeDir) ? "." : atlasHomeDir) +
File.separator + "models";
-
+        String typesDirName = (StringUtils.isEmpty(atlasHomeDir) ? System.getProperty("user.dir")
+ File.separator + "addons" : atlasHomeDir) + File.separator + "models";
         storeInitializer.initializeStore(this, typeRegistry, typesDirName);
     }
{noformat}


Below snippet confirms absence of addons dir on deployed cluster: 
{noformat}
atlas-testing-ui-1:~ # ls -lrt /usr/hdp/current/atlas-server/
total 60
-rwxr-xr-x 1 root  root     900 Nov 19 05:19 NOTICE
-rwxr-xr-x 1 root  root   13430 Nov 19 05:19 LICENSE
-rwxr-xr-x 1 root  root     698 Nov 19 05:19 DISCLAIMER.txt
drwxr-xr-x 3 root  root    4096 Nov 21 10:15 libext
drwxr-xr-x 3 root  root    4096 Nov 21 10:16 server
drwxr-xr-x 2 root  root    4096 Nov 21 10:16 models
drwxr-xr-x 2 root  root    4096 Nov 21 10:16 examples
drwxr-xr-x 6 root  root    4096 Nov 21 10:16 docs
drwxr-xr-x 2 root  root    4096 Nov 21 10:16 bin
lrwxrwxrwx 1 root  root      23 Nov 21 10:17 conf -> /etc/atlas/2.5.3.0-27/0
drwxr-xr-x 2 root  root    4096 Nov 21 10:18 hook-bin
drwxr-xr-x 7 root  root    4096 Nov 21 10:18 hook
drwxr-xr-x 2 atlas hadoop  4096 Nov 21 10:39 data
atlas-testing-ui-1:~ #
{noformat}


> Fix failure of Hive ITs
> -----------------------
>
>                 Key: ATLAS-1321
>                 URL: https://issues.apache.org/jira/browse/ATLAS-1321
>             Project: Atlas
>          Issue Type: Bug
>            Reporter: Vimal Sharma
>            Assignee: Vimal Sharma
>         Attachments: ATLAS-1321.patch
>
>
> Hive ITs fail because the Atlas TypeDef initializer cannot locate models directory. None
of the types defined in hive-model.json are regsitered.
> Below is the stack trace:
> Caused by: org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasBaseClient$APIInfo@6398e143
failed with status 400 (Bad Request) Response Body ({"error":"Unknown datatype: hive_db","stackTrace":"org.apache.atlas.typesystem.exception.TypeNotFoundException:
Unknown datatype: hive_db\n\tat org.apache.atlas.typesystem.types.TypeSystem.getDataType(TypeSystem.java:192)\n\tat
org.apache.atlas.services.DefaultMetadataService.validateTypeExists(DefaultMetadataService.java:519)\n\tat
org.apache.atlas.services.DefaultMetadataService.getEntityDefinitionReference(DefaultMetadataService.java:319)\n\tat
org.apache.atlas.services.DefaultMetadataService.getEntityDefinition(DefaultMetadataService.java:327)\n\tat
org.apache.atlas.web.resources.EntityResource.getEntityDefinitionByAttribute(EntityResource.java:525)\n\tat
org.apache.atlas.web.resources.EntityResource.getEntity(EntityResource.java:506)\n\tat sun.reflect.GeneratedMethodAccessor77.invoke(Unknown
Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:606)\n\tat com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n\tat
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\n\tat
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\n\tat
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n\tat com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)\n\tat
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)\n\tat
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)\n\tat com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)\n\tat
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)\n\tat
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)\n\tat
org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:73)\n\tat com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)\n\tat
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)\n\tat
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat
org.apache.atlas.web.filters.AtlasAuthorizationFilter.doFilter(AtlasAuthorizationFilter.java:154)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)\n\tat
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:232)\n\tat
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)\n\tat
org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:187)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:321)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.apache.atlas.web.filters.AtlasKnoxSSOAuthenticationFilter.doFilter(AtlasKnoxSSOAuthenticationFilter.java:113)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)\n\tat
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)\n\tat
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)\n\tat
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)\n\tat
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:499)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)\n\tat
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\n\tat java.lang.Thread.run(Thread.java:745)\n"})



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message