atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suma Shivaprasad" <sumasai.shivapra...@gmail.com>
Subject Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Date Mon, 21 Sep 2015 15:22:01 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99752
-----------------------------------------------------------



typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java (line 32)
<https://reviews.apache.org/r/38393/#comment156740>

    have we moved this from client to ypesystem?



typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java (line
121)
<https://reviews.apache.org/r/38393/#comment156741>

    if possible, can you pls fix the error message in other places where ValueCnversionException
is being thrown curently :) Right now, we dont ge to know which attribute is causing the issue


- Suma Shivaprasad


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
>     https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Hive hook sends notification messages (list of entities). The Notification consumer on
server side consumes these messages and registers the entities. The server handles de-duping
of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity
doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required
for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible

> 5. Moved security tests from integration tests to unit tests - as they were creating
issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem
module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification
is done synchronously. So, this adds to hive command execution delay. But this also makes
it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira
for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests.
So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
> 
> 
> Diffs
> -----
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba

>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
>   docs/src/site/resources/types-instance.png 6afca21 
>   docs/src/site/site.xml b412569 
>   docs/src/site/twiki/Architecture.twiki 5f10fde 
>   docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION 
>   docs/src/site/twiki/Configuration.twiki e2132cf 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   docs/src/site/twiki/QuickStart.twiki d899f95 
>   docs/src/site/twiki/TypeSystem.twiki 78c3503 
>   docs/src/site/twiki/index.twiki 40d2350 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION

>   notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
36a62f0 
>   notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
0951124 
>   notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35

>   notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc

>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293

>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de 
>   repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de

>   repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION

>   repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5

>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
cc10ea3 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65

>   repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db

>   repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88 
>   repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b

>   repository/src/test/java/org/apache/atlas/TestUtils.java de41793 
>   repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
127e6d7 
>   repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a

>   repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
487ac84 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
be99efe 
>   repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2

>   repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085 
>   repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8

>   repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf

>   repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020

>   repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1

>   repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION

>   repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886 
>   typesystem/pom.xml e4a1383 
>   typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION 
>   typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION 
>   typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java
309ab11 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java
31b5f47 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java
5cdee8d 
>   typesystem/src/main/resources/application.properties f7e2774 
>   typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION 
>   typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
7a681bf 
>   typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION

>   typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java
a8c58bd 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c

>   typesystem/src/test/resources/application.properties PRE-CREATION 
>   typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala
d14ac72 
>   typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala
ba4782c 
>   webapp/pom.xml 6085d68 
>   webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403 
>   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f

>   webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION

>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d 
>   webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
9248979 
>   webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915 
>   webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c

>   webapp/src/main/resources/atlas-log4j.xml 7827c1a 
>   webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java
PRE-CREATION 
>   webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387

>   webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de 
>   webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325

>   webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
a884b5f 
>   webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java
8664360 
>   webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e 
>   webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a

>   webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java
PRE-CREATION 
>   webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64 
>   webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION

>   webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185 
>   webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION 
>   webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54

>   webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625

>   webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION

>   webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38393/diff/
> 
> 
> Testing
> -------
> 
> UTs and ITs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message