atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shwetha GS" <sshivalingamur...@hortonworks.com>
Subject Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Date Mon, 21 Sep 2015 08:13:35 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------

(Updated Sept. 21, 2015, 8:13 a.m.)


Review request for atlas.


Changes
-------

Fixes for import-hive.sh in end to end set-up


Bugs: ATLAS-58
    https://issues.apache.org/jira/browse/ATLAS-58


Repository: atlas


Description
-------

Hive hook sends notification messages (list of entities). The Notification consumer on server
side consumes these messages and registers the entities. The server handles de-duping of entities
based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity
doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for
hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible 
5. Moved security tests from integration tests to unit tests - as they were creating issues
with server start as jetty already starts another server for integration tests
6. Removed some duplicate tests from repository module (the same tests exist in typesystem
module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending notification
is done synchronously. So, this adds to hive command execution delay. But this also makes
it reliable

Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for
this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So,
hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this


Diffs (updated)
-----

  addons/hive-bridge/pom.xml 914d8c6 
  addons/hive-bridge/src/bin/import-hive.sh 99300e8 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e

  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
cb8359e 
  addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba 
  client/pom.xml 2e27930 
  client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
  client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
  client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 
  client/src/test/resources/application.properties dbd6002 
  common/pom.xml PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
  dashboard/public/modules/details/detailsResource.js af2f249 
  dashboard/public/modules/search/searchController.js b5e6c37 
  distro/pom.xml d42d78f 
  distro/src/bin/atlas_start.py 7680f04 
  distro/src/conf/application.properties bf323a7 
  distro/src/main/assemblies/standalone-package.xml 625396a 
  docs/pom.xml 4535fe0 
  docs/src/site/resources/architecture.png 826df37 
  docs/src/site/resources/data-types.png 3aa1904 
  docs/src/site/resources/guide-class-diagram.png ca51239 
  docs/src/site/resources/guide-instance-graph.png a2c8f82 
  docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
  docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
  docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
  docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
  docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
  docs/src/site/resources/types-instance.png 6afca21 
  docs/src/site/site.xml b412569 
  docs/src/site/twiki/Architecture.twiki 5f10fde 
  docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION 
  docs/src/site/twiki/Configuration.twiki e2132cf 
  docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
  docs/src/site/twiki/QuickStart.twiki d899f95 
  docs/src/site/twiki/TypeSystem.twiki 78c3503 
  docs/src/site/twiki/index.twiki 40d2350 
  notification/pom.xml b036855 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION

  notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0

  notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124

  notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35

  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc 
  pom.xml e8d18a2 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de 
  repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de 
  repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION

  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
cc10ea3 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65 
  repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db 
  repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88 
  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b 
  repository/src/test/java/org/apache/atlas/TestUtils.java de41793 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
127e6d7 
  repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a

  repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
487ac84 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
be99efe 
  repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2 
  repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085 
  repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8

  repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf 
  repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020 
  repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1 
  repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION

  repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df 
  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886 
  typesystem/pom.xml e4a1383 
  typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION 
  typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION 
  typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11

  typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47

  typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c

  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java
5cdee8d 
  typesystem/src/main/resources/application.properties f7e2774 
  typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION 
  typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf

  typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION 
  typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd

  typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681 
  typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c 
  typesystem/src/test/resources/application.properties PRE-CREATION 
  typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72

  typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala
ba4782c 
  webapp/pom.xml 6085d68 
  webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403 
  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f 
  webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d 
  webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979

  webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915 
  webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c 
  webapp/src/main/resources/atlas-log4j.xml 7827c1a 
  webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION

  webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387

  webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de 
  webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325

  webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f

  webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java
8664360 
  webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e 
  webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a

  webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION

  webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64 
  webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION

  webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185 
  webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54 
  webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625

  webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION

  webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION


Diff: https://reviews.apache.org/r/38393/diff/


Testing
-------

UTs and ITs


Thanks,

Shwetha GS


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message