Return-Path: X-Original-To: apmail-atlas-dev-archive@minotaur.apache.org Delivered-To: apmail-atlas-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 920741730D for ; Tue, 22 Sep 2015 16:20:48 +0000 (UTC) Received: (qmail 52945 invoked by uid 500); 22 Sep 2015 16:20:48 -0000 Delivered-To: apmail-atlas-dev-archive@atlas.apache.org Received: (qmail 52904 invoked by uid 500); 22 Sep 2015 16:20:48 -0000 Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.incubator.apache.org Delivered-To: mailing list dev@atlas.incubator.apache.org Delivered-To: moderator for dev@atlas.incubator.apache.org Received: (qmail 69848 invoked by uid 99); 22 Sep 2015 06:56:25 -0000 X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.996 X-Spam-Level: *** X-Spam-Status: No, score=3.996 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-0.006, URIBL_BLOCKED=0.001] autolearn=disabled Content-Type: multipart/alternative; boundary="===============7598058384140352687==" MIME-Version: 1.0 Subject: Re: Review Request 38393: ATLAS-58 Make hive hook reliable From: "Shwetha GS" To: "atlas" , "Shwetha GS" Date: Tue, 22 Sep 2015 06:56:08 -0000 Message-ID: <20150922065608.24659.47606@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Shwetha GS" X-ReviewGroup: atlas X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/38393/ X-Sender: "Shwetha GS" References: <20150921152201.24659.40475@reviews.apache.org> In-Reply-To: <20150921152201.24659.40475@reviews.apache.org> Reply-To: "Shwetha GS" X-ReviewRequest-Repository: atlas --===============7598058384140352687== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit > On Sept. 21, 2015, 3:22 p.m., Suma Shivaprasad wrote: > > typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java, line 32 > > > > > > have we moved this from client to ypesystem? yes > On Sept. 21, 2015, 3:22 p.m., Suma Shivaprasad wrote: > > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java, line 121 > > > > > > if possible, can you pls fix the error message in other places where ValueCnversionException is being thrown curently :) Right now, we dont ge to know which attribute is causing the issue In the validation of actual attribute, the attribute name is not available. All attribute validation are called from Struct. So, this captures everything - Shwetha ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38393/#review99752 ----------------------------------------------------------- On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/38393/ > ----------------------------------------------------------- > > (Updated Sept. 21, 2015, 8:13 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-58 > https://issues.apache.org/jira/browse/ATLAS-58 > > > Repository: atlas > > > Description > ------- > > Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity > Big changes: > 1. Concept of service that are started and stopped at atlas start and stop > 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created > 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible > 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible > 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests > 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well) > 7. In webapp ITs, re-used the types defined > 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable > > Pending: > 1. Entity updates like alter table commands are not handlded. Will create another jira for this > 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this > > > Diffs > ----- > > addons/hive-bridge/pom.xml 914d8c6 > addons/hive-bridge/src/bin/import-hive.sh 99300e8 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446 > addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e > addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 > addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba > client/pom.xml 2e27930 > client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 > client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a > client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1 > client/src/test/resources/application.properties dbd6002 > common/pom.xml PRE-CREATION > common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION > common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION > dashboard/public/modules/details/detailsResource.js af2f249 > dashboard/public/modules/search/searchController.js b5e6c37 > distro/pom.xml d42d78f > distro/src/bin/atlas_start.py 7680f04 > distro/src/conf/application.properties bf323a7 > distro/src/main/assemblies/standalone-package.xml 625396a > docs/pom.xml 4535fe0 > docs/src/site/resources/architecture.png 826df37 > docs/src/site/resources/data-types.png 3aa1904 > docs/src/site/resources/guide-class-diagram.png ca51239 > docs/src/site/resources/guide-instance-graph.png a2c8f82 > docs/src/site/resources/images/twiki/architecture.png PRE-CREATION > docs/src/site/resources/images/twiki/data-types.png PRE-CREATION > docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION > docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION > docs/src/site/resources/images/twiki/notification.png PRE-CREATION > docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION > docs/src/site/resources/types-instance.png 6afca21 > docs/src/site/site.xml b412569 > docs/src/site/twiki/Architecture.twiki 5f10fde > docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION > docs/src/site/twiki/Configuration.twiki e2132cf > docs/src/site/twiki/InstallationSteps.twiki 3d0351c > docs/src/site/twiki/QuickStart.twiki d899f95 > docs/src/site/twiki/TypeSystem.twiki 78c3503 > docs/src/site/twiki/index.twiki 40d2350 > notification/pom.xml b036855 > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275 > notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION > notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0 > notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124 > notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35 > notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc > pom.xml e8d18a2 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293 > repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de > repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de > repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION > repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5 > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65 > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db > repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88 > repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b > repository/src/test/java/org/apache/atlas/TestUtils.java de41793 > repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7 > repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a > repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37 > repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84 > repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe > repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2 > repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085 > repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8 > repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf > repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020 > repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1 > repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION > repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df > repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886 > typesystem/pom.xml e4a1383 > typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION > typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11 > typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47 > typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5 > typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd > typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d > typesystem/src/main/resources/application.properties f7e2774 > typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION > typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf > typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION > typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd > typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681 > typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c > typesystem/src/test/resources/application.properties PRE-CREATION > typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72 > typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c > webapp/pom.xml 6085d68 > webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d > webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403 > webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f > webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d > webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979 > webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915 > webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c > webapp/src/main/resources/atlas-log4j.xml 7827c1a > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION > webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387 > webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de > webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325 > webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f > webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360 > webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e > webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a > webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION > webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64 > webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION > webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185 > webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION > webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54 > webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625 > webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION > webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION > > Diff: https://reviews.apache.org/r/38393/diff/ > > > Testing > ------- > > UTs and ITs > > > Thanks, > > Shwetha GS > > --===============7598058384140352687==--