Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@atlas.incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============3572644203695679003=="
MIME-Version: 1.0
Subject: Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB
 abstraction layer
From: Jeff Hagelberg <jnhagelberg@us.ibm.com>
To: Neeru Gupta <guptaneeru@us.ibm.com>, David Kantor <dkantor@us.ibm.com>
Cc: Jeff Hagelberg <jnhagelberg@us.ibm.com>,
 atlas <dev@atlas.incubator.apache.org>
Date: Mon, 06 Jun 2016 19:20:08 -0000
Message-ID: <20160606192008.8186.96491@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: Jeff Hagelberg <noreply@reviews.apache.org>
References: <20160527182029.1667.89302@reviews.apache.org>
In-Reply-To: <20160527182029.1667.89302@reviews.apache.org>
Reply-To: Jeff Hagelberg <jnhagelberg@us.ibm.com>
archived-at: Mon, 06 Jun 2016 19:20:20 -0000

--===============3572644203695679003==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
-----------------------------------------------------------

(Updated June 6, 2016, 7:19 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
-------

Rebase, apply latest changes to graph database abstraction layer.


Bugs: ATLAS-694
    https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
-------

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with the exception of the catalog, which was only updated minimally) has been updated to use the graph database abstraction layer.  In addition, there is now a required Atlas configuration property that specifies the class of the abstraction layer database to use.  I basically put all of the changes in here with the exception of the actual Titan 1 implementation of code.  This includes the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting the changes into Atlas.  Originally the TP3 changes were going to be brought in as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by hive-instances.json (which uses normal Atlas json syntax).  The data is saved with a new JSONImporter class.  This was needed because the graphson syntax used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager singleton to force index creation to happen before we execute any gremlin queries (which happens during the type system restoration triggered by the DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, we fall back to the previous behavior looking for JAVA_HOME and then checking the path.
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I could not figure out how to remove those.

Last rebase: 5/27/2016


Diffs (updated)
-----

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java ba8e0e712769afad72f7f8254ec626a1d17136e8 
  catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java PRE-CREATION 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java 2776b2242c4469dcae344616ccfe100633a79432 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java PRE-CREATION 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasSchemaViolationException.java fda83b807f3640c04a660001941376f41be6d9be 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java e027b6962cf27a8c2dfbbf6fcc9683fa19e00ae5 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java fd20a6524c4795b4212599c2f4ced69c2bbf14c0 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java a608eb27921d4be44e639f773ace818115f39ff2 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GremlinVersion.java e817cba200c6d3c4c593b242cb2317e141bab570 
  graphdb/pom.xml 666fe5a01e55d7b761d391e3ef6172f769488254 
  graphdb/titan0/pom.xml PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Edge.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Element.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphIndex.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphQuery.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0IndexQuery.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0VertexQuery.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/utils/EdgeToAtlasEdgeFunction.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/utils/VertexToAtlasVertexFunction.java PRE-CREATION 
  graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseTest.java PRE-CREATION 
  graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseValidationTest.java PRE-CREATION 
  graphdb/titan0/src/test/resources/atlas-application.properties PRE-CREATION 
  pom.xml eac95c8526011a3a86de8dd154545abec0f7bcf5 
  repository/pom.xml 533e48b8dbf8f1d80903b804ddc7da9818fe6702 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java b9689f4eb408819e91e8184d40534e983bf5d048 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 15201dd45913539b9b3bd430b46ce6cf30c327ac 
  repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java 39dde2af0aa86d7c9a3dc5f006c8e1660cd29d5c 
  repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java 90718edfd261c9f30ecc6b8f2424c232fbc9a66b 
  repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java f97b83d1b302dedc12c566a8bb4729d4cd68de75 
  repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b64fa5d38c8025e6faf86335864f77d1439 
  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 43e9f85163e8bb453601ca502d9e7a32c757b037 
  repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java d905c012a58ae4f53012f217253bf2495550da33 
  repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
  repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java 37cacb05e65d6b1f537bd208208c8c4b9b501645 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java 0d82d900bd750b1ca91c24623384fb2547790efe 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java c385df35c0ed37490033601487bf59eaf7dc677f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 4f6d0118072380342f5ea302d0aaf7902784849a 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphProvider.java f89bdf5c7e8596b11dafced6700d0a4245fd32cf 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java 6141927eb92bc7e681b43118fbaa399ada6c81f8 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java e240fb68ed9547ef6ef4cb773a12b2594dcffa09 
  repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java 36367913252b59f7adf7f42924a886365e60819f 
  repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java 25aa7c5844cd3fc39bd7cc9ee23a6c336ca1bfac 
  repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java 2cc1a50e2ea139c3a462c918b94945fff602cf17 
  repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java 4c1f5591f4adead41c3336e64b0bc956836f7edb 
  repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java 5ed9e02eb85ec25b63540cab40d148914d410786 
  repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java c1af0f691e9d26a3102fece20acc904bb85702d6 
  repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala c4621cd509ae049b30a08a9e5b3ace03f888a10c 
  repository/src/main/scala/org/apache/atlas/query/Expressions.scala 240575087468632c52512aea94759f8ace3943a0 
  repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala 34d101a2704e28a46426fdf6b67e7c36fde0507a 
  repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala 7de03c32edf16c1676db94179266e02928e01f35 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 73981c04d410a7292de357d9f63b6f3b3d241ffa 
  repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 60b57d9a4d0181cd1e3c45a4e6a928d15ed499db 
  repository/src/main/scala/org/apache/atlas/query/QueryProcessor.scala 0d2a908201f526aa487dd4cad926d308f53a04ba 
  repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 5a64c53bdb94174bf5699d9e034960f42450f696 
  repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java d1f9430124ab4071971950bf4acd2f09dd000da0 
  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 4195955051d842af973d3ad29602fb97c1e54ecd 
  repository/src/test/java/org/apache/atlas/TestUtils.java 345e874942cf047ae8b6467f2981c7cf2317707c 
  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 5e7de88559dbbf2460c819ba7470a2cc287ca8dc 
  repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java f6994048b7f92ba086a1b0949030897d1cacff41 
  repository/src/test/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepositoryTest.java 677eb392c4aa7674e27c48f86a44e953c80144c7 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java 449e066a167ba4546b118a77c8e3de5fd99f077b 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 2d1c33a9bef5dc9179225fa7ce7228eebc5fdf2b 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java cc6026461bcd38b83dde0bcec2e835f5824e894f 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java 90bb635126a02cd38c045bd5332499c4481d3751 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerMockTest.java 94dd29862cd37fd784a699fb45640e36992cff88 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerTest.java 3d9d45a1f6695a0ee1af8c42fb24dcfa41dfba79 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java 2f02ae28b2947ce7f7955c00bf9ca88ab3ca607a 
  repository/src/test/java/org/apache/atlas/repository/graph/TitanGraphProviderTest.java 6fc700823800eaa943a34afecb2c2e6fba50637a 
  repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java 961442bbcaff86b472a876e6c348aa9e65c1efca 
  repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java e6dd23081715458471d50f44f45ccd8374cbffda 
  repository/src/test/java/org/apache/atlas/services/DependencyTreeNode.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/services/JSONImporter.java PRE-CREATION 
  repository/src/test/resources/hive-instances.json PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala b23c0f60ace59fa15886c94f35e35d3113872783 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala f65cedbf44f5fa9e674833a20aae8de6d2537834 
  repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala 67ce12bf31dd6decf1a1323924303a066f90a4c1 
  repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 0e0ac8633a256dc7c8966c2b8d52eb942b52e1c6 
  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala b5faaf33cd5fd1a63bac4fd41d67588ef519dc11 
  repository/src/test/titan0-resources/atlas-application.properties PRE-CREATION 
  repository/src/test/titan1-resources/atlas-application.properties PRE-CREATION 
  src/build/checkstyle-suppressions.xml 0025360552a32ffa0a5aa5696f27921a4752513f 
  titan/pom.xml 09006d9dd5e21aa24f53524aa42a3fcd6f3d9fe6 
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/AdminMask.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/ConnectionMask.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseAdmin0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseAdmin1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat1_1.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompatLoader.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseKeyColumnValueStore.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseStoreManager.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseTransaction.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HConnection0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HConnection1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/TableMask.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/locking/LocalLockMediator.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/solr/Solr5Index.java  
  titan/src/test/java/com/thinkaurelius/titan/diskstorage/locking/LocalLockMediatorTest.java  
  typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 6fb2087a7cbeb6eefb0dd15b9a3c0c049b6b857b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java dc9cdf2567b45b35b324ecfbb5afcc7921409b8f 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDependencySorter.java aaec05c8c3913b6974332668ebe5af31332cd121 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/Multiplicity.java 06da32e807aa4b8e4d1f49b3929c9f3c9a8d4846 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java ef8448d9f278c45f0a9c7c6045338cd056d23d08 
  typesystem/src/main/resources/atlas-application.properties 3318ff306a8b8ab5e320129a8450d144215f009f 
  typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 5618938e4bd9f9ef934e89bf2bf3199073294706 
  typesystem/src/test/java/org/apache/atlas/typesystem/types/ClassTest.java daecdd78c952747964885d4cb8cdfb59727ffee5 
  typesystem/src/test/java/org/apache/atlas/typesystem/types/HierarchicalTypeDependencySorterTest.java 19bdccfc6baf503f3144fce8fe0b85914693d054 
  typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java 574e0f9811d195cd34e12a9f23b999568d54a1c7 
  typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala cfd4bdb26482c187c4e80f110b7349df8b524e6e 
  webapp/pom.xml cce6dd6f073b15732d2aa5d26fd4aed090ddf65d 
  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 010fa2aa979e12063a1d78c8db4ad7519c3ca973 
  webapp/src/main/java/org/apache/atlas/web/resources/BaseService.java 2a655383d6265213933beff2a884953e51fed889 
  webapp/src/main/java/org/apache/atlas/web/resources/RexsterGraphResource.java 150f913d881138c7afeb87881197ee0f1e7d3194 
  webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 08bb125241012b6a1c1852efc6443cc7a4ebecc3 

Diff: https://reviews.apache.org/r/47810/diff/


Testing
-------

Built entire Atlas project, ran all unit/integration tests.  No issues found.


Thanks,

Jeff Hagelberg


--===============3572644203695679003==--