atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Hagelberg <jnhagelb...@us.ibm.com>
Subject Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer
Date Tue, 07 Jun 2016 02:49:37 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
-----------------------------------------------------------

(Updated June 7, 2016, 2:49 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
-------

Added missing license headers.


Bugs: ATLAS-694
    https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
-------

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with the exception
of the catalog, which was only updated minimally) has been updated to use the graph database
abstraction layer.  In addition, there is now an optional Atlas configuration property that
specifies the class of the abstraction layer database to use.  I basically put all of the
changes in here with the exception of the actual Titan 1 implementation of code.  This includes
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting the changes
into Atlas.  Originally the TP3 changes were going to be brought in as part of the Titan 1
implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead of TitanGraph/Vertex/Edge,
etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, which was only
changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 when TP3 is being
used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database implementation is
determined from a new optional configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by hive-instances.json
(which uses normal Atlas json syntax).  The data is saved with a new JSONImporter class. 
This was needed because the graphson syntax used by HiveTitanSample is not compatible with
TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager singleton to force
index creation to happen before we execute any gremlin queries (which happens during the type
system restoration triggered by the DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  It now supports
having the java home in ATLAS_JAVA_HOME.  If that is not set, we fall back to the previous
behavior looking for JAVA_HOME and then checking the path.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I could not figure
out how to remove those.  You can, for the most part, ignore the new files under graphdb\titan0.
 There are a few minor changes that have been put in since ATLAS-693, but nothing very significant.
 Just changes to implement a couple new methods in the abstraction layer.

Last rebase: 6/6/2016


Diffs (updated)
-----

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java ac60294e328835ba0340e150799ddfb348ccdb52

  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java a28a32b692e61377479cdd432f39ba3c60bd5238

  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java ba8e0e712769afad72f7f8254ec626a1d17136e8

  catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 149134c8a79c802fe625d64c5e04af0d88174cfa

  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 71b577b5c2e8aee8fec816b9224672ebc2cad5b3

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java e7da1c9a5075004973ba68b8a36a2c95756e0e62

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 3c41693e531eb523f04a5c5781f1f123ab38ef07

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java PRE-CREATION

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
2776b2242c4469dcae344616ccfe100633a79432 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java f0dca9a28eebd8996d55d71dcc3254cce9cd4098

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java PRE-CREATION

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasSchemaViolationException.java
fda83b807f3640c04a660001941376f41be6d9be 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java e027b6962cf27a8c2dfbbf6fcc9683fa19e00ae5

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java fd20a6524c4795b4212599c2f4ced69c2bbf14c0

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java a608eb27921d4be44e639f773ace818115f39ff2

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GremlinVersion.java e817cba200c6d3c4c593b242cb2317e141bab570

  graphdb/pom.xml 666fe5a01e55d7b761d391e3ef6172f769488254 
  graphdb/titan0/pom.xml PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Edge.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Element.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphIndex.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphQuery.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0IndexQuery.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0VertexQuery.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/utils/EdgeToAtlasEdgeFunction.java PRE-CREATION

  graphdb/titan0/src/main/java/org/apache/atlas/utils/VertexToAtlasVertexFunction.java PRE-CREATION

  graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseTest.java
PRE-CREATION 
  graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseValidationTest.java
PRE-CREATION 
  graphdb/titan0/src/test/resources/atlas-application.properties PRE-CREATION 
  pom.xml eac95c8526011a3a86de8dd154545abec0f7bcf5 
  repository/pom.xml 533e48b8dbf8f1d80903b804ddc7da9818fe6702 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java b9689f4eb408819e91e8184d40534e983bf5d048

  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 15201dd45913539b9b3bd430b46ce6cf30c327ac

  repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java 39dde2af0aa86d7c9a3dc5f006c8e1660cd29d5c

  repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
90718edfd261c9f30ecc6b8f2424c232fbc9a66b 
  repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java
f97b83d1b302dedc12c566a8bb4729d4cd68de75 
  repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b64fa5d38c8025e6faf86335864f77d1439

  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 43e9f85163e8bb453601ca502d9e7a32c757b037

  repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
22d71df8af166bd88130f1f2e94ed3bbabbc16f5 
  repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java d905c012a58ae4f53012f217253bf2495550da33

  repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java PRE-CREATION

  repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 91f9bd008a6939dfe78656f5c1845637a30ee9eb

  repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java 37cacb05e65d6b1f537bd208208c8c4b9b501645

  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
0d82d900bd750b1ca91c24623384fb2547790efe 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
c385df35c0ed37490033601487bf59eaf7dc677f 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 4f6d0118072380342f5ea302d0aaf7902784849a

  repository/src/main/java/org/apache/atlas/repository/graph/GraphProvider.java f89bdf5c7e8596b11dafced6700d0a4245fd32cf

  repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java 6141927eb92bc7e681b43118fbaa399ada6c81f8

  repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
e240fb68ed9547ef6ef4cb773a12b2594dcffa09 
  repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java 36367913252b59f7adf7f42924a886365e60819f

  repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java 25aa7c5844cd3fc39bd7cc9ee23a6c336ca1bfac

  repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java 2cc1a50e2ea139c3a462c918b94945fff602cf17

  repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
4c1f5591f4adead41c3336e64b0bc956836f7edb 
  repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
5ed9e02eb85ec25b63540cab40d148914d410786 
  repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java c1af0f691e9d26a3102fece20acc904bb85702d6

  repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala c4621cd509ae049b30a08a9e5b3ace03f888a10c

  repository/src/main/scala/org/apache/atlas/query/Expressions.scala 240575087468632c52512aea94759f8ace3943a0

  repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala 34d101a2704e28a46426fdf6b67e7c36fde0507a

  repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala 7de03c32edf16c1676db94179266e02928e01f35

  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 73981c04d410a7292de357d9f63b6f3b3d241ffa

  repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 60b57d9a4d0181cd1e3c45a4e6a928d15ed499db

  repository/src/main/scala/org/apache/atlas/query/QueryProcessor.scala 0d2a908201f526aa487dd4cad926d308f53a04ba

  repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 5a64c53bdb94174bf5699d9e034960f42450f696

  repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java d1f9430124ab4071971950bf4acd2f09dd000da0

  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 4195955051d842af973d3ad29602fb97c1e54ecd

  repository/src/test/java/org/apache/atlas/TestUtils.java 345e874942cf047ae8b6467f2981c7cf2317707c

  repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
5e7de88559dbbf2460c819ba7470a2cc287ca8dc 
  repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
f6994048b7f92ba086a1b0949030897d1cacff41 
  repository/src/test/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepositoryTest.java
677eb392c4aa7674e27c48f86a44e953c80144c7 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
449e066a167ba4546b118a77c8e3de5fd99f077b 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
2d1c33a9bef5dc9179225fa7ce7228eebc5fdf2b 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
cc6026461bcd38b83dde0bcec2e835f5824e894f 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
90bb635126a02cd38c045bd5332499c4481d3751 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerMockTest.java
94dd29862cd37fd784a699fb45640e36992cff88 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerTest.java
3d9d45a1f6695a0ee1af8c42fb24dcfa41dfba79 
  repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
2f02ae28b2947ce7f7955c00bf9ca88ab3ca607a 
  repository/src/test/java/org/apache/atlas/repository/graph/TitanGraphProviderTest.java 6fc700823800eaa943a34afecb2c2e6fba50637a

  repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
961442bbcaff86b472a876e6c348aa9e65c1efca 
  repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java e6dd23081715458471d50f44f45ccd8374cbffda

  repository/src/test/java/org/apache/atlas/services/DependencyTreeNode.java PRE-CREATION

  repository/src/test/java/org/apache/atlas/services/JSONImporter.java PRE-CREATION 
  repository/src/test/resources/hive-instances.json PRE-CREATION 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala b23c0f60ace59fa15886c94f35e35d3113872783

  repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala f65cedbf44f5fa9e674833a20aae8de6d2537834

  repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala 67ce12bf31dd6decf1a1323924303a066f90a4c1

  repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 0e0ac8633a256dc7c8966c2b8d52eb942b52e1c6

  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala b5faaf33cd5fd1a63bac4fd41d67588ef519dc11

  repository/src/test/titan0-resources/atlas-application.properties PRE-CREATION 
  repository/src/test/titan1-resources/atlas-application.properties PRE-CREATION 
  src/build/checkstyle-suppressions.xml 0025360552a32ffa0a5aa5696f27921a4752513f 
  titan/pom.xml 09006d9dd5e21aa24f53524aa42a3fcd6f3d9fe6 
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/AdminMask.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/ConnectionMask.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseAdmin0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseAdmin1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat1_1.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompatLoader.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseKeyColumnValueStore.java
 
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseStoreManager.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseTransaction.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HConnection0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HConnection1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable0_98.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable1_0.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/TableMask.java  
  titan/src/main/java/com/thinkaurelius/titan/diskstorage/locking/LocalLockMediator.java 

  titan/src/main/java/com/thinkaurelius/titan/diskstorage/solr/Solr5Index.java  
  titan/src/test/java/com/thinkaurelius/titan/diskstorage/locking/LocalLockMediatorTest.java
 
  typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 6fb2087a7cbeb6eefb0dd15b9a3c0c049b6b857b

  typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java dc9cdf2567b45b35b324ecfbb5afcc7921409b8f

  typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDependencySorter.java
aaec05c8c3913b6974332668ebe5af31332cd121 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/Multiplicity.java 06da32e807aa4b8e4d1f49b3929c9f3c9a8d4846

  typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java ef8448d9f278c45f0a9c7c6045338cd056d23d08

  typesystem/src/main/resources/atlas-application.properties 3318ff306a8b8ab5e320129a8450d144215f009f

  typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 5618938e4bd9f9ef934e89bf2bf3199073294706

  typesystem/src/test/java/org/apache/atlas/typesystem/types/ClassTest.java daecdd78c952747964885d4cb8cdfb59727ffee5

  typesystem/src/test/java/org/apache/atlas/typesystem/types/HierarchicalTypeDependencySorterTest.java
19bdccfc6baf503f3144fce8fe0b85914693d054 
  typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java 574e0f9811d195cd34e12a9f23b999568d54a1c7

  typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala
cfd4bdb26482c187c4e80f110b7349df8b524e6e 
  webapp/pom.xml cce6dd6f073b15732d2aa5d26fd4aed090ddf65d 
  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 010fa2aa979e12063a1d78c8db4ad7519c3ca973

  webapp/src/main/java/org/apache/atlas/web/resources/BaseService.java 2a655383d6265213933beff2a884953e51fed889

  webapp/src/main/java/org/apache/atlas/web/resources/RexsterGraphResource.java 150f913d881138c7afeb87881197ee0f1e7d3194

  webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 08bb125241012b6a1c1852efc6443cc7a4ebecc3


Diff: https://reviews.apache.org/r/47810/diff/


Testing
-------

Built entire Atlas project, ran all unit/integration tests.  No issues found.


Thanks,

Jeff Hagelberg


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message