atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Hagelberg <jnhagelb...@us.ibm.com>
Subject Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer
Date Fri, 27 May 2016 15:47:24 GMT


> On May 25, 2016, 5:03 p.m., David Kantor wrote:
> > distro/src/bin/atlas_config.py, line 162
> > <https://reviews.apache.org/r/47810/diff/1/?file=1393023#file1393023line162>
> >
> >     Perhaps add comments here explaining the need/motivation for ATLAS_JAVA_HOME.

I added an ATLAS_JAVA_HOME as a convenient way to configure only Atlas to use a specific JVM.
 Many environments have multiple JVMs installed, and different software has different JVM
requirements.  In environments where the default JVM is not compatible with Atlas, this provides
a way to configure Atlas to use a specific JVM without affecting other applications.  This
is also a convenient way for developers to test Atlas with a specific JVM as well.


> On May 25, 2016, 5:03 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java,
line 18
> > <https://reviews.apache.org/r/47810/diff/1/?file=1393046#file1393046line18>
> >
> >     Please rename this, as plugin is an overloaded term.

plugin_ was renamed to graph_


> On May 25, 2016, 5:03 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java, line
470
> > <https://reviews.apache.org/r/47810/diff/1/?file=1393051#file1393051line470>
> >
> >     What is the use case for this method?

This was refactored out of DefaultMetadataService.  It is used when converting the json received
from clients into ITypedReferenceableInstances.  I refactored it so that it could be reused
by the JSONImporter class that I added.


- Jeff


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review134773
-----------------------------------------------------------


On May 27, 2016, 3:30 p.m., Jeff Hagelberg wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> -----------------------------------------------------------
> 
> (Updated May 27, 2016, 3:30 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
>     https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with the exception
of the catalog, which was only updated minimally) has been updated to use the graph database
abstraction layer.  In addition, there is now a required Atlas configuration property that
specifies the class of the abstraction layer database to use.  I basically put all of the
changes in here with the exception of the actual Titan 1 implementation of code.  This includes
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting the changes
into Atlas.  Originally the TP3 changes were going to be brought in as part of the Titan 1
implementation task.
> 
> Change Summary:
> 
>    - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead of TitanGraph/Vertex/Edge,
etc
>    - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, which was
only changed to use AtlasGraphProvider/AtlasGraph)
>    - updated DSL translation to generate Gremlin that is compliant with TP3 when TP3
is being used
>    - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database implementation
is determined from a new required configuration property
>    - search indexer enhanced to wait for all indices to be enabled
>    - HiveTitanSample is no longer used by tests.  It has been replaced by hive-instances.json
(which uses normal Atlas json syntax).  The data is saved with a new JSONImporter class. 
This was needed because the graphson syntax used by HiveTitanSample is not compatible with
TP3.  
>    - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager singleton to
force index creation to happen before we execute any gremlin queries (which happens during
the type system restoration triggered by the DefaultMetadataService constructor)
>    - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  It now
supports having the java home in ATLAS_JAVA_HOME.  If that is not set, we fall back to the
previous behavior looking for JAVA_HOME and then checking the path.
>    -
> Note - I did remove the spurious whitespace.  So far I have not figured out how to update
the diff.  It's kind of trickly since I only want the commits since the last diff generated
for ATLAS-693.
> 
> 
> Diffs
> -----
> 
>   addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
>   addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
>   addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java ac60294e328835ba0340e150799ddfb348ccdb52

>   addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
>   addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
>   addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
>   authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
>   catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java c2eabf70de57f2893a4c30a045fa8f10aaf0b837

>   catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 838d4c16579efcc2519d668149610edfe78f83c5

>   distro/pom.xml 22f70ae970d3ee4ce3d56d15f0e9df5a3fa21b94 
>   distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
>   distro/src/conf/atlas-application.properties bb12538197ea47aad0cad013529c6821e1b1a766

>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 71b577b5c2e8aee8fec816b9224672ebc2cad5b3

>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
e7da1c9a5075004973ba68b8a36a2c95756e0e62 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 3c41693e531eb523f04a5c5781f1f123ab38ef07

>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6

>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
PRE-CREATION 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
2776b2242c4469dcae344616ccfe100633a79432 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
PRE-CREATION 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasSchemaViolationException.java
fda83b807f3640c04a660001941376f41be6d9be 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java e027b6962cf27a8c2dfbbf6fcc9683fa19e00ae5

>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
fd20a6524c4795b4212599c2f4ced69c2bbf14c0 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java a608eb27921d4be44e639f773ace818115f39ff2

>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GremlinVersion.java e817cba200c6d3c4c593b242cb2317e141bab570

>   graphdb/pom.xml 666fe5a01e55d7b761d391e3ef6172f769488254 
>   graphdb/titan0/pom.xml PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Edge.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Element.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphIndex.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphQuery.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0IndexQuery.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0VertexQuery.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/utils/EdgeToAtlasEdgeFunction.java PRE-CREATION

>   graphdb/titan0/src/main/java/org/apache/atlas/utils/VertexToAtlasVertexFunction.java
PRE-CREATION 
>   graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseTest.java
PRE-CREATION 
>   graphdb/titan0/src/test/resources/atlas-application.properties PRE-CREATION 
>   pom.xml 6013cdc64a4c04c97d9c6b3e6b4fc10b9debd0ff 
>   repository/pom.xml 533e48b8dbf8f1d80903b804ddc7da9818fe6702 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java b9689f4eb408819e91e8184d40534e983bf5d048

>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 15201dd45913539b9b3bd430b46ce6cf30c327ac

>   repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java 39dde2af0aa86d7c9a3dc5f006c8e1660cd29d5c

>   repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
90718edfd261c9f30ecc6b8f2424c232fbc9a66b 
>   repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java
f97b83d1b302dedc12c566a8bb4729d4cd68de75 
>   repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b64fa5d38c8025e6faf86335864f77d1439

>   repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 43e9f85163e8bb453601ca502d9e7a32c757b037

>   repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java d905c012a58ae4f53012f217253bf2495550da33

>   repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java
PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 91f9bd008a6939dfe78656f5c1845637a30ee9eb

>   repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java 37cacb05e65d6b1f537bd208208c8c4b9b501645

>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
0d82d900bd750b1ca91c24623384fb2547790efe 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
c385df35c0ed37490033601487bf59eaf7dc677f 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 4f6d0118072380342f5ea302d0aaf7902784849a

>   repository/src/main/java/org/apache/atlas/repository/graph/GraphProvider.java f89bdf5c7e8596b11dafced6700d0a4245fd32cf

>   repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java
6141927eb92bc7e681b43118fbaa399ada6c81f8 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
e240fb68ed9547ef6ef4cb773a12b2594dcffa09 
>   repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java 36367913252b59f7adf7f42924a886365e60819f

>   repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java 25aa7c5844cd3fc39bd7cc9ee23a6c336ca1bfac

>   repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java
2cc1a50e2ea139c3a462c918b94945fff602cf17 
>   repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
5ed9e02eb85ec25b63540cab40d148914d410786 
>   repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java c1af0f691e9d26a3102fece20acc904bb85702d6

>   repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala c4621cd509ae049b30a08a9e5b3ace03f888a10c

>   repository/src/main/scala/org/apache/atlas/query/Expressions.scala 240575087468632c52512aea94759f8ace3943a0

>   repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala 34d101a2704e28a46426fdf6b67e7c36fde0507a

>   repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala 7de03c32edf16c1676db94179266e02928e01f35

>   repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 73981c04d410a7292de357d9f63b6f3b3d241ffa

>   repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 60b57d9a4d0181cd1e3c45a4e6a928d15ed499db

>   repository/src/main/scala/org/apache/atlas/query/QueryProcessor.scala 0d2a908201f526aa487dd4cad926d308f53a04ba

>   repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 5a64c53bdb94174bf5699d9e034960f42450f696

>   repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java d1f9430124ab4071971950bf4acd2f09dd000da0

>   repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 4195955051d842af973d3ad29602fb97c1e54ecd

>   repository/src/test/java/org/apache/atlas/TestUtils.java 345e874942cf047ae8b6467f2981c7cf2317707c

>   repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
5e7de88559dbbf2460c819ba7470a2cc287ca8dc 
>   repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
f6994048b7f92ba086a1b0949030897d1cacff41 
>   repository/src/test/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepositoryTest.java
677eb392c4aa7674e27c48f86a44e953c80144c7 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
449e066a167ba4546b118a77c8e3de5fd99f077b 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
2d1c33a9bef5dc9179225fa7ce7228eebc5fdf2b 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
cc6026461bcd38b83dde0bcec2e835f5824e894f 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
90bb635126a02cd38c045bd5332499c4481d3751 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerMockTest.java
94dd29862cd37fd784a699fb45640e36992cff88 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexerTest.java
3d9d45a1f6695a0ee1af8c42fb24dcfa41dfba79 
>   repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
2f02ae28b2947ce7f7955c00bf9ca88ab3ca607a 
>   repository/src/test/java/org/apache/atlas/repository/graph/TitanGraphProviderTest.java
6fc700823800eaa943a34afecb2c2e6fba50637a 
>   repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
961442bbcaff86b472a876e6c348aa9e65c1efca 
>   repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java e6dd23081715458471d50f44f45ccd8374cbffda

>   repository/src/test/java/org/apache/atlas/services/DependencyTreeNode.java PRE-CREATION

>   repository/src/test/java/org/apache/atlas/services/JSONImporter.java PRE-CREATION 
>   repository/src/test/resources/hive-instances.json PRE-CREATION 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala b23c0f60ace59fa15886c94f35e35d3113872783

>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala f65cedbf44f5fa9e674833a20aae8de6d2537834

>   repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala 67ce12bf31dd6decf1a1323924303a066f90a4c1

>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 0e0ac8633a256dc7c8966c2b8d52eb942b52e1c6

>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala b5faaf33cd5fd1a63bac4fd41d67588ef519dc11

>   repository/src/test/titan0-resources/atlas-application.properties PRE-CREATION 
>   repository/src/test/titan1-resources/atlas-application.properties PRE-CREATION 
>   titan/pom.xml 09006d9dd5e21aa24f53524aa42a3fcd6f3d9fe6 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/AdminMask.java  
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/ConnectionMask.java 

>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseAdmin0_98.java 

>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseAdmin1_0.java  
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat.java  
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat0_98.java
 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat1_0.java 

>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompat1_1.java 

>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseCompatLoader.java
 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseKeyColumnValueStore.java
 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseStoreManager.java
 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseTransaction.java
 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HConnection0_98.java
 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HConnection1_0.java 

>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable0_98.java  
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable1_0.java  
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/TableMask.java  
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/locking/LocalLockMediator.java
 
>   titan/src/main/java/com/thinkaurelius/titan/diskstorage/solr/Solr5Index.java  
>   titan/src/test/java/com/thinkaurelius/titan/diskstorage/locking/LocalLockMediatorTest.java
 
>   typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java
6fb2087a7cbeb6eefb0dd15b9a3c0c049b6b857b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java dc9cdf2567b45b35b324ecfbb5afcc7921409b8f

>   typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDependencySorter.java
aaec05c8c3913b6974332668ebe5af31332cd121 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/Multiplicity.java 06da32e807aa4b8e4d1f49b3929c9f3c9a8d4846

>   typesystem/src/main/resources/atlas-application.properties 3318ff306a8b8ab5e320129a8450d144215f009f

>   typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
5618938e4bd9f9ef934e89bf2bf3199073294706 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/ClassTest.java daecdd78c952747964885d4cb8cdfb59727ffee5

>   typesystem/src/test/java/org/apache/atlas/typesystem/types/HierarchicalTypeDependencySorterTest.java
19bdccfc6baf503f3144fce8fe0b85914693d054 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java 574e0f9811d195cd34e12a9f23b999568d54a1c7

>   typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala
cfd4bdb26482c187c4e80f110b7349df8b524e6e 
>   webapp/pom.xml 5b023ebfbd4f77f80b0ef676e0a160dda6576f5e 
>   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 010fa2aa979e12063a1d78c8db4ad7519c3ca973

>   webapp/src/main/java/org/apache/atlas/web/resources/RexsterGraphResource.java 150f913d881138c7afeb87881197ee0f1e7d3194

>   webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 08bb125241012b6a1c1852efc6443cc7a4ebecc3

> 
> Diff: https://reviews.apache.org/r/47810/diff/
> 
> 
> Testing
> -------
> 
> Built entire Atlas project, ran all unit/integration tests.  No issues found.
> 
> 
> Thanks,
> 
> Jeff Hagelberg
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message