atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Yamijala <yhema...@gmail.com>
Subject Re: Review Request 47382: ATLAS-491 : Business Catalog API
Date Tue, 17 May 2016 08:32:54 GMT


> On May 16, 2016, 6:59 a.m., Hemanth Yamijala wrote:
> > catalog/src/main/java/org/apache/atlas/catalog/query/AtlasEntityQuery.java, line
37
> > <https://reviews.apache.org/r/47382/diff/1/?file=1383456#file1383456line37>
> >
> >     If I understood this correctly, the basepipe for the Gremlin search for entities
begins by trying to identify all entity vertices (by virtue of using the full text search
property that is common to all vertices). This would still be a huge result set to begin with,
no?
> 
> John Speidel wrote:
>     This is currently the only way to identify an entity vertex with the current database
model.  It is a requirement to be able to search all entity types and to be able to apply
tags to entities across types. So, we need to be able to get all of the entity vertices, which
is the set that would be returned from the base query. The biggest issue with using this field
to determine the entity set is that it isn't indexed.  Also, the query starts with V(), which
is all vertices in the graph but this doesn't result in all vertices being returned but instead
adds them to the pipeline interator.  
>     
>     That being said, there are lots of improvements that can be made to improve query
performance.  This would include, changes to the database model so that we knew the propery
names ahead of time so that we could use an index instead of a filter.  Also, one optimization
that I was going to add which didn't make it in was to use the type if it was provided so
that indexes could be used as mentioned above.  For now, I don't think that we can improve
on this.

I thought we do index the ENTITY_TEXT_PROPERTY_KEY. This is done in GraphBackedSearchIndexer.createFullTextIndex.
Please see comment below on search and we can probably get this in and open follow-up JIRAs
to fix.


- Hemanth


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47382/#review133332
-----------------------------------------------------------


On May 14, 2016, 12:58 a.m., John Speidel wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47382/
> -----------------------------------------------------------
> 
> (Updated May 14, 2016, 12:58 a.m.)
> 
> 
> Review request for atlas and Hemanth Yamijala.
> 
> 
> Bugs: ATLAS-491
>     https://issues.apache.org/jira/browse/ATLAS-491
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Initial implementation of Catalog API.
> 
> Known Issues/Limitations:
> - No update/delete support for taxonomies/terms
>   -- this will be added next week
> - Some gaps in test coverage
>   - Need more in-depth test coverage that exercise the generated gremlin pipeline
> - some refactoring should still be done espcecially in the 'query' package
> - need to add more logging
> - need to finish some javadoc
> 
> 
> Diffs
> -----
> 
>   catalog/pom.xml PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/AtlasTypeSystem.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/BaseRequest.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/BaseResourceProvider.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/CollectionRequest.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/DefaultDateFormatter.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/DefaultPropertyMapper.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/EntityResourceProvider.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/EntityTagResourceProvider.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/InstanceRequest.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/JsonSerializer.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/PropertyMapper.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/PropertyValueFormatter.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/Request.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/ResourceComparator.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/ResourceProvider.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/Result.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/TaxonomyResourceProvider.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/TermPath.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/TermResourceProvider.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/TermVertexWrapper.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/VertexWrapper.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/definition/BaseResourceDefinition.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/definition/EntityResourceDefinition.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/definition/EntityTagResourceDefinition.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/definition/ResourceDefinition.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/definition/TaxonomyResourceDefinition.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/definition/TermResourceDefinition.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/exception/CatalogException.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/exception/CatalogRuntimeException.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/exception/InvalidPayloadException.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/exception/InvalidQueryException.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/exception/ResourceAlreadyExistsException.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/exception/ResourceNotFoundException.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/projection/GenericRelation.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/projection/Projection.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/projection/ProjectionResult.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/projection/Relation.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/projection/RelationProjection.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/projection/RelationSet.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/projection/TagRelation.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/projection/TraitRelation.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/AlwaysQueryExpression.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/AtlasEntityQuery.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/AtlasEntityTagQuery.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/AtlasQuery.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/query/AtlasTaxonomyQuery.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/AtlasTermQuery.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQueryExpression.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/BooleanQueryExpression.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/PrefixQueryExpression.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/ProjectionQueryExpression.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/query/QueryExpression.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/QueryFactory.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/RegexQueryExpression.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/TermQueryExpression.java PRE-CREATION

>   catalog/src/main/java/org/apache/atlas/catalog/query/TermRangeQueryExpression.java
PRE-CREATION 
>   catalog/src/main/java/org/apache/atlas/catalog/query/WildcardQueryExpression.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/CollectionRequestTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/DefaultDateFormatterTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/DefaultPropertyMapperTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/EntityResourceProviderTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/EntityTagResourceProviderTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/InstanceRequestTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/JsonSerializerTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/ResourceComparatorTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/TaxonomyResourceProviderTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/TermResourceProviderTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/VertexWrapperTest.java PRE-CREATION

>   catalog/src/test/java/org/apache/atlas/catalog/definition/EntityResourceDefinitionTest.java
PRE-CREATION 
>   catalog/src/test/java/org/apache/atlas/catalog/definition/EntityTagResourceDefinitionTest.java
PRE-CREATION 
>   catalog/src/test/java/org/apache/atlas/catalog/definition/TaxonomyResourceDefinitionTest.java
PRE-CREATION 
>   catalog/src/test/java/org/apache/atlas/catalog/definition/TermResourceDefinitionTest.java
PRE-CREATION 
>   catalog/src/test/java/org/apache/atlas/catalog/query/AlwaysQueryExpressionTest.java
PRE-CREATION 
>   catalog/src/test/java/org/apache/atlas/catalog/query/QueryFactoryTest.java PRE-CREATION

>   pom.xml 5e2871e 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java cccafc2

>   repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 5195cbe

>   server-api/src/main/java/org/apache/atlas/services/MetadataService.java 13d20d8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java
6fb2087 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypedStructHandler.java
b97669a 
>   webapp/pom.xml de48c15 
>   webapp/src/main/java/org/apache/atlas/web/resources/BaseService.java PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/resources/CatalogExceptionMapper.java PRE-CREATION

>   webapp/src/main/java/org/apache/atlas/web/resources/CatalogRuntimeExceptionMapper.java
PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityService.java PRE-CREATION

>   webapp/src/main/java/org/apache/atlas/web/resources/TaxonomyService.java PRE-CREATION

>   webapp/src/main/webapp/WEB-INF/web.xml f0b606e 
> 
> Diff: https://reviews.apache.org/r/47382/diff/
> 
> 
> Testing
> -------
> 
> Ran all existing unit tests and added new tests.
> 
> 
> Thanks,
> 
> John Speidel
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message