Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@atlas.incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============5296210490514846791=="
MIME-Version: 1.0
Subject: Re: Review Request 55813: Porting performance and stability changes
 made in 0.7 branch into master
From: Sarath Subramanian <sarath.kum4r@gmail.com>
To: Madhan Neethiraj <madhan@apache.org>
Cc: Sarath Subramanian <sarath.kum4r@gmail.com>,
 atlas <dev@atlas.incubator.apache.org>
Date: Mon, 23 Jan 2017 22:09:38 -0000
Message-ID: <20170123220938.13409.40815@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: Sarath Subramanian <noreply@reviews.apache.org>
References: <20170121021912.28415.33400@reviews.apache.org>
In-Reply-To: <20170121021912.28415.33400@reviews.apache.org>
Reply-To: Sarath Subramanian <sarath.kum4r@gmail.com>
archived-at: Mon, 23 Jan 2017 22:09:46 -0000

--===============5296210490514846791==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55813/
-----------------------------------------------------------

(Updated Jan. 23, 2017, 2:09 p.m.)


Review request for atlas, Madhan Neethiraj and Suma Shivaprasad.


Changes
-------

Implemented typeTestExpressionUsingInFilter() and made changes to use Gremlin2ExpressionFactory class


Bugs: ATLAS-1403
    https://issues.apache.org/jira/browse/ATLAS-1403


Repository: atlas


Description
-------

Currently DSL uses a fill function during Gremlin Translation to merge results by typeName and superTypeName and fill function loads the resulting vertices in memory. This causes significant memory usage and ATLAS server spends lot of time doing GC instead of useful work resulting in OOO sometimes ( when GC is not able to recover and search queries are run in parallel)
The proposal is to replace this with typeName checks along by finding all the subtypes for a given type and using an IN clause in the filter.
For eg:
Query = Person where (birthday < "1950-01-01T02:35:58.440Z") limit 40 offset 0
Optimized query
Gremlin Query = L:
{g.V.has("__typeName", T.in, ['Person','Manager']).and(_().has("Person.birthday", T.lt, -631142641560)) [0..<40].toList()}


Diffs (updated)
-----

  repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java fd5dba7 
  repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java 266f27c 
  repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java b637f90 
  repository/src/main/java/org/apache/atlas/gremlin/Gremlin2ExpressionFactory.java 41dc65f 
  repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java 3677544 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 889236c 
  repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala daef582 
  repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala a9dcdff 
  repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala ade4176 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 33513c5 

Diff: https://reviews.apache.org/r/55813/diff/


Testing
-------

Ran Unit Tests and was successful.


Thanks,

Sarath Subramanian


--===============5296210490514846791==--