atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (ATLAS-512) Decouple currently integrating components from availability of Atlas service for raising metadata events
Date Thu, 10 Mar 2016 13:18:40 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189265#comment-15189265
] 

Hemanth Yamijala edited comment on ATLAS-512 at 3/10/16 1:18 PM:
-----------------------------------------------------------------

[~suma.shivaprasad], I tested this patch today on my local machine with HBase set as the backend.


The first scenario had to create the schema, register base types and then register all reserved
types. The time it took for reserved type registration was 9seconds.

{code}
2016-03-10 18:26:20,638 INFO  - [main:] ~ Creating indexes for type name=falcon_process, definition=class
org.apache.atlas.typesystem.types.ClassType (GraphBackedSearchIndexer:149)
2016-03-10 18:26:29,464 INFO  - [main:] ~ Registered types in /Users/hyamijala/software/atlas/apache-atlas-0.7-incubating-SNAPSHOT/models/storm_model.json
(ReservedTypesRegistrar:75)
{code}

The second scenario was a restart of the server, where the schema and base types were registered.
In this case, note that the reserved types will be checked and if present, not be registered
again. The time for this checks is a little over 100ms.

{code}
2016-03-10 18:33:03,471 WARN  - [main:] ~ class falcon_process already registered, ignoring
types in /Users/hyamijala/software/atlas/apache-atlas-0.7-incubating-SNAPSHOT/models/falcon_model.json
(ReservedTypesRegistrar:77)
2016-03-10 18:33:03,593 WARN  - [main:] ~ class storm_topology already registered, ignoring
types in /Users/hyamijala/software/atlas/apache-atlas-0.7-incubating-SNAPSHOT/models/storm_model.json
(ReservedTypesRegistrar:77)
{code}

Just thinking through, in a failover scenario for e.g. the second scenario is what will play
out as the reserved types will already be registered, so we add a 100ms delay due to this
step.

Hope this answers your question.


was (Author: yhemanth):
[~suma.shivaprasad], I tested this patch today on my local machine with HBase set as the backend.


The first scenario had to create the schema, register base types and then register all reserved
types. The time it took for reserved type registration was 9seconds.

{code}
2016-03-10 18:26:20,638 INFO  - [main:] ~ Creating indexes for type name=falcon_process, definition=class
org.apache.atlas.typesystem.types.ClassType (GraphBackedSearchIndexer:149)
2016-03-10 18:26:29,464 INFO  - [main:] ~ Registered types in /Users/hyamijala/software/atlas/apache-atlas-0.7-incubating-SNAPSHOT/models/storm_model.json
(ReservedTypesRegistrar:75)
{code}

The second scenario was a restart of the server, where the schema and base types were registered.
In this case, note that the reserved types will be checked and if present, not be registered
again. The time for this checks is a little over 100ms.

{code}
2016-03-10 18:26:20,638 INFO  - [main:] ~ Creating indexes for type name=falcon_process, definition=class
org.apache.atlas.typesystem.types.ClassType (GraphBackedSearchIndexer:149)
2016-03-10 18:26:29,464 INFO  - [main:] ~ Registered types in /Users/hyamijala/software/atlas/apache-atlas-0.7-incubating-SNAPSHOT/models/storm_model.json
(ReservedTypesRegistrar:75)
{code}

Just thinking through, in a failover scenario for e.g. the second scenario is what will play
out as the reserved types will already be registered, so we add a 100ms delay due to this
step.

Hope this answers your question.

> Decouple currently integrating components  from availability of Atlas service for raising
metadata events
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: ATLAS-512
>                 URL: https://issues.apache.org/jira/browse/ATLAS-512
>             Project: Atlas
>          Issue Type: Sub-task
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>         Attachments: ATLAS-512.patch
>
>
> The components that currently integrate with Atlas (Hive, Sqoop, Falcon, Storm) all communicate
their metadata events using Kafka as a messaging layer. This effectively decouples these components
from the Atlas server. 
> However, all of these components have some initialization that checks if their respective
models are registered with Atlas. For components that integrate on the server, like HiveServer2
and Falcon, this initialization is a one time check and hence, is manageable. Others like
Sqoop, Storm and the Hive CLI are client side components and hence the initialization happens
for every run or session of these components. Invoking the initialization (and the one time
check) every time like this effectively means that the Atlas server should be always available.
> This JIRA is to try and remove this dependency and thus truly decouple these components.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message