atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <>
Subject [jira] [Commented] (ATLAS-512) Decouple currently integrating components from availability of Atlas service for raising metadata events
Date Mon, 29 Feb 2016 11:44:18 GMT


Hemanth Yamijala commented on ATLAS-512:

Uploaded a patch for review here:
Copying the description here for reference. Let's take code related conversation on the review

The changes are as follows:

* Introduces a build step that generates the JSON definitions of the reserved models for Hive,
Sqoop, Falcon and Storm and copies them into the binary distribution into a well defined folder
relative to ATLAS_HOME.
* Modifies the startup of Atlas server to load the JSON definitions and create the types during
load of DefaultMetadataService.
* Removes the hooks making calls to register types (thus removing the dependency on Atlas
service being available - which is the core issue being addressed).
* Implements unit tests for testing the functionality.

One change not implemented is the clean-up of dependencies on Hooks for the Atlas components.
This seems a bit more involved than simply removing the dependencies. I will take that up
in a separate JIRA.

> Decouple currently integrating components  from availability of Atlas service for raising
metadata events
> ---------------------------------------------------------------------------------------------------------
>                 Key: ATLAS-512
>                 URL:
>             Project: Atlas
>          Issue Type: Sub-task
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
> The components that currently integrate with Atlas (Hive, Sqoop, Falcon, Storm) all communicate
their metadata events using Kafka as a messaging layer. This effectively decouples these components
from the Atlas server. 
> However, all of these components have some initialization that checks if their respective
models are registered with Atlas. For components that integrate on the server, like HiveServer2
and Falcon, this initialization is a one time check and hence, is manageable. Others like
Sqoop, Storm and the Hive CLI are client side components and hence the initialization happens
for every run or session of these components. Invoking the initialization (and the one time
check) every time like this effectively means that the Atlas server should be always available.
> This JIRA is to try and remove this dependency and thus truly decouple these components.

This message was sent by Atlassian JIRA

View raw message