atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shwetha G S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-158) Provide Atlas Entity Change Notification
Date Wed, 04 Nov 2015 05:06:27 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988893#comment-14988893
] 

Shwetha G S commented on ATLAS-158:
-----------------------------------

Attaching the patch here as I won't have permission to update the patch on review board. Changes
done:
1. Removed dependency of repository module in notification. Moved listeners to common module
2. Added atlas.kafka.hook.group.id in distro/src/conf/application.properties - required for
atlas deployment. The consumer group id changed to 'atlas' as the value should represent the
consumer so that any debugging from kafka end can relate to the actual consumer. Also, removed
the property atlas.kafka.entities.group.id to force users to set this value
3. Replaced TraitInfo with IStruct
4. Changed EntityNotification interface. I think the new interface makes sense for the following
reasons:
   a. The entity returned IReferenceableInstance already has getTraits(). So, EntityNotification.getAllTraits()
can represent the flattened traits(including super types)
   b. Traits as map doesn't work - Assume traits t1, t2, t3. traits t2 and t3 extend t1. If
traits t2 and t3 are associated with the entity, getAllTraits() should return t1, t2, t1,
t3. So, it has to be a list and not map
   c. For trait_add/delete operation, the traits added/deleted won't be a single trait, it
will be multiple traits because of inheritance. And just the trait name is not enough as the
same super trait may be shared across multiple traits. In general, its better to just give
the whole entity and all its latest traits even in case of entity/trait add/update/delete.
This will work even for ranger as ranger maintains resource-tag mapping and they can just
replace all entity mappings.
5. Deleted AbstractNotificationConsumerTest and EntityNotificationImplTest as they weren't
compiling with the new changes. They didn't really test anything much. If you want, you can
add them back


The patch compiles and all the tests pass except for NotificationHookConsumerIT. Looks like
kafka doesn't stop/start clean. Pending are:
1. Debug NotificationHookConsumerIT failure
2. Add UT for EntityNotificationImpl.getAllTraits()
3. There are no tests at all for the end to end flow of entity notification. Please add ITs
for all operation types
4. Add documentation

I think without the ITs, its risky to commit the patch

2. 




> Provide Atlas Entity Change Notification
> ----------------------------------------
>
>                 Key: ATLAS-158
>                 URL: https://issues.apache.org/jira/browse/ATLAS-158
>             Project: Atlas
>          Issue Type: Task
>            Reporter: Tom Beerbower
>            Assignee: Tom Beerbower
>             Fix For: 0.6-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message