atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sarath Subramanian <sar...@apache.org>
Subject Re: Review Request 62874: Hive hook OOM fix
Date Tue, 10 Oct 2017 23:24:19 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62874/#review187604
-----------------------------------------------------------


Fix it, then Ship it!





addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
Line 592 (original), 579 (patched)
<https://reviews.apache.org/r/62874/#comment264630>

    we need to add split messages here => populateEventMessagesPerEntity(event, entities);


- Sarath Subramanian


On Oct. 10, 2017, 3:32 p.m., Apoorv Naik wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62874/
> -----------------------------------------------------------
> 
> (Updated Oct. 10, 2017, 3:32 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2198
>     https://issues.apache.org/jira/browse/ATLAS-2198
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> For the reported use case, Hive hook messages contains around 508 entities in a single
message.
> This causes the Base64 encoder to fail while memory allocation and the entire message
gets dropped because the hive process dies. 
> 
> This fix splits one giant message for all entity into N small messages per entity (Referenceable)
> 
> 
> Diffs
> -----
> 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java aca56454

> 
> 
> Diff: https://reviews.apache.org/r/62874/diff/1/
> 
> 
> Testing
> -------
> 
> Fired reported command via beeline and verified "view" and it's col lineage shows up
in Atlas UI.
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message