drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "abdelhakim deneche" <adene...@gmail.com>
Subject Re: Review Request 34541: DRILL-3147: tpcds-sf1-parquet query 73 causes memory leak
Date Thu, 21 May 2015 16:30:42 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34541/
-----------------------------------------------------------

(Updated May 21, 2015, 4:30 p.m.)


Review request for drill, Chris Westin and Jacques Nadeau.


Changes
-------

When DataServer.send() passes the ownership of the buffer to the fragment's allocator it will
reset the reference count.
FragmentContext.close() no longer needs to wait for the rpc layer to release the buffer


Bugs: DRILL-3147
    https://issues.apache.org/jira/browse/DRILL-3147


Repository: drill-git


Description
-------

- FragmentContext.close() waits 100ms before closing the allocator to give enough time to
the rpc layer to properly release any batch that was just transfered to this fragment's allocator
- each time a fragment A sends a "receiver finished" to fragment B, fragment B id will be
added to FragmentContext.ignoredSenders list
- refactored UnorderedReceiverBatch.informSenders() and MergingRecordBatch.informSenders()
by moving this method to FragmentContext
- DataServer.send() uses FragmentContext.ignoredSenders to decide if a batch should be passed
to the fragment or discarded right away
- BaseRawBatchBuffer methods enqueue() and kill() are now synchronized
- TestTpcdsSf1Leak test reproduces the leak, it's ignored by default because it requires a
large dataset


Diffs (updated)
-----

  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java 1cbe886 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java
baf9bda 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java
684f715 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataServer.java 80d2d6e 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/BaseRawBatchBuffer.java 11b6cc8

  exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/RootFragmentManager.java
b770a33 
  exec/java-exec/src/test/java/org/apache/drill/exec/server/TestTpcdsSf1Leaks.java PRE-CREATION


Diff: https://reviews.apache.org/r/34541/diff/


Testing
-------

unit tests and tpch100 are passing. Couldn't run functional yet


Thanks,

abdelhakim deneche


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message