drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cwestin <...@git.apache.org>
Subject [GitHub] drill pull request: Alloc nolocks
Date Wed, 26 Aug 2015 00:57:21 GMT
GitHub user cwestin opened a pull request:

    https://github.com/apache/drill/pull/132

    Alloc nolocks

    DRILL-1942-hygiene (another preparatory patch):
        - add AutoCloseable to some classes
        - minor fixes
        - formatting
    
    Unit tests pass
    Regression suite passed except for some window function queries (this branch needs to
get rebased to pick up that code), and the known fragment distribution timeout.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cwestin/incubator-drill alloc-nolocks

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #132
    
----
commit 51558954489e9015f604a0d017c519664eb7d55a
Author: Chris Westin <cwestin@yahoo.com>
Date:   2015-02-13T23:30:24Z

    DRILL-1942: Improve off-heap memory usage tracking
    This is a reimplementation of the allocators to make tracking better. The
    re-implementation includes support for JMX MBeans for on-the-fly memory
    statistics inspection. The new classes are BaseAllocator, RootAllocator, and
    ChildAllocator. The BufferAllocator interface has been modified a bit.
    
    In addition to that, the new allocator flattens the use of buffers in slices,
    so that a DrillBuf is always referring to an UnsafeDirectLittleEndian, and
    never to another DrillBuf; this simplifies figuring out what's going on when
    something goes wrong. The new allocator (BaseAllocator, and its derived
    classes RootAllocator and ChildAllocator) treats sharing buffers in the same
    way as slicing buffers, and sharing and ownership transfer of buffers and
    their slices both work.
    
    Record batch and value vector classes previously had a mandatory cleanup()
    method. These classes have been made to include AutoCloseable as a base class,
    and cleanup() has been renamed to close(). This was done to get additional
    help from tools to find places that we need to close() things.
    
    This allocator detects a lot more problems than the old one did; fixed many
    leaks in various locations.
    
    AbstractRecordBatch:
    - cleanup() -> close()
    - hygiene
    
    AbstractSingleRecordBatch:
    - cleanup() -> close()
    - hygiene
    
    Accounter:
    - hygiene
    - TODOs
    
    AllocatorLedger:
    - created private interface for child allocators to communicate with parents
    
    AllocationReservation:
    - created
    
    AtomicRemainder:
    - TODOs
    
    BaseAllocator:
    - created common base class for buffer allocators
    
    BaseDataValueVector
    - released the data buffer in close()
    
    BitVector
    - fixed a bug in splitAndTransferTo(), which was missing a retain() call after
      slicing a buffer for the transfer target.
    
    BootStrapContext:
    - hygiene
    
    BroadcastSenderRootExec:
    - cleanup() => close()
    - hygiene
    
    BufferAllocator:
    - replace preallocator with createReservation()
    - Closeable -> AutoCloseable, and add throws clause
      - required minor changes to several other files
        - created DrillCloseables and DrillAutoCloseables to make this easy
    - renamed the second form of BufferAllocator.takeOwnership to shareOwnership
    - clean up interface definition - no abstract needed
    - hygiene
    - some javadoc
    - TODOs
    
    ChildAllocator
    - created
    
    DataServer
    - handle failed data transfers due to cancelled closed fragments
    
    DrillAutoCloseables:
    - add comment that null argument is supported
    - fix a typo in a comment
    
    Drillbit:
    - sychhronize ShutdownThread.run()'s action against a static to avoid
      concurrent shutdown problems with metrics by serializing the shutdown
      of multiple drillbits running in the same JVM.
    
    DrillBuf:
    - add some finals
    - minor formatting
    - fix spelling mixtake
    - TODOs
    
    DrillConfig:
    - hygiene
    
    DrillRuleSets:
    - can't reuse query context from past queries
    
    ExecConstants
    - debugging property for memory to ExecConstants
    - added ALLOCATION_POLICY property name
    
    ExternalSortBatch:
    - only get allocator once, outside loop body
    - release buffers obtained from getBuffers()
    
    FakeAllocator:
    - switch from preallocator to AllocationReservation
    - private logger
    - TODOs
    
    FilterRecordBatch:
    - made some variables private
    
    FragmentContext:
    - getAllocator() shouldn't be @Deprecated
    - made logger private
    - TODOs
    
    ForemanException:
    - removed unused fromThrowable()
    - commented out unused logger
    - formatting
    
    FragmentSetupException:
    - leave out the finals
    
    HashAggTemplate:
    - cleanup() -> close()
    - hygiene
    
    MappifyUtility:
    - remove unused imports
    
    MergeJoinBatchBuilder:
    - replace preallocator with AllocationReservation
    - implement close() to clean up reservation
    
    MSortTemplate:
    - made some variables private
    
    NestedConfig:
    - hygiene
    
    OperatorContext:
    - removed unused applyFragmentLimits member
    - made logger private
    - TODOs
    
    OutOfMemoryException:
    - add serialVersionUID to silence warning
    - minor formatting
    
    OutOfMemoryRuntimeException:
    - add serialVersionUID to silence warning
    - minor formatting
    
    PartitionSenderRootExec:
    - cleanup() -> close()
    - hygiene
    
    PooledByteBufAllocatorL:
    - finals
    - formatting
    - make logger private (and then comment out because it is unused)
    
    PrintingResultsListener:
    - TODOs
    
    PriorityQueueCopierTemplate:
    - removed unnecessary use of the pre-allocator
    - made some members private
    
    PriorityQueueTemplate:
    - removed unnecessary use of preallocator
    - made some members private
    
    RecordBatch:
    - interface now extends AutoCloseable instead of declaring it's own cleanup()
    - hygiene
    - TODOs
    
    RootAllocator:
    - created replacement for TopLevelAllocator
    
    ScreenCreator:
    - clean() => close()
    - hygiene
    
    SimpleRootExec:
    - hygiene
    - fixup RecordBatch close()
    
    SingleSenderCreator:
    - cleanup() => close()
    - hygiene
    
    SortRecordBatchBuilder:
    - replace preallocation with AllocationReservation
    - implements AutoCloseable
    - implement close() to close reservation
    - clear batches list after closing items on it
    - formatting
    - hygiene
    
    StackTrace:
    - DRILL-2624: fixed numeric formatting for line numbers
    
    TestBaseAllocator:
    - created new test for allocators, unit tests them in isolation
    
    TopLevelAllocator:
    - replace preallocator with AllocationReservation implementation
    - eliminated @Deprecated constructors
      - changed many other files to use remaining constructor
    - eliminate private constructor
    - add finals, private
    - formatting
    - TODOs
    
    UnionAllRecordBatch:
    - added missing null check to clearCurrentRecordBatch()
    
    VectorAccessible:
    - hygiene (mostly comments)
    
    ExternalSortBatch:
    ExpressionInterpreterTest:
    FilterRecordBatch:
    HashAggBatch:
    HashJoinBatch:
    IteratorValidatorBatchIterator:
    LimitRecordBatch:
    MergeJoinBatch:
    MergingRecordBatch:
    OrderedPartitionRecordBatch:
    ProducerConsumerBatch:
    RemovingRecordBatch:
    ScanBatch:
    SortBatch:
    StreamingAggBatch:
    StreamingWindowFrameRecordBatch:
    TopNBatch:
    TraceRecordBatch:
    UnionAllRecordBatch:
    UnorderedReceiverBatch:
    WriterRecordBatch:
    - cleanup() -> close()
    - close builders that use AllocationReservation
    - hygiene
    
    TestAllocators
    TestSimpleFragmentRun
    TestSimpleFunctions
    TestStringFunctions
    TestLoad
    TestRepeated
    TestStringFunctions
    ... many others...
    - modify to use new allocator
    - some hygiene
    
    ValueVector and RecordWriter templates:
    - added clauses to release buffers before references are overwritten
    - eliminated common subexpressions and repeated getAccessor()/getMutator()
      calls in loops
    - hygiene

commit 169ef67ee3156cd7a3983ecc6cde5c72c86b69e5
Author: Chris Westin <cwestin@yahoo.com>
Date:   2015-07-29T20:16:16Z

    DRILL-1942: application of non-locking patch to rebased (20150729) locking implementation
(cwestin/alloc).
    - Some manual edits to handle hunks that did not apply cleanly
    - inability to allocate needs to throw an exception
      - makes TestAllocators run again
    - ensure preallocated space tracking is fully protected/transacted under
      debug mode in non-locking patch

commit e81b6f4acd914d66501657e558b6e39e41ac1ecc
Author: Chris Westin <cwestin@yahoo.com>
Date:   2015-08-19T16:35:16Z

    DRILL-1942-no-locks-retry-stats
    - collect stats on retries and report on them at shutdown

commit c6df22d361d79d3db0172f4d11fa98711f97d834
Author: Chris Westin <cwestin@yahoo.com>
Date:   2015-08-25T23:58:08Z

    DRILL-1942-no-locks-chunking:
    - add secondary chunking to allocation

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message