impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (JIRA)" <>
Subject [jira] [Created] (IMPALA-5706) Parallelise read I/O in sorter
Date Mon, 24 Jul 2017 18:04:00 GMT
Tim Armstrong created IMPALA-5706:

             Summary: Parallelise read I/O in sorter
                 Key: IMPALA-5706
             Project: IMPALA
          Issue Type: Sub-task
          Components: Backend
    Affects Versions: Impala 2.10.0
            Reporter: Tim Armstrong

IMPALA-3200 offers an opportunity to improve the spilling sort algorithm:

* Use the reliability of reservations to select the most efficient order to conduct merges
in (rather than greedily trying to maximise the fan-in of the current merge). We want to minimise
the depth of the merge tree, then structure the tree based on the preferred fan-in.
* Do multiple-buffering of the stream being written (this happens automatically if there are
free buffers in the BufferPool client).
* Do multiple-buffering of the streams being read, instead of blocking on read I/O frequently.

This message was sent by Atlassian JIRA

View raw message