impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Behm (JIRA)" <>
Subject [jira] [Resolved] (IMPALA-3905) Single-threaded scan node
Date Sun, 02 Jul 2017 04:35:00 GMT


Alexander Behm resolved IMPALA-3905.
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

commit 931bf49cd90e496df6bf260ae668ec6944f0016c
Author: Alex Behm <>
Date:   Thu Mar 9 10:06:54 2017 -0800

    IMPALA-3905: HdfsScanner::GetNext() for Avro, RC, and Seq scans.
    Implements HdfsScanner::GetNext() for the Avro, RC File, and
    Sequence File scanners. Changes ProcessSplit() to repeatedly call
    GetNext() to share the core scanning code between the legacy
    ProcessSplit() interface (ProcessSplit()) and the new GetNext()
    Summary of changes:
    - Slightly change code flow for initial scan range that
      only parses the file header. The new code sets
      'only_parsing_header_' in Open() and then honors
      that flag in GetNextInternal(). Before, all the logic
      was inside ProcessSpit().
    - Replace 'finished_' with 'eos_'.
    - Add a RowBatch parameter to various functions.
    - Change Close() to free all resources when a nullptr
      RowBatch is passed.
    - Exhaustive tests passed on debug
    - Core tests passed on asan
    - TODO: Perf testing on cluster
    Change-Id: Ie18f57b0d3fe0052a8ccd361b6a5fcdf979d0669
    Reviewed-by: Alex Behm <>
    Tested-by: Impala Public Jenkins

> Single-threaded scan node
> -------------------------
>                 Key: IMPALA-3905
>                 URL:
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>    Affects Versions: Impala 2.6.0
>            Reporter: Alexander Behm
>            Assignee: Alexander Behm
>            Priority: Minor
>             Fix For: Impala 2.10.0
> Since we will multi-thread query execution at the fragment level, we should rework our
scan nodes and scanners to only use a single thread (the one that's executing the fragment).

This message was sent by Atlassian JIRA

View raw message