impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Internal Jenkins (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-2.5.0_5.7.0) IMPALA-3106: Don't allow scanner threads to spin before ranges are issued
Date Wed, 02 Mar 2016 06:08:57 GMT
Internal Jenkins has submitted this change and it was merged.

Change subject: IMPALA-3106: Don't allow scanner threads to spin before ranges are issued
......................................................................


IMPALA-3106: Don't allow scanner threads to spin before ranges are issued

Scanner threads are started in HdfsScanNode::Open(), before the actual
scan ranges are issued in GetNext(). If there is a delay between these
methods, the scanner threads will spin on DiskIoMgr::GetNextRange()
which returns immediately thinking that all the scan ranges are
finished (rather than not yet issued).

This patch works around the issue by forcing scanner threads to wait
until the initial ranges are issued. That wait is limited to 20ms, after
which the thread will wake up to check if it should yield based on
contention for thread tokens; otherwise it waits again.

Fixing this drops CPU usage to nearly 0 during a large delay between
GetNext() and Open() (e.g. by artificially delaying runtime filter
delivery), where before nearly one CPU per scanner thread was consumed.

Change-Id: I064de1ae037b578c70d65503895a920461af877a
Reviewed-on: http://gerrit.cloudera.org:8080/2379
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: Internal Jenkins
---
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/util/counting-barrier.h
3 files changed, 32 insertions(+), 4 deletions(-)

Approvals:
  Marcel Kornacker: Looks good to me, approved
  Internal Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/2379
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I064de1ae037b578c70d65503895a920461af877a
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.5.0_5.7.0
Gerrit-Owner: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Marcel Kornacker <marcel@cloudera.com>
Gerrit-Reviewer: Skye Wanderman-Milne <skye@cloudera.com>

Mime
View raw message