drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5122) DrillBuf performs very expensive logging if JVM -ea option is set
Date Tue, 13 Dec 2016 01:04:58 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743735#comment-15743735
] 

ASF GitHub Bot commented on DRILL-5122:
---------------------------------------

GitHub user paul-rogers opened a pull request:

    https://github.com/apache/drill/pull/693

    DRILL-5122: DrillBuf performs expensive logging if -ea set

    In Drill, once assertions are enabled, the dominant cost of a query
    becomes DrillBuf logging. One example showed that DrillBuf logging
    increases query cost by 10x.
    
    Logging was enabled by the -ea option, which should be available for
    many other uses. Changed to require a specific option to be turned on.
    
    Enabled the option for memory tests (which require the option) but off
    by default and for all other tests.
    
    To do this, converted Surefire system options from command line args to
    system variables so that they can be customized in sub-project poms.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/paul-rogers/drill DRILL-5122

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #693
    
----
commit 25aa8ea844555806608f1a1e9c30202455219efa
Author: Paul Rogers <progers@maprtech.com>
Date:   2016-12-12T22:42:07Z

    DRILL-5122: DrillBuf performs expensive logging if -ea set
    
    In Drill, once assertions are enabled, the dominant cost of a query
    becomes DrillBuf logging. One example showed that DrillBuf logging
    increases query cost by 10x.
    
    Logging was enabled by the -ea option, which should be available for
    many other uses. Changed to require a specific option to be turned on.
    
    Enabled the option for memory tests (which require the option) but off
    by default and for all other tests.
    
    To do this, converted Surefire system options from command line args to
    system variables so that they can be customized in sub-project poms.

----


> DrillBuf performs very expensive logging if JVM -ea option is set
> -----------------------------------------------------------------
>
>                 Key: DRILL-5122
>                 URL: https://issues.apache.org/jira/browse/DRILL-5122
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components:  Server
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>
> The {{DrillBuf}} class is the foundation of Drill's direct memory model. It has a very
sophisticated allocator mechanism behind it. {{DrillBuf}} provides logging to help track down
memory leaks and other issues. Unfortunately, the trigger to turn on this option is based
on detecting if assertions are enabled in the JVM.
> Assertions are very handy tools: they are debug-only assertions not executed at run time.
They are turned on with the "-ea" option to the JVM. But, in production, the assertions don't
execute, avoiding production-time costs.
> In Drill, once assertions are enabled, the dominant cost of a query becomes {{DrillBuf}}
logging. One example showed that {{DrillBuf}} logging increases query cost by 10x.
> Debugging of {{DrillBuf}} is handy for those few engineers who need to do so. Over time,
{{DrillBuf}} has become quite stable so that, now, most developers, most of the time, do not
work on {{DrillBuf}}.
> One wonders: Drill uses Guava preconditions extensively. Is this perhaps due to the high
cost of enabling assertions? Since assertions were unavailable, production-time checking via
Guava may have become preferred. Assertions are, in fact, better because they incur no overhead
in production, while Guava preconditions always execute, even in production. (Here we assume
that the code errors that both check will find errors during development so that such errors
do not sneak through into production.)
> The code does have a way to turn off debugging, but only if a system property is set
on every run. Since Drill has hundreds of unit tests that people might run, this requires
creating custom run configurations for each.
> This enhancement proposes to modify {{DrillBuf}} to enable logging only when a) assertions
are enabled (so we know it is a debug environment) and b) when a boot-time configuration property
is set (so that logging can be turned off for most developers most of the time.) Optionally,
b) can be a system property that _enables_ (not disables) {{DrillBuf}} logging.
> The code in question:
> {code}
> class BaseAllocator ... {
>   public static final boolean DEBUG = AssertionUtil.isAssertionsEnabled()
>       || Boolean.parseBoolean(System.getProperty(DEBUG_ALLOCATOR, "false"));
> ...
> class DrillBuf ... {
>   public boolean release(int decrement) {
>     ...
>     if (BaseAllocator.DEBUG) {
>       historicalLog.recordEvent("release(%d). original value: %d", decrement, refCnt
+ decrement);
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message