flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Hogan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-4705) Instrument FixedLengthRecordSorter
Date Wed, 28 Sep 2016 18:24:20 GMT
Greg Hogan created FLINK-4705:

             Summary: Instrument FixedLengthRecordSorter
                 Key: FLINK-4705
                 URL: https://issues.apache.org/jira/browse/FLINK-4705
             Project: Flink
          Issue Type: Improvement
          Components: Local Runtime
    Affects Versions: 1.2.0
            Reporter: Greg Hogan
            Assignee: Greg Hogan

The {{NormalizedKeySorter}} sorts on the concatenation of (potentially partial) keys plus
an 8-byte pointer to the record. After sorting each pointer must be dereferenced, which is
not cache friendly.

The {{FixedLengthRecordSorter}} sorts on the concatentation of full keys followed by the remainder
of the record. The records can then be deserialized in sequence.

Instrumenting the {{FixedLengthRecordSorter}} requires implementing the comparator methods
{{writereadWithKeyNormalization}} and {{readWithKeyNormalization}}.

Testing {{JaccardIndex}} on an m4.16xlarge the scale 18 runtime dropped from 71.8 to 68.8
s (4.3% faster) and the scale 20 runtime dropped from 546.1 to 501.8 s (8.8% faster).

This message was sent by Atlassian JIRA

View raw message