kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6168) Connect Schema comparison is slow for large schemas
Date Sat, 04 Nov 2017 02:06:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16238704#comment-16238704
] 

ASF GitHub Bot commented on KAFKA-6168:
---------------------------------------

GitHub user tedyu opened a pull request:

    https://github.com/apache/kafka/pull/4176

    KAFKA-6168 Connect Schema comparison is slow for large schemas

    Re-arrange order of comparisons in equals() to evaluate non-composite fields first
    Cache hash code

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tedyu/kafka trunk

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/4176.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4176
    
----
commit 4bae62c8ad64408b20e4925977beadaa42b54619
Author: tedyu <yuzhihong@gmail.com>
Date:   2017-11-04T02:03:59Z

    KAFKA-6168 Connect Schema comparison is slow for large schemas

----


> Connect Schema comparison is slow for large schemas
> ---------------------------------------------------
>
>                 Key: KAFKA-6168
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6168
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: Randall Hauch
>            Assignee: Ted Yu
>            Priority: Critical
>         Attachments: 6168.v1.txt
>
>
> The {{ConnectSchema}} implementation computes the hash code every time its needed, and
{{equals(Object)}} is a deep equality check. This extra work can be expensive for large schemas,
especially in code like the {{AvroConverter}} (or rather {{AvroData}} in the converter) that
uses instances as keys in a hash map that then requires significant use of {{hashCode}} and
{{equals}}.
> The {{ConnectSchema}} is an immutable object and should at a minimum precompute the hash
code. Also, the order that the fields are compared in {{equals(...)}} should use the cheapest
comparisons first (e.g., the {{name}} field is one of the _last_ fields to be checked). Finally,
it might be worth considering having each instance precompute and cache a string or byte[]
representation of all fields that can be used for faster equality checking.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message