Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Date: Mon, 25 Jul 2016 08:01:20 +0000 (UTC)
From: "Ismael Juma (JIRA)" <jira@apache.org>
To: dev@kafka.apache.org
Message-ID: <JIRA.12990592.1468882580000.121010.1469433680736@Atlassian.JIRA>
In-Reply-To: <JIRA.12990592.1468882580000@Atlassian.JIRA>
References: <JIRA.12990592.1468882580000@Atlassian.JIRA> <JIRA.12990592.1468882580974@arcas>
Subject: [jira] [Comment Edited] (KAFKA-3973) Investigate feasibility of
 caching bytes vs. records
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 25 Jul 2016 08:01:28 -0000


    [ https://issues.apache.org/jira/browse/KAFKA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391502#comment-15391502 ] 

Ismael Juma edited comment on KAFKA-3973 at 7/25/16 8:01 AM:
-------------------------------------------------------------

Also, MemoryMeasurer has the following enum:

{code}
public static enum Guess {
        /* If instrumentation is not available, error when measuring */
        NEVER,
        /* If instrumentation is available, use it, otherwise guess the size using predefined specifications */
        FALLBACK_SPEC,
        /* If instrumentation is available, use it, otherwise guess the size using sun.misc.Unsafe */
        FALLBACK_UNSAFE,
        /* If instrumentation is available, use it, otherwise guess the size using sun.misc.Unsafe; if that is unavailable,
         * guess using predefined specifications.*/
        FALLBACK_BEST,
        /* Always guess the size of measured objects using predefined specifications*/
        ALWAYS_SPEC,
        /* Always guess the size of measured objects using sun.misc.Unsafe */
        ALWAYS_UNSAFE
    }
{code}

Which option did you test, the instrumentation one (that requires a Java agent to be configured)?


was (Author: ijuma):
Also, MemoryMeasurer has the following enum:

{code}
public static enum Guess {
        /* If instrumentation is not available, error when measuring */
        NEVER,
        /* If instrumentation is available, use it, otherwise guess the size using predefined specifications */
        FALLBACK_SPEC,
        /* If instrumentation is available, use it, otherwise guess the size using sun.misc.Unsafe */
        FALLBACK_UNSAFE,
        /* If instrumentation is available, use it, otherwise guess the size using sun.misc.Unsafe; if that is unavailable,
         * guess using predefined specifications.*/
        FALLBACK_BEST,
        /* Always guess the size of measured objects using predefined specifications*/
        ALWAYS_SPEC,
        /* Always guess the size of measured objects using sun.misc.Unsafe */
        ALWAYS_UNSAFE
    }
{code}

Which option did you test, the instrumentation one (that required a Java agent to be configured)?

> Investigate feasibility of caching bytes vs. records
> ----------------------------------------------------
>
>                 Key: KAFKA-3973
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3973
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: streams
>            Reporter: Eno Thereska
>            Assignee: Bill Bejeck
>             Fix For: 0.10.1.0
>
>         Attachments: CachingPerformanceBenchmarks.java, MemoryLRUCache.java
>
>
> Currently the cache stores and accounts for records, not bytes or objects. This investigation would be around measuring any performance overheads that come from storing bytes or objects. As an outcome we should know whether 1) we should store bytes or 2) we should store objects. 
> If we store objects, the cache still needs to know their size (so that it can know if the object fits in the allocated cache space, e.g., if the cache is 100MB and the object is 10MB, we'd have space for 10 such objects). The investigation needs to figure out how to find out the size of the object efficiently in Java.
> If we store bytes, then we are serialising an object into bytes before caching it, i.e., we take a serialisation cost. The investigation needs measure how bad this cost can be especially for the case when all objects fit in cache (and thus any extra serialisation cost would show).


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)