cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7030) Remove JEMallocAllocator
Date Mon, 14 Apr 2014 19:34:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968728#comment-13968728
] 

Benedict commented on CASSANDRA-7030:
-------------------------------------

[~vijay2win@yahoo.com] I think there was something wrong with your benchmark. Since it uses
a different random seed each time it's run, the end state that you check may vary a great
deal. Or possibly reporting total system memory isn't accurate in some way. As I find completely
the opposite effect: ~2.3Gb resident memory for the process working with unsafe, and ~2.7Gb
resident for JEMalloc.

{noformat}
UNSAFE:
Items: 1M
Elapsed: 4.373s
Allocated: 2046Mb
VM total:172
vsz: 3717
rsz: 2268

Items: 10M
Allocated: 2050Mb
VM total: 186
vsz: 3717
rsz: 2308

Items: 100M
Elapsed: 329.087s
Allocated: 2047Mb
VM total:186
vsz: 3717
rsz: 2308

JEMALLOC:

Items: 1M
Elapsed: 8.259s
Allocated: 2046Mb
VM total:161
vsz: 4128
rsz: 2651

Items: 10M
Allocated: 2050
VM total: 192
vsz: 4132
rsz: 2706

Items: 100M
Elapsed: 791.370s
Allocated: 2047Mb
VM total:192
vsz: 4136
rsz: 2710
{noformat}

{code}
    public static void main(String[] args) throws InterruptedException, IOException
    {
        String pid = ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
        System.out.println(pid);
        final IAllocator allocator = new JEMallocAllocator();
        final Random rand = new Random(0);
        final AtomicLong total = new AtomicLong();
        EvictionListener<UUID, Memory> listener = new EvictionListener<UUID, Memory>()
        {
            public void onEviction(UUID k, Memory mem)
            {
                total.addAndGet(-mem.size());
                mem.free(allocator);
            }
        };
        Map<UUID, Memory> map = new ConcurrentLinkedHashMap.Builder<UUID, Memory>().weigher(Weighers.<Memory>
singleton())
                                                                        .initialCapacity(8
* 65536).maximumWeightedCapacity(4 * 65536)
                                                                        .listener(listener).build();
        long start = System.nanoTime();
        byte[] keyBytes = new byte[16];
        for (int i = 0 ; i < 1000000 ; i++)
        {
            int size = rand.nextInt(128);
            if (size <= 0)
                continue;
            rand.nextBytes(keyBytes);
            Memory mem = new Memory(allocator, size * 128);
            mem.setMemory(0, mem.size(), (byte) 2);
            map.put(UUID.nameUUIDFromBytes(keyBytes), mem);
            total.addAndGet(size * 128);
        }
        long end = System.nanoTime();
        System.out.println(String.format("Elapsed: %.3fs", (end - start) * 0.000000001d));
        System.out.println(String.format("Allocated: %.0fMb", total.get() / (double) (1 <<
20)));
        System.out.println(String.format("VM total:%.0f", Runtime.getRuntime().totalMemory()
/ (double) (1 << 20)));
        memuse("vsz", pid);
        memuse("rsz", pid);
        Thread.sleep(100);
    }

    private static void memuse(String type, String pid) throws IOException
    {
        Process p = new ProcessBuilder().command("ps", "-o", type, pid).redirectErrorStream(true).start();
        BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
        reader.readLine();
        System.out.println(String.format("%s: %.0f", type, Integer.parseInt(reader.readLine())
/ 1024d));
    }
{code}

> Remove JEMallocAllocator
> ------------------------
>
>                 Key: CASSANDRA-7030
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7030
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Benedict
>            Assignee: Benedict
>            Priority: Minor
>              Labels: performance
>             Fix For: 2.1 beta2
>
>         Attachments: 7030.txt
>
>
> JEMalloc, whilst having some nice performance properties by comparison to Doug Lea's
standard malloc algorithm in principle, is pointless in practice because of the JNA cost.
In general it is around 30x more expensive to call than unsafe.allocate(); malloc does not
have a variability of response time as extreme as the JNA overhead, so using JEMalloc in Cassandra
is never a sensible idea. I doubt if custom JNI would make it worthwhile either.
> I propose removing it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message