From user-return-59326-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Tue Jan 9 19:31:16 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 1688C180718 for ; Tue, 9 Jan 2018 19:31:16 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 10B54160C2D; Tue, 9 Jan 2018 18:31:16 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 09D37160C17 for ; Tue, 9 Jan 2018 19:31:14 +0100 (CET) Received: (qmail 35601 invoked by uid 500); 9 Jan 2018 18:31:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 35591 invoked by uid 99); 9 Jan 2018 18:31:13 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jan 2018 18:31:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id AE994C148F for ; Tue, 9 Jan 2018 18:31:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.979 X-Spam-Level: * X-Spam-Status: No, score=1.979 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=scylladb-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id HqVi0TV51ahr for ; Tue, 9 Jan 2018 18:31:11 +0000 (UTC) Received: from mail-qk0-f174.google.com (mail-qk0-f174.google.com [209.85.220.174]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E42135F3E2 for ; Tue, 9 Jan 2018 18:31:10 +0000 (UTC) Received: by mail-qk0-f174.google.com with SMTP id d21so8753133qkj.3 for ; Tue, 09 Jan 2018 10:31:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scylladb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=m0MjVPjjJoeUDbEsZ6OOUJDOJMRSpgoac2H1RpJ+AFo=; b=hr5kOvewAkiL1ccVJ9yRcXI3t62g2ftPOK+OAsYjpMni9GqqUAdvfdiW118CI5Y8Cz ZtGQC+i4Olfh4Zrnmr1jVmjKnl1k8xrROGCz0OBXyjvC9uSsjWe79J1Wqs3QAk8JiESt fihrN7WZ42HeCAtAuEHvvts8/21G2xWvclyB7Fh1HTqEycE5AO1Y9j4QyWfrW/P+r7Ll GoooS2qLSbhG2lqonG1SY2j6AsLoIpA/TYjhQb3s9rjT4j2rCmSixkHXCVbQXwu8NEh7 t1PaPww633lUZW5Y+0xM7OHqk0wnYjzuTkyfZG2rSJtNHrUxB8/54d/ysNsvMHFUPS3K blGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=m0MjVPjjJoeUDbEsZ6OOUJDOJMRSpgoac2H1RpJ+AFo=; b=Wg+PKqyw7RiKZIVvQVOh/DRv7RaXgA/RmyfAXJl1UgmCpPE0OlLXH3bS6jNCxyQgrK yZMbw6UOaVZRvJVt8aUz3KnJ3rSKGzJHqUDJJKbYjmQmqSHWm4eu6mBcnLabQroQ3vEG 45CoyPVsyRCoLtIOxpTLtDbe/fHsEs81j41zpS8UbrCA4UpBk2Ve/qQgBTE4JpZZnWvt zPFDqseoZQpYZoP9WQmSTzkb+2hn6DNaczcgg7l9HetuDCisNi6fjStWBrFReA/qqAzl uyXJtdH8N9hSU4GTjq5mwVSXY/R7C7a0MUCbbpjOOq+yqgC/MgdaN1Br0t8SZ4PfW6vO vapw== X-Gm-Message-State: AKwxytffgeEZOA69jcUyYpLObLG6yRuweqD1hk4w7KY6C/omP1xwr3kx Tm1bWyOe3t59BQggmRSlZ6P7UmVA6fDgdXFr6xXcKZaa X-Google-Smtp-Source: ACJfBovGZP3TJUxEsiO2wvDYjGORYM2Xx0PPUApdm+ptdQKXn70HuGmUEXgEDMyQUqKF2Z8Md9QBqChl7rYABZ0RcLA= X-Received: by 10.55.73.86 with SMTP id w83mr21673282qka.332.1515522663685; Tue, 09 Jan 2018 10:31:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.200.47.164 with HTTP; Tue, 9 Jan 2018 10:30:23 -0800 (PST) In-Reply-To: References: From: Dor Laor Date: Tue, 9 Jan 2018 10:30:23 -0800 Message-ID: Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="001a114a804204d58005625c1e98" --001a114a804204d58005625c1e98 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Make sure you pick instances with PCID cpu capability, their TLB overhead flush overhead is much smaller On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas < thomas.steinmaurer@dynatrace.com> wrote: > Quick follow up. > > > > Others in AWS reporting/seeing something similar, e.g.: > https://twitter.com/BenBromhead/status/950245250504601600 > > > > So, while we have seen an relative CPU increase of ~ 50% since Jan 4, > 2018, we now also have applied a kernel update at OS/VM level on a single > node (loadtest and not production though), thus more or less double patch= ed > now. Additional CPU impact by OS/VM level kernel patching is more or less= negligible, > so looks highly Hypervisor related. > > > > Regards, > > Thomas > > > > *From:* Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com] > *Sent:* Freitag, 05. J=C3=A4nner 2018 12:09 > *To:* user@cassandra.apache.org > *Subject:* Meltdown/Spectre Linux patch - Performance impact on Cassandra= ? > > > > Hello, > > > > has anybody already some experience/results if a patched Linux kernel > regarding Meltdown/Spectre is affecting performance of Cassandra negative= ly? > > > > In production, all nodes running in AWS with m4.xlarge, we see up to a 50= % > relative (e.g. AVG CPU from 40% =3D> 60%) CPU increase since Jan 4, 2018, > most likely correlating with Amazon finished patching the underlying > Hypervisor infrastructure =E2=80=A6 > > > > Anybody else seeing a similar CPU increase? > > > > Thanks, > > Thomas > > > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 313 > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 313 > > --001a114a804204d58005625c1e98 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Make sure you pick instances with PCID cpu capability, the= ir TLB overhead flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM= , Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com= > wrote:

<= span lang=3D"EN-US" style=3D"color:#1f497d">Quick follow up.<= /span>

= =C2=A0

Others = in AWS reporting/seeing something similar, e.g.: https://twitter.com/BenBromhe= ad/status/950245250504601600

= =C2=A0

So, whi= le we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we now= also have applied a kernel update at OS/VM level on a single node (loadtes= t and not production though), thus more or less double patched now. Additional CPU impact by OS/VM level kernel pa= tching is more or less negligible, so looks hi= ghly Hypervisor related.

= =C2=A0

Regards= ,

Thomas<= /span>

=C2=A0

From: Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com]
Sent: Freitag, 05. J=C3=A4nner 2018 12:09
To: u= ser@cassandra.apache.org
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassan= dra?

=C2=A0

Hello,

=C2=A0

has anybody already some experi= ence/results if a patched Linux kernel regarding Meltdown/Spectre is affect= ing performance of Cassandra negatively?

=C2=A0

In production, all nodes runnin= g in AWS with m4.xlarge, we see up to a 50% relative (e.g. AVG CPU from 40%= =3D> 60%) CPU increase since Jan 4, 2018, most likely correlating with = Amazon finished patching the underlying Hypervisor infrastructure =E2=80=A6

=C2=A0

Anybody else seeing a similar C= PU increase?

=C2=A0

Thanks,

Thomas

=C2=A0

The contents of this e-mail are i= ntended for the named addressee only. It contains information that may be c= onfidential. Unless you are the named addressee or an authorized designee, you may not copy or use= it, or disclose it to anyone else. If you received it in error please noti= fy us immediately and then destroy it. Dynatrace Austria GmbH (registration= number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 3= 13

The contents of this e-mail are intended for the named addressee only. It c= ontains information that may be confidential. Unless you are the named addr= essee or an authorized designee, you may not copy or use it, or disclose it= to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Au= stria GmbH (registration number FN 91482h) is a company registered in Linz = whose registered office is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 313

--001a114a804204d58005625c1e98--