From user-return-59333-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Jan 10 08:31:06 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id DD68A18062E for ; Wed, 10 Jan 2018 08:31:05 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id CDA69160C2E; Wed, 10 Jan 2018 07:31:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C4CF3160C23 for ; Wed, 10 Jan 2018 08:31:04 +0100 (CET) Received: (qmail 94246 invoked by uid 500); 10 Jan 2018 07:31:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94235 invoked by uid 99); 10 Jan 2018 07:31:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Jan 2018 07:31:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 67118C100A for ; Wed, 10 Jan 2018 07:31:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.979 X-Spam-Level: * X-Spam-Status: No, score=1.979 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=scylladb-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Qu8dLv6-b4UC for ; Wed, 10 Jan 2018 07:30:58 +0000 (UTC) Received: from mail-qt0-f170.google.com (mail-qt0-f170.google.com [209.85.216.170]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7D4165FAEA for ; Wed, 10 Jan 2018 07:30:57 +0000 (UTC) Received: by mail-qt0-f170.google.com with SMTP id g10so20902746qtj.12 for ; Tue, 09 Jan 2018 23:30:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scylladb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=e1dQV5iBV2HnfR6mZwWgSUOAzrUGBTeLGgCqqRJpH6g=; b=k0/yThhWbSkKFAYE5CU2/SI1SUOaNEiMisR+E7bsc80S3rrMbervbAnQCFCKh2qbMM TqWAkhVWmTx+J5Qi7u3zsZAsD02lKJEv9GYzPTHEZhDumTSVR00K89cZVsUQIe4BvQ5L Ky8Wgmwt5LpjtxJ9zRDttQPA0Jk3vaoQ0tpTx1TWtNP7m6JRCuQUqO9f4KEvm7ltLxgU FzfqwKXy6cOXs6PGfXnhoeMf6OielTARQ/lzr271IW3cTqyPkcCRgxWTOTM8fzTs3cjI jIFAeT9q9CvsLCFXSttCJRJG4RBHrRvVnfptPepzJceew85lTwk8uanPS7EDGd2noa7Z eIGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=e1dQV5iBV2HnfR6mZwWgSUOAzrUGBTeLGgCqqRJpH6g=; b=rT6IurkxMwhqomxcK1Fhav9FkLCPLI25CtKPh/7Y0J57U5gGMCdVI7RKlxvvIjvfiE J4exxtEFKJi046HR6GZK/RxkbSpDF5+UliAp3gg9M84Fgg7OTR1YKPAfabm76pQ7vqQR w63cOAdtDWfGF//JjMDakbY3sCd7ifEm1Sayjuqp20Ky4CLU6EDEkr0hiWXhzODmQNmu gedV4b7kA7fdSri2p1EQf8IuTeahYCTuqtIA5vvhBJDAeP/yE2bOvjRWVVVivacuwDRX u2oed/tDirMuTMtZkA+cXRZ2KYIVxRi8znO8MLuwBszMjLIf0V0nD15UAkab9YQX2WEP LDOA== X-Gm-Message-State: AKwxyteff/jD8hWVQ9vi50C+RfelECu0Ld//cUz+PPVc1Usur652KQnc g8rfB2tWCN9v+4cqNCn3NMJEfMi40sAsu/Gd6U9fkwe+/to= X-Google-Smtp-Source: ACJfBoshi+E5ovnuTskcZrQ/jtHinVYNFEkcEtLKVLeGfZatny1D1173zCYsWoombx8d8aktHBqRfQpAhETqWxPFqWY= X-Received: by 10.237.55.226 with SMTP id j89mr23657261qtb.245.1515569456039; Tue, 09 Jan 2018 23:30:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.200.47.164 with HTTP; Tue, 9 Jan 2018 23:30:15 -0800 (PST) In-Reply-To: References: From: Dor Laor Date: Tue, 9 Jan 2018 23:30:15 -0800 Message-ID: Subject: Re: Meltdown/Spectre Linux patch - Performance impact on Cassandra? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="001a113946a40f5b52056267035c" --001a113946a40f5b52056267035c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hard to tell from the first 10 google search results which Intel CPUs has it so I went to ask my /proc/cpuinfo, turns out my >1 year Dell XPS laptop has it. AWS's i3 has it too. flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp On Tue, Jan 9, 2018 at 11:19 PM, daemeon reiydelle wrote: > Good luck with that. Pcid out since mid 2017 as I recall? > > > Daemeon (D=C3=A6m=C5=93n) Reiydelle > USA 1.415.501.0198 <(415)%20501-0198> > > On Jan 9, 2018 10:31 AM, "Dor Laor" wrote: > > Make sure you pick instances with PCID cpu capability, their TLB overhead > flush > overhead is much smaller > > On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas < > thomas.steinmaurer@dynatrace.com> wrote: > >> Quick follow up. >> >> >> >> Others in AWS reporting/seeing something similar, e.g.: >> https://twitter.com/BenBromhead/status/950245250504601600 >> >> >> >> So, while we have seen an relative CPU increase of ~ 50% since Jan 4, >> 2018, we now also have applied a kernel update at OS/VM level on a singl= e >> node (loadtest and not production though), thus more or less double patc= hed >> now. Additional CPU impact by OS/VM level kernel patching is more or les= s negligible, >> so looks highly Hypervisor related. >> >> >> >> Regards, >> >> Thomas >> >> >> >> *From:* Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com] >> *Sent:* Freitag, 05. J=C3=A4nner 2018 12:09 >> *To:* user@cassandra.apache.org >> *Subject:* Meltdown/Spectre Linux patch - Performance impact on >> Cassandra? >> >> >> >> Hello, >> >> >> >> has anybody already some experience/results if a patched Linux kernel >> regarding Meltdown/Spectre is affecting performance of Cassandra negativ= ely? >> >> >> >> In production, all nodes running in AWS with m4.xlarge, we see up to a >> 50% relative (e.g. AVG CPU from 40% =3D> 60%) CPU increase since Jan 4, = 2018, >> most likely correlating with Amazon finished patching the underlying >> Hypervisor infrastructure =E2=80=A6 >> >> >> >> Anybody else seeing a similar CPU increase? >> >> >> >> Thanks, >> >> Thomas >> >> >> >> The contents of this e-mail are intended for the named addressee only. I= t >> contains information that may be confidential. Unless you are the named >> addressee or an authorized designee, you may not copy or use it, or >> disclose it to anyone else. If you received it in error please notify us >> immediately and then destroy it. Dynatrace Austria GmbH (registration >> number FN 91482h) is a company registered in Linz whose registered offic= e >> is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 313 >> >> The contents of this e-mail are intended for the named addressee only. I= t >> contains information that may be confidential. Unless you are the named >> addressee or an authorized designee, you may not copy or use it, or >> disclose it to anyone else. If you received it in error please notify us >> immediately and then destroy it. Dynatrace Austria GmbH (registration >> number FN 91482h) is a company registered in Linz whose registered offic= e >> is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 313 >> >> > > > --001a113946a40f5b52056267035c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hard to tell from the first 10 google search results which= Intel CPUs
has it so I went to ask my /proc/cpuinfo, turns out my >= 1 year Dell XPS
laptop has it. AWS's i3 has it too.

flags : fpu vm= e de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush d= ts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constan= t_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid a= perfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ss= se3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_dead= line_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault = epb invpcid_single pti intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsb= ase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflush= opt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify h= wp_act_window hwp_epp


On Tue, Jan 9, 2018 at 11:19 PM, daeme= on reiydelle <daemeonr@gmail.com> wrote:
Good luck with that. Pcid out since = mid 2017 as I recall?=C2=A0

=
Daemeon (D=C3=A6m=C5=93n) Reiydelle
USA 1.415.501.0198

On Jan 9, 2018 10:31 AM, "Dor Laor" <dor@scylladb.com> wrote:
Make sure you pick instances with PCID cpu capability, their TLB overhead = flush
overhead is much smaller

On Tue, Jan 9, 2018 at 2:04 AM, Steinmaurer, Thomas &l= t;tho= mas.steinmaurer@dynatrace.com> wrote:

Quick follow up.

= =C2=A0

Others = in AWS reporting/seeing something similar, e.g.: https://twitter.com/BenBromhead/status/950245250504601600

= =C2=A0

So, whi= le we have seen an relative CPU increase of ~ 50% since Jan 4, 2018, we now= also have applied a kernel update at OS/VM level on a single node (loadtes= t and not production though), thus more or less double patched now. Additional CPU impact by OS/VM level kernel pa= tching is more or less negligible, so looks hi= ghly Hypervisor related.

= =C2=A0

Regards= ,

Thomas<= /span>

=C2=A0

From: Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com]
Sent: Freitag, 05. J=C3=A4nner 2018 12:09
To: u= ser@cassandra.apache.org
Subject: Meltdown/Spectre Linux patch - Performance impact on Cassan= dra?

=C2=A0

Hello,

=C2=A0

has anybody already some experi= ence/results if a patched Linux kernel regarding Meltdown/Spectre is affect= ing performance of Cassandra negatively?

=C2=A0

In production, all nodes runnin= g in AWS with m4.xlarge, we see up to a 50% relative (e.g. AVG CPU from 40%= =3D> 60%) CPU increase since Jan 4, 2018, most likely correlating with = Amazon finished patching the underlying Hypervisor infrastructure =E2=80=A6

=C2=A0

Anybody else seeing a similar C= PU increase?

=C2=A0

Thanks,

Thomas

=C2=A0

The contents of this e-mail are i= ntended for the named addressee only. It contains information that may be c= onfidential. Unless you are the named addressee or an authorized designee, you may not copy or use= it, or disclose it to anyone else. If you received it in error please noti= fy us immediately and then destroy it. Dynatrace Austria GmbH (registration= number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist=C3= =A4dterstra=C3=9Fe 313

The contents of this e-mail are intended for the named addressee only. It c= ontains information that may be confidential. Unless you are the named addr= essee or an authorized designee, you may not copy or use it, or disclose it= to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Au= stria GmbH (registration number FN 91482h) is a company registered in Linz = whose registered office is at 4040 Linz, Austria, Freist=C3=A4dterstra=C3=9Fe 313=



--001a113946a40f5b52056267035c--