From user-return-64538-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Fri Oct 4 10:38:35 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id BDCBA180651 for ; Fri, 4 Oct 2019 12:38:34 +0200 (CEST) Received: (qmail 4123 invoked by uid 500); 4 Oct 2019 10:38:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 4113 invoked by uid 99); 4 Oct 2019 10:38:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Oct 2019 10:38:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id EB0D91A32AF for ; Fri, 4 Oct 2019 10:38:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.11 X-Spam-Level: * X-Spam-Status: No, score=1.11 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=datastax.com header.b=dzqCZh8y; dkim=pass (1024-bit key) header.d=datastax.com header.b=rLSZGv6s Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 6O-0PYoI0x-I for ; Fri, 4 Oct 2019 10:38:26 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=148.163.151.30; helo=mx0a-002bad01.pphosted.com; envelope-from=cedrick.lunven@datastax.com; receiver= Received: from mx0a-002bad01.pphosted.com (mx0a-002bad01.pphosted.com [148.163.151.30]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 44A637E18F for ; Fri, 4 Oct 2019 10:38:25 +0000 (UTC) Received: from pps.filterd (m0121912.ppops.net [127.0.0.1]) by mx0a-002bad01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x94Aa9KA010560 for ; Fri, 4 Oct 2019 03:38:23 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datastax.com; h=mime-version : references : in-reply-to : from : date : message-id : subject : to : content-type; s=proofpoint20180122; bh=8yDxK+2Q55qs3SPJeGfVhYl6PD8Pp+d4zab4/N1szhw=; b=dzqCZh8yb4sXnAx9bsSrXzh3aOkDFQqjK1nFhu310xLtCR2FYXoaMOmSA56aFwNmo0Qh 01v9MOUSShiROa3BM9DgtGlX4y8L8+tyOfpvNXs7cUiHQt0BDL/Vd2xct9e6Hy//brWu +5sR4HTM7ix+gkwf2ZGlDTaj+g9WceS94EEhYAx/CHa3fUYGeik8Fd+dtqTMUTL9gxVi XgW99pO3+Eg5XRseFP4Jxd3pX1/Cyz+ppiOc7G9FpEyuIsF4ej9DbVqwlrIoEK5nA73Q Q0k04CN921lqfkeGcZFoxzYEO0JpuKwDONX2U+Brm6aTmxuPydnSadOgpuBqsx1sY/+z 6Q== Received: from mail-vk1-f198.google.com (mail-vk1-f198.google.com [209.85.221.198]) by mx0a-002bad01.pphosted.com with ESMTP id 2va65m862p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 04 Oct 2019 03:38:23 -0700 Received: by mail-vk1-f198.google.com with SMTP id u123so2433688vkf.8 for ; Fri, 04 Oct 2019 03:38:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datastax.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=8yDxK+2Q55qs3SPJeGfVhYl6PD8Pp+d4zab4/N1szhw=; b=rLSZGv6sASiKiZrTBcnCuq9B9egyqzSK+KekdJucwuKcPZQkqs93BIlTgFW0Ny+7eG 3hm+D4ooO6Ged3EqyLZqgla9Q0D5xTDob5emzEP9IO9db7cpeGzQcBEODXHKXMvIeUKb hamOiU6VxOkSRxHHDkS+zY1wAF8Gs41sbHEy8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=8yDxK+2Q55qs3SPJeGfVhYl6PD8Pp+d4zab4/N1szhw=; b=TAth7R4wSoV2EzfnGNaQnLKUHtbdrY2eRafRqRWWc+U/zc3i5MkqQ4ASekSVv/f+TO n8E5iIGJDj5B0MXKLQ7z3ScC+teEtTQTQ7yBuFZGF8UVR8S9vsYGz1txqOAJjNMIzWO0 QW1gxQsVpYcHcxQ774xHbny1YFe+g9qQVg47eDyXVm29D2b0AauQVxv+xQM9+sEPWFhZ 69puwPbhJdS5fGpKbhPuRZeFMwrt2E7R6fA7tzFMy4lHi/fj63oGPyMZI88HZ+M+C+fb Mj2caDuFjTddfo6NgK/lfPUYrI1plqTHY2NZt45d9eqpYv3R3ZBy6tlUeE0xt4mIAQml dhEg== X-Gm-Message-State: APjAAAViUTmVhTeJr/VJZl0yuj0uKR5/Xis1pzmWFlVGWiYnsvNeKBoO hOPI3VTzcQ68EfZkRxRX+MVrtyRVpeNUgyvD+Hdrh+ehjaDGeZ97Dz0fmtUp5BfGnw5mgEfRPZH rWWXMxKT1AmPSf5oPY64aX+sYejthxoC3zUUFeXSV X-Received: by 2002:a67:c181:: with SMTP id h1mr7048112vsj.195.1570185501928; Fri, 04 Oct 2019 03:38:21 -0700 (PDT) X-Google-Smtp-Source: APXvYqz2XoSqnae6EGPtupxNA7t2ACvdsAxw54lZ1ocsT2efKjKGhOFUOpBmLKPUhB0OJfOrb+1jEkrJ9kUiRj5tnm8= X-Received: by 2002:a67:c181:: with SMTP id h1mr7048103vsj.195.1570185501535; Fri, 04 Oct 2019 03:38:21 -0700 (PDT) MIME-Version: 1.0 References: <0D44F373-EF10-40FC-A79E-AE5C4236CD9B@gmail.com> In-Reply-To: From: Cedrick Lunven Date: Fri, 4 Oct 2019 12:38:10 +0200 Message-ID: Subject: Re: Cluster sizing for huge dataset To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="0000000000000d364c0594134d7d" X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,1.0.8 definitions=2019-10-04_06:2019-10-03,2019-10-04 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 malwarescore=0 mlxscore=0 impostorscore=0 bulkscore=0 suspectscore=1 lowpriorityscore=0 adultscore=0 clxscore=1011 mlxlogscore=999 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1908290000 definitions=main-1910040098 --0000000000000d364c0594134d7d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, If you are using DataStax Enterprise why not offloading cold data to DSEFS (HDFS implementation) with friendly analytics storage format like parquet, keep only OLTP in the Cassandra Tables. Recommended size for DSEFS can go up to 30TB a node. I am pretty sure you are already aware of this option and would be curious to get your think about this solution and limitations. Note: that would also probably help you with your init-load/TWCS issue . My2c. Cedrick On Tue, Oct 1, 2019 at 11:49 PM DuyHai Doan wrote: > The client wants to be able to access cold data (2 years old) in the > same cluster so moving data to another system is not possible > > However, since we're using Datastax Enterprise, we can leverage Tiered > Storage and store old data on Spinning Disks to save on hardware > > Regards > > On Tue, Oct 1, 2019 at 9:47 AM Julien Laurenceau > wrote: > > > > Hi, > > Depending on the use case, you may also consider storage tiering with > fresh data on hot-tier (Cassandra) and older data on cold-tier > (Spark/Parquet or Presto/Parquet). It would be a lot more complex, but ma= y > fit more appropriately the budget and you may reuse some tech already > present in your environment. > > You may even do subsampling during the transformation offloading data > from Cassandra in order to keep one point out of 10 for older data if > subsampling makes sense for your data signal. > > > > Regards > > Julien > > > > Le lun. 30 sept. 2019 =C3=A0 22:03, DuyHai Doan = a > =C3=A9crit : > >> > >> Thanks all for your reply > >> > >> The target deployment is on Azure so with the Nice disk snapshot > feature, replacing a dead node is easier, no streaming from Cassandra > >> > >> About compaction overhead, using TwCs with a 1 day bucket and removing > read repair and subrange repair should be sufficient > >> > >> Now the only remaining issue is Quorum read which triggers repair > automagically > >> > >> Before 4.0 there is no flag to turn it off unfortunately > >> > >> Le 30 sept. 2019 15:47, "Eric Evans" a > =C3=A9crit : > >> > >> On Sat, Sep 28, 2019 at 8:50 PM Jeff Jirsa wrote: > >> > >> [ ... ] > >> > >> > 2) The 2TB guidance is old and irrelevant for most people, what you > really care about is how fast you can replace the failed machine > >> > > >> > You=E2=80=99d likely be ok going significantly larger than that if y= ou use a > few vnodes, since that=E2=80=99ll help rebuild faster (you=E2=80=99ll str= eam from more > sources on rebuild) > >> > > >> > If you don=E2=80=99t want to use vnodes, buy big machines and run mu= ltiple > Cassandra instances in it - it=E2=80=99s not hard to run 3-4TB per instan= ce and > 12-16T of SSD per machine > >> > >> We do this too. It's worth keeping in mind though that you'll still > >> have a 12-16T blast radius in the event of a host failure. As the > >> host density goes up, consider steps to make the host more robust > >> (RAID, redundant power supplies, etc). > >> > >> -- > >> Eric Evans > >> john.eric.evans@gmail.com > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org > >> For additional commands, e-mail: user-help@cassandra.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org > For additional commands, e-mail: user-help@cassandra.apache.org > > --=20 Cedrick Lunven | EMEA Developer Advocate Manager =E2=9D=93Ask us your questions : *DataStax Community * =F0=9F=94=ACTest our new products : *DataStax Labs * --0000000000000d364c0594134d7d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

If you are usin= g DataStax Enterprise why not offloading cold data to DSEFS (HDFS implement= ation) with friendly analytics=C2=A0storage format like parquet, keep only = OLTP in the Cassandra Tables. Recommended size for DSEFS can go up to 30TB = a node.

I am pretty sure you are already awa= re of this option and would be curious to get your think about this solutio= n and limitations.

Note: that would also pro= bably help you with your init-load/TWCS issue .

My2c.
Cedrick

On Tue, Oct 1, 2019 at 11:49 PM DuyHai = Doan <doanduyhai@gmail.com&g= t; wrote:
The cl= ient wants to be able to access cold data (2 years old) in the
same cluster so moving data to another system is not possible

However, since we're using Datastax Enterprise, we can leverage Tiered<= br> Storage and store old data on Spinning Disks to save on hardware

Regards

On Tue, Oct 1, 2019 at 9:47 AM Julien Laurenceau
<j= ulien.laurenceau@pepitedata.com> wrote:
>
> Hi,
> Depending on the use case, you may also consider storage tiering with = fresh data on hot-tier (Cassandra) and older data on cold-tier (Spark/Parqu= et or Presto/Parquet). It would be a lot more complex, but may fit more app= ropriately the budget and you may reuse some tech already present in your e= nvironment.
> You may even do subsampling during the transformation offloading data = from Cassandra in order to keep one point out of 10 for older data if subsa= mpling makes sense for your data signal.
>
> Regards
> Julien
>
> Le lun. 30 sept. 2019 =C3=A0 22:03, DuyHai Doan <doanduyhai@gmail.com> a =C3= =A9crit :
>>
>> Thanks all for your reply
>>
>> The target deployment is on Azure so with the Nice disk snapshot f= eature, replacing a dead node is easier, no streaming from Cassandra
>>
>> About compaction overhead, using TwCs with a 1 day bucket and remo= ving read repair and subrange repair should be sufficient
>>
>> Now the only remaining issue is Quorum read which triggers repair = automagically
>>
>> Before 4.0=C2=A0 there is no flag to turn it off unfortunately
>>
>> Le 30 sept. 2019 15:47, "Eric Evans" <john.eric.evans@gmail.com> a =C3=A9crit :
>>
>> On Sat, Sep 28, 2019 at 8:50 PM Jeff Jirsa <jjirsa@gmail.com> wrote:
>>
>> [ ... ]
>>
>> > 2) The 2TB guidance is old and irrelevant for most people, wh= at you really care about is how fast you can replace the failed machine
>> >
>> > You=E2=80=99d likely be ok going significantly larger than th= at if you use a few vnodes, since that=E2=80=99ll help rebuild faster (you= =E2=80=99ll stream from more sources on rebuild)
>> >
>> > If you don=E2=80=99t want to use vnodes, buy big machines and= run multiple Cassandra instances in it - it=E2=80=99s not hard to run 3-4T= B per instance and 12-16T of SSD per machine
>>
>> We do this too.=C2=A0 It's worth keeping in mind though that y= ou'll still
>> have a 12-16T blast radius in the event of a host failure.=C2=A0 A= s the
>> host density goes up, consider steps to make the host more robust<= br> >> (RAID, redundant power supplies, etc).
>>
>> --
>> Eric Evans
>> joh= n.eric.evans@gmail.com
>>
>> ------------------------------------------------------------------= ---
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<= br> >> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org



--


Cedrick Lunven | EMEA Developer Advocate Manager

<= p style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt">

3D"" 3D"" 3D""=


=E2=9D=93Ask us your questions : DataStax Community

=F0=9F=94= =ACTest our new products : DataStax Labs





--0000000000000d364c0594134d7d--