Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0CD2C10D77 for ; Mon, 23 Feb 2015 17:59:53 +0000 (UTC) Received: (qmail 74656 invoked by uid 500); 23 Feb 2015 17:59:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 74612 invoked by uid 500); 23 Feb 2015 17:59:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74602 invoked by uid 99); 23 Feb 2015 17:59:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Feb 2015 17:59:49 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bluethundr@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Feb 2015 17:59:24 +0000 Received: by mail-wi0-f182.google.com with SMTP id l15so19452024wiw.3 for ; Mon, 23 Feb 2015 09:59:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Lgxb5WyKjSXzSsaZ7EKyiNMLxLQi6D4IbmZJSj92cHQ=; b=bXBR0lSOWwITNomT4fOFqhH9mJezUbEU9HP6otJH/pbGzmEMG/BwjBSJJJr2IIAntX DqIPbe7Gsd9v9eFoiVbBchK7h+8g/dYguCbwynsS1notANsoc6jRA8emlusKufB6amje RdUXFTwVHdofcfpkmMHgJEP+dIPBtmlyg1j6AOQ3kprEHHXZjEaphP2thFMZp1w4S0++ jB7E0UbYCpaCtn45gOAZB/igNHRJVhnFgTLcYGIse6FNww2yFB0mueUItCGUhr6DmPCJ Vu9S21Rq3uisjSkHNGgiBfMl3/t4iasTni3Qs/aO5sX/K5gPQWnsqiSX3gAmTMWQWH+r JaxQ== MIME-Version: 1.0 X-Received: by 10.194.85.129 with SMTP id h1mr24999018wjz.147.1424714362909; Mon, 23 Feb 2015 09:59:22 -0800 (PST) Received: by 10.194.158.33 with HTTP; Mon, 23 Feb 2015 09:59:22 -0800 (PST) In-Reply-To: References: Date: Mon, 23 Feb 2015 12:59:22 -0500 Message-ID: Subject: Re: run cassandra on a small instance From: Tim Dunphy To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=089e010d860a81fa18050fc529d6 X-Virus-Checked: Checked by ClamAV on apache.org --089e010d860a81fa18050fc529d6 Content-Type: text/plain; charset=UTF-8 > > You might also have some gains setting in_memory_compaction_limit_in_mb > to something very low to force Cassandra to use on disk compaction rather > than doing it in memory. Cool Ben.. thanks I'll add that to my config as well. Glad that helped. Thanks for reporting back! No problem, Nate! That's the least I can do. All I can hope is that this thread adds to the overall fund of knowledge for the list. Cheers, Tim On Mon, Feb 23, 2015 at 11:46 AM, Nate McCall wrote: > Glad that helped. Thanks for reporting back! > > On Sun, Feb 22, 2015 at 9:12 PM, Tim Dunphy wrote: > >> Nate, >> >> Definitely thank you for this advice. After leaving the new Cassandra >> node running on the 2GB instance for the past couple of days, I think I've >> had ample reason to report complete success in getting it stabilized on >> that instance! Here are the changes I've been able to make: >> >> I think manipulating the key cache and other stuff like concurrent >> writes and some of the other stuff I worked on based on that thread from >> the cassandra list definitely was key in getting Cassandra to work on the >> new instance. >> >> Check out the before and after (before working/ after working): >> >> Before in cassandra-env.sh: >> MAX_HEAP_SIZE="800M" >> HEAP_NEWSIZE="200M" >> >> After: >> MAX_HEAP_SIZE="512M" >> HEAP_NEWSIZE="100M" >> >> And before in the cassandra.yaml file: >> >> concurrent_writes: 32 >> compaction_throughput_mb_per_sec: 16 >> key_cache_size_in_mb: >> key_cache_save_period: 14400 >> # native_transport_max_threads: 128 >> >> And after: >> >> concurrent_writes: 2 >> compaction_throughput_mb_per_sec: 8 >> key_cache_size_in_mb: 4 >> key_cache_save_period: 0 >> native_transport_max_threads: 4 >> >> >> That really made the difference. I'm a puppet user, so these changes are >> in puppet. So any new 2GB instances I should bring up on Digital Ocean >> should absolutely work the way the first 2GB node does, there. But I was >> able to make enough sense of your chef recipe to adapt what you were >> showing me. >> >> Thanks again! >> Tim >> >> On Fri, Feb 20, 2015 at 10:31 PM, Tim Dunphy >> wrote: >> >>> The most important things to note: >>>> - don't include JNA (it needs to lock pages larger than what will be >>>> available) >>>> - turn down threadpools for transports >>>> - turn compaction throughput way down >>>> - make concurrent reads and writes very small >>>> I have used the above run a healthy 5 node clusters locally in it's own >>>> private network with a 6th monitoring server for light to moderate local >>>> testing in 16g of laptop ram. YMMV but it is possible. >>> >>> >>> Thanks!! That was very helpful. I just tried applying your suggestions >>> to my cassandra.yaml file. I used the info from your chef recipe. Well like >>> I've been saying typically it takes about 5 hours or so for this situation >>> to shake itself out. I'll provide an update to the list once I have a >>> better idea of how this is working. >>> >>> Thanks again! >>> Tim >>> >>> On Fri, Feb 20, 2015 at 9:37 PM, Nate McCall >>> wrote: >>> >>>> I frequently test with multi-node vagrant-based clusters locally. The >>>> following chef attributes should give you an idea of what to turn down in >>>> cassandra.yaml and cassandra-env.sh to build a decent testing cluster: >>>> >>>> :cassandra => {'cluster_name' => 'VerifyCluster', >>>> 'package_name' => 'dsc20', >>>> 'version' => '2.0.11', >>>> 'release' => '1', >>>> 'setup_jna' => false, >>>> 'max_heap_size' => '512M', >>>> 'heap_new_size' => '100M', >>>> 'initial_token' => server['initial_token'], >>>> 'seeds' => "192.168.33.10", >>>> 'listen_address' => server['ip'], >>>> 'broadcast_address' => server['ip'], >>>> 'rpc_address' => server['ip'], >>>> 'conconcurrent_reads' => "2", >>>> 'concurrent_writes' => "2", >>>> 'memtable_flush_queue_size' => "2", >>>> 'compaction_throughput_mb_per_sec' => "8", >>>> 'key_cache_size_in_mb' => "4", >>>> 'key_cache_save_period' => "0", >>>> 'native_transport_min_threads' => "2", >>>> 'native_transport_max_threads' => "4", >>>> 'notify_restart' => true, >>>> 'reporter' => { >>>> 'riemann' => { >>>> 'enable' => true, >>>> 'host' => '192.168.33.51' >>>> }, >>>> 'graphite' => { >>>> 'enable' => true, >>>> 'host' => '192.168.33.51' >>>> } >>>> } >>>> }, >>>> >>>> The most important things to note: >>>> - don't include JNA (it needs to lock pages larger than what will be >>>> available) >>>> - turn down threadpools for transports >>>> - turn compaction throughput way down >>>> - make concurrent reads and writes very small >>>> >>>> I have used the above run a healthy 5 node clusters locally in it's own >>>> private network with a 6th monitoring server for light to moderate local >>>> testing in 16g of laptop ram. YMMV but it is possible. >>>> >>> >>> >>> >>> -- >>> GPG me!! >>> >>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >>> >>> >> >> >> -- >> GPG me!! >> >> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B >> >> > > > -- > ----------------- > Nate McCall > Austin, TX > @zznate > > Co-Founder & Sr. Technical Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B --089e010d860a81fa18050fc529d6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You might also have some gains setting=C2=A0in_memory_compaction_limit_in_mb to something very= low to force Cassandra to use on disk compaction rather than doing it in m= emory.

Cool Ben.. thanks I'll ad= d that to my config as well.

Glad that helped. Thanks for reporting = back!

No problem, Nate! That's t= he least I can do. All I can hope is that this thread adds to the overall f= und of knowledge for the list.

Cheers,
T= im=C2=A0

=C2=A0

On Mon, Feb 23, 2015 at 11:46 AM, Nate M= cCall <nate@thelastpickle.com> wrote:
Glad that helped. Thanks for reporting ba= ck!

On Sun, Feb 22, 2015 at 9:12 PM, Tim Dunphy <bluethund= r@gmail.com> wrote:
Nate,

=C2=A0Definitely thank you for this advi= ce. After leaving the new Cassandra node running on the 2GB instance for th= e past couple of days, I think I've had ample reason to report complete= success in getting it stabilized on that instance! Here are the changes I&= #39;ve been able to make:

=C2=A0I think manipulating the = key cache and other stuff like concurrent writes and some of the other stuf= f I worked on based on that thread from the cassandra list definitely was k= ey in getting Cassandra to work on the new instance.=C2=A0

Check out= the before and after (before working/ after working):
<= div style=3D"font-size:12.8000001907349px">
Before in cassandra-env.sh:
=C2=A0 =C2=A0MAX_HEAP_SIZE=3D"800M"=
=C2=A0 =C2=A0HEAP_NEWSIZE=3D"200M"

After:
=C2= =A0 =C2=A0 MAX_HEAP_SIZE=3D"512M"
=C2=A0 =C2=A0 HEAP_NEWSIZE=3D"100M"

And before in the cassandra.yaml file:

=C2=A0 =C2=A0concurrent_writes: 32
=C2=A0 =C2=A0compaction_throughput_mb_per_sec: 16
=C2=A0 =C2=A0key_cache_size_in_mb= :
=C2=A0 =C2=A0key_cache_s= ave_period: 14400
=C2=A0 = =C2=A0# native_transport_max_threads: 128

And a= fter:

=C2=A0 =C2=A0=C2=A0concurrent_writes: 2
=C2=A0 =C2=A0 compaction_th= roughput_mb_per_sec: 8
=C2= =A0 =C2=A0 key_cache_size_in_mb: 4
=C2=A0 =C2=A0 key_cache_save_period: 0
=C2=A0 =C2=A0 native_transport_max_threads: 4


That really made the difference. I'm a puppet user, so these chan= ges are in puppet. So any new 2GB instances I should bring up on Digital Oc= ean should absolutely work the way the first 2GB node does, there.=C2=A0 Bu= t I was able to make enough sense of your chef recipe to adapt what you wer= e showing me.

Thanks again!
Tim

On Fri, Feb 20, 2015 at 10:31 PM, Tim Dunphy <<= a href=3D"mailto:bluethundr@gmail.com" target=3D"_blank">bluethundr@gmail.c= om> wrote:
The most important things to note:
- don't i= nclude JNA (it needs to lock pages larger than what will be available)
-= turn down threadpools for transports
- turn compaction throughput way d= own
- make concurrent reads and writes very small
I have used the abo= ve run a healthy 5 node clusters locally in it's own private network wi= th a 6th monitoring server for light to moderate local testing in 16g of la= ptop ram. YMMV but it is possible.=C2=A0

=
Thanks!! That was very helpful. I just tried applying your suggestions= to my cassandra.yaml file. I used the info from your chef recipe. Well lik= e I've been saying typically it takes about 5 hours or so for this situ= ation to shake itself out. I'll provide an update to the list once I ha= ve a better idea of how this is working.

Thanks ag= ain!
Tim
<= div class=3D"gmail_extra">

On Fri, = Feb 20, 2015 at 9:37 PM, Nate McCall <nate@thelastpickle.com><= /span> wrote:
I frequently test with multi-node vagrant-based clust= ers locally. The following chef attributes should give you an idea of what = to turn down in cassandra.yaml and cassandra-env.sh to build a decent testi= ng cluster:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 :cassandra =3D> {'cluster_name' =3D> 'VerifyCluster&#= 39;,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'package_name' =3D> 'dsc20= 9;,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'version' =3D> '2.0.11',<= /div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0'release' =3D> '1',
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0'setup_jna' =3D> false,
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0'max_heap_size' =3D> '512M',
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0'heap_new_size' =3D> '100M',
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0'initial_token' =3D> server['initial_token'],
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0'seeds' =3D> "192.168.33.10",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0'listen_address' =3D> server['ip'],=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0'broadcast_address' =3D> server['= ip'],
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'rpc_address' =3D> server[= 9;ip'],
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'conconcurrent_reads' =3D>= "2",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'concurrent_writes' =3D>= ; "2",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'memtable_flush_queue_size= 9; =3D> "2",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'compaction_through= put_mb_per_sec' =3D> "8",
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'k= ey_cache_size_in_mb' =3D> "4",
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0&#= 39;key_cache_save_period' =3D> "0",
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0'native_transport_min_threads' =3D> "2",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0'native_transport_max_threads' =3D> "4&= quot;,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'notify_restart' =3D> true,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0'reporter' =3D> {
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'riemann' =3D> {
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0'enable' =3D> true,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0'host' =3D> '192.168.33.51'
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0},
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'graphite' =3D> {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'enable' =3D> true,
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'host' =3D> '192.168.33.51= '
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0},

The most important things to = note:
- don't include JNA (it needs to lock pages larger than= what will be available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent = reads and writes very small

I have used the above = run a healthy 5 node clusters locally in it's own private network with = a 6th monitoring server for light to moderate local testing in 16g of lapto= p ram. YMMV but it is possible.=C2=A0



-= -
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F18619= 7B




--
GPG me!= !

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B




<= /div>--
-----------------
Nat= e McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Con= sultant
Apache Cassandra Consulting
http://www.thelastpickle.com


--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-k= eys F186197B

--089e010d860a81fa18050fc529d6--