Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
From: rock zhang <rock@alohar.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_3F80AB5D-C08E-4A9E-9FB7-16A7E0826CAD"
Message-Id: <2B143109-FABB-4583-9F19-443B510E96BE@alohar.com>
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Subject: Re: OOM when Adding host 
Date: Mon, 10 Aug 2015 14:21:59 -0700
References: <796B4E63-B95B-477D-99F2-2FC66FB252C4@alohar.com>
 <54804B16-1BF4-4591-BF6A-D4A3BA28D33F@alohar.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
In-Reply-To: <54804B16-1BF4-4591-BF6A-D4A3BA28D33F@alohar.com>


--Apple-Mail=_3F80AB5D-C08E-4A9E-9FB7-16A7E0826CAD
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

I logged the open files every 10 mins, last record is :=20

lsof -p $cassadnraPID | wc -l

74728

lsof |wc-l
5887913       # this is a very large number, don't know why.

After OOM the open file numbers back to few hundreds (lsof | wc -l ).=20


On Aug 10, 2015, at 9:59 AM, rock zhang <rock@alohar.com> wrote:

> My Cassandra version is 2.1.4.
>=20
> Thanks
> Rock=20
>=20
> On Aug 10, 2015, at 9:52 AM, rock zhang <rock@alohar.com> wrote:
>=20
>> Hi All,
>>=20
>> Currently i have three hosts. The data is not balanced, one is 79G, =
another two have 300GB. When I adding a new host, firstly I got "too =
many open files" error, then i changed file open limit from 100,000 to =
1, 000, 000. Then I got OOM error.
>>=20
>> Should I change the limits to 20,0000 instead of 1M?  My memory is =
33G, i am using EC2 c2*2xlarge.  Ideally even if the data is large, just =
slower, should not OOM, don't understand why .
>>=20
>> I actually got this error pretty often. I guess the reason is because =
my data is pretty large?  If cassandra try to split the data evenly on =
all host, then Cassandra need to copy around 200GB to the new host.=20
>>=20
>> =46rom my experience, An alternative way to solve this is add new =
host as seed, do not use "Add host", then data would be move, so so OOM. =
But not sure data will be lost or cannot be located.=20
>>=20
>> Thanks
>> Rock=20
>>=20
>=20


--Apple-Mail=_3F80AB5D-C08E-4A9E-9FB7-16A7E0826CAD
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div =
style=3D"margin: 0px; font-size: 11px; font-family: Menlo;">I logged the =
open files every 10 mins, last record is :&nbsp;</div><div =
style=3D"margin: 0px; font-size: 11px; font-family: =
Menlo;"><br></div><div style=3D"margin: 0px; font-size: 11px; =
font-family: Menlo;">lsof -p $cassadnraPID | wc -l</div><div =
style=3D"margin: 0px; font-size: 11px; font-family: =
Menlo;"><br></div><div style=3D"margin: 0px; font-size: 11px; =
font-family: Menlo;">74728</div><div style=3D"margin: 0px; font-size: =
11px; font-family: Menlo;"><br></div><div style=3D"margin: 0px; =
font-size: 11px; font-family: Menlo;">lsof |wc-l</div><div =
style=3D"margin: 0px; font-size: 11px; font-family: Menlo;"><div =
style=3D"margin: 0px;">5887913 &nbsp; &nbsp; &nbsp; # this is a very =
large number, don't know why.</div></div><div style=3D"margin: 0px; =
font-size: 11px; font-family: Menlo;"><br></div><div style=3D"margin: =
0px; font-size: 11px; font-family: Menlo;">After OOM the open file =
numbers back to few hundreds (lsof | wc -l ).&nbsp;</div><div =
style=3D"margin: 0px; font-size: 11px; font-family: =
Menlo;"><br></div><div style=3D"margin: 0px; font-size: 11px; =
font-family: Menlo;"><br></div><div style=3D"margin: 0px; font-size: =
11px; font-family: Menlo;"><br></div><div style=3D"margin: 0px; =
font-size: 11px; font-family: Menlo;"><br></div><div><div>On Aug 10, =
2015, at 9:59 AM, rock zhang &lt;<a =
href=3D"mailto:rock@alohar.com">rock@alohar.com</a>&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite">My =
Cassandra version is 2.1.4.<br><br>Thanks<br>Rock <br><br>On Aug 10, =
2015, at 9:52 AM, rock zhang &lt;<a =
href=3D"mailto:rock@alohar.com">rock@alohar.com</a>&gt; =
wrote:<br><br><blockquote type=3D"cite">Hi All,<br><br>Currently i have =
three hosts. The data is not balanced, one is 79G, another two have =
300GB. When I adding a new host, firstly I got "too many open files" =
error, then i changed file open limit from 100,000 to 1, 000, 000. Then =
I got OOM error.<br><br>Should I change the limits to 20,0000 instead of =
1M? &nbsp;My memory is 33G, i am using EC2 c2*2xlarge. &nbsp;Ideally =
even if the data is large, just slower, should not OOM, don't understand =
why .<br><br>I actually got this error pretty often. I guess the reason =
is because my data is pretty large? &nbsp;If cassandra try to split the =
data evenly on all host, then Cassandra need to copy around 200GB to the =
new host. <br><br>=46rom my experience, An alternative way to solve this =
is add new host as seed, do not use "Add host", then data would be move, =
so so OOM. But not sure data will be lost or cannot be located. =
<br><br>Thanks<br>Rock =
<br><br></blockquote><br></blockquote></div><br></body></html>=

--Apple-Mail=_3F80AB5D-C08E-4A9E-9FB7-16A7E0826CAD--