Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0E04218214 for ; Mon, 10 Aug 2015 16:58:53 +0000 (UTC) Received: (qmail 74410 invoked by uid 500); 10 Aug 2015 16:58:50 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 74375 invoked by uid 500); 10 Aug 2015 16:58:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74365 invoked by uid 99); 10 Aug 2015 16:58:50 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2015 16:58:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 1F3FD1A9BA1 for ; Mon, 10 Aug 2015 16:58:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.001 X-Spam-Level: * X-Spam-Status: No, score=1.001 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id qWvZtfzUvy7q for ; Mon, 10 Aug 2015 16:58:42 +0000 (UTC) Received: from mail-pa0-f47.google.com (mail-pa0-f47.google.com [209.85.220.47]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 2273B42994 for ; Mon, 10 Aug 2015 16:58:42 +0000 (UTC) Received: by pacrr5 with SMTP id rr5so107513524pac.3 for ; Mon, 10 Aug 2015 09:58:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:content-transfer-encoding:message-id:references:to; bh=YRBtdvb8ThFwb0cfSIGDJuiBmk7ZLAJ8fYqkoeQG49g=; b=Me8hDmmgr6AZnReagf+m8zRB+WmIBNSUYz7rSa0T1v/oq5mqjSsa4oihPDIn2SSFL5 W6lGZ5Ey39UIDMRPsJQCvFUA/UMaEZIqeAGZb5Qhi8EB1vgj5CUq9xhWaIJHcS+45IWM AdGfuKL64JzxD5vc6LTzWU2IhivFcRS6ycjCt5yMq4qSTMhDWni5io7AV07eLf9YDUVR Qfj3MK2thAXzyspmKEZuUKGU0gcD83usMyDqBEV2RwqMAEV+nVe6UccxN5gqhdhvCpX4 uqzwD3SQAcTtVERWbvtA7WLlvVb9Um9YL3R2jGpya87b85WIw835t+fOtjTU3lh65DMQ S/Mw== X-Gm-Message-State: ALoCoQl9Fn2fGLWbpmqFjqmajv8FrksXhu1EqC8jlsWjD4lThSR7QqRLm1pDNh89KRBQuykB8ESC X-Received: by 10.66.66.1 with SMTP id b1mr46209456pat.112.1439225914824; Mon, 10 Aug 2015 09:58:34 -0700 (PDT) Received: from [192.168.0.103] (c-73-202-71-33.hsd1.ca.comcast.net. [73.202.71.33]) by smtp.gmail.com with ESMTPSA id y2sm20572558pdc.91.2015.08.10.09.58.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 10 Aug 2015 09:58:34 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: OOM when Adding host From: rock zhang In-Reply-To: <796B4E63-B95B-477D-99F2-2FC66FB252C4@alohar.com> Date: Mon, 10 Aug 2015 09:59:43 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <54804B16-1BF4-4591-BF6A-D4A3BA28D33F@alohar.com> References: <796B4E63-B95B-477D-99F2-2FC66FB252C4@alohar.com> To: "user@cassandra.apache.org" X-Mailer: Apple Mail (2.1878.6) My Cassandra version is 2.1.4. Thanks Rock=20 On Aug 10, 2015, at 9:52 AM, rock zhang wrote: > Hi All, >=20 > Currently i have three hosts. The data is not balanced, one is 79G, = another two have 300GB. When I adding a new host, firstly I got "too = many open files" error, then i changed file open limit from 100,000 to = 1, 000, 000. Then I got OOM error. >=20 > Should I change the limits to 20,0000 instead of 1M? My memory is = 33G, i am using EC2 c2*2xlarge. Ideally even if the data is large, just = slower, should not OOM, don't understand why . >=20 > I actually got this error pretty often. I guess the reason is because = my data is pretty large? If cassandra try to split the data evenly on = all host, then Cassandra need to copy around 200GB to the new host.=20 >=20 > =46rom my experience, An alternative way to solve this is add new host = as seed, do not use "Add host", then data would be move, so so OOM. But = not sure data will be lost or cannot be located.=20 >=20 > Thanks > Rock=20 >=20