Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 41C578266 for ; Sun, 14 Aug 2011 16:24:21 +0000 (UTC) Received: (qmail 88103 invoked by uid 500); 14 Aug 2011 16:24:19 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 87995 invoked by uid 500); 14 Aug 2011 16:24:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 87987 invoked by uid 500); 14 Aug 2011 16:24:18 -0000 Delivered-To: apmail-incubator-cassandra-user@incubator.apache.org Received: (qmail 87984 invoked by uid 99); 14 Aug 2011 16:24:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Aug 2011 16:24:17 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of springrider@gmail.com designates 209.85.215.47 as permitted sender) Received: from [209.85.215.47] (HELO mail-ew0-f47.google.com) (209.85.215.47) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Aug 2011 16:24:11 +0000 Received: by ewy5 with SMTP id 5so1729047ewy.6 for ; Sun, 14 Aug 2011 09:23:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; bh=CmY+/9kZ5hDJHwDueSfR+U48Xdp8OV/9iIftcwCCO18=; b=MRTTFZ3dztXRMTf2fNEQAw4D5t6hGnMnC1hL6DlGatlzXBpksm/FtnIGQCApSt7ufT K+TISI3mveEVsfxAK5lcwFVSxa3Wg7ZbYRNSocQDX5YgrpXYguQUP1PfGwEWPTMRWggw c4u43n4r33DBQN6DO5GOieyWQFy0gKg5ANYPg= Received: by 10.213.30.7 with SMTP id s7mr1000090ebc.63.1313339030130; Sun, 14 Aug 2011 09:23:50 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.21.198 with HTTP; Sun, 14 Aug 2011 09:23:30 -0700 (PDT) From: Yan Chunlu Date: Mon, 15 Aug 2011 00:23:30 +0800 Message-ID: Subject: node restart taking too long To: cassandra-user@incubator.apache.org Content-Type: multipart/alternative; boundary=0015174bdf2e5c50cd04aa79924a --0015174bdf2e5c50cd04aa79924a Content-Type: text/plain; charset=ISO-8859-1 I got 3 nodes and RF=3, when I repairing ndoe3, it seems alot data generated. and server can not afford the load then crashed. after come back, node 3 can not return for more than 96 hours for 34GB data, the node 2 could restart and back online within 1 hour. I am not sure what's wrong with node3 and should I restart node 3 again? thanks! Address Status State Load Owns Token 113427455640312821154458202477256070484 node1 Up Normal 34.11 GB 33.33% 0 node2 Up Normal 31.44 GB 33.33% 56713727820156410577229101238628035242 node3 Down Normal 177.55 GB 33.33% 113427455640312821154458202477256070484 the log shows it is still going on, not sure why it is so slow: INFO [main] 2011-08-14 08:55:47,734 SSTableReader.java (line 154) Opening /cassandra/data/COMMENT INFO [main] 2011-08-14 08:55:47,828 ColumnFamilyStore.java (line 275) reading saved cache /cassandra/saved_caches/COMMENT-RowCache INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547) completed loading (1744370 ms; 200000 keys) row cache for COMMENT INFO [main] 2011-08-14 09:24:52,299 ColumnFamilyStore.java (line 275) reading saved cache /cassandra/saved_caches/COMMENT-RowCache INFO [CompactionExecutor:1] 2011-08-14 10:24:55,480 CacheWriter.java (line 96) Saved COMMENT-RowCache (200000 items) in 2535 ms --0015174bdf2e5c50cd04aa79924a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

I got 3 nodes and RF=3D3, when I repairing ndoe3, it seems alot dat= a generated.=A0 and server can not afford the load then crashed.
after c= ome back, node 3 can not return for more than 96 hours

for 34GB data= , the node 2 could restart and back online within 1 hour.

I am not sure what's wrong with node3 and should I restart node 3 a= gain? thanks!

Address=A0=A0=A0=A0=A0=A0=A0=A0 Status State=A0=A0 Loa= d=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Owns=A0=A0=A0 Token
=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 11= 3427455640312821154458202477256070484
node1 =A0=A0=A0 Up=A0=A0=A0=A0 Normal=A0 34.11 GB=A0=A0=A0=A0=A0=A0=A0 33.3= 3%=A0 0
node2 =A0=A0=A0 Up=A0=A0=A0=A0 Normal=A0 31.44 GB=A0=A0=A0=A0=A0= =A0=A0 33.33%=A0 56713727820156410577229101238628035242
node3 =A0=A0=A0 = Down=A0=A0 Normal=A0 177.55 GB=A0=A0=A0=A0=A0=A0 33.33%=A0 1134274556403128= 21154458202477256070484


the log shows it is still going on, not sure why it is so slow:
=

=A0INFO [main] 2011-08-14 08:55:47,734 SSTableReader.java (line 154= ) Opening /cassandra/data/COMMENT
=A0INFO [main] 2011-08-14 08:55:47,828= ColumnFamilyStore.java (line 275) reading saved cache /cassandra/saved_cac= hes/COMMENT-RowCache
=A0INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547) co= mpleted loading (1744370 ms; 200000 keys) row cache for COMMENT
=A0INFO = [main] 2011-08-14 09:24:52,299 ColumnFamilyStore.java (line 275) reading sa= ved cache /cassandra/saved_caches/COMMENT-RowCache
=A0INFO [CompactionExecutor:1] 2011-08-14 10:24:55,480 CacheWriter.java (li= ne 96) Saved COMMENT-RowCache (200000 items) in 2535 ms




--0015174bdf2e5c50cd04aa79924a--