Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of mvallebr@gmail.com designates
 209.85.219.41 as permitted sender)
MIME-Version: 1.0
Date: Thu, 22 Aug 2013 12:19:10 -0300
Message-ID: 
 <CABKQidufG9N-m0OsEYSOouf2C0VSofHRZVg17ag9-3DYZrvX5g@mail.gmail.com>
Subject: node dead after restart
From: Marcelo Elias Del Valle <mvallebr@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=e89a8fb1fe36d42f3c04e48acfa8

--e89a8fb1fe36d42f3c04e48acfa8
Content-Type: text/plain; charset=ISO-8859-1

Hello,

    I am having a problem with a node in a test environment I have at
amazon. I am using cassandra 1.2.3 in Amazon EC2. Here is my nodetool ring
output:

$ nodetool ring
Note: Ownership information does not include topology; for complete
information, specify a keyspace

Datacenter: us-east
==========
Address         Rack        Status State   Load            Owns
   Token

   113427455640312821154458202479064646084
10.0.0.76   1b          Up     Normal  31.34 MB        33.33%
 1808575600
10.0.0.146    1b          Up     Normal  34.24 MB        33.33%
 56713727820156410577229101240436610842
10.0.0.111   1b          Down   Normal  21.19 MB        33.33%
 113427455640312821154458202479064646084

     I logged in 10.0.0.111 machine and restarted cassandra, while looking
at the log. Gossip protocol is still up, but the node starts and goes down
just after it. Here is what I see in the logs:

sudo tail /var/log/cassandra/output.log
 INFO 12:16:23,084 Node /10.0.0.111 has restarted, now UP
 INFO 12:16:23,095 InetAddress /10.0.0.111 is now UP
 INFO 12:16:23,097 Node /10.0.0.111 state jump to normal
 INFO 12:16:23,105 Not starting native transport as requested. Use JMX
(StorageService->startNativeTransport()) to start it
 INFO 12:16:23,108 Binding thrift service to ip-10-0-0-146.ec2.internal/
10.0.0.146:9160
 INFO 12:16:23,137 Using TFramedTransport with a max frame size of 15728640
bytes.
 INFO 12:16:23,143 Using synchronous/threadpool thrift server on
ip-10-0-0-146.ec2.internal : 9160
 INFO 12:16:23,143 Listening for thrift clients...
 INFO 12:16:30,063 Saved local counter id:
76c1a930-a866-11e2-a3bd-831b111cd74c
 INFO 12:16:32,860 InetAddress /10.0.0.111 is now dead.

     I am having no clue of what is wrong. Any hint of what could I do to
look for the problem?

Best regards,
-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

--e89a8fb1fe36d42f3c04e48acfa8
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello,=A0<div><br></div><div>=A0 =A0 I am having a problem=
 with a node in a test environment I have at amazon. I am using cassandra 1=
.2.3 in Amazon EC2. Here is my nodetool ring output:</div><div><br></div><d=
iv><div>
$ nodetool ring</div><div>Note: Ownership information does not include topo=
logy; for complete information, specify a keyspace</div><div><br></div><div=
>Datacenter: us-east</div><div>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D</div><div>Add=
ress =A0 =A0 =A0 =A0 Rack =A0 =A0 =A0 =A0Status State =A0 Load =A0 =A0 =A0 =
=A0 =A0 =A0Owns =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Token =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 =A0 =A0113427455640312821154458202479064646084 =A0 =A0=A0</div><di=
v>10.0.0.76 =A0 1b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A031.34 MB =A0 =A0=
 =A0 =A033.33% =A0 =A0 =A0 =A0 =A0 =A0 =A01808575600 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0</div>
<div>10.0.0.146 =A0 =A01b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A034.24 MB =
=A0 =A0 =A0 =A033.33% =A0 =A0 =A0 =A0 =A0 =A0 =A056713727820156410577229101=
240436610842 =A0 =A0 =A0</div><div>10.0.0.111 =A0 1b =A0 =A0 =A0 =A0 =A0Dow=
n =A0 Normal =A021.19 MB =A0 =A0 =A0 =A033.33% =A0 =A0 =A0 =A0 =A0 =A0 =A01=
13427455640312821154458202479064646084 =A0 =A0=A0</div>
<div><br></div></div><div>=A0 =A0 =A0I logged in 10.0.0.111 machine and res=
tarted cassandra, while looking at the log. Gossip protocol is still up, bu=
t the node starts and goes down just after it. Here is what I see in the lo=
gs:</div>
<div><br></div><div><div>sudo tail /var/log/cassandra/output.log=A0</div><d=
iv>=A0INFO 12:16:23,084 Node /<a href=3D"http://10.0.0.111">10.0.0.111</a> =
has restarted, now UP</div><div>=A0INFO 12:16:23,095 InetAddress /<a href=
=3D"http://10.0.0.111">10.0.0.111</a> is now UP</div>
<div>=A0INFO 12:16:23,097 Node /<a href=3D"http://10.0.0.111">10.0.0.111</a=
> state jump to normal</div><div>=A0INFO 12:16:23,105 Not starting native t=
ransport as requested. Use JMX (StorageService-&gt;startNativeTransport()) =
to start it</div>
<div>=A0INFO 12:16:23,108 Binding thrift service to ip-10-0-0-146.ec2.inter=
nal/<a href=3D"http://10.0.0.146:9160">10.0.0.146:9160</a></div><div>=A0INF=
O 12:16:23,137 Using TFramedTransport with a max frame size of 15728640 byt=
es.</div>
<div>=A0INFO 12:16:23,143 Using synchronous/threadpool thrift server on ip-=
10-0-0-146.ec2.internal : 9160</div><div>=A0INFO 12:16:23,143 Listening for=
 thrift clients...</div><div>=A0INFO 12:16:30,063 Saved local counter id: 7=
6c1a930-a866-11e2-a3bd-831b111cd74c</div>
<div>=A0INFO 12:16:32,860 InetAddress /<a href=3D"http://10.0.0.111">10.0.0=
.111</a> is now dead.</div></div><div><br></div><div>=A0 =A0 =A0I am having=
 no clue of what is wrong. Any hint of what could I do to look for the prob=
lem?</div>
<div><br></div><div>Best regards,<br>-- <br>Marcelo Elias Del Valle<br><a h=
ref=3D"http://mvalle.com" target=3D"_blank">http://mvalle.com</a>=A0- @mval=
lebr
</div></div>

--e89a8fb1fe36d42f3c04e48acfa8--