Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.212.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=ZBto/H+DBF1eTYJ+u1YgbD3iKO1IRlyNCG6liP3kuTnfjBUspSmP005tLFhtIT0+e6
         /dWmENnOeg1jXdsvwqofDzqzYLGFsxtBjNzCkKtFxSnBJxyBNj12UMP02jJP4axBODTr
         rVMSpjxwVI+hpOJdHrmjAQxAon4tx01ZpH+ok=
MIME-Version: 1.0
In-Reply-To: <4D998BEC.7000402@zynga.co.jp>
References: <4D8C8256.20007@zynga.co.jp>
 <AANLkTikHeyoxzST7Dx4ZBe25QwUY6BtPVNhpMGQrSAbu@mail.gmail.com>
 <4D8FF02B.5070706@zynga.co.jp> <4D998BEC.7000402@zynga.co.jp>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Tue, 5 Apr 2011 08:47:02 -0500
Message-ID: <BANLkTimxPf3wW44cuHTakp-GoMENU8HEFA@mail.gmail.com>
Subject: Re: Error messages after rolling updating cassandra from 0.7.0 to
 0.7.2
To: Kazuo YAGI <kyagi@zynga.co.jp>
Cc: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Oops, I saw "EOFException" and jumped to "scrub."

But your EOF is coming from TCP. Something (almost certainly a
non-cassandra process) is connecting to the internal Cassandra
communication port (the one that defaults to 7000) and disconnecting.

On Mon, Apr 4, 2011 at 4:14 AM, Kazuo YAGI <kyagi@zynga.co.jp> wrote:
>>> Solution: upgrade to 0.7.4, run scrub
>
> Although I upgraded all my cassandra nodes from 0.7.0 to 0.7.4 and
> ran nodetool scrub to all keyspaces, this EOFException error messages
> didn't go away. Do you have any ideas how to deal with it next?
>
> Besides, it would be really useful if I could know whether or not
> these error messages are ignorable, because our application has been
> working well before and after upgrading.
>
> Thanks,
> Kazuo
>
> (11/03/28 11:19), Kazuo YAGI wrote:
>>
>> I really thank you for your information.
>> I'll try to upgrade 0.7.4 and run scrub.
>>
>> (11/03/25 22:39), Jonathan Ellis wrote:
>>>
>>> from NEWS.txt:
>>>
>>> Upgrading
>>> ---------
>>> - 0.7.1 and 0.7.2 shipped with a bug that caused incorrect row-level
>>> bloom filters to be generated when compacting sstables generated
>>> with earlier versions. This would manifest in IOExceptions during
>>> column name-based queries. 0.7.3 provides "nodetool scrub" to
>>> rebuild sstables with correct bloom filters, with no data lost.
>>> (If your cluster was never on 0.7.0 or earlier, you don't have to
>>> worry about this.) Note that nodetool scrub will snapshot your
>>> data files before rebuilding, just in case.
>>>
>>> Solution: upgrade to 0.7.4, run scrub
>>>
>>> 2011/3/25 Kazuo YAGI<kyagi@zynga.co.jp>:
>>>>
>>>> Hi everyone,
>>>>
>>>> I'm running 10 cassandra nodes in total and updated 5 nodes
>>>> from 0.7.0 to 0.7.2 one by one like below.
>>>>
>>>> # sudo /etc/init.d/cassandra stop
>>>> # sudo yum update cassandra-0.7.2-4
>>>> # sudo /etc/init.d/cassandra start
>>>>
>>>> After updating, those 5 updated nodes keep outputting error messages to
>>>> /var/lib/cassandra/log/system.log every exactly 5 minutes.
>>>>
>>>> Do you have any idea about this error message? Thanks for your support.
>>>>
>>>> /var/lib/cassandra/log/system.log
>>>> ------
>>>> ERROR [Thread-45] 2011-03-23 21:21:29,125
>>>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>>>> Thread[Thread-45,5,main]
>>>> java.io.IOError: java.io.EOFException
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
>>>>
>>>> Caused by: java.io.EOFException
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>>>>
>>>> ERROR [Thread-51] 2011-03-23 21:26:29,227
>>>> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
>>>> Thread[Thread-51,5,main]
>>>> java.io.IOError: java.io.EOFException
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:73)
>>>>
>>>> Caused by: java.io.EOFException
>>>> at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>>> at
>>>>
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:61)
>>>>
>>>> ------
>>>>
>>>> Best,
>>>> Kazuo YAGI
>>>>
>>>
>>>
>>>
>>
>>
>
>


-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com