incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <radu.manole...@barclays.com>
Subject Cassandra 1.2.2 | Unexpected Connection Pool Shutdown
Date Tue, 19 Mar 2013 17:49:33 GMT
We have recently upgraded to C* 1.2.2 from 1.0.2, and we have started seeing errors such as
the one below.
Our app collects changes and then flushes them out to C* in a batch.
Sometimes (at high volume) we see the following error:

The log shows this error repeated for each host in the ring (total: eight) all within the
same second:

[03/19/13 10:33:37.286 ERROR] Could not flush transport (to be expected if the pool is shutting
down) in close for client: CassandraClient<someHost.mycompany.com:9160-93> (HThriftClient.java:124)
in thread "MessageStorer-thread"
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
        at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
        at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
        at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:122)
        at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:38)
        at me.prettyprint.cassandra.connection.HConnectionManager.closeClient(HConnectionManager.java:324)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:272)
        at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113)
        at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
        at com.mycompany.some.package.DataWriter.handleInsert(DataWriter.java:283)
        at com.mycompany.some.package.DataWriter.writeObjectsColumns(DataWriter.java:233)
       at com.mycompany.some.package.DataWriter.persistFixMessages(DataWriter.java:140)
        at com.mycompany.some.package.MessageStorer$Storer.run(MessageStorer.java:151)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
        ... 12 more
[03/19/13 10:33:37.289 ERROR] MARK HOST AS DOWN TRIGGERED for host someHost.mycompany.com(so.me.ip.add):9160
(HConnectionManager.java:422) in thread "MessageStorer-thread"
[03/19/13 10:33:37.289 ERROR] Pool state on shutdown: <ConcurrentCassandraClientPoolByHost>:{someHost.mycompany.com(so.me.ip.add):9160};
IsActive?: true; Active: 1; Blocked: 0; Idle: 5; NumBeforeExhausted: 19 (HConnectionManager.java:426)
in thread "MessageStorer-thread"
[03/19/13 10:33:37.289 INFO ] Shutdown triggered on <ConcurrentCassandraClientPoolByHost>:{someHost.mycompany.com(so.me.ip.add):9160}
(ConcurrentHClientPool.java:162) in thread "MessageStorer-thread"
[03/19/13 10:33:37.302 INFO ] Shutdown complete on <ConcurrentCassandraClientPoolByHost>:{someHost.mycompany.com(so.me.ip.add):9160}
(ConcurrentHClientPool.java:170) in thread "MessageStorer-thread"
[03/19/13 10:33:37.302 INFO ] Host detected as down was added to retry queue: someHost.mycompany.com(so.me.ip.add):9160
(CassandraHostRetryService.java:68) in thread "MessageStorer-thread"
[03/19/13 10:33:37.302 INFO ] Client CassandraClient<someHost.mycompany.com:9160-93>
released to inactive or dead pool. Closing. (HConnectionManager.java:408) in thread "MessageStorer-thread"

Then the application abandons writing the batch, because it cannot write the changes (the
client pool has shut down).
On average, this involves abandoning 20k mutations, for a total of 14Mb of data.

[03/19/13 10:33:37.302 ERROR] DataWriter write failure -- count:21413 byteSize:14155488 (DataWriter.java:286)
in thread "MessageStorer-thread"
me.prettyprint.hector.api.exceptions.HectorTransportException: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
        at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:33)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
        at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113)
        at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
        at com.mycompany.some.package.DataWriter.handleInsert(DataWriter.java:283)
        at com.mycompany.some.package.DataWriter.writeObjectsColumns(DataWriter.java:233)
        at com.mycompany.some.package.DataWriter.persistMessages(DataWriter.java:140)
        at com.mycompany.some.package.MessageStorer$Storer.run(MessageStorer.java:151)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken
pipe
        at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
        at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
        at org.apache.cassandra.thrift.Cassandra$Client.send_batch_mutate(Cassandra.java:958)
        at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:949)
        at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
        at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
        at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104)
        at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
        ... 7 more
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
        ... 15 more

Immediately after shutting down, the pool restarts, so the application continues writing data,
but some data has been lost.
We have reduced the max size of each batch from 14.4Mb to 13.5Mb, but we are still seeing
the errors.
Should we reduce the size of the batch?

Our application is using the following JARs:
libthrift-0.7.0.jar
hector-core-1.1-2.jar
cassandra-thrift-1.2.1.jar
cassandra-javautils-0.7.1.jar
cassandra-all-1.2.0.jar

What is causing these errors, and how can we eliminate them?

Best regards
Radu Manolescu

_______________________________________________

This message may contain information that is confidential or privileged. If you are not an
intended recipient of this message, please delete it and any attachments, and notify the sender
that you have received it in error. Unless specifically stated in the message or otherwise
indicated, you may not uplicate, redistribute or forward this message or any portion thereof,
including any attachments, by any means to any other person, including any retail investor
or customer. This message is not a recommendation, advice, offer or solicitation, to buy/sell
any product or service, and is not an official confirmation of any transaction. Any opinions
presented are solely those of the author and do not necessarily represent those of Barclays.

This message is subject to terms available at: www.barclays.com/emaildisclaimer and, if received
from Barclays' Sales or Trading desk, the terms available at: www.barclays.com/salesandtradingdisclaimer/.
By messaging with Barclays you consent to the foregoing. Barclays Bank PLC is a company registered
in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.
This email may relate to or be sent from other members of the Barclays group.

_______________________________________________

Mime
View raw message