incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kani <javier.canil...@gmail.com>
Subject Re: Too many open files Exception + java.lang.ArithmeticException: / by zero
Date Thu, 16 Dec 2010 18:57:06 GMT
Ya, that happens when some operation throws a time out or any other sort of
operation (connection refuse, etc). There is a failback logic that will try
to discover all the nodes within the Cluster (not only the ones you
configured) in order to reach the  cluster and execution the operation.

Have you seen the log of Cassandra? You might be having a problem on your
client when a Compactation is kicked. Doing so, all your connections to the
node will be much slower. This may introduce a a file handling problem,
since your client will continue to stack up connections against Cassandra.

If you see that this is the problem that you are having, see how you can
manage to raise such limitation to a higher value that will let you get
through Cassandra compactation.

Kani

On Thu, Dec 16, 2010 at 2:48 PM, Germán Kondolf <german.kondolf@gmail.com>wrote:

> Indeed Hector has a connection pool behind it, I think it uses 50
> connectios per node.
> But also uses a node to discover the others, I assume that, as I saw
> connections from my app to nodes that I didn't configure in Hector.
>
> So, you may check the fds in OS level to see if there is a bottleneck
> there.
>
> On Thu, Dec 16, 2010 at 2:39 PM, Amin Sakka, Novapost
> <amin.sakka@novapost.fr> wrote:
> >
> > I'm using a unique client instance (using Hector) and a unique connection
> to
> > cassandra.
> > For each insertion I'm using a new mutator and then I release it.
> > I have 473  sstable "Data.db", the average size of each is 30Mo.
> >
> >
> >
> > 2010/12/16 Ryan King <ryan@twitter.com>
> >>
> >> Are you creating a new connection for each row you insert (and if so
> >> are you closing it)?
> >>
> >> -ryan
> >>
> >> On Wed, Dec 15, 2010 at 8:13 AM, Amin Sakka, Novapost
> >> <amin.sakka@novapost.fr> wrote:
> >> > Hello,
> >> > I'm using cassandra 0.7.0 rc1, a single node configuration,
> replication
> >> > factor 1, random partitioner, 2 GO heap size.
> >> > I ran my hector client to insert 5.000.000 rows but after a couple of
> >> > hours,
> >> > the following Exception occurs :
> >> >
> >> >  WARN [main] 2010-12-15 16:38:53,335 CustomTThreadPoolServer.java
> (line
> >> > 104)
> >> > Transport error occurred during acceptance of message.
> >> > org.apache.thrift.transport.TTransportException:
> >> > java.net.SocketException:
> >> > Too many open files
> >> > at
> >> >
> >> >
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
> >> > at
> >> >
> >> >
> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:67)
> >> > at
> >> >
> >> >
> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:38)
> >> > at
> >> >
> >> >
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
> >> > at
> >> >
> >> >
> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:98)
> >> > at
> >> >
> >> >
> org.apache.cassandra.thrift.CassandraDaemon.start(CassandraDaemon.java:120)
> >> > at
> >> >
> >> >
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:229)
> >> > at
> >> >
> >> >
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134)
> >> > Caused by: java.net.SocketException: Too many open files
> >> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> >> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> >> > at java.net.ServerSocket.implAccept(ServerSocket.java:453)
> >> > at java.net.ServerSocket.accept(ServerSocket.java:421)
> >> > at
> >> >
> >> >
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
> >> >
> >> > When I try to restart Cassandra, I have the following exception :
> >> >
> >> > ERROR 16:42:26,573 Exception encountered during startup.
> >> > java.lang.ArithmeticException: / by zero
> >> > at
> >> >
> >> >
> org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:233)
> >> > at
> >> >
> >> >
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:284)
> >> > at
> >> >
> >> >
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:200)
> >> > at
> >> >
> >> >
> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:225)
> >> > at
> >> >
> >> >
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:449)
> >> > at
> >> >
> >> >
> org.apache.cassandra.db.ColumnFamilyStore.addIndex(ColumnFamilyStore.java:306)
> >> > at
> >> >
> >> >
> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:246)
> >> > at
> >> >
> >> >
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:449)
> >> > at
> >> >
> >> >
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:437)
> >> > at org.apache.cassandra.db.Table.initCf(Table.java:341)
> >> > at org.apache.cassandra.db.Table.<init>(Table.java:283)
> >> > at org.apache.cassandra.db.Table.open(Table.java:114)
> >> > at
> >> >
> >> >
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138)
> >> > at
> >> >
> >> >
> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
> >> > at
> >> >
> >> >
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216)
> >> > at
> >> >
> >> >
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134)
> >> >
> >> > I am looking for advice on how to debug this.
> >> >
> >> > Thanks,
> >> > --
> >> >
> >> > Amin
> >> >
> >> >
> >> >
> >> >
> >> >
> >
> >
> >
> > --
> > Amin
> >
> >
> >
> >
>
>
>
> --
> //GK
> german.kondolf@gmail.com
> // sites
> http://twitter.com/germanklf
> http://www.facebook.com/germanklf
> http://ar.linkedin.com/in/germankondolf
>

Mime
View raw message