cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominic Williams (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-1423) Cassandra's internal state broke by getting column slices. Error org.apache.thrift.TApplicationException: Internal error processing get_slice afterwards.
Date Mon, 23 Aug 2010 22:25:33 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901631#action_12901631
] 

Dominic Williams commented on CASSANDRA-1423:
---------------------------------------------

Looks like this bug might hopefully be a straightforward case of not cleaning up file handles
(or at least, always opening files afresh, and then not handling the error when you can't
open any more). I should have checked logs. 

>From system.log:

ERROR [pool-1-thread-74] 2010-08-23 23:18:34,749 Cassandra.java (line 2651) Internal error
processing get_slice
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db (Too many open
files)
	at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:354)
	at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:297)
	at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:125)
	at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:231)
	at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:309)
	at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:270)
	at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:2643)
	at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2499)
	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db (Too many open
files)
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:350)
	... 11 more
Caused by: java.io.IOError: java.io.FileNotFoundException: /var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
	at org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:68)
	at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:545)
	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:71)
	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:48)
	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:64)
	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:76)
	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:956)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:851)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:826)
	at org.apache.cassandra.db.Table.getRow(Table.java:330)
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:71)
	at org.apache.cassandra.service.StorageProxy$weakReadLocalCallable.call(StorageProxy.java:816)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	... 3 more
Caused by: java.io.FileNotFoundException: /var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
	at java.io.RandomAccessFile.open(Native Method)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
	at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
	at org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:62)
	... 16 more


> Cassandra's internal state broke by getting column slices. Error org.apache.thrift.TApplicationException:
Internal error processing get_slice afterwards.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-1423
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1423
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7 beta 1
>         Environment: Ubuntu 9.04
>            Reporter: Dominic Williams
>            Priority: Critical
>             Fix For: 0.7 beta 2
>
>         Attachments: fmm-add-schools.zip
>
>
> The attached program (with data that it can import) causes large column slices to be
requested from Cassandra. 
> The program itself uploads school address data to a Cassyndex full text index, and then
allows you to search that. The program simulates someone typing a search into an active search
box, which shows you the matches for the current term as you type. Thus when you enter a search
term such as "cherwell school oxford" actually it performs the searches "c", "ch", "che",
"cher", "cherw" etc
> You can configure the delay between the "keystrokes". If your delay allows the searches
to complete sequentially, you are ok. But if you have a short delay, and searches are created
in parallel, pretty quickly this error will arise - "org.apache.thrift.TApplicationException:
Internal error processing get_slice".
> Once this has occurred all future attempts and getting slices of columns will return
the same error, and your'e only option is to restart Cassandra.
> This looks like some kind of concurrency edge condition bug caused by requesting sufficiently
large intersecting slices in parallel. It may be in other versions too.
> I've been testing on 0.7 B1 using an RP cluster. 
> The attached maven project should pull down the scale7 libraries but if interested you
can find the sources at http://github.com/s7

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message