cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alaa Zubaidi (PDF)" <alaa.zuba...@pdf.com>
Subject Re: Corrupt SSTABLE over and over
Date Fri, 12 Aug 2016 07:00:53 GMT
One more thing I noticed..
The corrupted SSTable is mentioned twice in the log file
[CompactionExecutor:10253] 2016-08-11 08:59:01,952.... - Compacting (.....)
[...la-1104-big-Data.db, ....]
[CompactionExecutor:10253] 2016-08-11 09:32:04,814.... - Compacting (.....)
[...la-1104-big-Data.db....]

Is it possible Cassandra is trying to compact the same file again while its
being compacted by another process?

Regards,
Alaa

On Thu, Aug 11, 2016 at 6:20 PM, Alaa Zubaidi (PDF) <alaa.zubaidi@pdf.com>
wrote:

> Hi,
>
> I have a 16 Node cluster, Cassandra 2.2.1 on Windows, local installation
> (NOT on the cloud)
>
> and I am getting
> Error [CompactionExecutor:2] 2016-08-12 06:51:52, 983 Cassandra
> Daemon.java:183 - Execption in thread Thread[CompactionExecutor:2,1main]
> org.apache.cassandra.io.FSReaderError: org.apache.cassandra.io.sstable.CorruptSSTableExecption:
> org.apache.cassandra.io.compress.CurrptBlockException:
> (E:\........\la-4886-big-Data.db): corruption detected, chunk at 4969092
> of length 10208.
>     at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:357)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> ....
> ....
> ERROR [CompactionExecutor:2] ....... FileUtils.java:463 - Existing
> forcefully due to file system exception on startup, disk failure policy
> "stop"
>
> I tried sstablescrub but it crashed with hs-err-pid-...
> I removed the corrupted file and started the Node again, after one day the
> corruption came back again, I removed the files, and restarted Cassandra,
> it worked for few days, then I ran "nodetool repair" after it finished,
> Cassandra failed again but with commitlog corruption, after removing the
> commitlog files, it failed again with another sstable corruption.
>
> I was also checking the HW, file system, and memory, the VMware logs
> showed no HW error, also the HW management logs showed NO problems or
> issues.
> Also checked the Windows Logs (Application and System) the only thing I
> found is on the system logs "Cassandra Service terminated with
> service-specific error Cannot create another system semaphore.
>
> I could not find any thing regarding that error, all comments point to
> application log.
>
> Any help is appreciated..
>
> --
>
> Alaa Zubaidi
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zubaidi@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.department@pdf.com* <legal.department@pdf.com>*.*

Mime
View raw message