cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Wee <peich...@gmail.com>
Subject Re: Corrupt SSTABLE over and over
Date Fri, 12 Aug 2016 06:02:15 GMT
cassandra run on virtual server (vmware)?

> I tried sstablescrub but it crashed with hs-err-pid-...
maybe try with larger heap allocated to sstablescrub

this sstable corrupt i ran into it as well (on cassandra 1.2), first i
try nodetool scrub, still persist, then offline sstablescrub still
persist, wipe the node and it happen again, then i change the hardware
(disk and mem). things went good.

hth

jason


On Fri, Aug 12, 2016 at 9:20 AM, Alaa Zubaidi (PDF)
<alaa.zubaidi@pdf.com> wrote:
> Hi,
>
> I have a 16 Node cluster, Cassandra 2.2.1 on Windows, local installation
> (NOT on the cloud)
>
> and I am getting
> Error [CompactionExecutor:2] 2016-08-12 06:51:52, 983 Cassandra
> Daemon.java:183 - Execption in thread Thread[CompactionExecutor:2,1main]
> org.apache.cassandra.io.FSReaderError:
> org.apache.cassandra.io.sstable.CorruptSSTableExecption:
> org.apache.cassandra.io.compress.CurrptBlockException:
> (E:\........\la-4886-big-Data.db): corruption detected, chunk at 4969092 of
> length 10208.
>     at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:357)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> ....
> ....
> ERROR [CompactionExecutor:2] ....... FileUtils.java:463 - Existing
> forcefully due to file system exception on startup, disk failure policy
> "stop"
>
> I tried sstablescrub but it crashed with hs-err-pid-...
> I removed the corrupted file and started the Node again, after one day the
> corruption came back again, I removed the files, and restarted Cassandra, it
> worked for few days, then I ran "nodetool repair" after it finished,
> Cassandra failed again but with commitlog corruption, after removing the
> commitlog files, it failed again with another sstable corruption.
>
> I was also checking the HW, file system, and memory, the VMware logs showed
> no HW error, also the HW management logs showed NO problems or issues.
> Also checked the Windows Logs (Application and System) the only thing I
> found is on the system logs "Cassandra Service terminated with
> service-specific error Cannot create another system semaphore.
>
> I could not find any thing regarding that error, all comments point to
> application log.
>
> Any help is appreciated..
>
> --
>
> Alaa Zubaidi
>
>
> This message may contain confidential and privileged information. If it has
> been sent to you in error, please reply to advise the sender of the error
> and then immediately permanently delete it and all attachments to it from
> your systems. If you are not the intended recipient, do not read, copy,
> disclose or otherwise use this message or any attachments to it. The sender
> disclaims any liability for such unauthorized use. PLEASE NOTE that all
> incoming e-mails sent to PDF e-mail accounts will be archived and may be
> scanned by us and/or by external service providers to detect and prevent
> threats to our systems, investigate illegal or inappropriate behavior,
> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
> concerns about this process, please contact us at legal.department@pdf.com.

Mime
View raw message