Hi, Aaron Morton:
   I thought it would be OK when I delete all the commit log.  After delete and restarting, it still failed. So I stop all the write/read client and restart it, it still throw an exception:
DEBUG 16:32:16,265 Resetting pool for /xx.xx.xx.xx
DEBUG 16:32:16,265 No bootstrapping, leaving or moving nodes, and no relocating tokens -> empty pending ranges for WebSearch
DEBUG 16:32:16,265 removing expire time for endpoint : /xx.xx.xx.xx
 INFO 16:32:16,265 InetAddress /xx.xx.xx.xx is now UP
ERROR 16:32:16,265 Exception in thread Thread[GossipStage:1,5,main]
java.lang.RuntimeException: The provided key was not UTF8 encoded.
    at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:170)
    at org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:42)
    at org.apache.cassandra.db.Table.apply(Table.java:380)
    at org.apache.cassandra.db.Table.apply(Table.java:354)
    at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:262)
    at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:119)
    at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:169)
    at org.apache.cassandra.db.SystemTable.updatePeerInfo(SystemTable.java:268)
    at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1173)
    at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1895)
    at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:805)
    at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:883)
    at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:43)
    at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
    at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
    at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:167)
    at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:124)
    at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:166)
    ... 16 more


My cluster info:
Keyspace: WebSearch:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
  Durable Writes: true
    Options: [replication_factor:2]
  Column Families:
    ColumnFamily: UrlCrawlStatsCF
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.BytesType
      GC grace seconds: 864000
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.2
      DC Local Read repair chance: 0.0
      Replicate on write: false
      Caching: KEYS_ONLY
      Bloom Filter FP chance: default
      Built indexes: []
      Compaction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrategy
      Compaction Strategy Options:
        sstable_size_in_mb: 256
      Compression Options:
        chunk_length_kb: 64
        sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor


I use c++ thrift client to write to cassandra cluster and the key is the hex string of 64bit unsiged int. It had been OK for a month.
Why suddenly there is a problem with the key? I am so confused with that.

Thank you!

On 03/19/2013 04:21 PM, aaron morton wrote:
There is a write in the commit log that has an invalid row key. The logs will say which file it was replaying, try removing that and restarting it. 

What client were you using to write to the cluster? 
What are you using for the keys ? 

Note: this could result in data loss on the one node because writes in the commit log may not be committed to sstables. If you have been using QUOURM / QUOURM this should not be a problem. 

    at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:170)
If you are just starting out I strongly encourage you to use the RandomPartitioner rather than the OderPreservingPartitioner. 

Cheers

 
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 19/03/2013, at 4:22 PM, 杨辉强 <huiqiangyang@yunrang.com> wrote:

Hi, all:

   I am a beginner of cassandra. I have a four node cassandra group.
   One of my cassandra group node had been running for a week. Recently because of too much writing and reading, it crashed.
I want to restart the node.
   While I start up cassandra, it ends up with throwing exception as follow every time:
   java.lang.RuntimeException: The provided key was not UTF8 encoded.
    at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:170)
    at org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:42)
    at org.apache.cassandra.db.Table.apply(Table.java:380)
    at org.apache.cassandra.db.Table.apply(Table.java:354)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:260)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

  I have googled and read the archive. But I can't find answer, please help me. Thank you very much.