Just had a good conversation with rcoli in chat. Wanted to clarify the steps for resolving this issue and see if there are any pitfalls I am missing.
Solution: Somehow I have to make Cassandra system's column family to forget about those old schemas with nanosecond timestamps. I have to do this either live or with a brief downtime. Please advice of any pitfalls or incorrectness in my steps. I am planning to automate them so please advice.
Within a short downtime, I have to do this:
1. Take all nodes out of service;
2. Run nodetool flush on each;
3. Stop cassandra on each node;
4. Remove /var/lib/cassandra/data/system
5. Remove /var/lib/cassandra/saved_caches/system-*
6. Start all nodes;
7. cassandra-cli < schema_definition_file on one node only. (includes create keyspace and create column familiy entries)
8. put the nodes back in service.
Please advice if I have got the steps correctly or if I am missing something.
Thanks in advance for you help.