On Tue, Nov 2, 2010 at 8:32 AM, Joe Alex <joe.m.alex@gmail.com> wrote:
Thanks, that clarifies why HH did not work, So have to use .7. Is
there .6.7 ? I am using .6.6 now.

You can use the 0.6 branch instead, it's generally more stable than trunk.
I did see the log entry in Node 4, adding a Hint for Node 2 and when
Node 2 came up noticed a log entry in Node 4 that 1 row Handed Off -
so I thought it was working.

This is why it was a pernicious little bug that got me.   Logs indicated it worked, it just...didn't.

About RR - a read in Node 1 or Node 2 indicated a RR (see log entry
above) was happening. But dont think that ever happened. Those entries
keep repeating every time a read in Node 1 and 2 and kept giving old

When you read at CL.ONE, RR is asynchronous -- it happens after your data is returned.  CL.QUORUM would return the right data after performing the RR.
AES assuming Anti-Entropy - Tried nodetool repair, this also did not
fix the issue.
That is odd.  Are you sure repair had finished before the test?  In 0.6, nodetool doesn't block until repair completes (but does in 0.7)
The only time I saw it being fixed was a read to Node 3 or Node 4
(mostly Node 3 which acted as responsible when Node 2 was down)  and
then 2nd reads to Node 2 started showing the latest data.

Right, node 3/4 was authoritative for CL.ONE since it had a copy of the data.  After you read it, RR worked and fixed node 2.
Do you think RR and AES should have definitely worked ?

I suspect RR worked, and you weren't waiting on AES long enough (it essentially performs a major compaction, and then streams whatever needs to be sent to the others, which can take a while depending on the volume of data.)