I've somehow now ended up in a very strange place...
If I ask '150' or '155' about the ring they report each other, but if I
ask the rest of the ring they have '155' but not '150' as members. All
of the storage-conf files are basically clones of each other with the
same ring masters.
$ nodetool -h 10.1.0.155 ring
Address Status Load
Range Ring
99811300882272441299595351868344045866
10.3.0.150 Up 3.08 TB
6436333895300580402214871779965756352 |<--|
10.1.0.155 Up 3.08 TB
99811300882272441299595351868344045866 |-->|
$ nodetool -h 10.2.0.174 ring
Address Status Load
Range Ring
144951579690133260853298391132870993575
10.2.0.115 Up 2.7 TB
55758122058160717108501182340054262660 |<--|
10.1.0.155 Up 3.08 TB
99811300882272441299595351868344045866 | ^
10.2.0.174 Up 1.32 TB
118283207506463595491596277948095451613 v |
10.3.0.151 Up 414.51 GB
127520031787005730998588483181387651399 | ^
10.3.0.152 Up 143.03 GB
132137258578258111824507171284723589567 v |
10.3.0.153 Up 245.51 GB
134446064220575108370358944111505967571 | ^
10.2.0.175 Up 3.16 TB
136754979922617117666448707835107404441 v |
10.2.0.114 Up 1.41 TB
144951579690133260853298391132870993575 |-->|
$ nodetool -h 10.3.0.150 streams
Mode: Normal
Not sending any streams.
Not receiving any streams.
$ nodetool -h 10.1.0.155 streams
Mode: Normal
Not sending any streams.
Not receiving any streams.
$ nodetool -h 10.2.0.115 streams
Mode: Normal
Not sending any streams.
Not receiving any streams.
On 5/11/10 6:30 AM, Jonathan Ellis wrote:
> s/keyspace/token/ and you've got it.
>
> On Mon, May 10, 2010 at 10:34 AM, David Koblas<koblas@extra.com> wrote:
>
>> Sounds great, will give it a go. However, just to make sure I understand
>> getting the keyspace correct.
>>
>> Lets say I've got:
>> A -- Node before overfull node in keyspace order
>> O -- Overfull node
>> B -- Node after O in keyspace order
>> N -- New empty node
>>
>> I'm going to assume that I should make the following assignment:
>> keyspace(N) = keyspace(A) + ( keyspace(O) - keyspace(A) ) / 2
>>
>> Or did I miss something else about keyspace ranges?
>> Thanks
>>
>>
>> On 5/7/10 1:25 PM, Jonathan Ellis wrote:
>>
>>> If you're using RackUnawareStrategy (the default replication strategy)
>>> then you can "bootstrap" manually fairly easily -- copy all the data
>>> (not system) sstables from an overfull machine to a new machine,
>>> assign the new one a token that gives it about half of the old node's
>>> range, then start it with autobootstrap OFF. Then run cleanup on both
>>> new and old nodes to remove the part of the data that belongs to the
>>> other.
>>>
>>> The downside vs real bootstrap is you can't do this safely while
>>> writes are coming in to the original node. You can reduce your
>>> read-only period by doing an intial scp, then doing a flush + rsync
>>> when you're ready to take it read only.
>>>
>>> (https://issues.apache.org/jira/browse/CASSANDRA-579 will make this
>>> problem obsolete for 0.7 but that doesn't help you on 0.6, of course.)
>>>
>>> On Fri, May 7, 2010 at 2:08 PM, David Koblas<koblas@extra.com> wrote:
>>>
>>>
>>>> I've got two (out of five) nodes on my cassandra ring that somehow got
>>>> too
>>>> full (e.g. over 60% disk space utilization). I've now gotten a few new
>>>> machines added to the ring, but evertime one of the overfull nodes
>>>> attempts
>>>> to stream its data it runs out of diskspace... I've tried half a dozen
>>>> different bad ideas of how to get things moving along a bit smoother, but
>>>> am
>>>> at a total loss at this point.
>>>>
>>>> Is there any good tricks to get cassandra to not need 2x the disk space
>>>> to
>>>> stream out, or is something else potentially going on that's causing me
>>>> problems?
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>
|