couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: [DISCUSS] _db_updates feed in FoundationDB
Date Wed, 10 Apr 2019 16:36:54 GMT
Thanks Alex - I need to get in the habit of posting more frequently over there :)

Adam

> On Apr 10, 2019, at 3:16 AM, Alex Miller <alexmiller@apple.com.INVALID> wrote:
> 
> 
>> On 2019/03/20 22:47:42, Adam Kocoloski <kocolosk@apache.org> wrote: 
>> I don’t know in detail what optimizations FoundationDB applies to atomic operations
(e.g. coalescing them at a layer above the storage engine). That’s worth checking into,
as otherwise I’d be concerned about introducing super-hot keys here.
> 
> 
> Inducing hot keys or hot shards would both be of concern.  There is logic that will try
to deal with write hotspots, but it splits shard based on write bandwidth to a particular
shard, which likely wouldn’t be triggered by a stream of atomic operations.
> 
> Clients will merge atomic operations together if you issue multiple in a transaction,
e.g. I believe three ATOMIC_ADDs on one key will become one ATOMIC_ADD of the sum.  There
isn’t particularly any logic that optimizes the handling of atomic operations once they
leave the client.  Storage servers only commit every ~250ms, so this shouldn’t translate
into a full page write on each atomic operation to the same key, but the storage server will
end up applying each atomic operation individually.
> 
> ( And I’ve gone and filed https://github.com/apple/foundationdb/issues/1450 to trigger
more discussion within FoundationDB on this topic. )
> 
> 
>> On Mar 25, 2019, at 11:42 AM, Adam Kocoloski <kocolosk@apache.org> wrote:
>>> On Mar 25, 2019, at 12:48 PM, Mike Rhodes <couchdb@dx13.co.uk> wrote:
>>> 
>>> I couldn't immediately see how we cleared out older entries from this potentially
very large queue. For example, the worker processing the queue to deduplicate might issue
range deletes after processing each "batch". Is this simple enough to do?
>> 
>> Yes, that’s the (implicit) idea. Simple to implement, not clear to me how well
the storage servers can handle the load. I think the “range clears are cheap” statement
largely refers to the transaction management system.
> 
> At the storage level, range clears are cheap in terms of immediate execution, and slow
in terms of total work performed.  You can find the details of this in https://forums.foundationdb.org/t/shards-are-not-splitted-into-smaller-ones/815/4
, and an example of what to be careful about with range clears on https://forums.foundationdb.org/t/used-disk-space-dramatically-increases-while-sum-of-key-value-sizes-is-constant/644
> 
> 
>> On Mar 27, 2019, at 11:07 AM, Ilya Khlopotov <iilyak@apache.org> wrote:
>> What if FDB would support a list type for a value and would have an atomic operation
to add the value to the list if it is missing. In this case we could store the data we need
as follows (under separate subspace TBD).
>> VersionStamp = (DbName, EventType)
>> DbName = [versionstamps]
>> ...
>> The question is how we would implement atomic addition of a value to a list. 
> 
> If I’ve understood your proposal correctly, I think it sounds easier to just use APPEND_IF_FITS
?
> 
> https://apple.github.io/foundationdb/javadoc/index.html?com/apple/foundationdb/MutationType.html
> 


Mime
View raw message