couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garren Smith <gar...@apache.org>
Subject Re: [DISCUSS] Implementing Mango Indexes for FoundationDB
Date Mon, 08 Apr 2019 13:50:24 GMT
On Tue, Apr 2, 2019 at 3:14 AM Adam Kocoloski <kocolosk@apache.org> wrote:

> Hi Will, great comments, I have replies to a couple of them.
>
> > On Apr 1, 2019, at 5:21 AM, Will Holley <willholley@gmail.com> wrote:
> >
> > 2. Does the ICU sort key have a bounded length? Mostly I'm wondering
> > whether we can guarantee that the generated keys will fit within the
> > maximum FDB key length or if there needs to be some thought as to the
> > failure mode / workaround. As Adam mentioned, it seems fine to store an
> > encoded key given Mango (currently) always fetches the associated
> document
> > / fields from the primary index to filter on anyway. It might even be
> > beneficial to have an additional layer of indirection and allow multiple
> > docs to be associated with each row so that we can maintain compact keys.
>
> Interesting thought on that layer of indirection; it reminds me of an
> optimization applied in the Record Layer’s text indexes. Would have to
> compare whether the extra reads needed to maintain the index that way are
> an acceptable tradeoff.
>
> Good point on the sort key sizes, I’ve not seen any way to place a
> reliably safe upper bound on the size of one that might be generated. The
> ICU folks have some hand-wavey guidance at
> http://userguide.icu-project.org/collation/architecture#TOC-Sort-key-size,
> but it seems like we might be able to dig a little deeper.
>
> I personally haven’t given much thought to a workaround where a
> user-defined index key exceeds 10 KB. We’ll definitely need to handle that
> failure mode safely even without the sort key complication — people try
> crazy things :)
>

For the 10 KB error, I think we should just return an error. As a
comparison, MongoDB has a 1024 Byte limit
https://docs.mongodb.com/manual/reference/limits/#Index-Key-Limit


> > 3. I don't immediately see how you clear previous values from the index
> > when a doc is updated, but I could easily be missing something obvious :)
>
> Ah yeah, this part wasn’t explicit, was it?
>
> I think the idea is that these are simple indexes on specific fields of a
> document, and we have a data model where those fields are already stored as
> their own keys in FDB, so there’s no need (in the case of Mango) to
> maintain a separate docid -> {viewid, [keys]} mapping like we do today in
> each view group. Rather, the flow would go something like
>
> 1) Check which fields are supposed to be indexed
> 2) Retrieve values for those fields in the ?DOCUMENTS space for the parent
> revision
> 3) Compare the parent values with the ones supplied in this transaction;
> if any indexed values change, clear the old ones and insert the new ones
>
> with some additional caveats around checking that the supplied edit is
> actually going to be winning (and therefore indexed) version after the
> commit succeeds.
>
> > 4. Regarding "Index on write" behaviour, is there something in the
> existing
> > design (Mango overlaying mrview / lucene) that would prevent this? I can
> > see some benefit for certain workloads (and headaches for others) but I
> > don't see that it's necessarily coupled to the Mango design given
> > background indexing of new/changed indexes needs to be supported anyway.
>
> I’m not sure I understand your question. In my mind the reason “index on
> write" is more applicable for Mango JSON than for generalized views is
> because in the view case batching is currently quite important to achieve
> good throughput to the JS system. You’re of course correct that we need to
> be able to re-generate Mango JSON indexes in the background as well.
>
> Adam
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message