lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Atomic Updates
Date Thu, 27 Apr 2017 23:41:39 GMT
Been there, done that, got the t-shirt. Thanks for closing it out!

Erick

On Thu, Apr 27, 2017 at 10:29 AM, Chris Ulicny <culicny@iq.media> wrote:
> While recreating it with a fresh schema, I realized that this was a case of
> a very, very stupid user error during configuring the cores.
>
> I setup the testing cores with the wrong configset, and then proceeded to
> edit the schema in the right configset. So, the field was actually stored
> by default, but I wasn't attempting to retrieve it so I never realized it
> was being stored since I ended up looking at the wrong schema.
>
> I switched the config sets and everything works as expected. Any atomic
> updates clear out the indexed values for the non-stored field.
>
> Thanks for bearing with me.
> Chris
>
>
> On Thu, Apr 27, 2017 at 11:23 AM Chris Ulicny <culicny@iq.media> wrote:
>
>> I'm sending commit=true with every update while testing. I'll write up the
>> tests and see if someone else can reproduce it.
>>
>> On Thu, Apr 27, 2017 at 10:54 AM Erick Erickson <erickerickson@gmail.com>
>> wrote:
>>
>>> bq: but is there any possibility that the values stick around until
>>> there is a segment merge for some strange reason
>>>
>>> There better not be or it's a bug. Things will stick around until
>>> you issue a commit, is there any chance that's the problem?
>>>
>>> If you can document the exact steps, maybe we can reproduce
>>> the issue and raise a JIRA.
>>>
>>> Best,
>>> Erick
>>>
>>> On Thu, Apr 27, 2017 at 6:03 AM, Chris Ulicny <culicny@iq.media> wrote:
>>> > Yeah, something's not quite right somewhere. We never even considered
>>> > in-place updates an option since it requires the fields to be
>>> non-indexed
>>> > and non-stored. Our schemas never have any field that satisfies those
>>> two
>>> > conditions let alone the other necessary ones.
>>> >
>>> > I went ahead and tested the atomic updates on different textfields, and
>>> I
>>> > still can't get the indexed but not-stored othertext_field to
>>> disappear. So
>>> > far set, add, and remove updates do not change it regardless of what the
>>> > fields are in the atomic update.
>>> >
>>> > It would be extraordinarily useful if this update behavior is now
>>> expected
>>> > (but not currently documented) functionality.
>>> >
>>> > I'm not too familiar with the nitty-gritty details of merging segment
>>> > files, but is there any possibility that the values stick around until
>>> > there is a segment merge for some strange reason?
>>> >
>>> > On Thu, Apr 27, 2017 at 12:59 AM Dorian Hoxha <dorian.hoxha@gmail.com>
>>> > wrote:
>>> >
>>> >> @Chris,
>>> >> According to doc-link-above, only INC,SET are in-place-updates. And
>>> only
>>> >> when they're not indexed/stored, while your 'integer-field' is. So
>>> still
>>> >> shenanigans in there somewhere (docs,your-code,your-test,solr-code).
>>> >>
>>> >> On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <culicny@iq.media>
>>> wrote:
>>> >>
>>> >> > That's probably it then. None of the atomic updates that I've tried
>>> have
>>> >> > been on TextFields. I'll give the TextField atomic update to verify
>>> that
>>> >> it
>>> >> > will clear the other field.
>>> >> >
>>> >> > Has this functionality been consistent since atomic updates were
>>> >> > introduced, or is this a side effect of some other change? It'd
be
>>> very
>>> >> > convenient for us to use this functionality as it currently works,
>>> but if
>>> >> > it's something that prevents us from upgrading versions in the
>>> future, we
>>> >> > should probably avoid expecting it to work.
>>> >> >
>>> >> > On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
>>> >> > ichattopadhyaya@gmail.com> wrote:
>>> >> >
>>> >> > > > Hmm, interesting. I can imagine that as long as you're
updating
>>> >> > > > docValues fields, the other_text field would be there.
But the
>>> >> instant
>>> >> > > > you updated a non-docValues field (text_field in your
example)
>>> the
>>> >> > > > other_text field would disappear
>>> >> > >
>>> >> > > I can confirm this. When in-place updates to DV fields are
done,
>>> the
>>> >> rest
>>> >> > > of the fields remain as they were.
>>> >> > >
>>> >> > > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <
>>> >> erickerickson@gmail.com
>>> >> > >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > Hmm, interesting. I can imagine that as long as you're
updating
>>> >> > > > docValues fields, the other_text field would be there.
But the
>>> >> instant
>>> >> > > > you updated a non-docValues field (text_field in your
example)
>>> the
>>> >> > > > other_text field would disappear.
>>> >> > > >
>>> >> > > > I DO NOT KNOW this for a fact, but I'm asking people
who do.
>>> >> > > >
>>> >> > > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <
>>> >> dorian.hoxha@gmail.com>
>>> >> > > > wrote:
>>> >> > > > > There are In Place Updates, but according to docs
they stll
>>> >> shouldn't
>>> >> > > > work
>>> >> > > > > in your case:
>>> >> > > > > https://cwiki.apache.org/confluence/display/solr/
>>> >> > > > Updating+Parts+of+Documents
>>> >> > > > >
>>> >> > > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny
>>> <culicny@iq.media>
>>> >> > > wrote:
>>> >> > > > >
>>> >> > > > >> That's the thing I'm curious about though. As
I mentioned in
>>> the
>>> >> > first
>>> >> > > > >> post, I've already tried a few tests, and the
value seems to
>>> still
>>> >> > be
>>> >> > > > >> present after an atomic update.
>>> >> > > > >>
>>> >> > > > >> I haven't exhausted all possible atomic updates,
but 'set' and
>>> >> 'add'
>>> >> > > > seem
>>> >> > > > >> to preserve the non-stored text field.
>>> >> > > > >>
>>> >> > > > >> Thanks,
>>> >> > > > >> Chris
>>> >> > > > >>
>>> >> > > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha
<
>>> >> > dorian.hoxha@gmail.com>
>>> >> > > > >> wrote:
>>> >> > > > >>
>>> >> > > > >> > You'll lose the data in that field. Try
doing a commit and
>>> it
>>> >> > should
>>> >> > > > >> > happen.
>>> >> > > > >> >
>>> >> > > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris
Ulicny
>>> <culicny@iq.media
>>> >> >
>>> >> > > > wrote:
>>> >> > > > >> >
>>> >> > > > >> > > Thanks Shawn, I didn't realize docValues
were enabled by
>>> >> default
>>> >> > > > now.
>>> >> > > > >> > > That's very convenient and probably
makes a lot of the
>>> schemas
>>> >> > > we've
>>> >> > > > >> been
>>> >> > > > >> > > making excessively verbose.
>>> >> > > > >> > >
>>> >> > > > >> > > This is on 6.3.0. Do you know what
the first version was
>>> that
>>> >> > they
>>> >> > > > >> added
>>> >> > > > >> > > the docValues by default for non-Text
field?
>>> >> > > > >> > >
>>> >> > > > >> > > However, that shouldn't apply to this
since I'm concerned
>>> >> with a
>>> >> > > > >> > non-stored
>>> >> > > > >> > > TextField without docValues enabled.
>>> >> > > > >> > >
>>> >> > > > >> > > Best,
>>> >> > > > >> > > Chris
>>> >> > > > >> > >
>>> >> > > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn
Heisey <
>>> >> > apache@elyograg.org
>>> >> > > >
>>> >> > > > >> > wrote:
>>> >> > > > >> > >
>>> >> > > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny
wrote:
>>> >> > > > >> > > > > Hello all,
>>> >> > > > >> > > > >
>>> >> > > > >> > > > > Suppose I have the following
fields in a document and
>>> >> > populate
>>> >> > > > all
>>> >> > > > >> 4
>>> >> > > > >> > > > fields
>>> >> > > > >> > > > > for every document.
>>> >> > > > >> > > > >
>>> >> > > > >> > > > > id: uniqueKey, indexed and
stored
>>> >> > > > >> > > > > integer_field: indexed and
stored
>>> >> > > > >> > > > > text_field: indexed and
stored
>>> >> > > > >> > > > > othertext_field: indexed
but not stored
>>> >> > > > >> > > > >
>>> >> > > > >> > > > > No default values, multivalues,
docvalues,
>>> copyfields, or
>>> >> > any
>>> >> > > > other
>>> >> > > > >> > > > > properties set.
>>> >> > > > >> > > >
>>> >> > > > >> > > > You didn't indicate the Solr
version.  In recent Solr
>>> >> > versions,
>>> >> > > > most
>>> >> > > > >> > > > field classes other than TextField
have docValues
>>> enabled by
>>> >> > > > default,
>>> >> > > > >> > > > even if the config is not mentioned
on the field, and in
>>> >> those
>>> >> > > > >> > versions,
>>> >> > > > >> > > > docValues will take the place
of stored if stored is
>>> false.
>>> >> > > > >> > > >
>>> >> > > > >> > > > Thanks,
>>> >> > > > >> > > > Shawn
>>> >> > > > >> > > >
>>> >> > > > >> > > >
>>> >> > > > >> > >
>>> >> > > > >> >
>>> >> > > > >>
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>>
>>

Mime
View raw message