lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar <susheel2...@gmail.com>
Subject Re: Dynamic fields vs parent child
Date Tue, 27 Jun 2017 12:39:35 GMT
Can you describe your use case in terms of what business functionality you
are looking to achieve.

Thanks,
Susheel

On Mon, Jun 26, 2017 at 4:26 PM, Saurabh Sethi <saurabh.sethi@sendgrid.com>
wrote:

> Number of dynamic fields will be in thousands (millions of users +
> thousands of events shared between subsets of users).
>
> We also thought about indexing in one field with value being
> fieldname_fieldvalue. Since we support range queries for dates and numbers,
> it won't work out of box.
>
> On Mon, Jun 26, 2017 at 1:05 PM, Erick Erickson <erickerickson@gmail.com>
> wrote:
>
> > How many distinct fields do you expect across _all_ documents? That
> > is, if doc1 has 10 dynamic fields and doc2 has 10 dynamic fields, will
> > there be exactly 10 fields total or more than 10 when you consider
> > both documents?
> >
> > 100s of fields total across all documents is a tractable problem.
> > thousands of dynamic fields total is going to be a problem.
> >
> > One technique that people do use is to index one field with a prefix
> > rather than N dynamic fields. So you have something like
> > dyn1_val1
> > dyn1_val2
> > dyn4_val67
> >
> > Only really works with string fields of course.
> >
> > Best,
> > Erick
> >
> > On Mon, Jun 26, 2017 at 10:11 AM, Saurabh Sethi
> > <saurabh.sethi@sendgrid.com> wrote:
> > > We have two requirements:
> > >
> > > 1. Indexing and storing event id and its timestamp.
> > > 2. Indexing and storing custom field name and value. The fields can be
> of
> > > any type, but for now lets say they are of types string, date and
> number.
> > >
> > > The events and custom fields for any solr document can easily be in
> > > hundreds.
> > >
> > > We are looking at two different approaches to handle these scenarios:
> > >
> > > 1. *Dynamic fields* - Have the fields name start with a particular
> > pattern
> > > like for string, the pattern could be like str_* and for event could be
> > > eventid_*
> > > 2. *Parent/child fields* - This seems to be an overkill for our use
> case
> > > since it's more for hierarchical data. Also, the parent and all its
> > > children need to be reindexed on update which defeats the purpose - we
> > are
> > > now reindexing multiple docs instead of one with dynamic fields. But it
> > > allows us to store custom field name along with its value unlike
> dynamic
> > > fields where we will have to map user supplied custom field to some
> other
> > > name based on type.
> > >
> > > Has anyone handled similar scenarios with Solr? If so, which approach
> > would
> > > you recommend based on your experience?
> > >
> > > We are using solr 6.6
> > >
> > > Thanks,
> > > Saurabh
> >
>
>
>
> --
> Saurabh Sethi
> Principal Engineer I | Engineering
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message