lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: Dynamic data model design questions
Date Sun, 21 Apr 2013 19:07:19 GMT
1. Relatively small numbers of dynamic fields are fine. 20-40 would be fine. 
1,000 would become problematic. "hundreds" would also likely be problematic, 
but it would depend on the application and its data.
2. Sparse dynamic fields are less problematic than larger numbers of fully 
populated fields.
3. Keep the field names to Java-like rules, no special characters, just 
letters, digits, and underscores.
4. Keep the field name length relative short - don't store large data values 
in the field names themselves.

Generally, "know your data". Static is preferable, but if your data gets too 
dynamic and you no longer have any clue as to how fields there are, how many 
are populated, and what the profile is for the field values, then you won't 
be able to keep on top of what's going on in your index.

-- Jack Krupansky

-----Original Message----- 
From: Marko Asplund
Sent: Saturday, April 20, 2013 4:16 PM
To: solr-user
Subject: Re: Dynamic data model design questions

Jack Krupansky wrote:

> In general, Solr is much more friendly towards static data models. Yes, 
> you
> can use dynamic fields, but use them in moderation. The more heavily you
> lean on them, the more likely that you will eventually become unhappy with
> Solr.

Can you concrete examples of what kinds of issues should I expect to
face when using a data model with only dynamic fields?
We've requirements that quite explicitly direct us into using dynamic
fields and I'd like to understand what kinds of problems we might end
up having.

> How many fields are we talking about here?

The data model is designed to be dynamic, so the number is not fixed,
but I'm expecting there'll be perhaps about 20-40 fields.

> The trick with Solr is not to brute-force flatten your data model (as you
> appear to be doing), but to REDESIGN your data model so that it is more
> amenable to a flat data model, and takes advantage of Solr's features. You
> can use multiple collections for different types of data. And you can
> simulate joins across tables by doing a sequence of queries (although it
> would be nice to have a SolrJ client-side method to do that in one API
> call.)

We're storing arbitrarily complex object graphs in a data store and
want to use Solr for implementing search property field search.
It may be difficult to use a flatter data model, but I'll consider
this option as well.

thanks!

marko 


Mime
View raw message