lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Rafalovitch <arafa...@gmail.com>
Subject Re: Enforce "reasonable" field names in Solr?
Date Thu, 12 Feb 2015 19:26:44 GMT
I wonder if the people who are using dynamic schema care about having
the fields indexed without _them_ doing pre-processing, but don't mind
if they have to use cleaned-up names during search. Like, when you
index from Tika and you just have no clue what possible metadata names
are in various files. So, you just want to throw the whole lot in,
prefixed.

In which case, this could be solved with a specialized
UpdateRequestProcessor step that will normalize the field names in a
consistent fashion.

Regards,
   Alex.
----
Sign up for my Solr resources newsletter at http://www.solr-start.com/


On 12 February 2015 at 13:02, Erick Erickson <erickerickson@gmail.com> wrote:
> Jack:
>
> re: your "little gotcha". I suspect there are enough of these lying
> around that it'd be a rat-hole to formally support them, and as a
> developer I'd at least like the choice to "fail early fail often".
>
> Your point about dynamic field names is well taken, sometimes there
> isn't total control of the field names. Which is why I suggested that
> the strict mode be the default, but overridable.
>
> So not only does the bit about verifying the field names need to take
> managed schema into account, but also dynamic field definition...
> Siiiggggh...
>
> That is, if we do anything about it.
>
>
> On Thu, Feb 12, 2015 at 8:35 AM, Jack Krupansky
> <jack.krupansky@gmail.com> wrote:
>> I used to be 100% in favor of strict names (well, plus the hyphen!), and in
>> general it is fine for statically declared fields. But then I started
>> encountering uses of numbers, spaces, slashes, and other punctuation, but
>> always in the context of dynamic fields. For example, somebody wants to
>> support a map-like field using dynamic fields with a dynamic field for each
>> map key, but their map keys are application-defined and not restricted to
>> Java name rules, such as a date with punctuation, or something that looks
>> like a part number with numbers and dashes, or a product name or person or
>> place name that has spaces and dashes and slashes and commas and periods and
>> parentheses.
>>
>> The big question is how might Solr depend on strict names, and then how to
>> properly escape improper field names. There are a lot of spaces that use
>> field names within some larger syntax, but no consistent escaping rules. For
>> example, the fl and qf parameters, and fielded queries.
>>
>> Maybe the real bottom line is to assure that the issue of field naming needs
>> to be clearly documented early on in tutorials and upfront in the doc,
>> rather than some relatively hidden fine print.
>>
>> Hmmm... what does Elasticsearch do? As long as the field name is simply a
>> single quoted string, then there is no issue.
>>
>> Oh, here's a great little gotcha: field names embedded in parameters that
>> are field-specific, like f.<field-name>.facet. URL escaping would be needed,
>> but are names with embedded dots supported? And does the URL query parameter
>> syntax support escaping of an equal sign in a query parameter name?
>>
>>
>> -- Jack Krupansky
>>
>> On Thu, Feb 12, 2015 at 10:30 AM, Erick Erickson <erickerickson@gmail.com>
>> wrote:
>>>
>>> I was commenting on SOLR-6997 about allowing hyphens in field names
>>> and started to wonder about whether we should try to push people to
>>> "good" names. The ref guide states:
>>>
>>> "Field names should consist of alphanumeric or underscore characters
>>> only and not start with a digit"
>>>
>>> and SOLR-6997 is a good example of why. I am _not_ at all interested
>>> in supporting the hyphen BTW.
>>>
>>> I realize we can't suddenly start enforcing this rule b/c it would
>>> break existing installations. What do people think about defaulting to
>>> throwing an error? Or posting a fat warning with a "deprecation"
>>> message?
>>>
>>> I'm envisioning a "strict_field_name" tag or some such that defaults
>>> to true, but could be set to false for back compat and just checking
>>> when parsing a schema.
>>>
>>> I'm not at all sure how that plays with the managed schema stuff though.
>>>
>>> Erick
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message