[ https://issues.apache.org/jira/browse/SOLR-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uri Boness updated SOLR-1625:
-----------------------------
Attachment: SOLR-1625.patch
Updated the patch to support the following changes (as discussed above):
- using "terms.regex" param (instead of "terms.regexp")
- using more explicit names for the regex flags
> Add regexp support for TermsComponent
> -------------------------------------
>
> Key: SOLR-1625
> URL: https://issues.apache.org/jira/browse/SOLR-1625
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Uri Boness
> Assignee: Noble Paul
> Priority: Minor
> Fix For: 1.5
>
> Attachments: SOLR-1625.patch, SOLR-1625.patch, SOLR-1625.patch
>
>
> At the moment the only way to filter the returned terms is by a prefix. It would be nice
it the filter could also be done by regular expression
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
OK, I'm fine w/ taking this type of approach, as opposed to the lookup mechanism I have. Of the two laid out below, there are pros and cons to both, as I see it. I'm inclined towards Option B. This keeps it "hidden" from the user, but doesn't require extra work for Solr. Let me code it up. On Dec 9, 2009, at 4:12 PM, Yonik Seeley wrote: > Proposal for handling points using only the field lookup mechanisms > currently in place in IndexSchema: > > Option A: dynamic fields used for subfields, those dynamic fields need > to be explicitly defined in the XML > ============================================================================ > // needed to essentially define the point type > <fieldType name="latlon" class="TrieDoubleFIeld" precisionStep="8"/> > <fieldType name="point" subFieldSuffix="_latlon" .../> > <dynamicField name="*_latlon" type="latlon" indexed="true" stored="false"/> > > // uses of the point type > <field name="home" type="point"/> > <dynamicField name="*_point" type="point"/> > > // subFieldSuffix is appended to the subFields indexed and thus those would be > home__0_latlon > home__1_latlon > > // And the indexed fields for dynamic field work_point would be > work_point__0_latlon > work_point__1_latlon > > // NOTE: this scheme works fine for subFields with different fieldTypes > > Option B: dynamic fields used for subfields, dynamic fields inserted > into schema automatically > ==================================================================== > // needed to essentially define the point type > <fieldType name="latlon" class="TrieDoubleFIeld" precisionStep="8"/> > <fieldType name="point" subFieldType="latlon"/> > > // uses of the point type > <field name="home" type="point"/> > <dynamicField name="*_point" type="point"/> > > // A dynamic field is inserted into the schema by the point class of > the form __<subFieldTypeName> by default. > // This could be changed via an optional subFieldSuffix param on the > point fieldType. double underscore used > // to minimize collisions with user-defined dynamic fields. > home_0__latlon > home_1__latlon > > // And the indexed fields for dynamic field work_point would be > work_point__0__latlon > work_point__1__latlon > > // NOTE: this scheme works fine for subFields with different fieldTypes
[ https://issues.apache.org/jira/browse/SOLR-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788319#action_12788319
]
Yonik Seeley commented on SOLR-1131:
------------------------------------
> Aside: it looks like the code for getFieldOrNull isn't right? Seems like it will return
a field with both the wrong type and the wrong name?
> > Hmmm, I think it should return the "owning" Schema Field, i.e. the one that exists
in the schema.xml file.
Those fields probably will be exposed at least internally to other parts of solr, so they
should really return the correct field / fieldType.
> Allow a single field type to index multiple fields
> --------------------------------------------------
>
> Key: SOLR-1131
> URL: https://issues.apache.org/jira/browse/SOLR-1131
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Assignee: Grant Ingersoll
> Fix For: 1.5
>
> Attachments: SOLR-1131-IndexMultipleFields.patch, SOLR-1131.patch, SOLR-1131.patch,
SOLR-1131.patch, SOLR-1131.patch, SOLR-1131.patch
>
>
> In a few special cases, it makes sense for a single "field" (the concept) to be indexed
as a set of Fields (lucene Field). Consider SOLR-773. The concept "point" may be best indexed
in a variety of ways:
> * geohash (sincle lucene field)
> * lat field, lon field (two double fields)
> * cartesian tiers (a series of fields with tokens to say if it exists within that region)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Proposal for handling points using only the field lookup mechanisms currently in place in IndexSchema: Option A: dynamic fields used for subfields, those dynamic fields need to be explicitly defined in the XML ============================================================================ // needed to essentially define the point type <fieldType name="latlon" class="TrieDoubleFIeld" precisionStep="8"/> <fieldType name="point" subFieldSuffix="_latlon" .../> <dynamicField name="*_latlon" type="latlon" indexed="true" stored="false"/> // uses of the point type <field name="home" type="point"/> <dynamicField name="*_point" type="point"/> // subFieldSuffix is appended to the subFields indexed and thus those would be home__0_latlon home__1_latlon // And the indexed fields for dynamic field work_point would be work_point__0_latlon work_point__1_latlon // NOTE: this scheme works fine for subFields with different fieldTypes Option B: dynamic fields used for subfields, dynamic fields inserted into schema automatically ==================================================================== // needed to essentially define the point type <fieldType name="latlon" class="TrieDoubleFIeld" precisionStep="8"/> <fieldType name="point" subFieldType="latlon"/> // uses of the point type <field name="home" type="point"/> <dynamicField name="*_point" type="point"/> // A dynamic field is inserted into the schema by the point class of the form __<subFieldTypeName> by default. // This could be changed via an optional subFieldSuffix param on the point fieldType. double underscore used // to minimize collisions with user-defined dynamic fields. home_0__latlon home_1__latlon // And the indexed fields for dynamic field work_point would be work_point__0__latlon work_point__1__latlon // NOTE: this scheme works fine for subFields with different fieldTypes -Yonik http://www.lucidimagination.com
[ https://issues.apache.org/jira/browse/SOLR-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Hatcher updated SOLR-1639:
-------------------------------
Summary: Misleading error message when dataimport.properties is not writable (was: Missleading
error message when dataimport.properties is not writable)
> Misleading error message when dataimport.properties is not writable
> -------------------------------------------------------------------
>
> Key: SOLR-1639
> URL: https://issues.apache.org/jira/browse/SOLR-1639
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Reporter: Hoss Man
>
> if dataimport.properties does not exist and/or is not writable, the resulting behavior
fro DIH is (evidiently) very confusing...
> http://old.nabble.com/Question-about-the-message-%22Indexing-failed.-Rolled-back-all--changes.%22-to26242714.html#a26459272
> DIH should make a best effort to create this file if it doesn't already eixst, and generate
a meaningful error message if it can't create/write to the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
On Dec 9, 2009, at 3:52 PM, Yonik Seeley wrote: > On Wed, Dec 9, 2009 at 3:49 PM, Grant Ingersoll <gsingers@apache.org> wrote: >> >> On Dec 9, 2009, at 3:47 PM, Yonik Seeley wrote: >>> >>> I thought I defined it well... hmmm. >>> I'll take another stab, outlining using dynamic fields in both >>> scenarios (explicitly defined dynamic fields, and automatically >>> defined as part of the creation of the point class). I think we >>> really do need to get concrete about our options at this point. >> >> Agreed, code would be good. > > I had code (untested) just using dynamic fields... you changed it :-P > But I meant actual fieldType and field definitions, and what fields > get indexed as a result, and how type lookups on those fields happens. > Fair enough!
On Wed, Dec 9, 2009 at 3:49 PM, Grant Ingersoll <gsingers@apache.org> wrote: > > On Dec 9, 2009, at 3:47 PM, Yonik Seeley wrote: >> >> I thought I defined it well... hmmm. >> I'll take another stab, outlining using dynamic fields in both >> scenarios (explicitly defined dynamic fields, and automatically >> defined as part of the creation of the point class). I think we >> really do need to get concrete about our options at this point. > > Agreed, code would be good. I had code (untested) just using dynamic fields... you changed it :-P But I meant actual fieldType and field definitions, and what fields get indexed as a result, and how type lookups on those fields happens. -Yonik http://www.lucidimagination.com
[ https://issues.apache.org/jira/browse/SOLR-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788307#action_12788307
]
Grant Ingersoll commented on SOLR-1131:
---------------------------------------
Note, I don't think the distance function queries will work w/ my patch yet.
> Allow a single field type to index multiple fields
> --------------------------------------------------
>
> Key: SOLR-1131
> URL: https://issues.apache.org/jira/browse/SOLR-1131
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Assignee: Grant Ingersoll
> Fix For: 1.5
>
> Attachments: SOLR-1131-IndexMultipleFields.patch, SOLR-1131.patch, SOLR-1131.patch,
SOLR-1131.patch, SOLR-1131.patch, SOLR-1131.patch
>
>
> In a few special cases, it makes sense for a single "field" (the concept) to be indexed
as a set of Fields (lucene Field). Consider SOLR-773. The concept "point" may be best indexed
in a variety of ways:
> * geohash (sincle lucene field)
> * lat field, lon field (two double fields)
> * cartesian tiers (a series of fields with tokens to say if it exists within that region)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
On Dec 9, 2009, at 3:47 PM, Yonik Seeley wrote: > > I thought I defined it well... hmmm. > I'll take another stab, outlining using dynamic fields in both > scenarios (explicitly defined dynamic fields, and automatically > defined as part of the creation of the point class). I think we > really do need to get concrete about our options at this point. Agreed, code would be good.
On Wed, Dec 9, 2009 at 3:21 PM, Grant Ingersoll <gsingers@apache.org> wrote: > Additionally, how do you deal w/ a point in a 3D (or n-D) space? I guess you would go back to the way you did it (0,1,etc). This was really just a naming variation, not really a different approach. > I just don't see why a user shouldn't be able to use the FieldType just like any other FieldType, dynamic or not. I think it is easy enough to detect name collisions and you still get all the flexibility of dynamic fields. > So, for example, say I was modeling a user and their employment history. Thus, I have a single home address plus multiple work addresses. One way of doing this would be: > > <field name="home" type="point"/> > <dynamicField name="work_*" type="point"/> > > And that should all just work. But it isn't that simple: you needed to define the point type, and that point type needed to reference/define another type. In the dynamicField proposal, you need to define a _latlon dynamic field once. It's also a separate decision from the lookup mechanism (dynamic field based, or add a new poly-field mechanism) - the point field type could choose to dynamically register *_latlon if it isn't already registered. [...] > How would you do this with what is proposed above? Seems like you'd have a whole proliferation of fields. I thought I defined it well... hmmm. I'll take another stab, outlining using dynamic fields in both scenarios (explicitly defined dynamic fields, and automatically defined as part of the creation of the point class). I think we really do need to get concrete about our options at this point. -Yonik
[ https://issues.apache.org/jira/browse/SOLR-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788298#action_12788298
]
Grant Ingersoll commented on SOLR-1131:
---------------------------------------
bq. OK... so the real issue is that this introduces a new mechanism to look up field types...
not necessarily a horrible thing, but we should definitely think twice before doing so.
Agreed. I'm not wedded to this approach, just want to see the discussion through. I do feel
strongly that the goal is such that an app designer should be able to use a FieldType just
as they always have, either dynamic or static. How we get to that I don't care so much as
long as it works and performs.
bq. But... that scheme seems to limit us to a single subField type (in addition to the other
downsides of requiring a new lookup mechanism).
I don't follow this. In this particular implementation, I have a single subFieldType, but
I don't see why a different implementation couldn't do something like:
{code}
<fieldType name="foo" type="solr.MultiSubPointType" dimension="3" subFieldTypes="double,tdouble,int"/>
{code}
bq. Aside: it looks like the code for getFieldOrNull isn't right? Seems like it will return
a field with both the wrong type and the wrong name?
Hmmm, I _think_ it should return the "owning" Schema Field, i.e. the one that exists in the
schema.xml file.
> Allow a single field type to index multiple fields
> --------------------------------------------------
>
> Key: SOLR-1131
> URL: https://issues.apache.org/jira/browse/SOLR-1131
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Assignee: Grant Ingersoll
> Fix For: 1.5
>
> Attachments: SOLR-1131-IndexMultipleFields.patch, SOLR-1131.patch, SOLR-1131.patch,
SOLR-1131.patch, SOLR-1131.patch, SOLR-1131.patch
>
>
> In a few special cases, it makes sense for a single "field" (the concept) to be indexed
as a set of Fields (lucene Field). Consider SOLR-773. The concept "point" may be best indexed
in a variety of ways:
> * geohash (sincle lucene field)
> * lat field, lon field (two double fields)
> * cartesian tiers (a series of fields with tokens to say if it exists within that region)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
On Dec 9, 2009, at 2:46 PM, Yonik Seeley wrote: > On Wed, Dec 9, 2009 at 2:41 PM, Yonik Seeley <yonik@lucidimagination.com> wrote: >> So... the question is, do we have a concrete alternative to this that >> is well fleshed out? > > I do, I do... just a little variant that is geo specific and hence > results in nicer names :-) > > <fieldType name="point" latSuffix="_lat" lonSuffix="_lon/> > <field name="home" type="point"/> > <dynamicField name="*_lat" type="tdouble" indexed="true" stored="false"/> > <dynamicField name="*_lon" type="tdouble" indexed="true" stored="false"/> > > <dynamicField name="*_point" type="point"/> > > home_lat > home_lon > > work_point_lat > work_point_lon > > Note: if you want the double or tripple underscore to help prevent > collisions... then you could use latSuffix="___lat" and define the > dynamic fields that way. Additionally, how do you deal w/ a point in a 3D (or n-D) space? I just don't see why a user shouldn't be able to use the FieldType just like any other FieldType, dynamic or not. I think it is easy enough to detect name collisions and you still get all the flexibility of dynamic fields. So, for example, say I was modeling a user and their employment history. Thus, I have a single home address plus multiple work addresses. One way of doing this would be: <field name="home" type="point"/> <dynamicField name="work_*" type="point"/> And that should all just work. The user would just ever deal w/ "home" or "work_*", but not have to deal w/ home___0 or whatever unless they really truly wanted to and even then I am not sure it is needed. How would you do this with what is proposed above? Seems like you'd have a whole proliferation of fields. Also, I don't see why a FieldType should have a dep. on a Field. Having a dependency on another FieldType seems reasonable, but I'm not sure about on a Field.
Missleading error message when dataimport.properties is not writable
--------------------------------------------------------------------
Key: SOLR-1639
URL: https://issues.apache.org/jira/browse/SOLR-1639
Project: Solr
Issue Type: Bug
Components: contrib - DataImportHandler
Reporter: Hoss Man
if dataimport.properties does not exist and/or is not writable, the resulting behavior fro
DIH is (evidiently) very confusing...
http://old.nabble.com/Question-about-the-message-%22Indexing-failed.-Rolled-back-all--changes.%22-to26242714.html#a26459272
DIH should make a best effort to create this file if it doesn't already eixst, and generate
a meaningful error message if it can't create/write to the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
On Wed, Dec 9, 2009 at 2:02 PM, Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov> wrote: >> What GIS tool could deal with a Solr XML response format w/o any other >> knowledge of everything else in the response? >> Are there some real use cases that using a namespace vs not for point >> make easier (an honest question... I don't know much about GIS stuff). > > Using standards enables standard tool development. We do use standards... lots of them :-) Let's be a bit more specific though - I was asking about using a namespace for the point type by *default*, and in isolation (i.e. the rest of solr xml isn't namespaced), and if/how that made things easier? At first blush it doesn't really seem to since any tool would need to deal with the Solr XML response in general. -Yonik http://www.lucidimagination.com
On Dec 9, 2009, at 11:11 AM, Mattmann, Chris A (388J) wrote: >> >> Any parser that does that is so broken that you should stop using it >> immediately. --wunder > > Walter, totally agree here. To elaborate my position: 1. Validation is a user option. The XML spec makes that very clear. We've had 10 years to get that right, and anyone who auto-validates is not paying attention. Validation is very useful when you are creating XML, rarely useful when reading it. 2. XML namespaces are string prefixes that use the URL syntax. They do not follow URI rules for anything but syntax and there is no guarantee that they can be resolved. In fact, an XML parser can't do anything standard with the result if they do resolve. Again, we've had 10 years to figure that out. Yes, this can be confusing, but if a parser author can't figure it out, don't use their parser because they are already getting the simple stuff wrong. wunder
On Wed, Dec 9, 2009 at 2:41 PM, Yonik Seeley <yonik@lucidimagination.com> wrote: > So... the question is, do we have a concrete alternative to this that > is well fleshed out? I do, I do... just a little variant that is geo specific and hence results in nicer names :-) <fieldType name="point" latSuffix="_lat" lonSuffix="_lon/> <field name="home" type="point"/> <dynamicField name="*_lat" type="tdouble" indexed="true" stored="false"/> <dynamicField name="*_lon" type="tdouble" indexed="true" stored="false"/> <dynamicField name="*_point" type="point"/> home_lat home_lon work_point_lat work_point_lon Note: if you want the double or tripple underscore to help prevent collisions... then you could use latSuffix="___lat" and define the dynamic fields that way. -Yonik http://www.lucidimagination.com On Wed, Dec 9, 2009 at 2:41 PM, Yonik Seeley <yonik@lucidimagination.com> wrote: > Here's an example of how everything could work with dynamic fields > (apologies if it it overlaps with examples already given by others in > this thread) : > > <fieldType name="point" fieldSuffix="_latlon" .../> // the subFields > for the points end in _latlon > <field name="home" type="point"/> > <dynamicField name="*_latlon" type="tdouble" indexed="true" stored="false"/> > > // And we also want to allow point dynamic fields > <dynamicField name="*_point" type="point"/> > > // Note: Grant make point more generic than geo, so it's 0 and 1 > instead of lat and lon > // OK, so now the indexed fields for home would be > home__0_latlon > home__1_latlon > > // And the indexed fields for dynamic field work_point would be > work_point__0_latlon > work_point__1_latlon > > Not the prettiest names... but I think everything is well defined (how > it would work with subFields of differing types.. have another param > specifying a different suffix, how it works with dynamic fields, etc). > > > -Yonik > http://www.lucidimagination.com >
Here's an example of how everything could work with dynamic fields (apologies if it it overlaps with examples already given by others in this thread) : <fieldType name="point" fieldSuffix="_latlon" .../> // the subFields for the points end in _latlon <field name="home" type="point"/> <dynamicField name="*_latlon" type="tdouble" indexed="true" stored="false"/> // And we also want to allow point dynamic fields <dynamicField name="*_point" type="point"/> // Note: Grant make point more generic than geo, so it's 0 and 1 instead of lat and lon // OK, so now the indexed fields for home would be home__0_latlon home__1_latlon // And the indexed fields for dynamic field work_point would be work_point__0_latlon work_point__1_latlon Not the prettiest names... but I think everything is well defined (how it would work with subFields of differing types.. have another param specifying a different suffix, how it works with dynamic fields, etc). So... the question is, do we have a concrete alternative to this that is well fleshed out? -Yonik http://www.lucidimagination.com
Hi All, >> >> : <fieldType name="point" type="solr.PointType" dimension="2" >> subFieldType="double"/> >> : <field name="home" type="point" indexed="true" stored="true"/> >> ... >> : And a new document of: >> : <doc> >> : <field name="point">39.0 -79.434</field> >> : </doc> >> : >> : There are three fields created: >> : home -- Contains the stored value >> : home___0 - Contains 39.0 indexed as a double (as in the "double" FieldType, >> not just a double precision) >> : home___1 - Contains -79.434 as a double >> >> Grant: All of this i understand -- the back and forth Mattmann and I have >> been having is specificly about the idea that the "__0" and __1" should be >> more transparent when declaring the schema. AS it stands right now, if i >> add this to my schema... >> >> <field name="home___0" type="int" indexed="true" stored="true"/> >> >> ...i can really break things. The odds of that happening are probably >> low, but it would still be very easy to make this type more transparent to >> schema creators by requring that PolyFields be declared as dynamicFields. >> so your previous example would become... >> >> : <fieldType name="point" type="solr.PointType" dimension="2" >> subFieldType="double"/> >> : <dynamicField name="home*" type="point" indexed="true" stored="true"/> >> >> ...now if i'm stupid enough to add <field name="home___0"/> it's my own >> damn fault (just like it is right now w/o having PolyFields in Solr) >> >> : > letting <dynamicField/> drive everything just seems a *lot* simpler ... >> : > both as far as implementation, and as far as maintaining the schema. >> : >> : I don't agree. It requires more configuration and more knowledge by the >> end user and doesn't hid the details. >> >> 1) My example requires 8 more characters then yours. > > It's not about the characters, obviously, it's about the mindset of the person > doing the modeling, hence... +1. > >> 2) The "end" user doesn't need to know it's a dynamic field, they still >> just deal with a field named "home" >> 3) my whole point is that we shouldn't be hiding these details from the >> person editing the schema.xml > > > I'm not sure I agree. I think people would expect to use a new Field Type in > exactly the same ways the use existing Field Types, namely anywhere they want > (dynamic or not). We could easily validate the schema at start up time to see > whether they have done the scenario you describe above and throw an exception. > +1 to that, as well. I had mentioned in an earlier thread about using the APIs and code to perform such a check. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: Chris.Mattmann@jpl.nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hey All, I think Eric is right on here and what I thought the intent of the patch was. Facilitating integration of Solr into environments where there is not "one true XML output". In addition, there shouldn't be "one true JSON output" for cases where your existing code already has a way it expects the JSON. Why not allow someone to write a JSON output that feeds directly into that tool without having to change that tool. This is what makes Solr so cool is because of its flexibility and to limit that would be a shame. None of this really has to limit the internal representation or what the Solr community builds to support it's format but don't unnecessarily relegate that functionality to XSLT. --Paul On 12/9/09 11:22 AM, "Eric Pugh" <epugh@opensourceconnections.com> wrote: Is this the opportunity of having more then one XML output type? I mean, XML is meant to be a transport medium for data, and maybe moving from a "one true XML output" for Solr to being able to support multiple outputs dependent on the consumer would be useful. I can see it making it easier to plug Solr into environments that expect data in certain formats, without doing an extra XSL transformation? Eric
Hi Hoss, > : ...unless things have changed since hte last time i looked, all of the > : "out of the box" response writers call "toString()" on any object they > : don't understand. So the best way to move forward in a flexible manner > : seems like it would be to add a new "GeoPoint" object to Solr, which > : toStrings to a simple "-34.56,67.89" for use by existing response writers > : as a string, but some newer smarter response writer could output it in > : some more sophisticated manner. > > The caveat to that, now that i've skimmed SOLR-1586, is that it currently > only applies to objects "added" to the SolrQueryResponse (or one of hte > containers in it) datastructure that the ResponseWriter's "walk" > themselves ... because of the back-ass-wards way we have FieldTypes write > their values directly to an XMLWriter or a TextWriter the idea of using an > object that stringifies itself as needed doesn't really apply very well I think it's rather powerful. You insulate the following variations into 1 single place to change them (FieldType): * output representation * indexing * validation To remove this from FieldType would be to strew the same functionality across multiple classes, which doesn't make sense IMHO. > ... and it won't unless we switch all of the ResponseWRiters to follow the > BinaryResponseWriter model of using FieldType.toObject(...) to get the > field value as an "obejct" that can be sent over the wire -- then the > existing XmlResponseWriter, and the Text ResponseWriters, can call > toString() on Objects they doesn't understand, and some > newer/hipper/cooler response writers that understand georss can do fancier > things with it. In the long run, this might be nice, and +1 on getting there in the long run. In the short, a compromise is to allow namespacing on fields in the existing XmlWriter, which is allowed anyways, whether by oversight or not. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: Chris.Mattmann@jpl.nasa.gov WWW: http://sunset.usc.edu/~mattmann/ Phone: +1 (818) 354-8810 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
XML is definitly one of those emotional issues in the tech world! Those who grok it don't understand why those who don't love it won't use it everywhere. And those who dislike it can't see the benefits often of XML because of their bad experiences. I know I just spent a week mucking around with an application where I couldn't start it up because of XML validation errors. Errors generated not because the XML was wrong, but because the validation process was borked up. It led me down a rat hole of frustration chasing Schemas and DTDs and validating parsers... I think that frustration is part of what has pushed people to embrace JSON, YML, and other approaches for encoding data. The biggest thing I love about Solr is "it just works...". It's simple. It's powerful. You don't have to commit months to understanding it. And yet if you want to do advanced things then Solr is fairly forgiving of that, and gives you the hooks/plugins to do it. Is this the opportunity of having more then one XML output type? I mean, XML is meant to be a transport medium for data, and maybe moving from a "one true XML output" for Solr to being able to support multiple outputs dependent on the consumer would be useful. I can see it making it easier to plug Solr into environments that expect data in certain formats, without doing an extra XSL transformation? Eric On Dec 9, 2009, at 2:11 PM, Mattmann, Chris A (388J) wrote: > Hi Yonik: > >> >> If you're forced to declare the namespace / put the URI, I'm just >> afraid of what clients / XML parsers out there may start trying to >> validate by default. > > And even if they did, it's valid XML so what's the problem? > >> And I'm still trying to figure out what we gain. > > * plugging into other standard GIS tools > (here's a list of georss ones: > > http://www.google.com/#hl=en&source=hp&fkt=1998&fsdt=4214&q=georss > +readers&a > q=f&aqi=g1&oq=&fp=b36c7832dbb01be6 > ) > > * understanding that a <point is not a <solr:point (which in your > examples > you're using a ',' to separate them while e.g., georss suggests a ' > ') but a > georss:point. From this you can: > - look up the field definition > - generate default values > - understand the unit restrictions > > There is a wealth of work in XML schema so I'm not sure I have to > justify > its use. > >> If one does want validation, it seems like we should have an >> (optional) schema for the XML response as a whole? > > I'm happy to provide this, for validation, but let's start small, > then grow > big. SOLR-1586 does _not_ break anything. > > Cheers, > Chris > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Senior Computer Scientist > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 171-266B, Mailstop: 171-246 > Email: Chris.Mattmann@jpl.nasa.gov > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Assistant Professor, Computer Science Department University of > Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > ----------------------------------------------------- Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com Co-Author: Solr 1.4 Enterprise Search Server available from http://www.packtpub.com/solr-1-4-enterprise-search-server Free/Busy: http://tinyurl.com/eric-cal
Hi Hoss, > : I think the initial geosearch feature can start off with > : <str>10,20</str> for a point. > > +1. Fundamentally, how is a string a point? > > The current XML format SOlr uses was designed to be extremely simple, very > JSON-esque, and easily parsable by *anyone* in any langauge, without > needing special knowledge of types . Whoah. I'm totally confused now. Why have FieldTypes then? When not just use Lucene? The use case for FieldTypes is _not_ just for indexing, or querying. It's also for representation? > It has been heavily advertised as > only containing a very small handful of tags, representing primitive types > (int, long, float, date, double, str) and basic collections (arr, lst, > doc) ... even if id neverh ad a formal shema/DTD. Which is leading to this confusion. Your argument is kind of weird too -- just because you never had or advertised a feature like this (which SOLR allowed for a while I think), why prevent it? Allowing namespaces does _not_ break anything. > adding new tags to that > -- name spaced or otherwise -- is a very VERY bad idea for clients who > have come to expect that they can use very simple parsing code to access > all the data. I disagree. I've got a number of projects here that could potentially use this across multiple domains (planetary science, cancer research, earth science, space science, etc.) and they all need this capability. Also what's "simple" have to do with anything? Even "simple" parsers will parse what SOLR-1586 outputs. > > introducing a new 'point" concept, wether as <point> or as > <georss:point/>, is going to break things for people. Show me an example, I fundamentally disagree with this. > > As discussed with Mattman in another thread -- some public methods in > XMLWriter have inadvertantly made it possible for plugin writers to add > their own XML tags -- but that doesn't mean we should do it in the core > Solr distribution. And why is that? Isn't the point of SOLR to expand to use cases brought up by users of the system? As long as those use cases can be principally supported, without breaking backwards compatibility (or in that case, if they do, with large blinking red text that says it), then you're shutting people out for 0 benefit? It's aesthetics we're talking about here. > If you write your own custom XMLWriter you aren't > allowed to be suprised when it contains new tags, but our "out of hte box" > users shouldn't have to deal with such suprises. What surprise -- their code won't break? > > As also discussed in that same thread thread: it makes a lot of sense > in the long run to start having Response Writers that can generate more > "rich" XML based responses and if there are already well defined standards > for some of these concepts (like georss) then by all means we should > support them -- but the existing XmlResponseWriter should NOT start > generating new tags. I agree with this, but rather than waiting for that to come 2-3 months down the road, why not buy into the need for this now, with what exists? > > The contract for SolrQueryResponse has always said: > >>>>>> A SolrQueryResponse may contain the following types of Objects >>>>>> generated by the SolrRequestHandler that processed the request. >>>>>> ... >>>>>> Other data types may be added to the SolrQueryResponse, but there is >>>>>> no guarantee that QueryResponseWriters will be able to deal with >>>>>> unexpected types. > > ...unless things have changed since hte last time i looked, all of the > "out of the box" response writers call "toString()" on any object they > don't understand. Actually most of them call some variation of #toExternal, regardless, which returns a String. Also, #toInternal returns the same type, a String. > So the best way to move forward in a flexible manner > seems like it would be to add a new "GeoPoint" object to Solr, which > toStrings to a simple "-34.56,67.89" for use by existing response writers > as a string, but some newer smarter response writer could output it in > some more sophisticated manner. I'm not convinced of that. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: Chris.Mattmann@jpl.nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I haven't followed this whole thread... but I wanted to point out that it probably intersects with the review of grant's latest patch that I did here: https://issues.apache.org/jira/browse/SOLR-1131 I did want to cut'n'paste something from that post: : I do want to separate these two issues though: : 1) field lookup mechanism (currently just exact name in schema followed by a dynamic field check) : 2) if and when fields or field types should be explicitly defined in the schema vs being created by the polyField My current thought on #1 is that we probably don't want to change the internal lookup mechanism used by IndexSchema unless we gain significant power by doing so. I'm not sure I currently see it. My thoughts on #2 is more on a case-by-case basis. For the simple case of a point class with two fields indexed separately, referencing a suffix that should be defined as a dynamic field vs referencing a type seem pretty close. The latter, while perhaps slightly simpler for the user, seem to introduce a lot of hidden complexities. I'm less concerned than Hoss is about name clashes, but much more concerned about those complexities. -Yonik http://www.lucidimagination.com
> Any parser that does that is so broken that you should stop using it > immediately. --wunder Walter, totally agree here. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: Chris.Mattmann@jpl.nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On Dec 9, 2009, at 2:04 PM, Chris Hostetter wrote: > > : <fieldType name="point" type="solr.PointType" dimension="2" subFieldType="double"/> > : <field name="home" type="point" indexed="true" stored="true"/> > ... > : And a new document of: > : <doc> > : <field name="point">39.0 -79.434</field> > : </doc> > : > : There are three fields created: > : home -- Contains the stored value > : home___0 - Contains 39.0 indexed as a double (as in the "double" FieldType, not just a double precision) > : home___1 - Contains -79.434 as a double > > Grant: All of this i understand -- the back and forth Mattmann and I have > been having is specificly about the idea that the "__0" and __1" should be > more transparent when declaring the schema. AS it stands right now, if i > add this to my schema... > > <field name="home___0" type="int" indexed="true" stored="true"/> > > ...i can really break things. The odds of that happening are probably > low, but it would still be very easy to make this type more transparent to > schema creators by requring that PolyFields be declared as dynamicFields. > so your previous example would become... > > : <fieldType name="point" type="solr.PointType" dimension="2" subFieldType="double"/> > : <dynamicField name="home*" type="point" indexed="true" stored="true"/> > > ...now if i'm stupid enough to add <field name="home___0"/> it's my own > damn fault (just like it is right now w/o having PolyFields in Solr) > > : > letting <dynamicField/> drive everything just seems a *lot* simpler ... > : > both as far as implementation, and as far as maintaining the schema. > : > : I don't agree. It requires more configuration and more knowledge by the end user and doesn't hid the details. > > 1) My example requires 8 more characters then yours. It's not about the characters, obviously, it's about the mindset of the person doing the modeling, hence... > 2) The "end" user doesn't need to know it's a dynamic field, they still > just deal with a field named "home" > 3) my whole point is that we shouldn't be hiding these details from the > person editing the schema.xml I'm not sure I agree. I think people would expect to use a new Field Type in exactly the same ways the use existing Field Types, namely anywhere they want (dynamic or not). We could easily validate the schema at start up time to see whether they have done the scenario you describe above and throw an exception.
Hi Yonik:
>
> If you're forced to declare the namespace / put the URI, I'm just
> afraid of what clients / XML parsers out there may start trying to
> validate by default.
And even if they did, it's valid XML so what's the problem?
> And I'm still trying to figure out what we gain.
* plugging into other standard GIS tools
(here's a list of georss ones:
http://www.google.com/#hl=en&source=hp&fkt=1998&fsdt=4214&q=georss+readers&a
q=f&aqi=g1&oq=&fp=b36c7832dbb01be6
)
* understanding that a <point is not a <solr:point (which in your examples
you're using a ',' to separate them while e.g., georss suggests a ' ') but a
georss:point. From this you can:
- look up the field definition
- generate default values
- understand the unit restrictions
There is a wealth of work in XML schema so I'm not sure I have to justify
its use.
> If one does want validation, it seems like we should have an
> (optional) schema for the XML response as a whole?
I'm happy to provide this, for validation, but let's start small, then grow
big. SOLR-1586 does _not_ break anything.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department University of
Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
: That's not how the Cartesian Field stuff works, but I think I see what : you are getting at and I would say I'm going to explicitly punt on that : right now. Ultimately, I think when such a case comes up, the FieldType : needs to be configured to be able to determine this information. I'm fine punting on it -- i don't understand half this stuff anyway -- i just wanted to raise the issue incase someone else said "Ack! ... yes that is a big oversight in the API that will cause problems with X, Y, Z." -Hoss
: <fieldType name="point" type="solr.PointType" dimension="2" subFieldType="double"/>
: <field name="home" type="point" indexed="true" stored="true"/>
...
: And a new document of:
: <doc>
: <field name="point">39.0 -79.434</field>
: </doc>
:
: There are three fields created:
: home -- Contains the stored value
: home___0 - Contains 39.0 indexed as a double (as in the "double" FieldType, not just a double
precision)
: home___1 - Contains -79.434 as a double
Grant: All of this i understand -- the back and forth Mattmann and I have
been having is specificly about the idea that the "__0" and __1" should be
more transparent when declaring the schema. AS it stands right now, if i
add this to my schema...
<field name="home___0" type="int" indexed="true" stored="true"/>
...i can really break things. The odds of that happening are probably
low, but it would still be very easy to make this type more transparent to
schema creators by requring that PolyFields be declared as dynamicFields.
so your previous example would become...
: <fieldType name="point" type="solr.PointType" dimension="2" subFieldType="double"/>
: <dynamicField name="home*" type="point" indexed="true" stored="true"/>
...now if i'm stupid enough to add <field name="home___0"/> it's my own
damn fault (just like it is right now w/o having PolyFields in Solr)
: > letting <dynamicField/> drive everything just seems a *lot* simpler ...
: > both as far as implementation, and as far as maintaining the schema.
:
: I don't agree. It requires more configuration and more knowledge by the end user and doesn't
hid the details.
1) My example requires 8 more characters then yours.
2) The "end" user doesn't need to know it's a dynamic field, they still
just deal with a field named "home"
3) my whole point is that we shouldn't be hiding these details from the
person editing the schema.xml
-Hoss
Hi Yonik, > > I've run across cases where I added a schema declaration to an XML > file and then things started failing. I think some parsers may > default to validating if it sees that it can? I've seen this too. But it won't affect the interaction we're talking about like I said, SOLR-1586 outputs valid XML, so this isn't an issue. > > Namespaces are to avoid name clashes. Solr XML is well defined and > not arbitrary... adding <point> if we wish to do so won't introduce > any clashes. > Actually there are quite a bit of use cases for namespacing beyond name clashes. Namespaces enable validation, understanding and definition for elements (understanding units, ranges, etc.). For instance, you and I both use the term "mass", but in my domain, mass refers to the planetary science definition of mass, but, in your domain you mean earth science. "mass" does not always mean the same thing (variation in units, representation, etc.) See here: http://www.w3.org/TR/2006/REC-xml-names11-20060816/ >> The only difference between what you call simple above and what I've >> proposed (and correct me if I'm wrong but others have too) is that your >> <point tag would include a namespace prefix and an xmlns attribute. What's >> the difference? >> >>> It is worth using standards when they buy you enough.... I'm not sure >>> this is one of those times. >>> I'm sure there are standards for numeric types like <int> too... but >>> using namespaces for that seems like overkill. >> >> There's a difference between a primitive type like int, and one like point. >> Also, it all comes down to your use case. If the only thing you're ever >> going to do with SOLR is have a SOLR client talk to it (Java, Ruby, whatever >> PL you want) then namespaces/etc. might be overkill. But why open up the >> response format then and advertise SOLR as something that provides REST-ful >> services for search? > > REST-ful doesn't say anything about customizing the response format. So are you saying that the intention is not to allow customization of the response format? Also you've released how many releases of SOLR that have the capability to do this and now you're suddenly going to change it? I'm sorry I disagree. > >> If that's the case, then users consuming those >> responses need the flexibility to customize them for their use case >> (validation, plugging into external GIS tools, etc.). So, I don't agree with >> this. > > What GIS tool could deal with a Solr XML response format w/o any other > knowledge of everything else in the response? > Are there some real use cases that using a namespace vs not for point > make easier (an honest question... I don't know much about GIS stuff). Using standards enables standard tool development. Unless you want everyone to develop their own custom tools for SOLR (or be tied to using whatever is provided by SOLR _only_), and I don't think that's the intent. I also don't think that's a very friendly, open strategy for users. What I'm proposing does _not_ break backwards compatibility, anywhere. If you've got an example, then speak up. > >> All I've done is use what already exists. There doesn't need to be any >> patches. XmlWriter#writePrim allowed you to do this before, see: > > Yeah, you can use that to output <long>false</long> too... but it will > cause certain clients to barf. That's a ResponseWriter issue. That's not a client issue. Clients don't arbitrarily connect to servers for which they don't speak the protocol language. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: Chris.Mattmann@jpl.nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
: > I'm not really understanding the value of an approach like that. for : > starters, what Lucene field names would ultimately be created in those : > examples? : : The first field would be named location__location. : The second field would be named location_home_location_home. : The third field would be named location_work_location_work. I'm not understanding your answer -- the whole point of this example is that this hypothetical PolyField example is that for each field the user knows about, it's builting up a "lat" and a "lon" field under the covers -- my question is what real, under the covers, names do you suggest be created from teh type of configuration you suggested? : > <field name="other_location" type=latlon"/> : > <dynamicField name="*_dynamic_location" type=latlon"/> : > : > ...then what field names would be created under the covers? : > : : In general, it would be FieldType#getPattern().stripOffEndRegexStarStuff() + : Field#getName(). ...that still feels a lot more obscure then letting the <field> declaration control things -- because as a schema creator i not only have to check that the field name i want to add doesn't conflict with an existing field name (or dynamicField name glob) but i also have to check that it doesn't conflict with a pattern in a PolyField <fieldType> declaration. why make people check too things when using dynamicFields is something they already understand? : Well if this feels wrong to you then I think the schema.xml file that ships : with SOLR should also feel wrong as well because it uses the exact same : pattern for defining field type variations. That is, differences between : FieldType representations for ints and tints are not stored as variations on : the SchemaField definition itself but they are stored as variation on the : FieldTypes (e.g., a different precisionStep in the case of int [0] versus : that of tint [8]). Based on what you are proposing, why isn't precisionStep : an attribute on <field, rather than <fieldType in those examples? There's a huge differnet there -- nothing in a <fieldTYpe/> declaration right now has any influence what so ever on the ultimate "name" of the fields used -- <field/> declaration's can inherit a lot of stuff from <fieldType/> but we've never let the <fieldType/> influence the name. In an existing solr schema, i can have a list of <fieldType/>s and i can have a list of <field/>s that refrence those fieldType's by name -- and i can tweak the settings on those things laregly independently (as long as i reindex) ... but i never have to wworry that tweaking the setting of fieldType might completley break an index by causing the underlyling name of some <field name="a".../> to suddenly collide with some other <field name="a__a" .../> : Possibly. It's also a lot less traceable. It's implicit versus explicit, : which I'm not sure leads to simplicity in the end. I feel the exact opposite actaully. Saying "these PolyField types will create multiple fields under the covers, and you have to use them as <dynamicFields/> do control what names they use" seems a lot more explict and easily tracable then "these PolyField types will create multiple fileds under the covers, so you specify a pattern when you declare them, and then when you declare fields or dynamicFields that use them the following rule(s) will be applied to generate the underlying field names, so remember this rule when naming other fields to prevent conflicts". -Hoss
[ https://issues.apache.org/jira/browse/SOLR-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788228#action_12788228
]
Yonik Seeley commented on SOLR-1131:
------------------------------------
OK... so the real issue is that this introduces a new mechanism to look up field types...
not necessarily a horrible thing, but we should definitely think twice before doing so.
home___0 and home___1 are not dynamic fields as I understand it (in that there is no ___0
dynamic field. The lookup is done by adding new support to the IndexSchema to strip off ___foo
off of any field and use that as it's type?
But... that scheme seems to limit us to a single subField type (in addition to the other downsides
of requiring a new lookup mechanism).
I do want to separate these two issues though:
1) field lookup mechanism (currently just exact name in schema followed by a dynamic field
check)
2) if and when fields or field types should be explicitly defined in the schema vs being created
by the polyField
Aside: it looks like the code for getFieldOrNull isn't right? Seems like it will return a
field with both the wrong type and the wrong name?
{code}
public SchemaField getFieldOrNull(String fieldName) {
SchemaField f = fields.get(fieldName);
@@ -1055,25 +1071,28 @@
for (DynamicField df : dynamicFields) {
if (df.matches(fieldName)) return df.makeSchemaField(fieldName);
}
-
+ int idx = fieldName.indexOf(FieldType.POLY_FIELD_SEPARATOR);
+ if (idx != -1){
+ String fn = fieldName.substring(0, idx);
+ f = getFieldOrNull(fn);
+ }
return f;
{code}
> Allow a single field type to index multiple fields
> --------------------------------------------------
>
> Key: SOLR-1131
> URL: https://issues.apache.org/jira/browse/SOLR-1131
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Assignee: Grant Ingersoll
> Fix For: 1.5
>
> Attachments: SOLR-1131-IndexMultipleFields.patch, SOLR-1131.patch, SOLR-1131.patch,
SOLR-1131.patch, SOLR-1131.patch, SOLR-1131.patch
>
>
> In a few special cases, it makes sense for a single "field" (the concept) to be indexed
as a set of Fields (lucene Field). Consider SOLR-773. The concept "point" may be best indexed
in a variety of ways:
> * geohash (sincle lucene field)
> * lat field, lon field (two double fields)
> * cartesian tiers (a series of fields with tokens to say if it exists within that region)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
: ...unless things have changed since hte last time i looked, all of the : "out of the box" response writers call "toString()" on any object they : don't understand. So the best way to move forward in a flexible manner : seems like it would be to add a new "GeoPoint" object to Solr, which : toStrings to a simple "-34.56,67.89" for use by existing response writers : as a string, but some newer smarter response writer could output it in : some more sophisticated manner. The caveat to that, now that i've skimmed SOLR-1586, is that it currently only applies to objects "added" to the SolrQueryResponse (or one of hte containers in it) datastructure that the ResponseWriter's "walk" themselves ... because of the back-ass-wards way we have FieldTypes write their values directly to an XMLWriter or a TextWriter the idea of using an object that stringifies itself as needed doesn't really apply very well ... and it won't unless we switch all of the ResponseWRiters to follow the BinaryResponseWriter model of using FieldType.toObject(...) to get the field value as an "obejct" that can be sent over the wire -- then the existing XmlResponseWriter, and the Text ResponseWriters, can call toString() on Objects they doesn't understand, and some newer/hipper/cooler response writers that understand georss can do fancier things with it. -Hoss
: I think the initial geosearch feature can start off with : <str>10,20</str> for a point. +1. The current XML format SOlr uses was designed to be extremely simple, very JSON-esque, and easily parsable by *anyone* in any langauge, without needing special knowledge of types . It has been heavily advertised as only containing a very small handful of tags, representing primitive types (int, long, float, date, double, str) and basic collections (arr, lst, doc) ... even if id neverh ad a formal shema/DTD. adding new tags to that -- name spaced or otherwise -- is a very VERY bad idea for clients who have come to expect that they can use very simple parsing code to access all the data. introducing a new 'point" concept, wether as <point> or as <georss:point/>, is going to break things for people. As discussed with Mattman in another thread -- some public methods in XMLWriter have inadvertantly made it possible for plugin writers to add their own XML tags -- but that doesn't mean we should do it in the core Solr distribution. If you write your own custom XMLWriter you aren't allowed to be suprised when it contains new tags, but our "out of hte box" users shouldn't have to deal with such suprises. As also discussed in that same thread thread: it makes a lot of sense in the long run to start having Response Writers that can generate more "rich" XML based responses and if there are already well defined standards for some of these concepts (like georss) then by all means we should support them -- but the existing XmlResponseWriter should NOT start generating new tags. The contract for SolrQueryResponse has always said: >>>>> A SolrQueryResponse may contain the following types of Objects >>>>> generated by the SolrRequestHandler that processed the request. >>>>> ... >>>>> Other data types may be added to the SolrQueryResponse, but there is >>>>> no guarantee that QueryResponseWriters will be able to deal with >>>>> unexpected types. ...unless things have changed since hte last time i looked, all of the "out of the box" response writers call "toString()" on any object they don't understand. So the best way to move forward in a flexible manner seems like it would be to add a new "GeoPoint" object to Solr, which toStrings to a simple "-34.56,67.89" for use by existing response writers as a string, but some newer smarter response writer could output it in some more sophisticated manner. -Hoss
this need to be reverted . there was data loss
On Wed, Dec 9, 2009 at 8:46 PM, Apache Wiki <wikidiffs@apache.org> wrote:
> Dear Wiki user,
>
> You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification.
>
> The "DataImportHandler" page has been changed by DNaber.
> http://wiki.apache.org/solr/DataImportHandler?action=diff&rev1=220&rev2=221
>
> --------------------------------------------------
>
> <dataConfig>
> <dataSource type="FileDataSource" />
> <document>
> + <entity name="f" processor="FileListEntityProcessor" baseDir="/some/path/tongle
implicit field called 'plainText'. The content is not parsed in any way, however you may add
transformers to manipulate the data within 'plainText' as needed or to create other additional
fields.
> - <entity name="f" processor="FileListEntityProcessor" baseDir="/some/path/to/files"
fileName=".*xml" newerThan="'NOW-3DAYS'" recursive="true" rootEntity="false" dataSource="null">
> - <entity name="x" processor="XPathEntityProcessor" forEach="/the/record/xpath"
url="${f.fileAbsolutePath}">
> - <field column="full_name" xpath="/field/xpath"/>
> - </entity>
> - </entity>
> - </document>
> - </dataConfig>
> - }}}
> - Do not miss the `rootEntity` attribute. The implicit fields generated by the !FileListEntityProcessor
are `fileAbsolutePath, fileSize, fileLastModified, fileName` and these are available for use
within the entity X as shown above. It should be noted that !FileListEntityProcessor returns
a list of pathnames and that the subsequent entity must use the !FileDataSource to fetch the
files content.
>
> + example:
> - === CachedSqlEntityProcessor ===
> - <<Anchor(cached)>>
> -
> - This is an extension of the !SqlEntityProcessor. This !EntityProcessor helps reduce
the no: of DB queries executed by caching the rows. It does not help to use it in the root
most entity because only one sql is run for the entity.
> -
> - Example 1.
> {{{
> - <entity name="x" query="select * from x">
> - <entity name="y" query="select * from y where xid=${x.id}" processor="CachedSqlEntityProcessor">
> - </entity>
> + <entity processor="PlainTextEntityProcessor" name="x" url="http://abc.com/a.txt"
dataSource="data-source-name">
> + <!-- copies the text to a field called 'text' in Solr-->
> + <field column="plainText" name="text"/>
> - <entity>
> + </entity>
> }}}
>
> - The usage is exactly same as the other one. When a query is run the results are stored
and if the same query is run again it is fetched from the cache and returned
> + Ensure that the dataSource is of type !DataSource<Reader> (!FileDataSource, URL!DataSource)
>
> - Example 2:
> - {{{
> - <entity name="x" query="select * from x">
> - <entity name="y" query="select * from y" processor="CachedSqlEntityProcessor"
where="xid=x.id">
> - </entity>
> - <entity>
> - }}}
> -
> - The difference with the previous one is the 'where' attribute. In this case the query
fetches all the rows from the table and stores all the rows in the cache. The magic is in
the 'where' value. The cache stores the values with the 'xid' value in 'y' as the key. The
value for 'x.id' is evaluated every time the entity has to be run and the value is looked
up in the cache an the rows are returned.
> -
> - In the where the lhs (the part before '=') is the column in y and the rhs (the part
after '=') is the value to be computed for looking up the cache.
> -
> - === PlainTextEntityProcessor ===
> + === LineEntityProcessor ===
> - <<Anchor(plaintext)>>
> + <<Anchor(LineEntityProcessor)>>
> <!> [[Solr1.4]]
>
> - This !EntityProcessor reads all content from the data source into an single implicit
field called 'plainText'. The content is not parsed in any way, however you may add transformers
to manipulate the data within 'plainText' as needed or to create other additional fields.
> + This !EntityProcessor reads all content from the data source on a line by line basis,
a field called 'rawLine' is returned for each line read. The content is not parsed in any
way, however you may add transformers to manipulate the data within 'rawLine' or to create
other additional fields.
> +
> + The lines read can be filtered by two regular expressions '''acceptLineRegex''' and
'''omitLineRegex'''.
> + This entities additional attributes are:
> + * '''`url`''' : a required attribute that specifies the location of the input file
in a way that is compatible with the configured datasource. If this value is relative and
you are using !FileDataSource or URL!DataSource, it assumed to be relative to '''baseLoc'''.
> + * '''`acceptLineRegex`''' :an optional attribute that if present discards any line
which does not match the regExp.
> + * '''`omitLineRegex`''' : an optional attribute that is applied after any acceptLineRegex
and discards any line which matches this regExp.
> + example:
> + {{{
> + <entity name="jc"
> + processor="LineEntityProcessor"
> + acceptLineRegex="^.*\.xml$"
> + omitLineRegex="/obsolete"
> + url="file:///Volumes/ts/files.lis"
> + rootEntity="false"
> + dataSource="myURIreader1"
> + transformer="RegexTransformer,DateFormatTransformer"
> + >
> + ...
> + }}}
> + While there are use cases where you might need to create a solr document per line read
from a file, it is expected that in most cases that the lines read will consist of a pathname
which is in turn consumed by another !EntityProcessor
> + such as X!PathEntityProcessor.
> +
> + == DataSource ==
> + <<Anchor(datasource)>>
> + A class can extend `org.apache.solr.handler.dataimport.DataSource` . [[http:/%ngle
implicit field called 'plainText'. The content is not parsed in any way, however you may add
transformers to manipulate the data within 'plainText' as needed or to create other additional
fields.
>
> example:
> {{{
> @@ -1026, +1026 @@
>
> {{attachment:interactive-dev-dataimporthandler.PNG}}
>
> = Where to find it? =
> - DataImportHandler is a new addition to Solr. You can either:
> + DataImportHandler was added to Solr in Solr 1.3. You can either:
> - * Download a nightly build of Solr from [[http://lucene.apache.org/solr/|Solr website]],
or
> + * Download a build of Solr from [[http://lucene.apache.org/solr/|Solr website]], or
> * Use the steps given in Full Import Example to try it out.
>
> For a history of development discussion related to DataImportHandler, please see [[http://issues.apache.org/jira/browse/SOLR-469|SOLR-469]]
in the Solr JIRA.
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com
Any parser that does that is so broken that you should stop using it immediately. --wunder On Dec 9, 2009, at 8:33 AM, Yonik Seeley wrote: > My gut feeling is that we should not be introducing namespaces by default. > It introduces a new requirement of XML parsers in clients, and some > parsers would start validating by default, and going out to the web to > retrieve the referenced namespace/schema, etc.
On Wed, Dec 9, 2009 at 12:40 PM, Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov> wrote: > <foo> > <zoo:bar xmlns:zoo="http://example.com/zoo">hi</zoo:bar> > </foo> If you're forced to declare the namespace / put the URI, I'm just afraid of what clients / XML parsers out there may start trying to validate by default. And I'm still trying to figure out what we gain. If one does want validation, it seems like we should have an (optional) schema for the XML response as a whole? -Yonik http://www.lucidimagination.com
Hi Yonik, > Should have tried this before... I just created a small XML file: > > <foo> > <bar>hi</bar> > </foo> > > I pointed both firefox and IE at this file and it displays as XML fine. > I then changed the file to this: > > <foo> > <zoo:bar>hi</zoo:bar> > </foo> Sure, of course it does. It's because that's not valid XML syntax. You have to declare the namespace for zoo. You can do it at the top of the XML file in the root XML tag. Or, you can do it inline (like I've done in SOLR). Try this: <foo> <zoo:bar xmlns:zoo="http://example.com/zoo">hi</zoo:bar> </foo> Cheers, Chris > > That made both of them barf. > That alone makes me lean pretty strongly against using a namespace for this. > > -Yonik > http://www.lucidimagination.com > > > > On Wed, Dec 9, 2009 at 12:28 PM, Yonik Seeley > <yonik@lucidimagination.com> wrote: >> On Wed, Dec 9, 2009 at 11:44 AM, Mattmann, Chris A (388J) >> <chris.a.mattmann@jpl.nasa.gov> wrote: >>> How does it introduce any new requirements? Namespaces are easily ignored by >>> any XML client as they are if they weren't present. In other words, unless >>> the XML client has setValidating=true, then this isn't an issue. >> >> I've run across cases where I added a schema declaration to an XML >> file and then things started failing. I think some parsers may >> default to validating if it sees that it can? >> >> Namespaces are to avoid name clashes. Solr XML is well defined and >> not arbitrary... adding <point> if we wish to do so won't introduce >> any clashes. >> >>> The only difference between what you call simple above and what I've >>> proposed (and correct me if I'm wrong but others have too) is that your >>> <point tag would include a namespace prefix and an xmlns attribute. What's >>> the difference? >>> >>>> It is worth using standards when they buy you enough.... I'm not sure >>>> this is one of those times. >>>> I'm sure there are standards for numeric types like <int> too... but >>>> using namespaces for that seems like overkill. >>> >>> There's a difference between a primitive type like int, and one like point. >>> Also, it all comes down to your use case. If the only thing you're ever >>> going to do with SOLR is have a SOLR client talk to it (Java, Ruby, whatever >>> PL you want) then namespaces/etc. might be overkill. But why open up the >>> response format then and advertise SOLR as something that provides REST-ful >>> services for search? >> >> REST-ful doesn't say anything about customizing the response format. >> >>> If that's the case, then users consuming those >>> responses need the flexibility to customize them for their use case >>> (validation, plugging into external GIS tools, etc.). So, I don't agree with >>> this. >> >> What GIS tool could deal with a Solr XML response format w/o any other >> knowledge of everything else in the response? >> Are there some real use cases that using a namespace vs not for point >> make easier (an honest question... I don't know much about GIS stuff). >> >>> All I've done is use what already exists. There doesn't need to be any >>> patches. XmlWriter#writePrim allowed you to do this before, see: >> >> Yeah, you can use that to output <long>false</long> too... but it will >> cause certain clients to barf. >> >> -Yonik >> http://www.lucidimagination.com >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: Chris.Mattmann@jpl.nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Should have tried this before... I just created a small XML file: <foo> <bar>hi</bar> </foo> I pointed both firefox and IE at this file and it displays as XML fine. I then changed the file to this: <foo> <zoo:bar>hi</zoo:bar> </foo> That made both of them barf. That alone makes me lean pretty strongly against using a namespace for this. -Yonik http://www.lucidimagination.com On Wed, Dec 9, 2009 at 12:28 PM, Yonik Seeley <yonik@lucidimagination.com> wrote: > On Wed, Dec 9, 2009 at 11:44 AM, Mattmann, Chris A (388J) > <chris.a.mattmann@jpl.nasa.gov> wrote: >> How does it introduce any new requirements? Namespaces are easily ignored by >> any XML client as they are if they weren't present. In other words, unless >> the XML client has setValidating=true, then this isn't an issue. > > I've run across cases where I added a schema declaration to an XML > file and then things started failing. I think some parsers may > default to validating if it sees that it can? > > Namespaces are to avoid name clashes. Solr XML is well defined and > not arbitrary... adding <point> if we wish to do so won't introduce > any clashes. > >> The only difference between what you call simple above and what I've >> proposed (and correct me if I'm wrong but others have too) is that your >> <point tag would include a namespace prefix and an xmlns attribute. What's >> the difference? >> >>> It is worth using standards when they buy you enough.... I'm not sure >>> this is one of those times. >>> I'm sure there are standards for numeric types like <int> too... but >>> using namespaces for that seems like overkill. >> >> There's a difference between a primitive type like int, and one like point. >> Also, it all comes down to your use case. If the only thing you're ever >> going to do with SOLR is have a SOLR client talk to it (Java, Ruby, whatever >> PL you want) then namespaces/etc. might be overkill. But why open up the >> response format then and advertise SOLR as something that provides REST-ful >> services for search? > > REST-ful doesn't say anything about customizing the response format. > >> If that's the case, then users consuming those >> responses need the flexibility to customize them for their use case >> (validation, plugging into external GIS tools, etc.). So, I don't agree with >> this. > > What GIS tool could deal with a Solr XML response format w/o any other > knowledge of everything else in the response? > Are there some real use cases that using a namespace vs not for point > make easier (an honest question... I don't know much about GIS stuff). > >> All I've done is use what already exists. There doesn't need to be any >> patches. XmlWriter#writePrim allowed you to do this before, see: > > Yeah, you can use that to output <long>false</long> too... but it will > cause certain clients to barf. > > -Yonik > http://www.lucidimagination.com >
On Wed, Dec 9, 2009 at 11:44 AM, Mattmann, Chris A (388J) <chris.a.mattmann@jpl.nasa.gov> wrote: > How does it introduce any new requirements? Namespaces are easily ignored by > any XML client as they are if they weren't present. In other words, unless > the XML client has setValidating=true, then this isn't an issue. I've run across cases where I added a schema declaration to an XML file and then things started failing. I think some parsers may default to validating if it sees that it can? Namespaces are to avoid name clashes. Solr XML is well defined and not arbitrary... adding <point> if we wish to do so won't introduce any clashes. > The only difference between what you call simple above and what I've > proposed (and correct me if I'm wrong but others have too) is that your > <point tag would include a namespace prefix and an xmlns attribute. What's > the difference? > >> It is worth using standards when they buy you enough.... I'm not sure >> this is one of those times. >> I'm sure there are standards for numeric types like <int> too... but >> using namespaces for that seems like overkill. > > There's a difference between a primitive type like int, and one like point. > Also, it all comes down to your use case. If the only thing you're ever > going to do with SOLR is have a SOLR client talk to it (Java, Ruby, whatever > PL you want) then namespaces/etc. might be overkill. But why open up the > response format then and advertise SOLR as something that provides REST-ful > services for search? REST-ful doesn't say anything about customizing the response format. > If that's the case, then users consuming those > responses need the flexibility to customize them for their use case > (validation, plugging into external GIS tools, etc.). So, I don't agree with > this. What GIS tool could deal with a Solr XML response format w/o any other knowledge of everything else in the response? Are there some real use cases that using a namespace vs not for point make easier (an honest question... I don't know much about GIS stuff). > All I've done is use what already exists. There doesn't need to be any > patches. XmlWriter#writePrim allowed you to do this before, see: Yeah, you can use that to output <long>false</long> too... but it will cause certain clients to barf. -Yonik http://www.lucidimagination.com
[ https://issues.apache.org/jira/browse/SOLR-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788170#action_12788170
]
Grant Ingersoll edited comment on SOLR-1131 at 12/9/09 5:18 PM:
----------------------------------------------------------------
{quote}
<fieldType name="xy" class="solr.PointType" dimension="2" subFieldType="double"/>
<field name="home" type="xy" indexed="true" stored="true"/>
{quote}
Two indexed fields
home___0
home___1
One stored field:
home
was (Author: gsingers):
{quote}
<fieldType name="xy" class="solr.PointType" dimension="2" subFieldType="double"/>
<field name="home" type="xy" indexed="true" stored="true"/>
{quote}
home___0
home___1
> Allow a single field type to index multiple fields
> --------------------------------------------------
>
> Key: SOLR-1131
> URL: https://issues.apache.org/jira/browse/SOLR-1131
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Assignee: Grant Ingersoll
> Fix For: 1.5
>
> Attachments: SOLR-1131-IndexMultipleFields.patch, SOLR-1131.patch, SOLR-1131.patch,
SOLR-1131.patch, SOLR-1131.patch, SOLR-1131.patch
>
>
> In a few special cases, it makes sense for a single "field" (the concept) to be indexed
as a set of Fields (lucene Field). Consider SOLR-773. The concept "point" may be best indexed
in a variety of ways:
> * geohash (sincle lucene field)
> * lat field, lon field (two double fields)
> * cartesian tiers (a series of fields with tokens to say if it exists within that region)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.