Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAH6s0fy-Mgh8AFbg7pYMYhoN6TPFCCaY_shuB38=-qoWQhg_Wg@mail.gmail.com>
References: 
 <CAH6s0fwzx_sddnnLbYX3Uq+g-HckrX-ui4cj+E7uiQMX3213uA@mail.gmail.com>
 <CAPe6Lt2SoHE9P73cwcdqPwHHXG+mfTp_Gqm38NieZK9_Vh_BhA@mail.gmail.com>
 <alpine.DEB.2.02.1302221243200.9095@frisbee>
 <CAPe6Lt2tt5bB30PCn-+EMfZXn6RECOjCZvV-cN9fMXrXkoYyuw@mail.gmail.com>
 <CAH6s0fw8N3PAnkQgSv1OohZ74d5Do6sQuTRrL6Gf=94jzo9vtw@mail.gmail.com>
 <CAH6s0fy-Mgh8AFbg7pYMYhoN6TPFCCaY_shuB38=-qoWQhg_Wg@mail.gmail.com>
From: Michael Della Bitta <michael.della.bitta@appinions.com>
Date: Sun, 24 Feb 2013 16:16:12 -0500
Message-ID: 
 <CAPe6Lt3feJn+Pjiv469GC00UvCgNH8pW=8GmQYvSPv3+SspTjg@mail.gmail.com>
Subject: Re: Interesting issue with "special characters" in a string field
 value
To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hello Jack,

I'm not sure if this is an option for you, but if you submit and
retrieve your documents using only SolrJ, you won't have to worry
about escaping them for encoding into a particular document format.
SolrJ would handle that for you.

Michael Della Bitta

------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn=92t a Game


On Sun, Feb 24, 2013 at 12:29 AM, Jack Park <jackpark@topicquests.org> wrot=
e:
> Ok. I have revisited this issue as deeply as possible using simplistic
> unit tests, tossing out indexes, and starting fresh.
>
> A typical Solr document might have a label, e.g. the string inside the
> quotes: "Node Type".  That would be queried, according to what I've
> been able to read, as a Phrase Query, which means, include the quotes
> around the text.
>
> When I use the admin query panel with this query:
> label:"Node Type"
> A fragment of the full document is returned. it is this:
>
>   <doc>
>     <str name=3D"locator">NodeType</str>
>     <arr name=3D"label">
>       <str>Node Type</str>
>     </arr>
>
> In my code using SolrJ, I have printlines just as the "escaped" query
> string comes in, and one which shows what the SolrQuery looks like
> after setting it up to go online. I then show what came back:
>
> Solr3Client.runQuery- label:"Node Type" 0 10
> Solr3Client.runQuery-1 q=3Dlabel%3A%22Node+Type%22&start=3D0&rows=3D10
> ZZZZ {numFound=3D1,start=3D0,docs=3D[SolrDocument{locator=3DNodeType,
> smallIcon=3Dcogwheel.png, subOf=3DClassType, details=3DThe TopicQuests
> typology node type., isPrivate=3Dfalse, creatorId=3DSystemUser, label=3DN=
ode
> Type, largeIcon=3Dcogwheel.png, lastEditDate=3DSat Feb 23 20:43:22 PST
> 2013, createdDate=3DSat Feb 23 20:43:22 PST 2013,
> _version_=3D1427826019119661056}]}
>
> What that says is that SolrQuery inserted a + inside the query string,
> and that it found 1 document, but did not return it.
>
> In the largest picture, I have returned to using XMLResponseParser on
> the theory that I will now be able to take advantage of partialUpdates
> on multi-valued fields (List<String>) but haven't tested that yet. I
> am not yet escaping such things as "<" or ">" but just escaping those
> things mentioned in the Solr documents which are reserved characters.
>
> So, the current update is this: learning about phrase queries, and
> judicious escaping of reserved characters seems to be helping. Next up
> entails two issues: more robust testing of escaped characters, and
> trying to discover what is the best approach to dealing with
> characters that must be escaped to get past XML, e.g. '<', '>', and
> others.
>
> Many thanks
> Jack
>
>
> On Fri, Feb 22, 2013 at 2:44 PM, Jack Park <jackpark@topicquests.org> wro=
te:
>> Michael,
>> I don't think you misunderstood. I will soon give a full response here, =
but
>> am on the road at the moment.
>>
>> Many thanks
>> Jack
>>
>>
>> On Friday, February 22, 2013, Michael Della Bitta
>> <michael.della.bitta@appinions.com> wrote:
>>> My mistake, I misunderstood the problem.
>>>
>>> Michael Della Bitta
>>>
>>> ------------------------------------------------
>>> Appinions
>>> 18 East 41st Street, 2nd Floor
>>> New York, NY 10017-6271
>>>
>>> www.appinions.com
>>>
>>> Where Influence Isn=92t a Game
>>>
>>>
>>> On Fri, Feb 22, 2013 at 3:55 PM, Chris Hostetter
>>> <hossman_lucene@fucit.org> wrote:
>>>>
>>>> : If you're submitting documents as XML, you're always going to have t=
o
>>>> : escape meaningful XML characters going in. If you ask for them back =
as
>>>> : XML, you should be prepared to unescape special XML characters as
>>>>
>>>> that still wouldn't explain the discrepency he's claiming to see betwe=
en
>>>> the json & xml resmonses (the json containing an empty string
>>>>
>>>> Jack: please elaborate with specifics about your solr version, field,
>>>> field type, how you indexed your doc, and what the request urls & raw
>>>> responses that you get are (ie: don't trust the XML you see in your
>>>> browser, it may be unescaping escaped sequences in element text to be
>>>> "helpful" .. use something like curl)
>>>>
>>>> For example...
>>>>
>>>> ----BEGIN GOOD EXAMPLE OF SPECIFICS---
>>>>
>>>> I'm using Solr 4.x with the 4.x example schema which has the following
>>>> field...
>>>>
>>>>    <field name=3D"cat" type=3D"string" indexed=3D"true" stored=3D"true=
"
>>>> multiValued=3D"true"/>
>>>>    <fieldType name=3D"string" class=3D"solr.StrField" sortMissingLast=
=3D"true"
>>>> />
>>>>
>>>> I indexed a doc like this...
>>>>
>>>> $ curl "http://localhost:8983/solr/update?commit=3Dtrue" -H
>>>> 'Content-type:application/json' -d '[{"id":"hoss", "cat":"<Something t=
o use
>>>> as a source node>" } ]'
>>>>
>>>> And this is what i get from the following requests...
>>>>
>>>> $ curl
>>>> "http://localhost:8983/solr/select?q=3Did:hoss&wt=3Dxml&indent=3Dtrue&=
omitHeader=3Dtrue"
>>>> <?xml version=3D"1.0" encoding=3D"UTF-8"?>
>>>> <response>
>>>>
>>>> <result name=3D"response" numFound=3D"1" start=3D"0">
>>>>   <doc>
>>>>     <str name=3D"id">hoss</str>
>>>>     <arr name=3D"cat">
>>>>       <str>&lt;Something to use as a source node&gt;</str>
>>>>     </arr>
>>>>     <long name=3D"_version_">1427705631375097856</long></doc>
>>>> </result>
>>>> </response>
>>>>
>>>> $ curl
>>>> "http://localhost:8983/solr/select?q=3Did:hoss&wt=3Djson&indent=3Dtrue=
&omitHeader=3Dtrue"
>>>> {
>>>>   "response":{"numFound":1,"start":0,"docs":[
>>>>       {
>>>>         "id":"hoss",
>>>>         "cat":["<Something to use as a source node>"],
>>>>         "_version_":1427705631375097856}]
>>>>   }}
>>>>
>>>> $ curl
>>>> "http://localhost:8983/solr/select?q=3Dcat:%22<Something+to+use+as+a+s=
ource+node>%22&wt=3Djson&indent=3Dtrue&omitHeader=3Dtrue"
>>>> {
>>>>   "response":{"numFound":1,"start":0,"docs":[
>>>>       {
>>>>         "id":"hoss",
>>>>         "cat":["<Something to use as a source node>"],
>>>>         "_version_":1427705631375097856}]
>>>>   }}
>>>>
>>>> ----END GOOD EXAMPLE OF SPECIFICS---
>>>>
>>>> : > Even more curious, if I use this query at the console:
>>>> : >
>>>> : > details:<Something to use as a source node>
>>>> : >
>>>> : > I get nothing back.
>>>>
>>>> note in my last example above the importance of using quotes (or the
>>>> {!term} qparser) to query string fields that contain special character=
s
>>>> like whitespace -- whitespace is syntacally meaningul to the lucene qu=
ery
>>>> parser, it seperates clauses of a boolean query.
>>>>
>>>>
>>>> -Hoss
>>>