lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Farrell <gsf...@gmail.com>
Subject Re: Wiki Error JSON syntax
Date Mon, 25 Jul 2011 17:00:08 GMT
On Mon, Jul 25, 2011 at 12:24 PM, Stefan Matheis
<matheis.stefan@googlemail.com> wrote:
> Hi Remy,
>
> so you may open an Issue for this on the github Project? i mean .. just
> creating another client, because i have one problem, does not sound like a
> good plan?

Agreed, and thanks for calling my attention to this thread, Stefan.

> Regards
> Stefan
>
> Am 25.07.2011 10:56, schrieb Remy Loubradou:
>>
>> Hey Stephan,
>>
>> Thanks, but I already used this solr client and I got an error when I add
>> too much documents "FATAL ERROR: JS Allocation failed - process out of
>> memory".
>> I didn't find the source of the problem in the solr client. So I decided
>> to
>> write my own without this error hopefully and also I'm using JSON
>> documents
>> and not XML documents. I read a post saying that I can get better
>> performance using JSON documents.
>>
>> I will release this client as an npm module.

How many documents are you attempting to add at once when you get that
error? Would it possible to chunk them into smaller groups?

I'm happy to work with you on enhancing node-solr to meet your needs.
The only reason updates are via XML rather than JSON is that 3.1 was
new or not yet released (don't quite remember which) when I first
wrote node-solr. Even now I imagine many people may still be using a
version of Solr that doesn't handle JSON updates. Maybe a flag
parameter could be added to the Client object to switch from XML to
JSON?

The node-solr client has fairly complete test coverage, a history of
commits from the Node community, and  versions aligning with several
versions of Node. I would appreciate your contributions, either via
issues or pull requests.

>> Regards,
>> Remy
>>
>> 2011/7/25 Stefan Matheis<matheis.stefan@googlemail.com>
>>
>>> Remy,
>>>
>>> didn't use it myself .. but you know about https://github.com/gsf/node-**
>>> solr<https://github.com/gsf/node-solr>  ?
>>>
>>> Regards
>>> Stefan
>>>
>>> Am 20.07.2011 20:05, schrieb Remy Loubradou:
>>>
>>>  I think I can trust you but this is weird.
>>>>
>>>> Funny things if you try to validate on http://jsonlint.com/ this JSON,
>>>> duplicates keys are automatically removed. But the thing is, how can you
>>>> possibly generate this json with Javascript Object?
>>>>
>>>> It will be really nice to combine both ways that you show on the page.
>>>> Something like:
>>>>
>>>> {
>>>>     "add": [
>>>>         {
>>>>             "doc": {
>>>>                 "id": "DOC1",
>>>>                 "my_boosted_field": {
>>>>                     "boost": 2.3,
>>>>                     "value": "test"
>>>>                 },
>>>>                 "my_multivalued_field": [
>>>>                     "aaa",
>>>>                     "bbb"
>>>>                 ]
>>>>             }
>>>>         },
>>>>         {
>>>>             "commitWithin": 5000,
>>>>             "overwrite": false,
>>>>             "boost": 3.45,
>>>>             "doc": {
>>>>                 "f1": "v2"
>>>>             }
>>>>         }
>>>>     ],
>>>>     "commit": {},
>>>>     "optimize": {
>>>>         "waitFlush": false,
>>>>         "waitSearcher": false
>>>>     },
>>>>     "delete": [
>>>>         {
>>>>             "id": "ID"
>>>>         },
>>>>         {
>>>>             "query": "QUERY"
>>>>         }
>>>>     ]
>>>> }
>>>>
>>>> Thanks you for you previous response Yonik.
>>>>
>>>> 2011/7/20 Yonik
>>>> Seeley<yonik@lucidimagination.**com<yonik@lucidimagination.com>
>>>>>
>>>>
>>>>  On Wed, Jul 20, 2011 at 12:16 PM, Remy Loubradou
>>>>>
>>>>> <remyloubradou@gmail.com>   wrote:
>>>>>
>>>>>> Hi,
>>>>>> I was writing a Solr Client API for Node and I found an error on
this
>>>>>>
>>>>> page
>>>>>
>>>>>>
>>>>>> http://wiki.apache.org/solr/**UpdateJSON<http://wiki.apache.org/solr/UpdateJSON>,on
>>>>>> the section "Update Commands"
>>>>>>
>>>>> the
>>>>>
>>>>>> JSON is not valid because there are duplicate keys and two times
with
>>>>>>
>>>>> "add"
>>>>>
>>>>>> and "delete".
>>>>>>
>>>>>
>>>>> It's a common misconception that it's invalid JSON.  Duplicate keys
>>>>> are in fact legal.

I can't resist addressing this side conversation.

While I understand the desire for a straightforward mapping between
the XML and JSON update formats, I think the use of duplicate keys is
a bad idea. As noted in the spec
(http://www.ietf.org/rfc/rfc4627.txt), "The names within an object
SHOULD be unique." I'm not sure the reasons here justify ignoring that
recommendation.

The fact that you need to keep reminding people that duplicate names
are legal is a sign that it's more trouble than it's worth. Also, most
JSON parsers just punt on duplicate names (see the third paragraph in
"A word about design" at
http://planet.plt-scheme.org/package-source/dherman/json.plt/3/0/planet-docs/json/index.html
for one take on the situation). I really don't want to write a new
JavaScript JSON parser just for node-solr.

Mime
View raw message