Return-Path: X-Original-To: apmail-lucene-commits-archive@www.apache.org Delivered-To: apmail-lucene-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3483910F51 for ; Tue, 9 Jul 2013 21:05:01 +0000 (UTC) Received: (qmail 76988 invoked by uid 500); 9 Jul 2013 21:05:01 -0000 Mailing-List: contact commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list commits@lucene.apache.org Received: (qmail 76981 invoked by uid 99); 9 Jul 2013 21:05:01 -0000 Received: from cwiki-vm.apache.org (HELO cwiki-vm) (140.211.11.69) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jul 2013 21:05:01 +0000 Received: from cwiki-vm (localhost [127.0.0.1]) by cwiki-vm (Postfix) with ESMTP id AE282A262C for ; Tue, 9 Jul 2013 21:05:00 +0000 (UTC) Date: Tue, 9 Jul 2013 21:05:00 +0000 (UTC) From: "Hoss Man (Confluence)" To: commits@lucene.apache.org Message-ID: <916068990.3943.1373403900005.JavaMail.confluence@cwiki-vm> Subject: [CONF] Apache Solr Reference Guide > Uploading Data with Index Handlers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Auto-Submitted: auto-generated Space: Apache Solr Reference Guide (https://cwiki.apache.org/confluence/display/solr) Page: Uploading Data with Index Handlers (https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers) Change Comment: --------------------------------------------------------------------- fix formating of delete w/ version example Edited by Hoss Man: --------------------------------------------------------------------- {section} {column:width=75%} Index Handlers are Update Handlers designed to add, delete and update documents to the index. Solr includes several of these to allow indexing documents in XML, CSV and JSON. The example URLs given here reflect the handler configuration in the supplied {{solrconfig.xml}}. If the name associated with the handler is changed then the URLs will need to be modified. It is quite possible to access the same handler using more than one name, which can be useful if you wish to specify different sets of default options. New {{UpdateProcessors}} now default to the {{uniqueKey}} field if it is the appropriate type for configured fields. The processors automatically add fields with new UUIDs and Timestamps to {{SolrInputDocuments}}. These work similarly to the option in {{schema.xml}}, but are applied in the {{UpdateProcessorChain}}. They may be used prior to other {{UpdateProcessors}}, or to generate a {{uniqueKey}} field value when using the {{DistributedUpdateProcessor}} (i.e., SolrCloud), {{TimestampUpdateProcessorFactory}}, {{UUIDUpdateProcessorFactory}}, and {{DefaultValueUpdateProcessorFactory}}. {column} {column:width=25%} {panel} Index Handlers covered in this section: {toc:minLevel=2|maxLevel=2} {panel} {column} {section} h2. Combined UpdateRequestHandlers For the separate XML, CSV, JSON, and javabin update request handlers explained below, Solr provides a single {{RequestHandler}}, and chooses the appropriate {{ContentStreamLoader}} based on the the {{Content-Type}} header, entered as the {{qt}} (query type) parameter matching the name of registered handlers. The "standard" request handler is the default and will be used if {{qt}} is not specified in the request. {code:lang=xml|borderStyle=solid|borderColor=#666666} {code} h3. Configuring Shard Handlers for Distributed Searches Inside the RequestHandler, you can configure and specify the shard handler used for distributed search. You can also plug in custom shard handlers as well. Configuring the standard handler, set up the configuration as in this example: {code:lang=xml|borderStyle=solid|borderColor=#666666} 1000 5000 {code} The parameters that can be specified are as follows: || Parameter || Default || Explanation || | socketTimeout | default: 0 (use OS default) | The amount of time in ms that a socket is allowed to wait | | connTimeout | default: 0 (use OS default) | The amount of time in ms that is accepted for binding / connection a socket | | maxConnectionsPerHost | default: 20 | The maximum number of connections that is made to each individual shard in a distributed search | | corePoolSize | default: 0 | The retained lowest limit on the number of threads used in coordinating distributed search | | maximumPoolSize | default: Integer.MAX_VALUE | The maximum number of threads used for coordinating distributed search | | maxThreadIdleTime | default: 5 seconds | The amount of time to wait for before threads are scaled back in response to a reduction in load | | sizeOfQueue | default: \-1 | If specified, the thread pool will use a backing queue instead of a direct handoff buffer. High throughput systems will want to configure this to be a direct hand off (with \-1). Systems that desire better latency will want to configure a reasonable size of queue to handle variations in requests. | | fairnessPolicy | default: false | Chooses the JVM specifics dealing with fair policy queuing. If enabled, distributed searches will be handled in a First in - First out method at a cost to throughput. If disabled, throughput will be favored over latency. | {topofpage} h2. XMLUpdateRequestHandler for XML-formatted Data h3. Configuration The default configuration file has the update request handler configured by default. {code:lang=xml|borderStyle=solid|borderColor=#666666} {code} h3. Adding Documents Documents are added to the index by sending an XML message to the update handler. The XML schema recognized by the update handler is very straightforward: * The {{}} element introduces one more more documents to be added. * The {{}} element introduces the fields making up a document. * The {{}} element presents the content for a specific field. For example: {code:lang=xml|borderStyle=solid|borderColor=#666666} Patrick Eagar Sports 796.35 128 12.40 Summer of the all-rounder: Test and championship cricket in England 1982 0002166313 1982 Collins ... {code} If the document schema defines a unique key, then an {{/update}} operation silently replaces a document in the index with the same unique key, unless the {{}} element sets the {{allowDups}} attribute to {{true}}. If no unique key has been defined, indexing performance is somewhat faster, as no search has to be made for an existing document. Each element has certain optional attributes which may be specified. || Command || Command Description || Optional Parameter || Parameter Description || | | Introduces one or more documents to be added to the index. | commitWithin=_number_ | Add the document within the specified number of milliseconds | | | Introduces the definition of a specific document. | boost=_float_ | Default is 1.0. Sets a boost value for the document.To learn more about boosting, see [Searching]. | | | Defines a field within a document. | boost=_float_ | Default is 1.0. Sets a boost value for the field. | {note} Other optional parameters for {{}}, including {{allowDups}}, {{overwritePending}}, and {{overwriteCommitted}}, are now deprecated. However, you can specify {{overwrite=false}} for XML updates to avoid overwriting. {note} h3. Commit and Optimize Operations The {{}} operation writes all documents loaded since the last commit to one or more segment files on the disk. Before a commit has been issued, newly indexed content is not visible to searches. The commit operation opens a new searcher, and triggers any event listeners that have been configured. Commits may be issued explicitly with a {{}} message, and can also be triggered from {{}} parameters in {{solrconfig.xml}}. The {{}} operation requests Solr to merge internal data structures in order to improve search performance. For a large index, optimization will take some time to complete, but by merging many small segment files into a larger one, search performance will improve. If you are using Solr's replication mechanism to distribute searches across many systems, be aware that after an optimize, a complete index will need to be transferred. In contrast, post-commit transfers are usually much smaller. The {{}} and {{}} elements accept these optional attributes: || Optional Attribute || Description || | maxSegments | Default is 1. Optimizes the index to include no more than this number of segments. | | waitFlush | Default is true. Blocks until index changes are flushed to disk. | | waitSearcher | Default is true. Blocks until a new searcher is opened and registered as the main query searcher, making the changes visible. | | expungeDeletes | Default is false. Merges segments and removes deleted documents. | Here are examples of and using optional attributes: {code:lang=xml|borderStyle=solid|borderColor=#666666} {code} h3. Delete Operations Documents can be deleted from the index in two ways. "Delete by ID" deletes the document with the specified ID, and can be used only if a UniqueID field has been defined in the schema. "Delete by Query" deletes all documents matching a specified query, although {{commitWithin}} is ignored for a Delete by Query. A single delete message can contain multiple delete operations. {code:lang=xml|borderStyle=solid|borderColor=#666666} 0002166313 0031745983 subject:sport publisher:penguin {code} h3. Rollback Operations The rollback command rolls back all add and deletes made to the index since the last commit. It neither calls any event listeners nor creates a new searcher. Its syntax is simple: {{}}. h3. Using {{curl}} to Perform Updates with the Update Request Handler. You can use the {{curl}} utility to perform any of the above commands, using its {{\--data-binary}} option to append the XML message to the {{curl}} command, and generating a HTTP POST request. For example: {code:lang=xml|borderStyle=solid|borderColor=#666666} curl http://localhost:8983/update -H "Content-Type: text/xml" --data-binary ' Patrick Eagar Sports 796.35 0002166313 1982 Collins ' {code} For posting XML messages contained in a file, you can use the alternative form: {code:borderStyle=solid|borderColor=#666666} curl http://localhost:8983/update -H "Content-Type: text/xml" --data-binary @myfile.xml {code} Short requests can also be sent using a HTTP GET command, URL-encoding the request, as in the following. Note the escaping of "<" and ">": {code:borderStyle=solid|borderColor=#666666} curl http://localhost:8983/update?stream.body=%3Ccommit/%3E {code} Responses from Solr take the form shown here: {code:borderStyle=solid|borderColor=#666666} 0 127 {code} The status field will be non-zero in case of failure. The servlet container will generate an appropriate HTML-formatted message in the case of an error at the HTTP layer. h3. A Simple Cross-Platform Posting Tool For demo purposes, the file {{$SOLR/example/exampledocs/post.jar}} includes a cross-platform Java tool for POST-ing XML documents. Open a window and run: {code:borderStyle=solid|borderColor=#666666} java -jar post.jar {code} By default, this will contact the server at {{localhost:8983}}. The "-help" option outputs the following information on its usage: {code:borderStyle=solid|borderColor=#666666} SimplePostTool: version 1.2 {code} This is a simple command line tool for POSTing raw XML to a Solr port. XML data can be read from files specified as command line args; as raw commandline arg strings; or via STDIN. Examples: {code:borderStyle=solid|borderColor=#666666} java -Ddata=files -jar post.jar *.xml java -Ddata=args -jar post.jar '42' java -Ddata=stdin -jar post.jar < hd.xml {code} Other options controlled by System Properties include the Solr URL to POST to, and whether a commit should be executed. These are the defaults for all System Properties. {code:borderStyle=solid|borderColor=#666666} -Ddata=files -Durl=[http://localhost:8983/solr/update|http://localhost:8983/solr/update] -Dcommit=yes {code} For more information about the XML Update Request Handler, see [https://wiki.apache.org/solr/UpdateXmlMessages]. {topofpage} h2. XSLTRequestHandler to Transform XML Content h3. Configuration The default configuration file has the update request handler configured by default, although the "lazy load" flag is set. The XSLTRequestHandler allows you to index any XML data with the [XML {{}} command|http://xmlstar.sourceforge.net/doc/UG/ch04s02.html]. You must have an XSLT stylesheet in the solr/conf/xslt directory that can transform the incoming data to the expected {{}} format. {code:lang=xml|borderStyle=solid|borderColor=#666666} {code} Here is an example XSLT stylesheet: {code:borderStyle=solid|borderColor=#666666} {code} Attaching the stylesheet "updateXml.xsl" transforms a search result to Solr's {{UpdateXml}} syntax. One example is to copy a Solr1.3 index (which does not have CSV response writer) into a format which can be indexed into another Solr file (provided that all fields are stored): {code} http://localhost:8983/solr/select?q=*:*&wt=xslt&tr=updateXml.xsl&rows=1000 {code} You can also use the stylesheet in {{XsltUpdateRequestHandler}} to transform an index when updating: {code} curl "http://localhost:8983/solr/update/xslt?commit=true&tr=updateX {code} {topofpage} h2. CSVRequestHandler for CSV Content h3. Configuration The default configuration file has the update request handler configured by default, although the "lazy load" flag is set. {code:lang=xml|borderStyle=solid|borderColor=#666666} {code} h3. Parameters The CSV handler allows the specification of many parameters in the URL in the form: {{f.}}{{{}{_}parameter{_}{}}}{{.}}{{{}{_}optional_fieldname{_}{}}}{{=}}{{{}{_}value{_}}}. The table below describes the parameters for the update handler. || Parameter || Usage || Global (g) or Per Field (f) || Example || | separator | Character used as field separator; default is "," | g,(f: see split) | separator=% | | trim | If true, remove leading and trailing whitespace from values. Default=false. | g,f | f.isbn.trim=true \\ trim=false | | header | Set to true if first line of input contains field names. These will be used if the *field_name* parameter is absent. | g | | | field_name | Comma separated list of field names to use when adding documents. | g | field_name=isbn,price,title | | literal. | Comma separated list of field names to use when processing literal values. | g | literal.color=red,blue,black | | skip | Comma separated list of field names to skip. | g | skip=uninteresting,shoesize | | skipLines | Number of lines to discard in the input stream before the CSV data starts, including the header, if present. Default=0. | g | skipLines=5 | | encapsulator | The character optionally used to surround values to preserve characters such as the CSV separator or whitespace. This standard CSV format handles the encapsulator itself appearing in an encapsulated value by doubling the encapsulator. | g,(f: see split) | encapsulator=" | | escape | The character used for escaping CSV separators or other reserved characters. If an escape is specified, the encapsulator is not used unless also explicitly specified since most formats use either encapsulation or escaping, not both | g | escape=\ \\ | | keepEmpty | Keep and index zero length (empty) fields. Default=false. | g,f | f.price.keepEmpty=true | | map | Map one value to another. Format is value:replacement (which can be empty.) | g,f | map=left:right \\ f.subject.map=history:bunk | | split | If true, split a field into multiple values by a separate parser. | f | | | overwrite | If true (the default), check for and overwrite duplicate documents, based on the uniqueKey field declared in the Solr schema. If you know the documents you are indexing do not contain any duplicates then you may see a considerable speed up setting this to false. | g | | | commit | Issues a commit after the data has been ingested. | g | | | commitWithin | Add the document within the specified number of milliseconds. | g | commitWithin=10000 | | rowid | Map the rowid (line number) to a field specified by the value of the parameter, for instance if your CSV doesn't have a unique key and you want to use the row id as such. | g | rowid=id| | rowidOffset | Add the given offset (as an int) to the rowid before adding it to the document. Default is 0 | g | rowidOffset=10| For more information on the CSV Update Request Handler, see [https://wiki.apache.org/solr/UpdateCSV]. {topofpage} h2. Using the JSONRequestHandler for JSON Content JSON formatted update requests may be sent to Solr using the {{/solr/update/json}} URL. All of the normal methods for uploading content are supported. h3. Configuration The default configuration file has the update request handler configured by default, although the "lazy load" flag is set. {code:lang=xml|borderStyle=solid|borderColor=#666666} {code} h3. Examples There is a sample JSON file at {{example/exampledocs/books.json}} that you can use to add documents to the Solr example server. {code:borderStyle=solid|borderColor=#666666} cd example/exampledocs curl 'http://localhost:8983/solr/update/json?commit=true' --data-binary @books.json -H 'Content-type:application/json' {code} Adding {{commit=true}} to the URL makes the documents immediately searchable. You should now be able to query for the newly added documents: {{[http://localhost:8983/solr/select?q=title:monsters&wt=json&indent=true]}} returns: {code:borderStyle=solid|borderColor=#666666} { "responseHeader":{ "status":0, "QTime":2, "params":{ "indent":"true", "wt":"json", "q":"title:monsters"}}, "response":{"numFound":1,"start":0,"docs":[ { "id":"978-1423103349", "author":"Rick Riordan", "series_t":"Percy Jackson and the Olympians", "sequence_i":2, "genre_s":"fantasy", "inStock":true, "price":6.49, "pages_i":304, "title":[ "The Sea of Monsters"], "cat":["book","paperback"]}] } } {code} h3. Update Commands The JSON update handler accepts all of the update commands that the XML update handler supports, through a straightforward mapping. Multiple commands may be contained in one message: {code:borderStyle=solid|borderColor=#666666} { "add": { "doc": { "id": "DOC1", "my_boosted_field": { /* use a map with boost/value for a boosted field */ "boost": 2.3, "value": "test" }, "my_multivalued_field": [ "aaa", "bbb" ] /* use an array for a multi-valued field */ } }, "add": { "commitWithin": 5000, /* commit this document within 5 seconds */ "overwrite": false, /* don't check for existing documents with the same uniqueKey */ "boost": 3.45, /* a document boost */ "doc": { "f1": "v1", "f1": "v2" } }, "commit": {}, "optimize": { "waitFlush":false, "waitSearcher":false }, "delete": { "id":"ID" }, /* delete by ID */ "delete": { "query":"QUERY" } /* delete by query */ } {code} {note} Comments are not allowed JSON, but duplicate names are. {note} As with other update handlers, parameters such as {{commit}}, {{commitWithin}}, {{optimize}}, and {{overwrite}} may be specified in the URL instead of in the body of the message. The JSON update format allows for a simple delete-by-id. The value of a {{delete}} can be an array which contains a list of zero or more specific document id's (not a range) to be deleted. For example: {code:borderStyle=solid|borderColor=#666666} "delete":"myid" {code} {code:borderStyle=solid|borderColor=#666666} "delete":["id1","id2"] {code} The value of a "delete" can be an array which contains a list of zero or more id's to be deleted. It is not a range (start and end). You can also specify {{\_version\_}} with each "delete": {code:borderStyle=solid|borderColor=#666666} String str = "{'delete':'id':50, '_version_':12345}" {code} You can specify the version of deletes in the body of the update request as well. For more information about the JSON Update Request Handler, see [https://wiki.apache.org/solr/UpdateJSON]. {topofpage} h2. Updating Only Part of a Document Solr supports several modifiers that atomically update values of a document. || Modifier || Usage || | set | Set or replace a particular value, or remove the value if null is specified as the new value. | | add | Adds an additional value to a list. | | inc | Increments a numeric value by a specific amount. | {note} All original source fields must be stored for field modifiers to work correctly, which is the Solr default. {note} For example: {code:borderStyle=solid|borderColor=#666666} {"id":"mydoc", "f1"{"set":10}, "f2"{"add":20}} {code} This example results in field {{f1}} being set to "10", and field {{f2}} having an additional value of "20" added. All other existing fields from the original document remain unchanged. {topofpage} h2. Using SimplePostTool This is a simple command line tool for POSTing raw data to a Solr port. Data can be read from files specified as command line arguments, as raw command line argument strings, or via {{STDIN}}. Options controlled by System Properties include the Solr URL to post to, the {{Content-Type}} of the data, whether a commit or optimize should be executed, and whether the response should be written to {{STDOUT}}. If {{auto=yes}} the tool will try to guess the type and set {{Content-Type}} and the URL automatically. When posting rich documents the file name will be propagated as {{resource.name}} and also used as {{literal.id}}. You may override these or any other request parameter through the {{\-Dparams}} property Supported System Properties and their defaults: | *Parameter* | *Values* | *Default* | | \-Ddata | yes, no | default=files | | \-Dtype | | default=application/xml | | \-Durl | | default=[http://localhost:8983/solr/update] | | \-Dauto | yes, no | default=no | | \-Drecursive | yes, no | default=no | | \-Dfiletypes | \[,,..\] | default=xml, json, csv, pdf, doc, docx, ppt, pptx, xls, xlsx, odt, odp, ods, rtf, htm, html | | \-Dparams | "=\[&=...\]" | values must be URL-encoded | | \-Dcommit | yes, no | default=yes | | \-Doptimize | yes, no | default=no | | \-Dout | yes,no | default=no | Examples: {code:borderStyle=solid|borderColor=#666666} java -jar post.jar *.xml java -Ddata=args -jar post.jar '42' java -Ddata=stdin -jar post.jar < hd.xml java -Dtype=text/csv -jar post.jar *.csv java -Dtype=application/json -jar post.jar *.json java -Durl=[http://localhost:8983/solr/update/extract] -Dparams=literal.id=a -Dtype=application/pdf -jar post.jar a.pdf java -Dauto=yes -jar post.jar a.pdf java -Dauto=yes -Drecursive=yes -jar post.jar afolder java -Dauto=yes -Dfiletypes=ppt,html -jar post.jar afolder {code} In the above example: | *\-Dauto=yes* | Will guess file type from file name suffix, and set type and url accordingly. It also sets the ID and file name automatically. | | *\-Drecursive=yes* | Will recurse into sub-folders and index all files. | | *\-Dfiletypes* | Specifies the file types to consider when indexing folders. | | *\-Dparams* | HTTP GET params to add to the request, so you don't need to write the whole URL again. | h2. Indexing Using SolrJ Use of the the SolrJ client library is covered in the section on [solr:Using SolrJ]. {scrollbar} Stop watching space: https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=solr Change email notification preferences: https://cwiki.apache.org/confluence/users/editmyemailsettings.action