incubator-connectors-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Lucene Connector Framework > Programmatic Operation of LCF
Date Fri, 16 Jul 2010 17:07:00 GMT
Space: Lucene Connector Framework (https://cwiki.apache.org/confluence/display/CONNECTORS)
Page: Programmatic Operation of LCF (https://cwiki.apache.org/confluence/display/CONNECTORS/Programmatic+Operation+of+LCF)


Edited by Karl Wright:
---------------------------------------------------------------------
h1. Programmatic Operation of LCF

A certain subset of LCF users want to think of LCF as an engine that they can poke from whatever
other system they are developing.  While LCF is not precisely a document indexing engine per
se, it can certainly be controlled programmatically.  Right now, there are three principle
ways of achieving this control.

h3. Control by Servlet API

LCF provides a servlet-based JSON API that gives you the complete ability to define connections
and jobs, and control job execution.  You can read about JSON [here|http://www.json.org].
 The API can be called with either GET, POST, or multipart POST methods.  The format of the
servlet URL is as follows:

http\[s\]://_<server_and_port>_/lcf-api/json/_<command>_\[?object=_<json_argument>_\]

The servlet either returns an error response code (either 400 or 500) with an appropriate
explanatory message, or a 200 response code and a JSON object.  The _json_argument_ parameter
can be passed in either as part of the URL, or in POST data, whichever is most convenient.
 Bear in mind that URLs are limited by specification to 4096 characters, so for large payloads
you will want to use multipart form data rather than encoding arguments on the URL.

The actual available commands are as follows:



|| Command || What it does || Argument format || Response format ||
| outputconnector/list | List all registered output connectors | N/A | \{"outputconnector":\[_<list_of_output_connector_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| authorityconnector/list | List all registered authority connectors | N/A | \{"authorityconnector":\[_<list_of_authority_connector_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| repositoryconnector/list | List all registered repository connectors | N/A | \{"repositoryconnector":\[_<list_of_repository_connector_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| outputconnection/list | List all output connections | N/A | \{"outputconnection":\[_<list_of_output_connection_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| outputconnection/get | Get a specific output connection | \{"connection_name":_<connection_name>_\}
| \{"outputconnection":_<output_connection_object>_\} *OR* \{ \} *OR* \{"error":_<error_text>_\}
|
| outputconnection/save | Save or create an output connection | \{"outputconnection":_<output_connection_object>_\}
| \{"connection_name":_<connection_name>_\} *OR* \{"error":_<error_text>_\} |
| outputconnection/delete | Delete an output connection | \{"connection_name":_<connection_name>_\}
| \{ \} *OR* \{"error":_<error_text>_\} |
| outputconnection/checkstatus | Check the status of an output connection | \{"connection_name":_<connection_name>_\}
| \{"check_result":_<message>_\} *OR* \{"error":_<error_text>_\} |
| authorityconnection/list | List all authority connections | N/A | \{"authorityconnection":\[_<list_of_authority_connection_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| authorityconnection/get | Get a specific authority connection | \{"connection_name":_<connection_name>_\}
| \{"authorityconnection":_<authority_connection_object>_\} *OR* \{ \} *OR* \{"error":_<error_text>_\}
|
| authorityconnection/save | Save or create an authority connection | \{"authorityconnection":_<authority_connection_object>_\}
| \{"connection_name":_<connection_name>_\} *OR* \{"error":_<error_text>_\} |
| authorityconnection/delete | Delete an authority connection | \{"connection_name":_<connection_name>_\}
| \{ \} *OR* \{"error":_<error_text>_\} |
| authorityconnection/checkstatus | Check the status of an authority connection | \{"connection_name":_<connection_name>_\}
| \{"check_result":_<message>_\} *OR* \{"error":_<error_text>_\} |
| repositoryconnection/list | List all repository connections | N/A | \{"repositoryconnection":\[_<list_of_repository_connection_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| repositoryconnection/get | Get a specific repository connection | \{"connection_name":_<connection_name>_\}
| \{"repositoryconnection":_<repository_connection_object>_\} *OR* \{ \} *OR* \{"error":_<error_text>_\}
|
| repositoryconnection/save | Save or create a repository connection | \{"repositoryconnection":_<repository_connection_object>_\}
| \{"connection_name":_<connection_name>_\} *OR* \{"error":_<error_text>_\} |
| repositoryconnection/delete | Delete a repository connection | \{"connection_name":_<connection_name>_\}
| \{ \} *OR* \{"error":_<error_text>_\} |
| repositoryconnection/checkstatus | Check the status of a repository connection | \{"connection_name":_<connection_name>_\}
| \{"check_result":_<message>_\} *OR* \{"error":_<error_text>_\} |
| job/list | List all job definitions | N/A | \{"job":\[_<list_of_job_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| job/get | Get a specific job definition | \{"job_id":_<job_identifier>_\} | \{"job":_<job_object_>\}
*OR* \{ \} *OR* \{"error":_<error_text>_\} |
| job/save | Save or create a job definition | \{"job":_<job_object>_\} | \{"job_id":_<job_identifier>_\}
*OR* \{"error":_<error_text>_\} |
| job/delete | Delete a job definition | \{"job_id":_<job_identifier>_\} | \{ \} *OR*
\{"error":_<error_text>_\} |
| jobstatus/list | List all jobs and their status | N/A | \{"job":\[_<list_of_job_status_objects>_\]\}
*OR* \{"error":_<error_text>_\} |
| jobstatus/get | Get a specific job's status | \{"job_id":_<job_identifier>_\} | \{"jobstatus":_<job_status_object>\}
*OR* \{ \} *OR* \{"error":_<error_text>_\}  |
| jobstatus/start | Start a specified job manually | \{"job_id":_<job_identifier>_\}
| \{ \} *OR* \{"error":_<error_text>_\} |
| jobstatus/abort | Abort a specified job | \{"job_id":_<job_identifier>_\} | \{ \}
*OR* \{"error":_<error_text>_\} |
| jobstatus/restart | Stop and start a specified job | \{"job_id":_<job_identifier>_\}
| \{ \} *OR* \{"error":_<error_text>_\} |
| jobstatus/pause | Pause a specified job | \{"job_id":_<job_identifier>_\} | \{ \}
*OR* \{"error":_<error_text>_\} |
| jobstatus/resume | Resume a specified job | \{"job_id":_<job_identifier>_\} | \{ \}
*OR* \{"error":_<error_text>_\} |

Other commands having to do with reports have been planned, but not yet been implemented.

h5. Output connector objects

The JSON fields an output connector object has are as follows:

|| Field || Meaning ||
| "description" | The optional description of the connector |
| "class_name" | The class name of the class implementing the connector |

h5. Authority connector objects

The JSON fields an authority connector object has are as follows:

|| Field || Meaning ||
| "description" | The optional description of the connector |
| "class_name" | The class name of the class implementing the connector |

h5. Repository connector objects

The JSON fields a repository connector object has are as follows:

|| Field || Meaning ||
| "description" | The optional description of the connector |
| "class_name" | The class name of the class implementing the connector |

h5. Output connection objects

The JSON fields an output connection object has are as follows:

|| Field || Meaning ||
| "name" | The unique name of the connection |
| "description" | The description of the connection |
| "class_name" | The java class name of the class implementing the connection |
| "max_connections" | The total number of outstanding connections allowed to exist at a time
|
| "configuration" | The configuration object for the connection, which is specific to the
connection class |

h5. Authority connection objects

The JSON fields for an authority connection object are as follows:

|| Field || Meaning ||
| "name" | The unique name of the connection |
| "description" | The description of the connection |
| "class_name" | The java class name of the class implementing the connection |
| "max_connections" | The total number of outstanding connections allowed to exist at a time
|
| "configuration" | The configuration object for the connection, which is specific to the
connection class |

h5. Repository connection objects

The JSON fields for a repository connection object are as follows:

|| Field || Meaning ||
| "name" | The unique name of the connection |
| "description" | The description of the connection |
| "class_name" | The java class name of the class implementing the connection |
| "max_connections" | The total number of outstanding connections allowed to exist at a time
|
| "configuration" | The configuration object for the connection, which is specific to the
connection class |
| "acl_authority" | The (optional) name of the authority that will enforce security for this
connection |
| "throttle" | An array of throttle objects, which control how quickly documents can be requested
from this connection |

Each throttle object has the following fields:

|| Field || Meaning ||
| "match" | The regular expression which is used to match a document's bins to determine if
the throttle should be applied |
| "match_description" | Optional text describing the meaning of the throttle |
| "rate" | The maximum fetch rate to use if the throttle applies, in fetches per minute |

h5. Job objects

The JSON fields for a job are is as follows:

|| Field || Meaning ||
| "id" | The job's identifier, if present.  If not present, LCF will create one (and will
also create the job when saved). |
| "description" | Text describing the job |
| "repository_connection" | The name of the repository connection to use with the job |
| "output_connection" | The name of the output connection to use with the job |
| "document_specification" | The document specification object for the job, whose format is
repository-connection specific |
| "output_specification" | The output specification object for the job, whose format is output-connection
specific |
| "start_mode" | The start mode for the job, which can be one of "schedule window start",
"schedule window anytime", or "manual" |
| "run_mode" | The run mode for the job, which can be either "continuous" or "scan once" |
| "hopcount_mode" | The hopcount mode for the job, which can be either "accurate", "no delete",
"never delete" |
| "priority" | The job's priority, typically "5" |
| "recrawl_interval" | The default time between recrawl of documents (if the job is "continuous"),
in milliseconds, or "infinite" for infinity |
| "expiration_interval" | The time until a document expires (if the job is "continuous"),
in milliseconds, or "infinite" for infinity |
| "reseed_interval" | The time between reseeding operations (if the job is "continuous"),
in milliseconds, or "infinite" for infinity |
| "hopcount" | An array of hopcount objects, describing the link types and associated maximum
hops permitted for the job |
| "schedule" | An array of schedule objects, describing when the job should be started and
run |

Each hopcount object has the following fields:

|| Field || Meaning ||
| "link_type" | The connection-type-dependent type of a link for which a hop count restriction
is specified |
| "count" | The maximum number of hops allowed for the associated link type, starting at a
seed |

Each schedule object has the following fields:

|| Field || Meaning ||
| "timezone" | The optional time zone for the schedule object; if not present the default
server time zone is used |
| "duration" | The optional length of the described time window, in milliseconds; if not present,
duration is considered infinite |
| "dayofweek" | The optional day-of-the-week enumeration object |
| "monthofyear" | The optional month-of-the-year enumeration object |
| "dayofmonth" | The optional day-of-the-month enumeration object |
| "year" | The optional year enumeration object |
| "hourofday" | The optional hour-of-the-day enumeration object |
| "minutesofhour" | The optional minutes-of-the-hour enumeration object |

Each enumeration object describes an array of integers using the form:

\{"value":\[_<integer_list>_\]\}

Each integer is a zero-based index describing which entity is being specified.  For example,
for "dayofweek", 0 corresponds to Sunday, etc., and thus "dayofweek":\{"value":\[0,6\]\} would
describe Saturdays and Sundays.

h5. Job status objects

The JSON fields of a job status object are as follows:

|| Field || Meaning ||
| "job_id" | The job identifier |
| "status" | The job status, having the possible values: "not yet run", "running", "paused",
"done", "waiting", "starting up", "cleaning up", "error", "aborting", "restarting", "running
no connector", and "terminating" |
| "error_text" | The error text, if the status is "error" |
| "start_time" | The job start time, in milliseconds since Jan 1, 1970 |
| "end_time" | The job end time, in milliseconds since Jan 1, 1970 |
| "documents_in_queue" | The total number of documents in the queue for the job |
| "documents_outstanding" | The number of documents for the job that are currently considered
'active' |
| "documents_processed" | The number of documents that in the queue for the job that have
been processed at least once |

h5. Connection-type-specific objects

As you may note when trying to use the above JSON API methods, you cannot get very far in
defining connections or jobs without knowing the JSON format of a connection's configuration
information, or a job's connection-specific document specification and output specification
information.  The form of these objects is controlled by the Java implementation of the underlying
connector, and is translated directly into JSON, so if you write your own connector you should
be able to figure out what it will be in the API.  For connectors already part of LCF, it
remains an ongoing task to document these connector-specific objects.  This task is not yet
underway.

Luckily, it is pretty easy to learn a lot about the objects in question by simply creating
connections and jobs in the LCF crawler UI, and then inspecting the resulting JSON objects
through the API.  In this way, it should be possible to do a decent job of coding most API-based
integrations.  The one place where difficulties will certainly occur will be if you try to
completely replace the LCF crawler UI with one of your own.  This is because most connectors
have methods that communicate with their respective back-ends in order to allow the user to
select appropriate values.  For example, the path drill-down that is presented by the LiveLink
connector requires that the connector interrogate the appropriate LiveLink repository in order
to populate its path selection pull-downs.  There is, at this time, no way to accomplish the
same job using the API.  The connectors in question generate HTML code designed to work within
the LCF UI, and they have no current mechanism in place for supporting other UI's in a similar
manner.


h3. Control via Commands

For script writers, there currently exist a number of LCF execution commands.  These commands
are primarily rich in the area of definition of connections and jobs, controlling jobs, and
running reports.  The following table lists the current suite.

|| Command || What it does ||
| org.apache.lcf.agents.DefineOutputConnection | Create a new output connection |
| org.apache.lcf.agents.DeleteOutputConnection | Delete an existing output connection |
| org.apache.lcf.authorities.ChangeAuthSpec | Modify an authority's configuration information
|
| org.apache.lcf.authorities.CheckAll | Check all authorities to be sure they are functioning
|
| org.apache.lcf.authorities.DefineAuthorityConnection | Create a new authority connection
|
| org.apache.lcf.authorities.DeleteAuthorityConnection | Delete an existing authority connection
|
| org.apache.lcf.crawler.AbortJob | Abort a running job |
| org.apache.lcf.crawler.AddScheduledTime | Add a schedule record to a job |
| org.apache.lcf.crawler.ChangeJobDocSpec | Modify a job's specification information |
| org.apache.lcf.crawler.DefineJob | Create a new job |
| org.apache.lcf.crawler.DefineRepositoryConnection | Create a new repository connection |
| org.apache.lcf.crawler.DeleteJob | Delete an existing job |
| org.apache.lcf.crawler.DeleteRepositoryConnection | Delete an existing repository connection
|
| org.apache.lcf.crawler.ExportConfiguration | Write the complete list of all connection definitions
and job specifications to a file |
| org.apache.lcf.crawler.FindJob | Locate a job identifier given a job's name |
| org.apache.lcf.crawler.GetJobSchedule | Find a job's schedule given a job's identifier |
| org.apache.lcf.crawler.ImportConfiguration | Import configuration as written by a previous
ExportConfiguration command |
| org.apache.lcf.crawler.ListJobStatuses | List the status of all jobs |
| org.apache.lcf.crawler.ListJobs | List the identifiers for all jobs |
| org.apache.lcf.crawler.PauseJob | Given a job identifier, pause the specified job |
| org.apache.lcf.crawler.RestartJob | Given a job identifier, restart the specified job |
| org.apache.lcf.crawler.RunDocumentStatus | Run a document status report |
| org.apache.lcf.crawler.RunMaxActivityHistory | Run a maximum activity report |
| org.apache.lcf.crawler.RunMaxBandwidthHistory | Run a maximum bandwidth report |
| org.apache.lcf.crawler.RunQueueStatus | Run a queue status report |
| org.apache.lcf.crawler.RunResultHistory | Run a result history report |
| org.apache.lcf.crawler.RunSimpleHistory | Run a simply history report |
| org.apache.lcf.crawler.StartJob | Start a job |
| org.apache.lcf.crawler.WaitForJobDeleted | After a job has been deleted, wait until the
delete has completed |
| org.apache.lcf.crawler.WaitForJobInactive | After a job has been started or aborted, wait
until the job ceases all activity |
| org.apache.lcf.crawler.WaitJobPaused | After a job has been paused, wait for the pause to
take effect |

h3. Control by direct code

Control by direct java code is quite a reasonable thing to do.  The sources of the above commands
should give a pretty clear idea how to proceed, if that's what you want to do.


h3. Caveats

The existing commands know nothing about the differences between connection types.  Instead,
they deal with configuration and specification information in the form of XML documents. 
Normally, these XML documents are hidden from a system integrator, unless they happen to look
into the database with a tool such as psql.  But the API commands above often will require
such XML documents to be included as part of the command execution.

This has one major consequence.  Any application that would manipulate connections and jobs
directly cannot be connection-type independent - these applications must know the proper form
of XML to submit to the command.  So, it is not possible to use these command APIs to write
one's own UI wrapper, without sacrificing some of the generality that LCF by itself maintains.


Change your notification preferences: https://cwiki.apache.org/confluence/users/viewnotifications.action
  

Mime
View raw message