lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mario Corchero (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-10888) almost self-generating python script(s) to access V2 APIs
Date Sat, 07 Oct 2017 11:48:04 GMT

    [ https://issues.apache.org/jira/browse/SOLR-10888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195659#comment-16195659
] 

Mario Corchero commented on SOLR-10888:
---------------------------------------

I've made some changes to the base script to:
- Use URL + body instead of passing the arguments in the URL (that did not work, commands
and parameters are part of the body)
- Add type documentation
- Ability to add new types and the mapping between them
- Defaults for unmappable types/commands.

At the moment the script just prints a curl command to shoot and get the desired output but
it can be just executed if we want to.

If this looks interesting there are still a bunch of "TODO" like:
- Supporting non-command actions (just retrieve the list of collections for example)
- Support other endpoints "cores", etc.
- Automagically discover sub-URLs (Specific operations on a collection for example)
- Add more types like array

This is how using it looks like:

{code}
$ ./solr_cli.py --help
usage: solr_cli.py [-h]
                   {collection:create,collection:create-alias,collection:delete-alias,backup-collection,restore-collection}
                   ...

CLI tool to interact with Solr Admin REST API.

optional arguments:
  -h, --help            show this help message and exit

collections:
  List of commands to manipulate collections

  {collection:create,collection:create-alias,collection:delete-alias,backup-collection,restore-collection}
    collection:create   Create a collection.
                        https://lucene.apache.org/solr/guide/collections-
                        api.html#create
    collection:create-alias
                        Allows one or more collections to be known by another
                        name. If this command is used on an existing alias,
                        the existing alias will be replaced with the new
                        collection details.
                        https://lucene.apache.org/solr/guide/collections-
                        api.html#createalias
    collection:delete-alias
                        Deletes a collection alias
                        https://lucene.apache.org/solr/guide/collections-
                        api.html#deletealias
    backup-collection   Backup Solr indexes and configurations for a specific
                        collection. One copy of the indexes will be taken from
                        each shard, and the config set for the collection will
                        also be copied.
                        https://lucene.apache.org/solr/guide/collections-
                        api.html#backup
    restore-collection  Restore Solr indexes and configurations from a backup.
                        You cannot restore into the same collection you took
                        the backup from. The target collection must not exist
                        before calling this command, as it will be created by
                        the restore action. The new collection will have the
                        same number of shards and replicas as the original
                        collection, and all routing strategies will be
                        retained.
                        https://lucene.apache.org/solr/guide/collections-
                        api.html#restore

All options and parameters are generated dynamically. Check the Solr Reference
guide for further documentation on the API.

$ ./solr_cli.py collection:create -h
usage: solr_cli.py collection:create [-h] [--config str]
                                     [--router json-object] [--numShards int]
                                     [--shards str] [--replicationFactor int]
                                     [--nrtReplicas int] [--tlogReplicas int]
                                     [--pullReplicas int]
                                     [--nodeSet solr-array]
                                     [--shuffleNodes bool]
                                     [--maxShardsPerNode int]
                                     [--autoAddReplicas bool]
                                     [--rule solr-array] [--snitch solr-array]
                                     [--properties json-object] [--async str]
                                     [--http_method {POST}]
                                     str

positional arguments:
  str                   The name of the collection to be created.

optional arguments:
  -h, --help            show this help message and exit
  --config str          The name of the configuration set (which must already
                        be stored in ZooKeeper) to use for this collection. If
                        not provided, Solr will default to the collection name
                        as the configuration set name.
  --router json-object  These properties define how to distribute documents
                        across a collection's shards.
  --numShards int       The number of shards to be created as part of the
                        collection. Shards are logical partitions of a single
                        collection. Each shard has at least one replica, but
                        more replicas for each shard can be defined with the
                        replicationFactor property. This is a required
                        parameter when using the 'compositeId' router.
  --shards str          A comma-separated list of shard names, e.g.,
                        shard-x,shard-y,shard-z. This is a required parameter
                        when using the 'implicit' router.
  --replicationFactor int
                        The number of NRT replicas to be created for each
                        shard. Replicas are physical copies of each shard,
                        acting as failover for the shard.
  --nrtReplicas int     The number of NRT replicas to be created for each
                        shard. Replicas are physical copies of each shard,
                        acting as failover for the shard. Replicas of type NRT
                        will be updated with each document that is added to
                        the cluster, and can use "softCommits" to get a new
                        view of the index in Near Real Time. This parameter
                        works in the same way as 'replicationFactor'
  --tlogReplicas int    The number of TLOG replicas to be created for each
                        shard. TLOG replicas update their transaction log for
                        every update to the cluster, but only the shard leader
                        updates the local index, other TLOG replicas will use
                        segment replication and copy the latest index files
                        from the leader.
  --pullReplicas int    The number of PULL replicas to be created for each
                        shard. PULL replicas don't receive copies of the
                        documents on update requests, they just replicate the
                        latest segments periodically from the shard leader.
                        PULL replicas can't become shard leaders, and need at
                        least one active TLOG(recommended) or NRT replicas in
                        the shard to replicate from.
  --nodeSet solr-array  Defines nodes to spread the new collection across. If
                        not provided, the collection will be spread across all
                        live Solr nodes. The names to use are the 'node_name',
                        which can be found by a request to the cluster/nodes
                        endpoint. A special value of EMPTY will create no
                        shards or replicas for the new collection. In this
                        case, shards and replicas can be added later with the
                        add-replica command available on the
                        /collections/{collection}/shards endpoint.
  --shuffleNodes bool   Controls whether or not the shard-replicas created for
                        this collection will be assigned to the nodes
                        specified by the nodeSet property in a sequential
                        manner, or if the list of nodes should be shuffled
                        prior to creating individual replicas. A 'false' value
                        makes the results of a collection creation predictable
                        and gives more exact control over the location of the
                        individual shard-replicas, but 'true' can be a better
                        choice for ensuring replicas are distributed evenly
                        across nodes. This property is ignored if nodeSet is
                        not also specified.
  --maxShardsPerNode int
                        When creating collections, the shards and/or replicas
                        are spread across all available, live, nodes, and two
                        replicas of the same shard will never be on the same
                        node. If a node is not live when the collection is
                        created, it will not get any parts of the new
                        collection, which could lead to too many replicas
                        being created on a single live node. Defining
                        maxShardsPerNode sets a limit on the number of
                        replicas can be spread to each node. If the entire
                        collection can not be fit into the live nodes, no
                        collection will be created at all.
  --autoAddReplicas bool
                        When set to true, enables auto addition of replicas on
                        shared file systems (such as HDFS). See
                        https://lucene.apache.org/solr/guide/running-solr-on-
                        hdfs.html for more details on settings and overrides.
  --rule solr-array     Defines rules for where replicas should be located in
                        a cluster.
  --snitch solr-array
  --properties json-object
                        Allows adding core.properties for the collection. Some
                        examples of core properties you may want to modify
                        include the config set, the node name, the data
                        directory, among others.
  --async str           Defines a request ID that can be used to track this
                        action after it's submitted. The action will be
                        processed asynchronously.
  --http_method {POST}  HTTP Method to use.

$ ./solr_cli.py collection:create example_collection --numShards=1
curl http://localhost:8983/v2/collections/collection:create -X POST -d '{'collection:create':
{'name': 'example_collection', 'numShards': 1}}'
{code}

All "known" types are validated. Even complex ones. Example:

{code}
$ ./solr_cli.py collection:create example_collection --numShards=1 --properties="not_an_object"
usage: solr_cli.py collection:create [-h] [--config str]
                                     [--router json-object] [--numShards int]
                                     [--shards str] [--replicationFactor int]
                                     [--nrtReplicas int] [--tlogReplicas int]
                                     [--pullReplicas int]
                                     [--nodeSet solr-array]
                                     [--shuffleNodes bool]
                                     [--maxShardsPerNode int]
                                     [--autoAddReplicas bool]
                                     [--rule solr-array] [--snitch solr-array]
                                     [--properties json-object] [--async str]
                                     [--http_method {POST}]
                                     str
solr_cli.py collection:create: error: argument --properties: 'not_an_object' is not valid
JSON

$ ./solr_cli.py collection:create example_collection --numShards=1 --properties="{}"
curl http://localhost:8983/v2/collections/collection:create -X POST -d '{'collection:create':
{'name': 'example_collection', 'numShards': 1, 'properties': '{}'}}'

$ ./solr_cli.py collection:create example_collection --numShards=1 --properties="{}"
{code}

> almost self-generating python script(s) to access V2 APIs
> ---------------------------------------------------------
>
>                 Key: SOLR-10888
>                 URL: https://issues.apache.org/jira/browse/SOLR-10888
>             Project: Solr
>          Issue Type: New Feature
>          Components: v2 API
>            Reporter: Christine Poerschke
>            Priority: Minor
>         Attachments: SOLR-10888.patch
>
>
> The V2 API supports introspection and the results of such introspection(s) can be used
to automatically on-the-fly generate a (nested) {{argparse.ArgumentParser}} in a python script
and then to again automatically transform the script arguments into a url and http call to
the V2 API.
> Illustrative patch and sample output to follow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message