manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CONNECTORS-1567) export of web connection bandwidth throttling
Date Tue, 08 Jan 2019 21:51:00 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737559#comment-16737559
] 

Karl Wright commented on CONNECTORS-1567:
-----------------------------------------

The output code is in common for all connections, and looks correct:

{code}
    String[] throttles = connection.getThrottles();
    j = 0;
    while (j < throttles.length)
    {
      String match = throttles[j++];
      String description = connection.getThrottleDescription(match);
      float rate = connection.getThrottleValue(match);
      child = new ConfigurationNode(CONNECTIONNODE_THROTTLE);
      ConfigurationNode throttleChildNode;

      throttleChildNode = new ConfigurationNode(CONNECTIONNODE_MATCH);
      throttleChildNode.setValue(match);
      child.addChild(child.getChildCount(),throttleChildNode);

      if (description != null)
      {
        throttleChildNode = new ConfigurationNode(CONNECTIONNODE_MATCHDESCRIPTION);
        throttleChildNode.setValue(description);
        child.addChild(child.getChildCount(),throttleChildNode);
      }

      throttleChildNode = new ConfigurationNode(CONNECTIONNODE_RATE);
      throttleChildNode.setValue(new Float(rate).toString());
      child.addChild(child.getChildCount(),throttleChildNode);

      connectionNode.addChild(connectionNode.getChildCount(),child);
    }
{code}

Note that the throttles are an array, so if there are no throttles, one should expect null
or an empty array to be output.  Checking the reading side now.

> export of web connection bandwidth throttling
> ---------------------------------------------
>
>                 Key: CONNECTORS-1567
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1567
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Web connector
>    Affects Versions: ManifoldCF 2.11, ManifoldCF 2.12
>            Reporter: Tim Steenbeke
>            Assignee: Karl Wright
>            Priority: Major
>             Fix For: ManifoldCF 2.13
>
>
> When exporting the web connector using the API, it doesn't export the bandwidth throttling.
>  Than when importing this connector to a clean manifoldcf it creates the connector with
basic bandwidth.
>  When using the connector in a job it works properly.
> The issue here is that the connector isn't created with correct bandwidth throttling.
>  And the connector gives issues in the UI when trying to view or edit.
> (related to issue: [CONNECTORS-1568|https://issues.apache.org/jira/projects/CONNECTORS/issues/CONNECTORS-1568])
> e.g.:
> {code:java}
> {
>   "name": "test_web",
>   "configuration": null,
>     "_PARAMETER_": [
>       {
>         "_attribute_name": "Email address",
>         "_value_": "tim.steenbeke@formica.digital"
>       },
>       {
>         "_attribute_name": "Robots usage",
>         "_value_": "all"
>       },
>       {
>         "_attribute_name": "Meta robots tags usage",
>         "_value_": "all"
>       },
>       {
>         "_attribute_name": "Proxy host",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy port",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication domain",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication user name",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication password",
>         "_value_": ""
>       }
>     ]
>   },
>   "description": "Website repository standard settup",
>   "throttle": null,
>   "max_connections": 10,
>   "class_name": "org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector",
>   "acl_authority": null
> }{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message