manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CONNECTORS-1567) export of web connection bandwidth throttling
Date Tue, 08 Jan 2019 22:43:00 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737605#comment-16737605
] 

Karl Wright commented on CONNECTORS-1567:
-----------------------------------------

I just modified a test to set a throttle and read it back from a file system connection. 
The result does not align at all with the ticket description.  Here's the JSON:

{code}
    [junit] Reading back JSON = '{"repositoryconnection":{"_children_":[{"_type_":"isnew","_value_":"false"},{"_type_":"name","_value_":"File
Connection"},{"_type_":"class_name","_value_":"org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector"},{"_type_":"max_connections","_value_":"100"},{"_type_":"description","_value_":"File
Connection"},{"_type_":"configuration"},{"_type_":"throttle","_children_":[{"_type_":"match","_value_":"blah"},{"_type_":"rate","_value_":"50.0"}]}]}}'
{code}

Note that each node is spelled out in long form, not compact form.  This is because the generation
of the compact form was found to be happening in situations where order dependency was important.
 The fix meant that long form is used much more than it used to be.

Note that we indeed see an array of throttles, NOT null.

Furthermore, I verified that what went into the connection came back out in the same form.
 So I think that the source of this problem was the user exporting via the API on an older
version of ManifoldCF that did not have the fix.


> export of web connection bandwidth throttling
> ---------------------------------------------
>
>                 Key: CONNECTORS-1567
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1567
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Web connector
>    Affects Versions: ManifoldCF 2.11, ManifoldCF 2.12
>            Reporter: Tim Steenbeke
>            Assignee: Karl Wright
>            Priority: Major
>             Fix For: ManifoldCF 2.13
>
>
> When exporting the web connector using the API, it doesn't export the bandwidth throttling.
>  Than when importing this connector to a clean manifoldcf it creates the connector with
basic bandwidth.
>  When using the connector in a job it works properly.
> The issue here is that the connector isn't created with correct bandwidth throttling.
>  And the connector gives issues in the UI when trying to view or edit.
> (related to issue: [CONNECTORS-1568|https://issues.apache.org/jira/projects/CONNECTORS/issues/CONNECTORS-1568])
> e.g.:
> {code:java}
> {
>   "name": "test_web",
>   "configuration": null,
>     "_PARAMETER_": [
>       {
>         "_attribute_name": "Email address",
>         "_value_": "tim.steenbeke@formica.digital"
>       },
>       {
>         "_attribute_name": "Robots usage",
>         "_value_": "all"
>       },
>       {
>         "_attribute_name": "Meta robots tags usage",
>         "_value_": "all"
>       },
>       {
>         "_attribute_name": "Proxy host",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy port",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication domain",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication user name",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication password",
>         "_value_": ""
>       }
>     ]
>   },
>   "description": "Website repository standard settup",
>   "throttle": null,
>   "max_connections": 10,
>   "class_name": "org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector",
>   "acl_authority": null
> }{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message