manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CONNECTORS-1567) export of web connection bandwidth throttling
Date Wed, 09 Jan 2019 08:02:00 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737950#comment-16737950
] 

Karl Wright commented on CONNECTORS-1567:
-----------------------------------------

The best way to construct an API request for any connection or job is to create it in the
UI and then export it.  The documentation is correct but it is hard to pick through all the
details, and the UI is easier.  So that is what I would do if I were trying to verify everything
worked.

Unfortunately, because ManifoldCF was forced to remove a JSON jar we depended on due to a
legal ruling by the Board, we had to retrofit a different (and not as good) JSON jar in place
a few years back, and that had all sorts of downstream effects on the API JSON format.  We
did not need to change the specification, but we did need to change how we output certain
constructs to JSON to not use the syntactic sugar we earlier could use.  I fixed a bug in
this area in either MCF 2.10 or 2.11, so anything output before that time might not have reimported
faithfully.  Hope that helps with the explanation.


> export of web connection bandwidth throttling
> ---------------------------------------------
>
>                 Key: CONNECTORS-1567
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1567
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Web connector
>    Affects Versions: ManifoldCF 2.11, ManifoldCF 2.12
>            Reporter: Tim Steenbeke
>            Assignee: Karl Wright
>            Priority: Major
>             Fix For: ManifoldCF 2.13
>
>         Attachments: bandwidth.png, bandwidth_test_abc.png
>
>
> When exporting the web connector using the API, it doesn't export the bandwidth throttling.
>  Than when importing this connector to a clean manifoldcf it creates the connector with
basic bandwidth.
>  When using the connector in a job it works properly.
> The issue here is that the connector isn't created with correct bandwidth throttling.
>  And the connector gives issues in the UI when trying to view or edit.
> (related to issue: [CONNECTORS-1568|https://issues.apache.org/jira/projects/CONNECTORS/issues/CONNECTORS-1568])
> e.g.:
> {code:java}
> {
>   "name": "test_web",
>   "configuration": null,
>     "_PARAMETER_": [
>       {
>         "_attribute_name": "Email address",
>         "_value_": "tim.steenbeke@formica.digital"
>       },
>       {
>         "_attribute_name": "Robots usage",
>         "_value_": "all"
>       },
>       {
>         "_attribute_name": "Meta robots tags usage",
>         "_value_": "all"
>       },
>       {
>         "_attribute_name": "Proxy host",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy port",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication domain",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication user name",
>         "_value_": ""
>       },
>       {
>         "_attribute_name": "Proxy authentication password",
>         "_value_": ""
>       }
>     ]
>   },
>   "description": "Website repository standard settup",
>   "throttle": null,
>   "max_connections": 10,
>   "class_name": "org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector",
>   "acl_authority": null
> }{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message