kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4388) Connect key and value converters are listed without default values
Date Sun, 25 Jun 2017 01:08:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062178#comment-16062178
] 

Ewen Cheslack-Postava commented on KAFKA-4388:
----------------------------------------------

What I meant was that the following:

{code}
$ curl -X PUT -H "Content-Type: application/json" --data '{"name": "foo", "connector.class":
"FileStreamSourceConnector"}' http://localhost:8083/connector-plugins/FileStreamSourceConnector/config/validate
| jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4021  100  3958  100    63   318k   5184 --:--:-- --:--:-- --:--:--  322k
{
  "name": "FileStreamSourceConnector",
  "error_count": 1,
  "groups": [
    "Common",
    "Transforms",
    "Common",
    "Transforms"
  ],
  "configs": [
    {
      "definition": {
        "name": "topic",
        "type": "STRING",
        "required": true,
        "default_value": null,
        "importance": "HIGH",
        "documentation": "The topic to publish data to",
        "group": null,
        "width": "NONE",
        "display_name": "topic",
        "dependents": [],
        "order": -1
      },
      "value": {
        "name": "topic",
        "value": null,
        "recommended_values": [],
        "errors": [
          "Missing required configuration \"topic\" which has no default value."
        ],
        "visible": true
      }
    },
    {
      "definition": {
        "name": "file",
        "type": "STRING",
        "required": false,
        "default_value": null,
        "importance": "HIGH",
        "documentation": "Source filename. If not specified, the standard input will be used",
        "group": null,
        "width": "NONE",
        "display_name": "file",
        "dependents": [],
        "order": -1
      },
      "value": {
        "name": "file",
        "value": null,
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    },
    {
      "definition": {
        "name": "value.converter",
        "type": "CLASS",
        "required": false,
        "default_value": null,
        "importance": "LOW",
        "documentation": "Converter class used to convert between Kafka Connect format and
the serialized form that is written to Kafka. This controls the format of the values in messages
written to or read from Kafka, and since this is independent of connectors it allows any connector
to work with any serialization format. Examples of common formats include JSON and Avro.",
        "group": "Common",
        "width": "SHORT",
        "display_name": "Value converter class",
        "dependents": [],
        "order": 5
      },
      "value": {
        "name": "value.converter",
        "value": null,
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    },
    {
      "definition": {
        "name": "name",
        "type": "STRING",
        "required": true,
        "default_value": null,
        "importance": "HIGH",
        "documentation": "Globally unique name to use for this connector.",
        "group": "Common",
        "width": "MEDIUM",
        "display_name": "Connector name",
        "dependents": [],
        "order": 1
      },
      "value": {
        "name": "name",
        "value": "foo",
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    },
    {
      "definition": {
        "name": "tasks.max",
        "type": "INT",
        "required": false,
        "default_value": "1",
        "importance": "HIGH",
        "documentation": "Maximum number of tasks to use for this connector.",
        "group": "Common",
        "width": "SHORT",
        "display_name": "Tasks max",
        "dependents": [],
        "order": 3
      },
      "value": {
        "name": "tasks.max",
        "value": "1",
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    },
    {
      "definition": {
        "name": "connector.class",
        "type": "STRING",
        "required": true,
        "default_value": null,
        "importance": "HIGH",
        "documentation": "Name or alias of the class for this connector. Must be a subclass
of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector,
you can either specify this full name,  or use \"FileStreamSink\" or \"FileStreamSinkConnector\"
to make the configuration a bit shorter",
        "group": "Common",
        "width": "LONG",
        "display_name": "Connector class",
        "dependents": [],
        "order": 2
      },
      "value": {
        "name": "connector.class",
        "value": "FileStreamSourceConnector",
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    },
    {
      "definition": {
        "name": "key.converter",
        "type": "CLASS",
        "required": false,
        "default_value": null,
        "importance": "LOW",
        "documentation": "Converter class used to convert between Kafka Connect format and
the serialized form that is written to Kafka. This controls the format of the keys in messages
written to or read from Kafka, and since this is independent of connectors it allows any connector
to work with any serialization format. Examples of common formats include JSON and Avro.",
        "group": "Common",
        "width": "SHORT",
        "display_name": "Key converter class",
        "dependents": [],
        "order": 4
      },
      "value": {
        "name": "key.converter",
        "value": null,
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    },
    {
      "definition": {
        "name": "transforms",
        "type": "LIST",
        "required": false,
        "default_value": null,
        "importance": "LOW",
        "documentation": "Aliases for the transformations to be applied to records.",
        "group": "Transforms",
        "width": "LONG",
        "display_name": "Transforms",
        "dependents": [],
        "order": 6
      },
      "value": {
        "name": "transforms",
        "value": null,
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    }
  ]
}
{code}

Did not used to provide output for key.converter and value.converter. That has since been
fixed incidentally as the validation code has been updated. I think the one thing remaining
is that we aren't currently providing any recommended values -- those should probably be suggested
based on the new Plugins class that was added as part of the new classloader/plugin support.
That'd probably be a pretty simple change to make except that it will require passing an instance
of the Plugins class through whereas previously getting the ConfigDef for a ConnectorConfig
did not require that.

> Connect key and value converters are listed without default values
> ------------------------------------------------------------------
>
>                 Key: KAFKA-4388
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4388
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 0.10.1.0
>            Reporter: Ewen Cheslack-Postava
>            Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> KIP-75 added per connector converters. This exposes the settings on a per-connector basis
via the validation API. However, the way this is specified for each connector is via a config
value with no default value. This means the validation API implies there is no setting unless
you provide one.
> It would be much better to include the default value extracted from the WorkerConfig
instead so it's clear you shouldn't need to override the default.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message