kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phillip Mann <pm...@trulia.com>
Subject Creating a connector with Kafka Connect Distributed returning 500 error
Date Tue, 06 Dec 2016 19:22:24 GMT
I am working on migrating from Camus to Kafka Connect. I am working on the implementation of
Kafka Connect and specifically focused on distributed mode. I am able to start a worker successfully
on my local machine which I assume communicates with my Kafka cluster. I am further able to
run two GET commands such as / and /connector-plugins which return the correct JSON. However,
when I try to POST a command to create a connector, I receive a 500 error and a time out.
Specifically, I use this command to POST for testing:

curl -X POST -H "Content-Type: application/json" --data '{"name": "local-file-sink", "config":
{"connector.class":"FileStreamSinkConnector", "tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic"
}}' localhost:8083/connectors

and eventually I get this response:

{"error_code": 500, "message": "Request timed out"}

I am lost as to what is going on. The logs from my Kafka Connect distributed worker show this:

[2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32 +0000] "GET /connector-plugins
HTTP/1.1" 200 315  2 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25 +0000] "GET /connector-plugins
HTTP/1.1" 200 315  3 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28 +0000] "GET /connector-plugins
HTTP/1.1" 200 315  2 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08 +0000] "GET /connectors
HTTP/1.1" 500 48  90003 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44 +0000] "GET /connector-plugins
HTTP/1.1" 200 315  1 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36 +0000] "POST /connectors
HTTP/1.1" 500 48  90003 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08 +0000] "POST /connectors
HTTP/1.1" 500 48  90005 (org.apache.kafka.connect.runtime.rest.RestServer:60)

where you can see the error codes and the commands.

I guess my main questions and issues are:

  1.  How can I better debug Kafka Connect so I can try and fix this?
  2.  Is there anything that I'm doing that is glaringly wrong?
  3.  Is there any step-by-step documentation or blog posts on getting a Kafka Connect distributed
worker and connector to run? I have not really seen anything or even best practices kinds
of documentation? Maybe I am just way too early of an adopter.

I look forward to hearing back from the community and thank you for your help!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message