Return-Path: X-Original-To: apmail-kafka-dev-archive@www.apache.org Delivered-To: apmail-kafka-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9954210D8B for ; Mon, 3 Nov 2014 20:09:34 +0000 (UTC) Received: (qmail 58876 invoked by uid 500); 3 Nov 2014 20:09:34 -0000 Delivered-To: apmail-kafka-dev-archive@kafka.apache.org Received: (qmail 58822 invoked by uid 500); 3 Nov 2014 20:09:34 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 58806 invoked by uid 99); 3 Nov 2014 20:09:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2014 20:09:34 +0000 Date: Mon, 3 Nov 2014 20:09:34 +0000 (UTC) From: "Ewen Cheslack-Postava (JIRA)" To: dev@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (KAFKA-1748) Decouple system test cluster resources definition from service definitions MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KAFKA-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195013#comment-14195013 ] Ewen Cheslack-Postava commented on KAFKA-1748: ---------------------------------------------- The patches I submitted make the suggested changes, including some support for pulling dynamic configurations (from Vagrant). For the local default config, this shouldn't have any effect, but will require config changes if you're overriding those settings -- the cluster resources go in cluster.json and just contain the hostnames, optional usernames, ssh args, java home and kafka home. In the Vagrant case, I defaulted KAFKA_HOME to /opt/kafka to match the setup in KAFKA-1173 since there's no place for the user to override that setting. > Decouple system test cluster resources definition from service definitions > -------------------------------------------------------------------------- > > Key: KAFKA-1748 > URL: https://issues.apache.org/jira/browse/KAFKA-1748 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.8.1.1 > Reporter: Ewen Cheslack-Postava > Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1748.patch, KAFKA-1748_2014-11-03_12:04:18.patch > > > Currently the system tests use JSON files that specify the set of services for each test and where they should run (i.e. hostname). These currently assume that you already have SSH keys setup, use the same username on the host running the tests and the test cluster, don't require any additional ssh/scp/rsync flags, and assume you'll always have a fixed set of compute resources (or that you'll spend a lot of time editing config files). > While we don't want a whole cluster resource manager in the system tests, a bit more flexibility would make it easier to, e.g., run tests against a local vagrant cluster or on dynamically allocated EC2 instances. We can separate out the basic resource spec (i.e. json specifying how to access machines) from the service definition (i.e. a broker should run with settings x, y, z). Restricting to a very simple set of mappings (i.e. map services to hosts with round robin, optionally restricting to no reuse of hosts) should keep things simple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)