Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Date: Sat, 21 Jan 2017 20:04:26 +0000 (UTC)
From: "postmaster@inn.ru (JIRA)" <jira@apache.org>
To: dev@kafka.apache.org
Message-ID: <JIRA.12689059.1389811935000.70819.1485029066951@Atlassian.JIRA>
In-Reply-To: <JIRA.12689059.1389811935000@Atlassian.JIRA>
References: <JIRA.12689059.1389811935000@Atlassian.JIRA> <JIRA.12689059.1389811935466@jira-lw-us.apache.org>
Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 21 Jan 2017 20:04:42 -0000


    [ https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833128#comment-15833128 ] 

postmaster@inn.ru commented on KAFKA-1207:
------------------------------------------

Delivery is delayed to these recipients or groups:

ed@inn.ru<mailto:ed@inn.ru>

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 hours and 51 minutes. You'll be notified if the message can't be delivered by that time.


Diagnostic information for administrators:

Generating server: lc-exch-04.inn.local
Receiving server: inn.ru (109.105.153.25)

ed@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/21/2017 7:52:39 PM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 Error communicating with target host: "Failed to connect. Winsock error code: 10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-04.inn.local
 (10.64.37.99) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sat, 21
 Jan 2017 18:54:32 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sat, 21 Jan 2017 18:54:32 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; helo=spamd2-us-west.apache.org; envelope-from=jira@apache.org; receiver=ed@inn.ru
X-Envelope-From: <jira@apache.org>
Received: from spamd2-us-west.apache.org (pnap-us-west-generic-nat.apache.org [209.188.14.142])
        by lc-asp-02.inn.ru (Postfix) with ESMTP id E9A71400C3
        for <ed@inn.ru>; Sat, 21 Jan 2017 16:54:30 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
        by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 00E991A04DE
        for <ed@inn.ru>; Sat, 21 Jan 2017 15:54:31 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
        tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
        autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
        by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024)
        with ESMTP id INFelhK5MSM1 for <ed@inn.ru>;
        Sat, 21 Jan 2017 15:54:29 +0000 (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139])
        by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 0B7005FC4D
        for <ed@inn.ru>; Sat, 21 Jan 2017 15:54:29 +0000 (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
        by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8A93FE0285
        for <ed@inn.ru>; Sat, 21 Jan 2017 15:54:27 +0000 (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
        by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9AEDC2528D
        for <ed@inn.ru>; Sat, 21 Jan 2017 15:54:26 +0000 (UTC)
Date: Sat, 21 Jan 2017 15:54:26 +0000
From: "postmaster@inn.ru (JIRA)" <jira@apache.org>
To: <ed@inn.ru>
Message-ID: <JIRA.12689059.1389811935000.70245.1485014066632@Atlassian.JIRA>
In-Reply-To: <JIRA.12689059.1389811935000@Atlassian.JIRA>
References: <JIRA.12689059.1389811935000@Atlassian.JIRA> <JIRA.12689059.1389811935466@jira-lw-us.apache.org>
Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: E9A71400C3.A541C
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: jira@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485618871.32963@ssDzvWvOZRNAi0++bLGNEQ
Return-Path: jira@apache.org
X-OrganizationHeadersPreserved: lc-exch-04.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-04.inn.local


> Launch Kafka from within Apache Mesos
> -------------------------------------
>
>                 Key: KAFKA-1207
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1207
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Joe Stein
>              Labels: mesos
>         Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to this.
> 1) The Framework:  This is going to be responsible for starting up and managing the fail over of brokers within the mesos cluster.  This will have to get some Kafka focused paramaters for launching new replica brokers, moving topics and partitions around based on what is happening in the grid through time.
> 2) The Scheduler: This is what is going to ask for resources for Kafka brokers (new ones, replacement ones, commissioned ones) and other operations such as stopping tasks (decommissioning brokers).  I think this should also expose a user interface (or at least a rest api) for producers and consumers so we can have producers and consumers run inside of the mesos cluster if folks want (just add the jar)
> 3) The Executor : This is the task launcher.  It launches tasks kills them off.
> 4) Sharing data between Scheduler and Executor: I looked at the a few implementations of this.  I like parts of the Storm implementation but think using the environment variable ExectorInfo.CommandInfo.Enviornment.Variables[] is the best shot.  We can have a command line bin/kafka-mesos-scheduler-start.sh that would build the contrib project if not already built and support conf/server.properties to start.
> The Framework and operating Scheduler would run in on an administrative node.  I am probably going to hook Apache Curator into it so it can do it's own failure to a another follower.  Running more than 2 should be sufficient as long as it can bring back it's state (e.g. from zk).  I think we can add this in after once everything is working.
> Additional detail can be found on the Wiki page https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=38570672


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)