kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-1754) KOYA - Kafka on YARN
Date Thu, 06 Nov 2014 07:02:34 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199918#comment-14199918

Arun C Murthy commented on KAFKA-1754:

bq. My main concern is whether its possible to prevent YARN from spawning new container when
a broker goes down.

[~gwenshap] - YARN doesn't spawn a new container on its own... the Kafka-AM would get a notification
that a particular broker (YARN container) has failed. The Kafka-AM can then decide when/where
to request & launch a new broker.

> KOYA - Kafka on YARN
> --------------------
>                 Key: KAFKA-1754
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1754
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Thomas Weise
>         Attachments: DT-KOYA-Proposal- JIRA.pdf
> YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads, emerging
as distributed operating system for big data applications. Initiatives are on the way to bring
long running services under the YARN umbrella, leveraging it for centralized resource management
and operations ([YARN-896] and examples such as HBase, Accumulo or Memcached through Slider).
This JIRA is to propose KOYA (Kafka On Yarn), a YARN application master to launch and manage
Kafka clusters running on YARN. Brokers will use resources allocated through YARN with support
for recovery, monitoring etc. Please see attached for more details.

This message was sent by Atlassian JIRA

View raw message