Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of neha.narkhede@gmail.com
 designates 209.85.213.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <611B2319-62A3-4512-A045-6CBF39B2371E@gmail.com>
References: <133DF7E2-A2EA-4975-81D7-2199968849CA@gmail.com>
	<CAOG_4Qb0ALmjNgXR843c-SL=mPFtZ_hCRkXnJo_G7vN+VCQQ2w@mail.gmail.com>
	<611B2319-62A3-4512-A045-6CBF39B2371E@gmail.com>
Date: Wed, 1 Aug 2012 09:07:48 -0700
Message-ID: 
 <CAOG_4QYoFhqt8XiVagWgk4X9hqSvaZuYM2VboShXU2LUmxVZdQ@mail.gmail.com>
Subject: Re: Log4jAppender backoff if server is down
From: Neha Narkhede <neha.narkhede@gmail.com>
To: kafka-dev@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

David,

>From the logs, seems like the box you ran a Kafka server on, ran out
of disk space. That's why the Kafka server starts up but shuts itself
down immediately. Please can you try starting Kafka on a box with
enough disk space ?

Thanks,
Neha

On Tue, Jul 31, 2012 at 5:08 PM, David Arthur <mumrah@gmail.com> wrote:
> Here are some log snippets:
>
> Kafka server logs: https://gist.github.com/c440ada8daa629e337e2
> Solr logs: https://gist.github.com/42624c901fc7967fd137
>
> In this case, I am sending all the "org.apache.solr" logs to Kafka, so ea=
ch document update in Solr produces a log message. Each update to Solr prod=
uced an exception like this which caused things to slow way down.
>
> My earlier statement about Kafka being up but unable to write logs was in=
correct. During this, it seems Kafka was simply down (our supervisor that r=
estarts Kafka gave up after a few tries). So in the case that Kafka is down=
, what should the client behavior be like?
>
> Ideally, to me, the client could know about what brokers are available th=
rough ZK watches and just refuse to attempt a send/produce if no one is ava=
ilable.
>
> What do you guys think? I'm not saying this is necessarily a Kafka issue,=
 I'm just not sure what's the best thing to do here.
>
> Cheers
> -David
>
> On Jul 31, 2012, at 5:48 PM, Neha Narkhede wrote:
>
>> David,
>>
>> Would you mind sending around the error stack traces ? That will help
>> determine the right fix.
>>
>> Thanks,
>> Neha
>>
>> On Tue, Jul 31, 2012 at 1:09 PM, David Arthur <mumrah@gmail.com> wrote:
>>> Greetings all,
>>>
>>> I'm using the KafkaLog4jAppender with Solr and ran into an interesting =
issue recently. The disk filled up on my Kafka broker (just a single broker=
, this is a dev environment) and Solr slowed down to a near halt. My best e=
stimation is that each log4j log message created was incurring quite a bit =
of overhead dealing with exceptions coming back from the Kafka broker.
>>>
>>> So I'm wondering, would it make sense to implement some back off strate=
gy for this client if it starts getting exceptions from the server? Alterna=
tively, could maybe the Kafka broker mark it self as "down" in ZooKeeper if=
 it gets into certain situations (like disk full). I guess this really coul=
d apply to any client, not just the log4j appender.
>>>
>>> Thanks!
>>> -David
>