Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B9573DF6A for ; Sun, 26 Aug 2012 18:40:09 +0000 (UTC) Received: (qmail 70095 invoked by uid 500); 26 Aug 2012 18:40:04 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 70025 invoked by uid 500); 26 Aug 2012 18:40:04 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 70017 invoked by uid 99); 26 Aug 2012 18:40:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Aug 2012 18:40:04 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Aug 2012 18:39:59 +0000 Received: by iagt4 with SMTP id t4so7739907iag.35 for ; Sun, 26 Aug 2012 11:39:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:content-type:x-gm-message-state; bh=C5br275aER68Q1lEmuDdr2/sM+hep1FIzslriDLOuQ4=; b=Nt5rWXcUJMHXoxnanVz4m0wbl7gIW7bYNG9sYLTUIlDcWG9FQp45IKqeFRT7VwNdrl j5GgCjGuGf+SPCnX74r5DkUbwV+xVy7txWnl3OKDPB1V5mN8a5BaQU8qwYoRhS4SS5NK gebst1SA1oD7v81zKZGTtWdKDQkJT8LesPDlhuzD9Tl5FGawobzzXPo6DgoUszZO1UVt M66Vd00dtf1J+Zkar8PbH+DEsOOiQ6D1X0ti5WsRoKEAsR/1oBjWj58OZ/e7D6IkKjl9 pxxuAWeXeLfRNCaaHkMNATqBV6WVRLeKVQqTwn6bOsGemCfzeNfE33Reawhr2ufAmOdr +NDw== MIME-Version: 1.0 Received: by 10.42.180.201 with SMTP id bv9mr9124570icb.43.1346006378428; Sun, 26 Aug 2012 11:39:38 -0700 (PDT) Received: by 10.231.7.3 with HTTP; Sun, 26 Aug 2012 11:39:38 -0700 (PDT) X-Originating-IP: [207.237.178.191] In-Reply-To: References: Date: Sun, 26 Aug 2012 14:39:38 -0400 Message-ID: Subject: Re: Is there a way to turn off MAPREDUCE-2415? From: Koert Kuipers To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=90e6ba6e8d680d3a7004c82f88cf X-Gm-Message-State: ALoCoQlaEUMxbG7IzNWFxPPtearn8vCSfqqAu9t0meI1K/7gtEr9ccgejw3l6GSsYzAMvmOGkzrj X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba6e8d680d3a7004c82f88cf Content-Type: text/plain; charset=ISO-8859-1 Harsh, I see the problem as follows: Usually we want to have people log what they want, as long as they don't threaten the stability of the system. However every once in a while somebody will submit a job that is overly verbose and will generate many gigabytes of logs in minutes. This is typically a honest mistake, and the person doesn't realize what is going on (why is my job so slow?). Limiting the general logging levels for everyone to deal with these mistakes seems ineffective. Telling the person to change the logging level for his job will not work either since he/she doesn't realize what is going on and certainly didn't know in advance. So all i really want is a very high and hard limit on the log size per job, to protect the system. Say many hundreds of megabytes or even gigabytes. But when this limit is reached i want to logging to stop from that point on, or even the job to be killed. mapred.userlog.limit.kb seems the wrong tool for the job. Before the logging got moved to the mapred.local.dir i had a limit simply by limiting the size of the partition that logging went to. Anyhow, looks like i will have to wait for MAPRED-1100 Have a good day! Koert On Sun, Aug 26, 2012 at 2:21 PM, Harsh J wrote: > Yes that is true, it does maintain N events in memory and then flushes > them down to disk upon closure. With a reasonable size (2 MB of logs > say) I don't see that causing any memory fill-up issues at all, since > it does cap (and discard at tail). > > The other alternative may be to switch down the log level on the task, > via mapred.map.child.log.level and/or mapred.reduce.child.log.level > set to WARN or ERROR. > > On Sun, Aug 26, 2012 at 11:37 PM, Koert Kuipers wrote: > > Looks like mapred.userlog.limit.kb is implemented by keeping some list in > > memory, and the logs are not writting to disk until the job finishes or > is > > killed. That doesn't sound acceptable to me. > > > > Well i am not the only one with this problem. See MAPREDUCE-1100 > > > > > > On Sun, Aug 26, 2012 at 1:58 PM, Harsh J wrote: > >> > >> Hi Koert, > >> > >> On Sun, Aug 26, 2012 at 11:20 PM, Koert Kuipers > wrote: > >> > Hey Harsh, > >> > Thanks for responding! > >> > Would limiting the logging for each task via mapred.userlog.limit.kb > be > >> > strictly enforced (while the job is running)? That would solve my > issue > >> > of > >> > runaway logging on a job filling up the datanode disks. I would set > the > >> > limit high since in general i do want to retain logs, just not in > case a > >> > single rogue job starts producing many gigabytes of logs. > >> > Thanks! > >> > >> It is not strictly enforced such as counter limits are. Exceeding it > >> wouldn't fail the task, only cause the extra logged events to not > >> appear at all (thereby limiting the size). > >> > >> > On Sun, Aug 26, 2012 at 1:44 PM, Harsh J wrote: > >> >> > >> >> Hi Koert, > >> >> > >> >> To answer on point, there is no turning off this feature. > >> >> > >> >> Since you don't seem to care much for logs from tasks persisting, > >> >> perhaps consider lowering the mapred.userlog.retain.hours to a lower > >> >> value than 24 hours (such as 1h)? Or you may even limit the logging > >> >> from each task to a certain amount of KB via mapred.userlog.limit.kb, > >> >> which is unlimited by default. > >> >> > >> >> Would either of these work for you? > >> >> > >> >> On Sun, Aug 26, 2012 at 11:02 PM, Koert Kuipers > >> >> wrote: > >> >> > We have smaller nodes (4 to 6 disks), and we used to write logs to > >> >> > the > >> >> > same > >> >> > disk as where the OS is. So if that disks goes then i don't really > >> >> > care > >> >> > about tasktrackers failing. Also, the fact that logs were written > to > >> >> > a > >> >> > single partition meant that i could make sure they would not grow > too > >> >> > large > >> >> > in case someone had too verbose logging on a large job. With > >> >> > MAPREDUCE-2415 > >> >> > a job that does massive amount of logging can fill up all the > >> >> > mapred.local.dir, which in our case are on the same partition as > the > >> >> > hdfs > >> >> > data dirs, so now faulty logging can fill up hdfs storage, which i > >> >> > really > >> >> > don't like. Any ideas? > >> >> > > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Harsh J > >> > > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J > --90e6ba6e8d680d3a7004c82f88cf Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Harsh,

I see the problem as fol= lows: Usually we want to have people log what they want, as long as they do= n't threaten the stability of the system.

However every once in= a while somebody will submit a job that is overly verbose and will generat= e many gigabytes of logs in minutes. This is typically a honest mistake, an= d the person doesn't realize what is going on (why is my job so slow?).= Limiting the general logging levels for everyone to deal with these mistak= es seems ineffective. Telling the person to change the logging level for hi= s job will not work either since he/she doesn't realize what is going o= n and certainly didn't know in advance.

So all i really want is a very high and hard limit on the log size per = job, to protect the system. Say many hundreds of megabytes or even gigabyte= s. But when this limit is reached i want to logging to stop from that point= on, or even the job to be killed. mapred.userlog.limit.kb seems the wrong = tool for the job.

Before the logging got moved to the mapred.local.dir i had a limit simp= ly by limiting the size of the partition that logging went to.

Anyho= w, looks like i will have to wait for MAPRED-1100

Have a good day! K= oert

On Sun, Aug 26, 2012 at 2:21 PM, Hars= h J <harsh@cloudera.com> wrote:
Yes that is true, it does maintain N events in memory and then flushes
them down to disk upon closure. With a reasonable size (2 MB of logs
say) I don't see that causing any memory fill-up issues at all, since it does cap (and discard at tail).

The other alternative may be to switch down the log level on the task,
via mapred.map.child.log.level and/or mapred.reduce.child.log.level
set to WARN or ERROR.

On Sun, Aug 26, 2012 at 11:37 PM, Koert Kuipers <koert@tresata.com> wrote:
> Looks like mapred.userlog.limit.kb is implemented by keeping some list= in
> memory, and the logs are not writting to disk until the job finishes o= r is
> killed. That doesn't sound acceptable to me.
>
> Well i am not the only one with this problem. See MAPREDUCE-1100
>
>
> On Sun, Aug 26, 2012 at 1:58 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Hi Koert,
>>
>> On Sun, Aug 26, 2012 at 11:20 PM, Koert Kuipers <koert@tresata.com> wrote:
>> > Hey Harsh,
>> > Thanks for responding!
>> > Would limiting the logging for each task via mapred.userlog.l= imit.kb be
>> > strictly enforced (while the job is running)? That would solv= e my issue
>> > of
>> > runaway logging on a job filling up the datanode disks. I wou= ld set the
>> > limit high since in general i do want to retain logs, just no= t in case a
>> > single rogue job starts producing many gigabytes of logs.
>> > Thanks!
>>
>> It is not strictly enforced such as counter limits are. Exceeding = it
>> wouldn't fail the task, only cause the extra logged events to = not
>> appear at all (thereby limiting the size).
>>
>> > On Sun, Aug 26, 2012 at 1:44 PM, Harsh J <harsh@cloudera.com> wrote:
>> >>
>> >> Hi Koert,
>> >>
>> >> To answer on point, there is no turning off this feature.=
>> >>
>> >> Since you don't seem to care much for logs from tasks= persisting,
>> >> perhaps consider lowering the mapred.userlog.retain.hours= to a lower
>> >> value than 24 hours (such as 1h)? Or you may even limit t= he logging
>> >> from each task to a certain amount of KB via mapred.userl= og.limit.kb,
>> >> which is unlimited by default.
>> >>
>> >> Would either of these work for you?
>> >>
>> >> On Sun, Aug 26, 2012 at 11:02 PM, Koert Kuipers <koert@tresata.com>
>> >> wrote:
>> >> > We have smaller nodes (4 to 6 disks), and we used to= write logs to
>> >> > the
>> >> > same
>> >> > disk as where the OS is. So if that disks goes then = i don't really
>> >> > care
>> >> > about tasktrackers failing. Also, the fact that logs= were written to
>> >> > a
>> >> > single partition meant that i could make sure they w= ould not grow too
>> >> > large
>> >> > in case someone had too verbose logging on a large j= ob. With
>> >> > MAPREDUCE-2415
>> >> > a job that does massive amount of logging can fill u= p all the
>> >> > mapred.local.dir, which in our case are on the same = partition as the
>> >> > hdfs
>> >> > data dirs, so now faulty logging can fill up hdfs st= orage, which i
>> >> > really
>> >> > don't like. Any ideas?
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



--
Harsh J

--90e6ba6e8d680d3a7004c82f88cf--