Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BB82ED3A3 for ; Fri, 23 Nov 2012 08:46:37 +0000 (UTC) Received: (qmail 86164 invoked by uid 500); 23 Nov 2012 08:46:32 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 86073 invoked by uid 500); 23 Nov 2012 08:46:32 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 86055 invoked by uid 99); 23 Nov 2012 08:46:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Nov 2012 08:46:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_FONT_SIZE_LARGE,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dwivedishashwat@gmail.com designates 209.85.214.176 as permitted sender) Received: from [209.85.214.176] (HELO mail-ob0-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Nov 2012 08:46:24 +0000 Received: by mail-ob0-f176.google.com with SMTP id un3so10087250obb.35 for ; Fri, 23 Nov 2012 00:46:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=AHo/c/Z1SRZTv3yByrIYxyS+3dflXPrG0Y96UoAezZ4=; b=B04KQBXpaRHtogkbQkWnwryos6ugBbi5prgpS0Ci+4jvy2XH/fj5qPQy8aPOO1v08H 8iwsjwrls6ZmzyLS9bGYOIsX2tW+VjtMm7+9EfUm7waSbZR7+DVfslOEwT9P1MafujtQ snl1igjKNKsl5SJTqxoLFHa/kMsF9O3jlCIzM1vBacjsqEn26HMKGXdI4mzUVOEUlz04 Jl6ZHpY5YzQRCI2xUZ+FWQ+a/0C5BSPwqO+eAi8daoggyS/YBVd163lyuFEDfD04Fdjk F4nuxPGvW7EoWx2kjP988DT/XIYsxsticbukfkkzcEc3g41VOvDT/1HdqRaLAhuW1N3P 5Uhg== Received: by 10.60.32.193 with SMTP id l1mr2262344oei.114.1353660363574; Fri, 23 Nov 2012 00:46:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.76.112.43 with HTTP; Fri, 23 Nov 2012 00:45:43 -0800 (PST) In-Reply-To: References: From: shashwat shriparv Date: Fri, 23 Nov 2012 14:15:43 +0530 Message-ID: Subject: Re: Log files occupy lot of Disk size To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=e89a8fb1f6ca1e0c4a04cf259d87 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8fb1f6ca1e0c4a04cf259d87 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable When you run a hive query it internally runs lot of map reduce tasks, which intern generates lot of temporary files, so your disk uses grows, so can you tell which folder is taking most of the spaces? =E2=88=9E Shashwat Shriparv On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq wrote: > Harsh has got a point. I was thinking the same, but then I thought maybe > you need all these log files. If not then do as Harsh has suggested. And > deleting log files won't affect your Hdfs working, but it will not write > logs for any operation until the next Hdfs restart. > > Regards, > Mohammad Tariq > > > > On Fri, Nov 23, 2012 at 1:12 PM, Harsh J wrote: > >> Lower your log levels if you do not need all that verbosity. You can >> control log retention, max sizes to keep, max number of files to keep, >> and logging levels, etc. via each components' log4j.properties file. >> >> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games >> wrote: >> > If i delete the log file without stopping the cluster won't it >> > terminate the session. >> > >> > >> > >> > On 11/23/12, Mohammad Tariq wrote: >> >> Hi there, >> >> >> >> You can write a small job or some script which periodically check= s >> for >> >> the log growth and performs the delete after certain threshold. >> >> >> >> Regards, >> >> Mohammad Tariq >> >> >> >> >> >> >> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games < >> >> funnlearnforkids@gmail.com> wrote: >> >> >> >>> Hi, >> >>> >> >>> Everytime i query hbase or hive ,there is a significant growth in my >> >>> log files and it consumes lot of space from my hard disk....(Approx = 40 >> >>> gb) >> >>> So i stop the cluster ,delete all the logs and free the space and th= en >> >>> again start the cluster to start my work. >> >>> >> >>> Is there any other solution coz i cannot restart the cluster everyda= y. >> >>> >> >> >> >> >> >> -- >> Harsh J >> > > --e89a8fb1f6ca1e0c4a04cf259d87 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable When you run a hive query it internally runs lot of map reduce tasks, which= intern generates lot of temporary files, so your disk uses grows, so can y= ou tell which folder is taking most of the spaces?

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=20 =09 =09 =09 =09

=E2=88=9E

Shashwat Shriparv




On Fri, Nov 23, 2012 at 1:24 PM, Mohamma= d Tariq <dontariq@gmail.com> wrote:
Harsh has got a point. I was thinking the same, but then I thought maybe yo= u need all these log files. If not then do as Harsh has suggested. And dele= ting log files won't affect your Hdfs working, but it will not write lo= gs for any operation until the next Hdfs restart.

Regards,
=C2=A0=C2=A0 =C2=A0Mohammad Tariq
=



On Fri, Nov 23, 2012 at 1:12 PM, Harsh J= <harsh@cloudera.com> wrote:
Lower your log levels if you do not need all that verbosity. You can
control log retention, max sizes to keep, max number of files to keep,
and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
<funnlearnforkids@gmail.com> wrote:
> If i delete the log file without stopping the cluster =C2=A0won't = it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <dontariq@gmail.com> wrote:
>> Hi there,
>>
>> =C2=A0 =C2=A0 You can write a small job or some script which perio= dically checks for
>> the log growth and performs the delete after certain threshold. >>
>> Regards,
>> =C2=A0 =C2=A0 Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> fu= nnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth= in my
>>> log files and it consumes lot of space from my hard disk....(A= pprox 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space = and then
>>> again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster e= veryday.
>>>
>>



--
Harsh J


--e89a8fb1f6ca1e0c4a04cf259d87--