hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shashwat shriparv <dwivedishash...@gmail.com>
Subject Re: Log files occupy lot of Disk size
Date Fri, 23 Nov 2012 08:45:43 GMT
When you run a hive query it internally runs lot of map reduce tasks, which
intern generates lot of temporary files, so your disk uses grows, so can
you tell which folder is taking most of the spaces?



∞
Shashwat Shriparv




On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> Harsh has got a point. I was thinking the same, but then I thought maybe
> you need all these log files. If not then do as Harsh has suggested. And
> deleting log files won't affect your Hdfs working, but it will not write
> logs for any operation until the next Hdfs restart.
>
> Regards,
>     Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <harsh@cloudera.com> wrote:
>
>> Lower your log levels if you do not need all that verbosity. You can
>> control log retention, max sizes to keep, max number of files to keep,
>> and logging levels, etc. via each components' log4j.properties file.
>>
>> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
>> <funnlearnforkids@gmail.com> wrote:
>> > If i delete the log file without stopping the cluster  won't it
>> > terminate the session.
>> >
>> >
>> >
>> > On 11/23/12, Mohammad Tariq <dontariq@gmail.com> wrote:
>> >> Hi there,
>> >>
>> >>     You can write a small job or some script which periodically checks
>> for
>> >> the log growth and performs the delete after certain threshold.
>> >>
>> >> Regards,
>> >>     Mohammad Tariq
>> >>
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> >> funnlearnforkids@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Everytime i query hbase or hive ,there is a significant growth in my
>> >>> log files and it consumes lot of space from my hard disk....(Approx
40
>> >>> gb)
>> >>> So i stop the cluster ,delete all the logs and free the space and then
>> >>> again start the cluster to start my work.
>> >>>
>> >>> Is there any other solution coz i cannot restart the cluster everyday.
>> >>>
>> >>
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Mime
View raw message