hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject RE: Any reason Hadoop logs cant be directed to a separate filesystem?
Date Sat, 25 Jun 2011 10:58:19 GMT

Yes, and its called using cron and writing a simple ksh script to clear out any files that
are older than 15 days. 
There may be another way, but that's really the easiest.


> Date: Thu, 23 Jun 2011 02:44:48 +0530
> From: jagaran_das@yahoo.co.in
> Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
> To: common-user@hadoop.apache.org
> 
> Hi,
> 
> Can I limit the log file duration ?
> I want to keep files for last 15 days only.
> 
> Regards,
> Jagaran 
> 
> 
> 
> ________________________________
> From: Jack Craig <JCraig@carrieriq.com>
> To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
> Sent: Wed, 22 June, 2011 2:00:23 PM
> Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
> 
> Thx to both respondents.
> 
> Note i've not tried this redirection as I have only production grids available.
> 
> Our grids are growing and with them, log volume.
> 
> As until now that log location has been in the same fs as the grid data,
> so running out of space due log bloat is a growing problem.
> 
> From your replies, sounds like I can relocate my logs, Cool!
> 
> But now the tough question, if i set up a too small partition and it runs out of 
> space,
> will my grid become unstable if hadoop can no longer write to its logs?
> 
> Thx again, jackc...
> 
> ------------------------------------
> Jack Craig, Operations
> CarrierIQ.com<http://CarrierIQ.com>
> 1200 Villa Ct, Suite 200
> Mountain View, CA. 94041
> 650-625-5456
> 
> On Jun 22, 2011, at 1:09 PM, Harsh J wrote:
> 
> Jack,
> 
> I believe the location can definitely be set to any desired path.
> Could you tell us the issues you face when you change it?
> 
> P.s. The env var is used to set the config property hadoop.log.dir
> internally. So as long as you use the regular scripts (bin/ or init.d/
> ones) to start daemons, it would apply fine.
> 
> On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig 
> <JCraig@carrieriq.com<mailto:JCraig@carrieriq.com>> wrote:
> Hi Folks,
> 
> In the hadoop-env.sh, we find, ...
> 
> # Where log files are stored.  $HADOOP_HOME/logs by default.
> # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
> 
> is there any reason this location could not be a separate filesystem on the name 
> node?
> 
> Thx, jackc...
> ------------------------------------
> Jack Craig, Operations
> CarrierIQ.com<http://CarrierIQ.com>
> 1200 Villa Ct, Suite 200
> Mountain View, CA. 94041
> 650-625-5456
> 
> 
> 
> 
> 
> --
> Harsh J
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message