Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5BFFEE7DE for ; Thu, 24 Jan 2013 13:10:42 +0000 (UTC) Received: (qmail 51370 invoked by uid 500); 24 Jan 2013 12:57:17 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 50316 invoked by uid 500); 24 Jan 2013 12:57:13 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 50279 invoked by uid 99); 24 Jan 2013 12:57:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jan 2013 12:57:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cembree@gmail.com designates 209.85.216.51 as permitted sender) Received: from [209.85.216.51] (HELO mail-qa0-f51.google.com) (209.85.216.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jan 2013 12:57:07 +0000 Received: by mail-qa0-f51.google.com with SMTP id dx4so318971qab.10 for ; Thu, 24 Jan 2013 04:56:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:reply-to:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=Z4wpYSuvteREGjVp01FO3L0XfXjx32b0uMatPVAa9i0=; b=jfUZGu7DjWEQkGSSDUlvfugE7comOblyTnFk4rZk4MxhSIBekylnvgkCyOLo2JtYj0 4lT7pktRuTiAk5yBFXwW2PmCE22XvOCE/b48+WMlTCQ+SujEo9JJjBfY1iyXfNk3BTTA Y7r60mt2VUVCHh9HPzODaZOnVFjCAGo5FlgZGecArdMw/aHeKFoWXKI7eEFiQzcENMtk JPr60ftRyvt2IoMMBg9ozccZXUWx7KbQsOX+6B62COiEoQ1A3ck72aEYTUnuh9UFKo7r pm/L37j8lnu4jgt5SJUDLCrZU9UZA6rl1vK1kw+dxIO3wAEbmnMmrc6BYpQ5muTpW2sp RAAA== MIME-Version: 1.0 X-Received: by 10.49.3.234 with SMTP id f10mr2149134qef.52.1359032206729; Thu, 24 Jan 2013 04:56:46 -0800 (PST) Received: by 10.49.106.170 with HTTP; Thu, 24 Jan 2013 04:56:46 -0800 (PST) Reply-To: chris@embree.us In-Reply-To: References: Date: Thu, 24 Jan 2013 07:56:46 -0500 Message-ID: Subject: Re: hdfs du periodicity and hdfs not respond at that time From: Chris Embree To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7bb03e68eba28204d4085717 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bb03e68eba28204d4085717 Content-Type: text/plain; charset=ISO-8859-1 What type of FS are you using under HDFS? XFS, ext3, ext4? The type and configuration of the underlying FS will impact performance. Most notably, ext3 has a lock-up effect when flushing disk cache. On Thu, Jan 24, 2013 at 2:54 AM, Xibin Liu wrote: > Thanks, http://search-hadoop.com/m/LLBgUiH0Bg2 is my issue , but I still > dont't know how to solve this problem, 3 minutes not respond once an hour > is a big problem for me, any clue for this? > > > 2013/1/24 Harsh J > >> Hi, >> >> HDFS does this to estimate space reports. Perhaps the discussion here >> may help you: http://search-hadoop.com/m/LLBgUiH0Bg2 >> >> On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu >> wrote: >> > hi all, >> > I found hdfs du periodicity(one hour), and because my disk is big, the >> > smallest one is 15T, so when hdfs exec du, datanode will not respond for >> > about 3 minuts because of io loading, this cause a lot of problem, >> anybody >> > knows why hdfs doing this and how to disable it? >> > >> > -- >> > Thanks & Regards >> > Xibin Liu >> > >> >> >> >> -- >> Harsh J >> > > > > -- > Thanks & Best Regards > Xibin Liu > > --047d7bb03e68eba28204d4085717 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable What type of FS are you using =A0under HDFS? =A0XFS, ext3, ext4? =A0The typ= e and configuration of the underlying FS will impact performance.
Most = notably, ext3 has a lock-up effect when flushing disk cache.

On Thu, Jan 24, 2013 at 2:54 AM, Xibin Liu <xibin.liu.cn@gmail.com> wrote:
Thanks,=A0http://search-hadoop.com/m/LLBgUiH0Bg2=A0is my issue=A0, but I stil= l dont't know how to solve this problem, 3 minutes not respond once an = hour =A0is a big problem for me, any clue for this?


2013/1/24 Harsh J <harsh@cloudera.com&= gt;
Hi,

HDFS does this to estimate space reports. Perhaps the discussion here
may help you: http://search-hadoop.com/m/LLBgUiH0Bg2

On Thu, Jan 24, 2013 at 12:51 PM, Xibin Liu <xibin.liu.cn@gmail.com> wrote:
> hi all,
> I found hdfs du periodicity(one hour), and because my disk is big, the=
> smallest one is 15T, so when hdfs exec du, datanode will not respond f= or
> about 3 minuts because of io loading, this cause a lot of problem, any= body
> knows why hdfs doing this and how to disable it?
>
> --
> Thanks & =A0Regards
> Xibin Liu
>



--
Harsh J



--
Thanks & B= est Regards
Xibin Liu


--047d7bb03e68eba28204d4085717--