Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 688F01042E for ; Wed, 23 Apr 2014 04:03:32 +0000 (UTC) Received: (qmail 39748 invoked by uid 500); 23 Apr 2014 04:03:23 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 39153 invoked by uid 500); 23 Apr 2014 04:03:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 39142 invoked by uid 99); 23 Apr 2014 04:03:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Apr 2014 04:03:19 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of amitkabraiiit@gmail.com designates 209.85.214.176 as permitted sender) Received: from [209.85.214.176] (HELO mail-ob0-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Apr 2014 04:03:14 +0000 Received: by mail-ob0-f176.google.com with SMTP id wp4so439079obc.7 for ; Tue, 22 Apr 2014 21:02:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=YZoQoAfdrguQtRu4Mxz/MF3bCoDK5LJZ+Vz50HbG/vw=; b=BS9wL1YWRlQUrpciJdXXjyOzclTwkFZXU8hmz2XA4MhST3XFe1QxmFodshhDkJjRe2 vT5OyQA2hN4bFnbvUJuTY3cZ2Y4VXtgleyKVpnlXd7c5+tkie6hAyFDWPiRTJtRBlSc7 3Ml3G4Lyy/arXTKrx9wsFn+4ZU23FdcC4xrHNYe/6ubtTT3rxMMYdARDs/5SBboTQj5l IpGOYO3z7cLC8BIZz6gfARm/cY+Uqs9/AObIXYh9cLpwLhMoDlN6ypSxnYtRbvN8oG02 g4WzkMyNqxeHJ1U1kichCO8Xwybxzq6TzcHT6LDNJqal8i66RbWvxz+FULsX9Mv8aM9G yrZg== X-Received: by 10.60.229.228 with SMTP id st4mr40581334oec.16.1398225773576; Tue, 22 Apr 2014 21:02:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.171.65 with HTTP; Tue, 22 Apr 2014 21:02:32 -0700 (PDT) In-Reply-To: References: <4133BAA7-D6D1-4D7D-AF87-FB41A26A22E9@gmail.com> From: Amit Kabra Date: Wed, 23 Apr 2014 09:32:32 +0530 Message-ID: Subject: Re: All datanodes are bad. Aborting ... To: user@hadoop.apache.org Cc: sudhakara st Content-Type: text/plain; charset=UTF-8 X-Virus-Checked: Checked by ClamAV on apache.org fsck showed cluster to be healthy that time. On Wed, Apr 23, 2014 at 8:25 AM, Shumin Guo wrote: > Did you do fsck? And what's the result? > > > On Sun, Apr 20, 2014 at 12:14 PM, Amit Kabra > wrote: >> >> 1) ulimit -a >> >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> scheduling priority (-e) 0 >> file size (blocks, -f) unlimited >> pending signals (-i) 513921 >> max locked memory (kbytes, -l) 64 >> max memory size (kbytes, -m) unlimited >> open files (-n) 65536 >> pipe size (512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> real-time priority (-r) 0 >> stack size (kbytes, -s) 10240 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 32000 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> >> 2) dfs.datanode.max.xcievers = 4096 >> >> 3) dfs.datanode.max.transfer.threads = 4096 >> >> >> >> On Sun, Apr 20, 2014 at 10:36 PM, sudhakara st >> wrote: >> > check with open file descriptor limit in data nodes and namenode. >> > >> > $ ulimit -a >> > >> > and >> > check with 'dfs.datanode.max.xcievers or >> > dfs.datanode.max.transfer.threads' >> > property in hdfs-site.xml >> > >> > >> > >> > >> > On Sun, Apr 20, 2014 at 9:40 PM, Amit Kabra >> > wrote: >> >> >> >> Yes, error logs here : http://pastebin.com/RBdN5Euf >> >> >> >> On Sun, Apr 20, 2014 at 8:14 PM, Serge Blazhievsky >> >> >> >> wrote: >> >> > Do you see any errors in datanodes logs? >> >> > >> >> > Sent from my iPhone >> >> > >> >> >> On Apr 20, 2014, at 2:57, Amit Kabra >> >> >> wrote: >> >> >> >> >> >> number >> > >> > >> > >> > >> > -- >> > >> > Regards, >> > ...sudhakara >> > > >