Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 058AE101EF for ; Wed, 23 Apr 2014 02:56:40 +0000 (UTC) Received: (qmail 31026 invoked by uid 500); 23 Apr 2014 02:56:32 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 30710 invoked by uid 500); 23 Apr 2014 02:56:31 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 30703 invoked by uid 99); 23 Apr 2014 02:56:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Apr 2014 02:56:30 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gsmsteve@gmail.com designates 209.85.213.46 as permitted sender) Received: from [209.85.213.46] (HELO mail-yh0-f46.google.com) (209.85.213.46) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Apr 2014 02:56:25 +0000 Received: by mail-yh0-f46.google.com with SMTP id b6so333310yha.33 for ; Tue, 22 Apr 2014 19:56:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=eAPC8svfEbeaDJsrgPs5r2a9yVlKd0PFvQxQqRpd9V0=; b=olOPxkzSn1i3VSdSVXFQ6TdD+7AAckUfkHxjGljzMxZ9c4TO/S8DBqfQ1RyEMIRE6Y VbAAr+0VHlvuMMCH0LtwqnqZnjF1JrdqcfsMJdIBkimrKxFR61cm/+snjSuVCVMhabNP Zl+8Kqhv/UPyazxjos2+WesEofnPwRatQrSvmc6w6vC6TzdaWNaqD4JV+Dy1gr0d2jVJ rYxaLbDRDXBdJ7mvrJhZuzblYmIbm4CU/6yqLTNAukV4RCfCI8gSqQPKG+CmggS9hY8D VXHf4mkjk4P8fXN/ZpSr3wa8IvIeulCYtFMqSZNgKOupVXhgv6+CJVtwkN2lJmG2+DdL p00w== X-Received: by 10.236.7.47 with SMTP id 35mr66471017yho.23.1398221762278; Tue, 22 Apr 2014 19:56:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.170.158.3 with HTTP; Tue, 22 Apr 2014 19:55:42 -0700 (PDT) In-Reply-To: References: <4133BAA7-D6D1-4D7D-AF87-FB41A26A22E9@gmail.com> From: Shumin Guo Date: Tue, 22 Apr 2014 21:55:42 -0500 Message-ID: Subject: Re: All datanodes are bad. Aborting ... To: user@hadoop.apache.org Cc: sudhakara.st@gmail.com Content-Type: multipart/alternative; boundary=001a1134036a7539ba04f7acdf74 X-Virus-Checked: Checked by ClamAV on apache.org --001a1134036a7539ba04f7acdf74 Content-Type: text/plain; charset=UTF-8 Did you do fsck? And what's the result? On Sun, Apr 20, 2014 at 12:14 PM, Amit Kabra wrote: > 1) ulimit -a > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 513921 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 65536 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 32000 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > 2) dfs.datanode.max.xcievers = 4096 > > 3) dfs.datanode.max.transfer.threads = 4096 > > > > On Sun, Apr 20, 2014 at 10:36 PM, sudhakara st > wrote: > > check with open file descriptor limit in data nodes and namenode. > > > > $ ulimit -a > > > > and > > check with 'dfs.datanode.max.xcievers or > dfs.datanode.max.transfer.threads' > > property in hdfs-site.xml > > > > > > > > > > On Sun, Apr 20, 2014 at 9:40 PM, Amit Kabra > wrote: > >> > >> Yes, error logs here : http://pastebin.com/RBdN5Euf > >> > >> On Sun, Apr 20, 2014 at 8:14 PM, Serge Blazhievsky > > >> wrote: > >> > Do you see any errors in datanodes logs? > >> > > >> > Sent from my iPhone > >> > > >> >> On Apr 20, 2014, at 2:57, Amit Kabra > wrote: > >> >> > >> >> number > > > > > > > > > > -- > > > > Regards, > > ...sudhakara > > > --001a1134036a7539ba04f7acdf74 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Did you do fsck? And what's the result?=C2=A0


On Sun, Apr 20, = 2014 at 12:14 PM, Amit Kabra <amitkabraiiit@gmail.com>= wrote:
1) ulimit -a

core file size =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(blocks, -c) 0
data seg size =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (kbytes, -d) unlimited
scheduling priority =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (-e) 0
file size =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (blocks, -f) unl= imited
pending signals =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (-i= ) 513921
max locked memory =C2=A0 =C2=A0 =C2=A0 (kbytes, -l) 64
max memory size =C2=A0 =C2=A0 =C2=A0 =C2=A0 (kbytes, -m) unlimited
open files =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0(-n) 65536
pipe size =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(512 bytes, -p) 8
POSIX message queues =C2=A0 =C2=A0 (bytes, -q) 819200
real-time priority =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(-r) 0 stack size =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(kbytes, -s) 102= 40
cpu time =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (seconds, -t) unl= imited
max user processes =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(-u) 320= 00
virtual memory =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(kbytes, -v) unlimited
file locks =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0(-x) unlimited

2) dfs.datanode.max.xcievers =3D 4096

3) dfs.datanode.max.transfer.threads =3D 4096



On Sun, Apr 20, 2014 at 10:36 PM, sudhakara st <sudhakara.st@gmail.com> wrote:
> check with =C2=A0open file descriptor limit in data nodes and namenode= .
>
> $ ulimit -a
>
> and
> check with 'dfs.datanode.max.xcievers or dfs.datanode.max.transfer= .threads'
> property in hdfs-site.xml
>
>
>
>
> On Sun, Apr 20, 2014 at 9:40 PM, Amit Kabra <amitkabraiiit@gmail.com> wrote:
>>
>> Yes, error logs here : http://pastebin.com/RBdN5Euf
>>
>> On Sun, Apr 20, 2014 at 8:14 PM, Serge Blazhievsky <hadoop.ca@gmail.com>
>> wrote:
>> > Do you see any errors in datanodes logs?
>> >
>> > Sent from my iPhone
>> >
>> >> On Apr 20, 2014, at 2:57, Amit Kabra <amitkabraiiit@gmail.com> wrote:
>> >>
>> >> number
>
>
>
>
> --
>
> Regards,
> ...sudhakara
>

--001a1134036a7539ba04f7acdf74--