Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of dontariq@gmail.com designates
 209.85.216.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <7F07D5EF988D499E92435EA0AFAC251A@gallotta.co.za>
References: <337C6F38C62740D5BB5893CA2B2F9D0A@gallotta.co.za>
 <CAF1+Vs9FkLsAK-3in8xCuLAfY_UCB+Okqd0V+DKTo0UcyVTZ8Q@mail.gmail.com>
 <680A0D625EFF468FBDF81FAFC00CE25A@gallotta.co.za>
 <CAF1+Vs8GY7Hc9q5E771Y24YSE=A3LpAHFUB-Ycwt7K5Lqq7ywg@mail.gmail.com>
 <48F495EE31BB468AB38B38423C03BF43@gallotta.co.za>
 <CAMVC6RPKeAEY38oXkOSsu-sA50GN5tMrJL_QsXA2-_B+A+xDnA@mail.gmail.com>
 <CAF1+Vs_pUVZQpFt6tPuRBHou69MqVza-ENmshsyaOs41zvMsRA@mail.gmail.com>
 <30CC9891A5D64924A97EE65FE70074D6@gallotta.co.za>
 <CAF1+Vs9dPd8zHFrcLc8P9W9H1hYNXvvpEZtWqscaiM7Gfmhvxw@mail.gmail.com>
 <62DCDB5612384C89AF514CD04F1E36DE@gallotta.co.za>
 <CAMVC6RPDEx6j3706HaX982AC2+PADctdQN-rDWv-ACmjA8e99g@mail.gmail.com>
 <CAMVC6RP_qCaxD0BwzCgtGp2uVuFiDF-fXcVTXtNEdASWjwEfHw@mail.gmail.com>
 <7F07D5EF988D499E92435EA0AFAC251A@gallotta.co.za>
From: Mohammad Tariq <dontariq@gmail.com>
Date: Sat, 11 Aug 2012 05:20:41 +0530
Message-ID: 
 <CAMVC6RN32mhTUDJvMKSz3EPUjw7TbTA54NQq0AjM3fPM3=ELQA@mail.gmail.com>
Subject: Re: Table listed in "list", but not in .META.
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
point your web browser to http://localhost:9000. It'll show all the
details of your HDFS.

Regards,
    Mohammad Tariq


On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <marco@gallotta.co.za> wrote:
> How do you check that?
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>
>> This is pretty strange. I mean everything seems to be in place, but we
>> are stuck. Please make a check once if your Hdfs is in safemode.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
>> > What about fs.default.name (http://fs.default.name)?????
>> >
>> > Regards,
>> > Mohammad Tariq
>> >
>> >
>> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)> wrote:
>> > > It's in /var which is persistent across reboots.
>> > >
>> > > --
>> > > Marco Gallotta | Mountain View, California
>> > > Software Engineer, Infrastructure | Loki Studios
>> > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> > >
>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > >
>> > >
>> > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>> > >
>> > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
>> > > > rebooted your machined then you will have problems.
>> > > >
>> > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>> > > >
>> > > > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> > > > > gathering more data.
>> > > > >
>> > > > > I get the following error when running hbck -repair, and then it stalls:
>> > > > >
>> > > > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> > > > > after fix...
>> > > > > Version: 0.94.0
>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> > > > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> > > > > HDFS...
>> > > > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> > > > > at
>> > > > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>> > > > > at
>> > > > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> > > > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Marco Gallotta | Mountain View, California
>> > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>> > > > >
>> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > >
>> > > > >
>> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> > > > >
>> > > > > > Is it a standalone installation or pseudo-distributed?
>> > > > > > I faced a similar problem a few days back in a distributed cluster and
>> > > > > >
>> > > > >
>> > > > >
>> > > > > used
>> > > > > > hbck -repair option. You might give it a try.
>> > > > > >
>> > > > > > ~Anil
>> > > > > >
>> > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>> > > > > dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>> > > > > >
>> > > > > > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > > > > > compaction and see if ti works.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > > Mohammad Tariq
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
>> > > > > > > wrote:
>> > > > > > > > It's not a distributed cluster. I'm not processing enough data yet.
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > So
>> > > > > > >
>> > > > > > > the reference to localhost is correct.
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > 417-3313
>> > > > > > > >
>> > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > > > > > >
>> > > > > > > > > Are you running a distributed cluster?
>> > > > > > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > > > > > >
>> > > > > > > > > You are getting reference to localhost in hbck output:
>> > > > > > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > region
>> > > > > > > > > server
>> > > > > > > > >
>> > > > > > > > > ~Anil
>> > > > > > > > >
>> > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > > > > > >
>> > > > > > > > > > Here's the output from hbck -details:
>> > > > > http://pastebin.com/ZxVZEctY
>> > > > > > > > > >
>> > > > > > > > > > Extract:
>> > > > > > > > > >
>> > > > > > > > > > 6 inconsistencies detected.
>> > > > > > > > > > Status: INCONSISTENT
>> > > > > > > > > >
>> > > > > > > > > > 6 is the number of tables that appear in "list" but cannot be
>> > > > > > > operated on
>> > > > > > > > > > (which btw, includes not being able to run disable/drop on them -
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > both ops
>> > > > > > > > > > say table not found). I also just noticed "foo" does not occur
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > in a
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > table
>> > > > > > > > > > list, although I did create it at one point but was able to
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > clear it
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > from
>> > > > > > > > > > .META. when it also was reporting table not found when trying to
>> > > > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > table
>> > > > > > > > > > creation when I was trying to get lzo compression working and
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > table
>> > > > > > > > > > creation was hanging.
>> > > > > > > > > >
>> > > > > > > > > > Is there any way to repair this? I see hbck has repair options,
>> > > > > but I
>> > > > > > > want
>> > > > > > > > > > to proceed with caution.
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 417-3313
>> > > > > > > > > >
>> > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Marco,
>> > > > > > > > > > >
>> > > > > > > > > > > Did anything disastrous happen to cluster?
>> > > > > > > > > > > Can you try using hbck utility of HBase.
>> > > > > > > > > > > Run: 'hbase hbck -help' to get all the available options.
>> > > > > > > > > > >
>> > > > > > > > > > > ~Anil
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>> > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>> > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi there
>> > > > > > > > > > > >
>> > > > > > > > > > > > I have a few tables which show up in a "list" in the shell,
>> > > > > but
>> > > > > > > produce
>> > > > > > > > > > > > "table not found" when performing any operation on them.
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > There is
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > no
>> > > > > > > > > > > > reference of them in the .META. table. It seems to be
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > resulting in
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > some of
>> > > > > > > > > > > > the hbase services being killed every so often.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Here are some logs from master (foo is one of the tables not
>> > > > > > > found):
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > > > > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > > > > > Master server abort: loaded coprocessors are: []
>> > > > > > > > > > > > 2012-08-09 20:40:44,301 FATAL
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > org.apache.hadoop.hbase.master.HMaster:
>> > > > > > > > > > > > Unexpected state :
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > > > > > > state=PENDING_OPEN, ts=1344570044277,
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > ..
>> > > > > > > > > > > > Cannot transit it to OFFLINE.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > There are also a number of the following types of error logs:
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2012-08-09 20:10:04,308 ERROR
>> > > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > assignment in:
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > due to
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > > > > > > > > > > Received:OPEN for the
>> > > > > > > > > > > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > ,which we
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > are
>> > > > > > > > > > > > already trying to OPEN.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Any ideas how to find and remove any references to these
>> > > > > > > non-existent
>> > > > > > > > > > > > tables?
>> > > > > > > > > > > >
>> > > > > > > > > > > > --
>> > > > > > > > > > > > Marco Gallotta | Mountain View, California
>> > > > > > > > > > > > Software Engineer, Infrastructure | Loki Studios
>> > > > > > > > > > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > twitter.com/marcog (http://twitter.com/marcog)
>> > > > > > > > > > > > marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > (650)
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 417-3313
>> > > > > > > > > > > >
>> > > > > > > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > Thanks & Regards,
>> > > > > > > > > > > Anil Gupta
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Thanks & Regards,
>> > > > > > > > > Anil Gupta
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Thanks & Regards,
>> > > > > > Anil Gupta
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Thanks & Regards,
>> > > > Anil Gupta
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>>
>
>