hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anil Gupta <anilgupt...@gmail.com>
Subject Re: Table listed in "list", but not in .META.
Date Sat, 11 Aug 2012 00:09:34 GMT
Can you try to reboot the machine and run repair again. Might not sound logical but I would
give it a shot.

PS: My personal experience is that hbase and hadoop has never been reliable in my standalone
environment. I always trust the distributed cluster environment. AFAIK, these things are tested
extensively in distributed mode.

Best Regards,
Anil

On Aug 10, 2012, at 5:00 PM, Marco Gallotta <marco@gallotta.co.za> wrote:

> Nope, not in safe mode. Gar, this is going nowhere. :/ Thanks for the help so far though!
> 
> Configured Capacity: 211474616320 (196.95 GB)
> Present Capacity: 169354764288 (157.72 GB)
> DFS Remaining: 162401554432 (151.25 GB)
> DFS Used: 6953209856 (6.48 GB)
> DFS Used%: 4.11%
> Under replicated blocks: 436
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> 
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
> 
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 211474616320 (196.95 GB)
> DFS Used: 6953209856 (6.48 GB)
> Non DFS Used: 42119852032 (39.23 GB)
> DFS Remaining: 162401554432(151.25 GB)
> DFS Used%: 3.29%
> DFS Remaining%: 76.79%
> Last contact: Fri Aug 10 16:59:26 PDT 2012
> 
> 
> 
> -- 
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> marco@gallotta.co.za | +1 (650) 417-3313
> 
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> 
> 
> On Friday 10 August 2012 at 4:52 PM, Mohammad Tariq wrote:
> 
>> If your HDFS is in safemode you'll get something like this :
>> 
>> cluster@ubuntu:~/hadoop-1.0.3$ bin/hadoop dfsadmin -report
>> Safe mode is ON
>> Configured Capacity: 31111143424 (28.97 GB)
>> Present Capacity: 5309755392 (4.95 GB)
>> DFS Remaining: 4799320064 (4.47 GB)
>> DFS Used: 510435328 (486.79 MB)
>> DFS Used%: 9.61%
>> Under replicated blocks: 1
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>> 
>> -------------------------------------------------
>> Datanodes available: 1 (1 total, 0 dead)
>> 
>> Name: 127.0.0.1:50010
>> Decommission Status : Normal
>> Configured Capacity: 31111143424 (28.97 GB)
>> DFS Used: 510435328 (486.79 MB)
>> Non DFS Used: 25801388032 (24.03 GB)
>> DFS Remaining: 4799320064(4.47 GB)
>> DFS Used%: 1.64%
>> DFS Remaining%: 15.43%
>> Last contact: Sat Aug 11 05:19:18 IST 2012
>> 
>> See the line in red color.
>> 
>> Regards,
>> Mohammad Tariq
>> 
>> 
>> On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)>
wrote:
>>> You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
>>> point your web browser to http://localhost:9000. It'll show all the
>>> details of your HDFS.
>>> 
>>> Regards,
>>> Mohammad Tariq
>>> 
>>> 
>>> On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <marco@gallotta.co.za (mailto:marco@gallotta.co.za)>
>> wrote:
>>>> How do you check that?
>>>> 
>>>> --
>>>> Marco Gallotta | Mountain View, California
>>>> Software Engineer, Infrastructure | Loki Studios
>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650) 417-3313
>>>> 
>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>> 
>>>> 
>>>> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>>>> 
>>>>> This is pretty strange. I mean everything seems to be in place, but we
>>>>> are stuck. Please make a check once if your Hdfs is in safemode.
>>>>> 
>>>>> Regards,
>>>>> Mohammad Tariq
>>>>> 
>>>>> 
>>>>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <dontariq@gmail.com
(mailto:dontariq@gmail.com)(mailto:
>> dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>>>>>> What about fs.default.name (http://fs.default.name)?????
>>>>>> 
>>>>>> Regards,
>>>>>> Mohammad Tariq
>>>>>> 
>>>>>> 
>>>>>> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <marco@gallotta.co.za
(mailto:marco@gallotta.co.za)(mailto:
>> marco@gallotta.co.za (mailto:marco@gallotta.co.za))> wrote:
>>>>>>> It's in /var which is persistent across reboots.
>>>>>>> 
>>>>>>> --
>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) | +1 (650)
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 417-3313
>>>>>>> 
>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>> 
>>>>>>> 
>>>>>>> On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
>>>>>>> 
>>>>>>>> Where are you storing your hdfs data? Is it /tmp? If it's
/tmp
>> and you have
>>>>>>>> rebooted your machined then you will have problems.
>>>>>>>> 
>>>>>>>> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)>wrote:
>>>>>>>> 
>>>>>>>>> It's a pseudo-distributed cluster, as I plan to add more
nodes
>> as we start
>>>>>>>>> gathering more data.
>>>>>>>>> 
>>>>>>>>> I get the following error when running hbck -repair,
and then
>> it stalls:
>>>>>>>>> 
>>>>>>>>> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms
before
>> re-checking
>>>>>>>>> after fix...
>>>>>>>>> Version: 0.94.0
>>>>>>>>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos
HDFS
>>>>>>>>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase
regioninfo
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> from
>>>>>>>>> HDFS...
>>>>>>>>> Exception in thread "main"
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> java.util.concurrent.RejectedExecutionException
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956)
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:1059)
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:504)
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:304)
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>>>>>>>>> at
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za) |
+1 (650)
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 417-3313
>>>>>>>>> 
>>>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>>>>>>>> 
>>>>>>>>>> Is it a standalone installation or pseudo-distributed?
>>>>>>>>>> I faced a similar problem a few days back in a distributed
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> cluster and
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> used
>>>>>>>>>> hbck -repair option. You might give it a try.
>>>>>>>>>> 
>>>>>>>>>> ~Anil
>>>>>>>>>> 
>>>>>>>>>> On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
>> dontariq@gmail.com (mailto:dontariq@gmail.com)(mailto:
>>>>>>>>> dontariq@gmail.com (mailto:dontariq@gmail.com))> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Could you please share your /etc/hosts file??Meantime,
do a
>> manual
>>>>>>>>>>> compaction and see if ti works.
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Mohammad Tariq
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta
<
>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za))>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> It's not a distributed cluster. I'm not processing
enough
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> data yet.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> So
>>>>>>>>>>> 
>>>>>>>>>>> the reference to localhost is correct.
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>>>>> Software Engineer, Infrastructure | Loki
Studios
>>>>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta)
|
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)
| +1
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> (650)
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 417-3313
>>>>>>>>>>>> 
>>>>>>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Friday 10 August 2012 at 3:30 PM, anil
gupta wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Are you running a distributed cluster?
>>>>>>>>>>>>> If yes, do you have localhost in /etc/hosts
file?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> You are getting reference to localhost
in hbck output:
>>>>>>>>>>>>> ERROR: Region { meta => null, hdfs
=>
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>>>>>>>>>>>>> deployed => } on HDFS, but not listed
in META or
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> deployed on any
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> region
>>>>>>>>>>>>> server
>>>>>>>>>>>>> 
>>>>>>>>>>>>> ~Anil
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Aug 10, 2012 at 3:08 PM, Marco
Gallotta <
>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za))>wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Here's the output from hbck -details:
>>>>>>>>> http://pastebin.com/ZxVZEctY
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Extract:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 6 inconsistencies detected.
>>>>>>>>>>>>>> Status: INCONSISTENT
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 6 is the number of tables that appear
in "list" but
>> cannot be
>>>>>>>>>>> operated on
>>>>>>>>>>>>>> (which btw, includes not being able
to run
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> disable/drop on them -
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> both ops
>>>>>>>>>>>>>> say table not found). I also just
noticed "foo" does
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> not occur
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> in a
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> table
>>>>>>>>>>>>>> list, although I did create it at
one point but was
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> able to
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> clear it
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> from
>>>>>>>>>>>>>> .META. when it also was reporting
table not found
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> when trying to
>>>>>>>>>>>>>> disable/drop it. All these come from
when I ^C'ed
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> (i.e. killed)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> table
>>>>>>>>>>>>>> creation when I was trying to get
lzo compression
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> working and
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> table
>>>>>>>>>>>>>> creation was hanging.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Is there any way to repair this?
I see hbck has
>> repair options,
>>>>>>>>> but I
>>>>>>>>>>> want
>>>>>>>>>>>>>> to proceed with caution.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>>>>>>> Software Engineer, Infrastructure
| Loki Studios
>>>>>>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta)
|
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)
|
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> +1 (650)
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 417-3313
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Friday 10 August 2012 at 2:49
PM, anil gupta wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Marco,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Did anything disastrous happen
to cluster?
>>>>>>>>>>>>>>> Can you try using hbck utility
of HBase.
>>>>>>>>>>>>>>> Run: 'hbase hbck -help' to get
all the available
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> options.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> ~Anil
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Aug 10, 2012 at 2:22
PM, Marco Gallotta <
>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)(mailto:
>>>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> ))>wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi there
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have a few tables which
show up in a "list" in
>> the shell,
>>>>>>>>> but
>>>>>>>>>>> produce
>>>>>>>>>>>>>>>> "table not found" when performing
any operation
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> on them.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> There is
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> no
>>>>>>>>>>>>>>>> reference of them in the
.META. table. It seems
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> to be
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> resulting in
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> some of
>>>>>>>>>>>>>>>> the hbase services being
killed every so often.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Here are some logs from master
(foo is one of the
>> tables not
>>>>>>>>>>> found):
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2012-08-09 20:40:44,301 FATAL
>>>>>>>>>>> org.apache.hadoop.hbase.master.HMaster:
>>>>>>>>>>>>>>>> Master server abort: loaded
coprocessors are: []
>>>>>>>>>>>>>>>> 2012-08-09 20:40:44,301 FATAL
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> org.apache.hadoop.hbase.master.HMaster:
>>>>>>>>>>>>>>>> Unexpected state :
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>>>>>>>>>>>>>>> state=PENDING_OPEN, ts=1344570044277,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ..
>>>>>>>>>>>>>>>> Cannot transit it to OFFLINE.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> There are also a number of
the following types of
>> error logs:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2012-08-09 20:10:04,308 ERROR
>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.master.AssignmentManager:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> Failed
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> assignment in:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> due to
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>>>>>>>>>>>>>>>> Received:OPEN for the
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ,which we
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> already trying to OPEN.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Any ideas how to find and
remove any references
>> to these
>>>>>>>>>>> non-existent
>>>>>>>>>>>>>>>> tables?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Marco Gallotta | Mountain
View, California
>>>>>>>>>>>>>>>> Software Engineer, Infrastructure
| Loki Studios
>>>>>>>>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> |
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> twitter.com/marcog (http://twitter.com/marcog)
>>>>>>>>>>>>>>>> marco@gallotta.co.za (mailto:marco@gallotta.co.za)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> | +1
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> (650)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 417-3313
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Sent with Sparrow (
>> http://www.sparrowmailapp.com/?sig)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>>> Anil Gupta
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>> Anil Gupta
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Anil Gupta
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Thanks & Regards,
>>>>>>>> Anil Gupta
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> 
> 
> 

Mime
View raw message