hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-826) delete table followed by recreation results in honked table
Date Fri, 15 Aug 2008 07:47:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622826#action_12622826
] 

stack commented on HBASE-826:
-----------------------------

More testing would seem to indicate the above described fix works for at least the small testcase
where small case was upload of 2M rows into a table, delete it, then try same upload.  Was
failing reliably before this fix.

Trying with a bigger table.

Debugging this issue, I made getStores public in HRegion and then in HStores, I made getReader
also public.  Doing this, I was able to do a little script that could enumerate all keys in
the .META. table in a few lines.  I think we should make these methods public to facilitate
future easier debugging.

Script looked like this:

{code}
# Script that outputs all keys in .META. store files.
# HBase MUST be shutdown when you run this script.
#
# To run it, do: ${HBASE_HOME}/bin/hbase org.jruby.Main iteratemeta.rb
include Java
import org.apache.hadoop.hbase.util.MetaUtils
import org.apache.hadoop.hbase.HRegionInfo
import org.apache.hadoop.hbase.HConstants
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.HStoreKey
import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.hbase.regionserver.HLogEdit

# Get configuration to use.
c = HBaseConfiguration.new()
# Set hadoop filesystem configuration using the hbase.rootdir.
# Otherwise, we'll always use localhost though the hbase.rootdir
# might be pointing at hdfs location.
c.set("fs.default.name", c.get(HConstants::HBASE_DIR));
meta = utils.getMetaRegion(HRegionInfo::FIRST_META_REGIONINFO)
begin
  for columnDescriptor in meta.getRegionInfo().getTableDesc().getFamilies()
    store = meta.getStore(columnDescriptor.getName())  
    puts "Store: %s" % [store.toString()]
    for reader in store.getReaders()
      puts "Store: %s, reader %s" % [store.toString(), reader.toString()]
      key = HStoreKey.new()
      value = ImmutableBytesWritable.new()
      while (reader.next(key, value))
        puts "%s %s" % [key, (HLogEdit.isDeleted(value.get()) ? "d": "")]
      end 
    end
  end
ensure
  utils.shutdown()
end
{code}

> delete table followed by recreation results in honked table
> -----------------------------------------------------------
>
>                 Key: HBASE-826
>                 URL: https://issues.apache.org/jira/browse/HBASE-826
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.2.1, 0.3.0
>
>
> Daniel Leffel suspected that delete and then recreate causes issues.  I tried it on our
little cluster.  I'm doing a MR load up into the newly created table and after a few million
rows, the MR job just hangs.  Its looking for a region that doesn't exist:
> {code}
> 2008-08-13 03:32:36,840 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics with processName=MAP, sessionId=
> 2008-08-13 03:32:36,940 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
> 2008-08-13 03:32:37,420 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED
=> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
> 2008-08-13 03:32:37,541 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in .META.
> 2008-08-13 03:32:37,541 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of TestTable,0008388608,99999999999999
> 2008-08-13 03:32:37,544 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED
=> 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true',
FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS
=> '1', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}]}}
> 2008-08-13 03:32:47,605 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in .META.
> 2008-08-13 03:32:47,606 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of TestTable,0008388608,99999999999999
> ....
> {code}
> My guess is that its a region that was in the tables' previous incarnation with ghosts
left over down inside .META.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message