hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Jeltema <bdjelt...@gmail.com>
Subject Re: regions in transition
Date Tue, 22 Dec 2015 17:46:21 GMT
> 
> You should really find out where you hmaster ui lives (there is a master UI
> for every node provided by the apache project) because it gives you
> information on the state of your system,

I’m familiar with the HMaster UI. I’m looking at it now. It does not contain
the information you describe. There is a list of region servers and an
a menu bar that contains: Home    Table Details    Local Logs   Degug Dump    Metrics Dump
   HBase Configuration

If I click on the Table Details item, I get a list of the tables. If I click on a table, there
is a Tasks section that says
No tasks currently runining on this node.

The region server logs do not contain any records relating to RITs, or really even regions.
The master UI does not contain any information about RITs
Version:  HDP 2.2 -> HBase 0.98.4

The zookeeper node /hbase-unsecure/regions-in-transition contains a long list of items
that are not removed when I restart the service. I think this is a side-effect of problems
I had when I did the HDP 2.1 -> HDP 2.2 upgrade, which did not go well. 

I would like to remove or clear the /hbase-unsecure/region-in-transition node
as an experiment. I’m just looking for guidance on whether that is a safe thing to do.

Brian

> but if you want to skip all that,
> here are the instructions for OfflineRepair, without knowing what is
> happening with your system (logs, master ui info) you can try this but at
> your own risk.
> 
> OfflineMetaRepair.
> Description Below:
> This code is used to rebuild meta off line from file system data. If there
>   * are any problem detected, it will fail suggesting actions for the user
> to do
>   * to "fix" problems. If it succeeds, it will backup the previous
> hbase:meta and
>   * -ROOT- dirs and write new tables in place.
> 
> Stop HBase
> zookeeper-client rmr /hbase
> HADOOP_USER_NAME=hbase hbase
> org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair
> start hbase
> 
> ^ This has worked for me in some situations where I understood HDFS and
> Zookeeper disagreed on region locations, but keep in mind I have tried this
> on hbase 1.0.0 and your mileage may vary.
> 
> We don't have your hbase version (you can even find this on the hbase shell)
> We don't have logs msgs
> We don't have master's view of your RITs
> 
> 
> On Tue, Dec 22, 2015 at 11:52 AM, Brian Jeltema <bdjeltema@gmail.com> wrote:
> 
>> I’m running Ambari 2.0.2 and HPD 2.2. I don’t see any of this displayed at
>> master:60010.
>> 
>> I really think this problem is the result of cruft in ZooKeeper. Does
>> anybody know
>> if it’s safe to delete the node?
>> 
>> 
>>> On Dec 22, 2015, at 11:40 AM, Geovanie Marquez <
>> geovanie.marquez@gmail.com> wrote:
>>> 
>>> check hmaster:60010 under TASKS (between Software Attributes and Tables)
>>> you will see if you have regions in transition. This will tell you which
>>> regions are transitioning and you can go to those region server logs and
>>> check them, I've run into a couple of these and every time they've talk
>> to
>>> me about their problem.
>>> 
>>> Also, under Software Attributes you can check the HBase version.
>>> 
>>> On Tue, Dec 22, 2015 at 11:29 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> 
>>>> From RegionListTmpl.jamon :
>>>> 
>>>> <%if (onlineRegions != null && onlineRegions.size() > 0) %>
>>>> ...
>>>> <%else>
>>>>   <p>Not serving regions</p>
>>>> </%if>
>>>> 
>>>> The message means that there was no region online on the underlying
>> server.
>>>> 
>>>> FYI
>>>> 
>>>> On Tue, Dec 22, 2015 at 7:18 AM, Brian Jeltema <bdjeltema@gmail.com>
>>>> wrote:
>>>> 
>>>>> Following up, if I look at the MBase Master UI in the Ambari console
I
>>>> see
>>>>> links to
>>>>> all of the region servers. If I click on those links, the Region Server
>>>>> page comes
>>>>> up and in the Regions section, is displays ‘Not serving regions’.
I’m
>> not
>>>>> sure
>>>>> if that means something is disabled, or it just doesn’t have any
>> regions
>>>>> to server.
>>>>> 
>>>>>> On Dec 22, 2015, at 6:19 AM, Brian Jeltema <bdjeltema@gmail.com>
>>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> Can you pick a few regions stuck in transition and check related
>>>> region
>>>>>>> server logs to see why they couldn't be assigned ?
>>>>>> 
>>>>>> I don’t see anything in the region logs relating any regions.
>>>>>> 
>>>>>>> 
>>>>>>> Which release were you using previously ?
>>>>>> 
>>>>>> HDP 2.1 -> HDP 2.2
>>>>>> 
>>>>>> So is it safe to stop HBase and delete the ZK node?
>>>>>> 
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> On Mon, Dec 21, 2015 at 3:54 PM, Brian Jeltema <bdjeltema@gmail.com>
>>>>> wrote:
>>>>>>> 
>>>>>>>> I am doing a cluster upgrade to the HDP 2.2 stack. For some
reason,
>>>>> after
>>>>>>>> the upgrade HBase
>>>>>>>> cannot find any regions for existing tables. I believe the
HDFS file
>>>>>>>> system is OK. But looking at the ZooKeeper
>>>>>>>> nodes, I noticed that many (maybe all) of the regions were
listed in
>>>>> the
>>>>>>>> ZooKeeper
>>>>>>>> /hbase-unsecure/region-in-transition node. I suspect this
could be
>>>>> causing
>>>>>>>> a problem. Is it
>>>>>>>> safe to stop HBase and delete that node?
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Brian
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 


Mime
View raw message