From dev-return-783-apmail-tephra-dev-archive=tephra.apache.org@tephra.incubator.apache.org Tue Feb 7 07:50:48 2017 Return-Path: X-Original-To: apmail-tephra-dev-archive@minotaur.apache.org Delivered-To: apmail-tephra-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0D650190B2 for ; Tue, 7 Feb 2017 07:50:48 +0000 (UTC) Received: (qmail 44587 invoked by uid 500); 7 Feb 2017 07:50:48 -0000 Delivered-To: apmail-tephra-dev-archive@tephra.apache.org Received: (qmail 44550 invoked by uid 500); 7 Feb 2017 07:50:47 -0000 Mailing-List: contact dev-help@tephra.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@tephra.incubator.apache.org Delivered-To: mailing list dev@tephra.incubator.apache.org Received: (qmail 44485 invoked by uid 99); 7 Feb 2017 07:50:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Feb 2017 07:50:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 733C9C22F0 for ; Tue, 7 Feb 2017 07:50:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.999 X-Spam-Level: X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id cB0PHBHRNsMb for ; Tue, 7 Feb 2017 07:50:46 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 9AA6960DB7 for ; Tue, 7 Feb 2017 07:50:45 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D0501E05C4 for ; Tue, 7 Feb 2017 07:50:43 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 3E66525298 for ; Tue, 7 Feb 2017 07:50:42 +0000 (UTC) Date: Tue, 7 Feb 2017 07:50:42 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: dev@tephra.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (TEPHRA-214) Provide a way to debug Transaction Pruning progress MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/TEPHRA-214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855505#comment-15855505 ] ASF GitHub Bot commented on TEPHRA-214: --------------------------------------- Github user poornachandra commented on a diff in the pull request: https://github.com/apache/incubator-tephra/pull/31#discussion_r99751362 --- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java --- @@ -0,0 +1,191 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.tephra.hbase.txprune; + +import com.google.common.collect.MinMaxPriorityQueue; +import com.google.gson.Gson; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.tephra.TxConstants; +import org.apache.tephra.txprune.RegionPruneInfo; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.io.PrintWriter; +import java.util.Comparator; +import java.util.LinkedList; +import java.util.List; +import java.util.Queue; +import javax.annotation.Nullable; + +/** + * Invalid List Pruning Debug Tool. + */ +public class InvalidListPruningDebug { + private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class); + private static final Gson GSON = new Gson(); + private DataJanitorState dataJanitorState; + + /** + * Initialize the Invalid List Debug Tool. + * @param conf {@link Configuration} + * @throws IOException + */ + public void initialize(final Configuration conf) throws IOException { + LOG.info("InvalidListPruningDebugMain : initialize method called"); + final Connection connection = ConnectionFactory.createConnection(conf); + dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() { + @Override + public Table get() throws IOException { + return connection.getTable(TableName.valueOf( + conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE, + TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE))); + } + }); + } + + /** + * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds. + * If -1 is passed in, all the regions and their prune upper bound will be returned. + * + * @param numRegions number of regions + * @return Map of region name and its prune upper bound + */ + public Queue getIdleRegions(Integer numRegions) throws IOException { + List regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null); + if (regionPruneInfos.isEmpty()) { + return new LinkedList<>(); + } + + if (numRegions < 0) { + numRegions = regionPruneInfos.size(); + } + + Queue lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator() { + @Override + public int compare(RegionPruneInfo o1, RegionPruneInfo o2) { + return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound()); + } + }).maximumSize(numRegions).create(); + + for (RegionPruneInfo pruneInfo : regionPruneInfos) { + lowestPrunes.add(pruneInfo); + } + return lowestPrunes; + } + + /** + * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet, + * it will return a null. + * + * @param regionId region id + * @return {@link RegionPruneInfo} of the region + * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo} + */ + @Nullable + public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException { + return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId)); + } + + /** + * + * @param time Given a time, provide the {@link TimeRegions} at or before that time + * @return time regions or null if no time regions are present at or before the given time + * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions} + */ + @Nullable + public String getRegionsOnOrBeforeTime(Long time) throws IOException { + TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time); + if (timeRegions == null) { + return null; + } + return timeRegions.toString(); --- End diff -- If `timeRegions` is returned as object from here, then it can be printed as JSON later > Provide a way to debug Transaction Pruning progress > --------------------------------------------------- > > Key: TEPHRA-214 > URL: https://issues.apache.org/jira/browse/TEPHRA-214 > Project: Tephra > Issue Type: Improvement > Reporter: Gokul Gunasekaran > Assignee: Gokul Gunasekaran > > It would be good to get information about what regions are lagging that might be useful to debug why transaction pruning might not be happening. We do print debug information but this method could be invoked on a need-basis outside of the regular pruning operation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)