gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kiyonari Harigae (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (GORA-506) Investigate timing of HBase tests
Date Tue, 07 Mar 2017 01:20:33 GMT

    [ https://issues.apache.org/jira/browse/GORA-506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897023#comment-15897023
] 

Kiyonari Harigae edited comment on GORA-506 at 3/7/17 1:20 AM:
---------------------------------------------------------------

I have focused on the difference between the test at the time of 0.6.1 and the current test.
I tried to test locally, result as follows

0.6.1:
Total: 126 sec

current test:
Total: 634 sec

The most occupation was HBaseClusterSingleton#truncateAllTables in current test,
whose percentage was about 74%.

I think that because of the implementation of HBaseTestingUtility#truncateTable differs between
0.98.8 and 1.2.3.
At 0.98.8 the implementation of this method was delete each rows, whereas 1.2.3 implemented
as disableTable -> Truncate.
Instead, at 1.2.3 can be satisfy the requirement by using deleteTableData.
I tried to use deleteTableData then, improvement to 2min40sec.
It is close to at gora-0.6.1.

HBaseTestingUtility#truncateTable
(hbase-server-0.98.8-hadoop2-tests)
{code}
    public HTable truncateTable(byte[] tableName)
      throws IOException
    {
      return truncateTable(TableName.valueOf(tableName));
    }
    
    public HTable truncateTable(TableName tableName)
      throws IOException
    {
      HTable table = new HTable(getConfiguration(), tableName);
      Scan scan = new Scan();
      ResultScanner resScan = table.getScanner(scan);
      for (Result res : resScan)
      {
        Delete del = new Delete(res.getRow());
        table.delete(del);
      }
      resScan = table.getScanner(scan);
      resScan.close();
      return table;
    }
{code}

HBaseTestingUtility#truncateTable
(hbase-server-1.2.3-tests)

{code}
   public HTable truncateTable(TableName tableName, boolean preserveRegions)
     throws IOException
   {
     Admin admin = getHBaseAdmin();
     if (!admin.isTableDisabled(tableName)) {
       admin.disableTable(tableName);
     }
     admin.truncateTable(tableName, preserveRegions);
     return new HTable(getConfiguration(), tableName);
   }
   
   public HTable truncateTable(TableName tableName)
     throws IOException
   {
     return truncateTable(tableName, false);
   }
   
   public HTable truncateTable(byte[] tableName, boolean preserveRegions)
     throws IOException
   {
     return truncateTable(TableName.valueOf(tableName), preserveRegions);
   }
   
   public HTable truncateTable(byte[] tableName)
     throws IOException
   {
     return truncateTable(tableName, false);
   }
{code}

Instead can be use follow:

{code}
    public HTable deleteTableData(byte[] tableName)
      throws IOException
    {
      return deleteTableData(TableName.valueOf(tableName));
    }
    
    public HTable deleteTableData(TableName tableName)
      throws IOException
    {
      HTable table = new HTable(getConfiguration(), tableName);
      Scan scan = new Scan();
      ResultScanner resScan = table.getScanner(scan);
      for (Result res : resScan)
      {
        Delete del = new Delete(res.getRow());
        table.delete(del);
      }
      resScan = table.getScanner(scan);
      resScan.close();
      return table;
    }
{code}



was (Author: cloudysunny14):
I have focused on the difference between the test at the time of 0.6.1 and the current test.
I tried to test locally, result as follows

0.6.1:
Total: 126 sec

current test:
Total: 634 sec

The most occupation was HBaseClusterSingleton#truncateAllTables in current test,
whose percentage was about 74%.

I think that because of the implementation of HBaseTestingUtility#truncateTable differs between
0.98.8 and 1.2.3.
At 0.98.8 the implementation of this method was delete each rows, whereas 1.2.3 implemented
as disableTable -> Truncate.
Instead, at 1.2.3 can be satisfy the requirement by using deleteTableData.
I tried to use deleteTableData then, improvement to 2min40sec.
It is close to at gora-0.6.1.

HBaseTestingUtility#truncateTable
(hbase-server-0.98.8-hadoop2-tests)
{code}
    public HTable truncateTable(byte[] tableName)
      throws IOException
    {
      return truncateTable(TableName.valueOf(tableName));
    }
    
    public HTable truncateTable(TableName tableName)
      throws IOException
    {
      HTable table = new HTable(getConfiguration(), tableName);
      Scan scan = new Scan();
      ResultScanner resScan = table.getScanner(scan);
      for (Result res : resScan)
      {
        Delete del = new Delete(res.getRow());
        table.delete(del);
      }
      resScan = table.getScanner(scan);
      resScan.close();
      return table;
    }
{code}

HBaseTestingUtility#truncateTable
(hbase-server-1.2.3-tests)

{code}
   public HTable truncateTable(TableName tableName, boolean preserveRegions)
     throws IOException
   {
     Admin admin = getHBaseAdmin();
     if (!admin.isTableDisabled(tableName)) {
       admin.disableTable(tableName);
     }
     admin.truncateTable(tableName, preserveRegions);
     return new HTable(getConfiguration(), tableName);
   }
   
   public HTable truncateTable(TableName tableName)
     throws IOException
   {
     return truncateTable(tableName, false);
   }
   
   public HTable truncateTable(byte[] tableName, boolean preserveRegions)
     throws IOException
   {
     return truncateTable(TableName.valueOf(tableName), preserveRegions);
   }
   
   public HTable truncateTable(byte[] tableName)
     throws IOException
   {
     return truncateTable(tableName, false);
   }
{code}

Instead use follow:

{code}
    public HTable deleteTableData(byte[] tableName)
      throws IOException
    {
      return deleteTableData(TableName.valueOf(tableName));
    }
    
    public HTable deleteTableData(TableName tableName)
      throws IOException
    {
      HTable table = new HTable(getConfiguration(), tableName);
      Scan scan = new Scan();
      ResultScanner resScan = table.getScanner(scan);
      for (Result res : resScan)
      {
        Delete del = new Delete(res.getRow());
        table.delete(del);
      }
      resScan = table.getScanner(scan);
      resScan.close();
      return table;
    }
{code}


> Investigate timing of HBase tests
> ---------------------------------
>
>                 Key: GORA-506
>                 URL: https://issues.apache.org/jira/browse/GORA-506
>             Project: Apache Gora
>          Issue Type: Test
>          Components: gora-hbase
>    Affects Versions: 0.7
>            Reporter: Lewis John McGibbney
>            Priority: Critical
>
> GORA-443 * fixed * an upgrade to HBase 1.2.3, there is a significant amount of time however
spent on the tests.
> We need to investigate why and fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message