gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kiyonari Harigae (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GORA-506) Investigate timing of HBase tests
Date Mon, 06 Mar 2017 10:11:32 GMT

    [ https://issues.apache.org/jira/browse/GORA-506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897023#comment-15897023
] 

Kiyonari Harigae commented on GORA-506:
---------------------------------------

I tried to test locally, result as follows

Total: 634 sec
HBaseClusterSingleton#truncateAllTables: 472 sec

The most occupation was HBaseClusterSingleton#truncateAllTables,
whose percentage was about 74%.

I think that because of the implementation of HBaseTestingUtility#truncateTable differs between
0.98.8 and 1.2.3.
At 0.98.8 the implementation of this method was delete each rows, whereas 1.2.3 implemented
as disableTable -> Truncate.
Instead, at 1.2.3 can be satisfy the requirement by using deleteTableData.
Improvement to 2min40sec by using deleteTableData.
It is close to at gora-0.6.1.

HBaseTestingUtility#truncateTable
(hbase-server-0.98.8-hadoop2-tests)

  public HTable truncateTable(byte[] tableName)
    throws IOException
  {
    return truncateTable(TableName.valueOf(tableName));
  }
  
  public HTable truncateTable(TableName tableName)
    throws IOException
  {
    HTable table = new HTable(getConfiguration(), tableName);
    Scan scan = new Scan();
    ResultScanner resScan = table.getScanner(scan);
    for (Result res : resScan)
    {
      Delete del = new Delete(res.getRow());
      table.delete(del);
    }
    resScan = table.getScanner(scan);
    resScan.close();
    return table;
  }

HBaseTestingUtility#truncateTable
(hbase-server-1.2.3-tests)

  public HTable truncateTable(TableName tableName, boolean preserveRegions)
    throws IOException
  {
    Admin admin = getHBaseAdmin();
    if (!admin.isTableDisabled(tableName)) {
      admin.disableTable(tableName);
    }
    admin.truncateTable(tableName, preserveRegions);
    return new HTable(getConfiguration(), tableName);
  }
  
  public HTable truncateTable(TableName tableName)
    throws IOException
  {
    return truncateTable(tableName, false);
  }
  
  public HTable truncateTable(byte[] tableName, boolean preserveRegions)
    throws IOException
  {
    return truncateTable(TableName.valueOf(tableName), preserveRegions);
  }
  
  public HTable truncateTable(byte[] tableName)
    throws IOException
  {
    return truncateTable(tableName, false);
  }

Instead use follow:

  public HTable deleteTableData(byte[] tableName)
    throws IOException
  {
    return deleteTableData(TableName.valueOf(tableName));
  }
  
  public HTable deleteTableData(TableName tableName)
    throws IOException
  {
    HTable table = new HTable(getConfiguration(), tableName);
    Scan scan = new Scan();
    ResultScanner resScan = table.getScanner(scan);
    for (Result res : resScan)
    {
      Delete del = new Delete(res.getRow());
      table.delete(del);
    }
    resScan = table.getScanner(scan);
    resScan.close();
    return table;
  }


> Investigate timing of HBase tests
> ---------------------------------
>
>                 Key: GORA-506
>                 URL: https://issues.apache.org/jira/browse/GORA-506
>             Project: Apache Gora
>          Issue Type: Test
>          Components: gora-hbase
>    Affects Versions: 0.7
>            Reporter: Lewis John McGibbney
>            Priority: Critical
>
> GORA-443 * fixed * an upgrade to HBase 1.2.3, there is a significant amount of time however
spent on the tests.
> We need to investigate why and fix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message