hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12626) Archive cleaner cannot keep up; it maxes out at about 400k deletes/hour
Date Wed, 03 Dec 2014 23:30:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233669#comment-14233669
] 

stack commented on HBASE-12626:
-------------------------------

Ideas:

+ Rather than a single thread doing a delete every 9ms, run multiple threads to do deletes
in // either in Master process or by farming out deletes so RS ran them.
+ Add bulk delete function to NN so we could pass it batches to remove.
+ Files are grouped by column family and then by regions. When lots of files, delete whole
region dirs rather than individual files if all past the TTL (this would give us a bit of
NN 'batching').  See below:

{code}
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/17ac14014ffaf06eec8f4ca8f8a8093c
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/17ac14014ffaf06eec8f4ca8f8a8093c/meta
-rw-r--r--   3 stack supergroup   53257814 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/17ac14014ffaf06eec8f4ca8f8a8093c/meta/82b3d9f0ed0e4787a75036cf3e3e7165
-rw-r--r--   3 stack supergroup   53144343 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/17ac14014ffaf06eec8f4ca8f8a8093c/meta/8e3652741bc044bb8de90d3da44e73da
-rw-r--r--   3 stack supergroup  155406596 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/17ac14014ffaf06eec8f4ca8f8a8093c/meta/c81d8bd73eba41049a24065fe8a92335
-rw-r--r--   3 stack supergroup   40131260 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/17ac14014ffaf06eec8f4ca8f8a8093c/meta/e10bdea983744cbe9499a7f2a1ac73af
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/1addf0a136f3a461323dad87a5229838
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/1addf0a136f3a461323dad87a5229838/meta
-rw-r--r--   3 stack supergroup   53229613 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/1addf0a136f3a461323dad87a5229838/meta/2928456b642d4bed8432ff215c3adc28
-rw-r--r--   3 stack supergroup   53134686 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/1addf0a136f3a461323dad87a5229838/meta/515453c6dbd54e228932dd1ca4a3cd66
-rw-r--r--   3 stack supergroup  155390925 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/1addf0a136f3a461323dad87a5229838/meta/61d591035d6f4950a0fd974223e1defb
-rw-r--r--   3 stack supergroup   40082545 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/1addf0a136f3a461323dad87a5229838/meta/a03366c8862143d4bf77b81a83c73872
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/4fb408938e6e26021159b6f66239edf3
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/4fb408938e6e26021159b6f66239edf3/meta
-rw-r--r--   3 stack supergroup   53305889 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/4fb408938e6e26021159b6f66239edf3/meta/1132adf2d4a2422f8781502e1055d6d1
-rw-r--r--   3 stack supergroup   53146535 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/4fb408938e6e26021159b6f66239edf3/meta/2e988c9287a0412194f1f9050ded934b
-rw-r--r--   3 stack supergroup   40005595 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/4fb408938e6e26021159b6f66239edf3/meta/3fdd18de2eb2488f84f90846c0f6f578
-rw-r--r--   3 stack supergroup  155409996 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/4fb408938e6e26021159b6f66239edf3/meta/84750360a0bc463c9fa8023355da37ad
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/967b45186c5cf3e8e61ad9babd61768e
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/967b45186c5cf3e8e61ad9babd61768e/meta
-rw-r--r--   3 stack supergroup   36320770 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/967b45186c5cf3e8e61ad9babd61768e/meta/6d4baec2118b4aa482e283f4972f4b65
-rw-r--r--   3 stack supergroup  157346642 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/967b45186c5cf3e8e61ad9babd61768e/meta/7b1e80e2c26848e3bdb43abc7377bd42
-rw-r--r--   3 stack supergroup   55010361 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/967b45186c5cf3e8e61ad9babd61768e/meta/8f91251906c0407ba0d26f6c5e78baef
-rw-r--r--   3 stack supergroup   53156631 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/967b45186c5cf3e8e61ad9babd61768e/meta/d51d9d2f7fab407eb196b08555304ffc
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/e8441872044807da79dab2130c1337b2
drwxr-xr-x   - stack supergroup          0 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/e8441872044807da79dab2130c1337b2/meta
-rw-r--r--   3 stack supergroup   40012654 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/e8441872044807da79dab2130c1337b2/meta/30395d1e04b2427aa9665817b85ca593
-rw-r--r--   3 stack supergroup   53248791 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/e8441872044807da79dab2130c1337b2/meta/36785dda11f048c2ab4215329aed189d
-rw-r--r--   3 stack supergroup  155438135 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/e8441872044807da79dab2130c1337b2/meta/5e811190c0b546598ce46f470a5d0d17
-rw-r--r--   3 stack supergroup   53139133 2014-12-03 12:46 /hbase/archive/data/default/IntegrationTestBigLinkedList/e8441872044807da79dab2130c1337b2/meta/bb1b9c541bde41f7a2c287c62e0e3caa
{code}

> Archive cleaner cannot keep up; it maxes out at about 400k deletes/hour
> -----------------------------------------------------------------------
>
>                 Key: HBASE-12626
>                 URL: https://issues.apache.org/jira/browse/HBASE-12626
>             Project: HBase
>          Issue Type: Improvement
>          Components: master, scaling
>    Affects Versions: 0.94.25
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>
> On big clusters, it is possible to overrun the archive cleaning thread.  Make it able
to do more work per cycle when needed.
> We saw this on a user's cluster. The rate at which files were being moved to the archive
exceeded our delete rate such that the archive had tens of millions of files putting a friction
on all cluster ops.
> The cluster had ~500 nodes.  It that was RAM constrained (other processes on box also
need RAM). Over a period of days, the loading was thrown off kilter because it started taking
double writes going from one schema to another (Cluster was running hot before the double
loading).  The master was deleting an archived file every 9ms on average, about 400k deletes
an hour.  The constrained RAM and their having 4-5 column famiilies had them creating files
in excess of this rate so we backed up.
> For some helpful background/input, see the dev thread http://search-hadoop.com/m/DHED4UYSF9



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message