hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michał Podsiadłowski <podsiadlow...@gmail.com>
Subject Re: Can't Put Row with Same ID Twice (if using custom timestamp)
Date Tue, 02 Feb 2010 12:24:19 GMT
Ryan,
is there a way to truncate whole table using similar (tombstone) mechanism?
What I need is to remove very quickly content of a table. Tables can be
significantly large - few 10s of K. so normal truncate with disabling
regions,
droping and then recreating table is not acceptable because thread is
blocked for that time and user will wait ages for response.
What I thought when I've read you post is I could put a row that would work
similarly to the tombstone.
During get I would just retrieve also that row and if empty or row timestamp
> then market timestamp then return value otherwise return null.
But problem is that i would have to run a periodical mechanism to do real
deletes and this should be done before major compaction.
Second concern is that each access requires 2 get operations so efficiently
would be to do this inside region servers.
Can you advice something?

Thanks,
Michal



2010/2/2 Ryan Rawson <ryanobjc@gmail.com>

> This is expected...
>
> To understand why, we need to look at how deletes are handled in
> HBase.  Since files in HDFS are immutable, we don't actually go
> through and remove data when you ask for a 'delete'.  Instead we
> insert a delete marker, at a given timestamp that says 'everything
> older than this time is gone'.  This delete marker (also known as
> tombstones in other systems) is an explicit entry and does not go away
> for a while (until the next major compaction).  During reads, we use
> the delete markers to suppress 'deleted data'.
>
> When you insert a row with a timestamp that overlaps with a delete
> marker like this, the effect is as you see.
>
> One way to "fix" this is:
> put
> delete
> major_compact 'table'
> put
>
> during a major compaction, we prune all delete records and their
> suppressed data leaving a nice and clean file with no deleted data nor
> markers.  But normally major compaction is run at most 1x a day, since
> on a larger cluster is is very heavy-weight - it must rewrite the
> entire region of data!
>
> Good luck!
> -ryan
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message