kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Martynov <mr.xk...@gmail.com>
Subject Re: WAL size estimation
Date Wed, 19 Jun 2019 07:49:01 GMT
Hi Todd, thanks for the answer!

> Any chance you've done something like copy the files away and back that
might cause them to lose their sparseness?

No, I don't think so. Recently we experienced some problems with stability
with Kudu, and ran rebalance a couple of times, if this related. But we
never used fs commands like cp/mv against Kudu dirs.

I ran du on all-WALs dir:
# du -sh /mnt/data01/kudu-tserver-wal/
12G     /mnt/data01/kudu-tserver-wal/

# du -sh --apparent-size /mnt/data01/kudu-tserver-wal/
25G     /mnt/data01/kudu-tserver-wal/

And on WAL with a many indexes:
# du -sh --apparent-size
/mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f
306M    /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f

# du -sh /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f
296M    /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f


> Also, any chance you're using XFS here?

Yes, exactly XFS. We use CentOS 7.6.

What is interesting, there are no many holes in index files in
/mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f (WAL dir
that I mention before). Only single hole in single index file (of 13 files):
# xfs_bmap -v index.000000120
index.000000120:
 EXT: FILE-OFFSET      BLOCK-RANGE            AG AG-OFFSET          TOTAL
   0: [0..4231]:       1176541248..1176545479  2 (4429888..4434119)  4232
   1: [4232..9815]:    1176546592..1176552175  2 (4435232..4440815)  5584
   2: [9816..11583]:   1176552832..1176554599  2 (4441472..4443239)  1768
   3: [11584..13319]:  1176558672..1176560407  2 (4447312..4449047)  1736
   4: [13320..15239]:  1176565336..1176567255  2 (4453976..4455895)  1920
   5: [15240..17183]:  1176570776..1176572719  2 (4459416..4461359)  1944
   6: [17184..18999]:  1176575856..1176577671  2 (4464496..4466311)  1816
   7: [19000..20927]:  1176593552..1176595479  2 (4482192..4484119)  1928
   8: [20928..22703]:  1176599128..1176600903  2 (4487768..4489543)  1776
   9: [22704..24575]:  1176602704..1176604575  2 (4491344..4493215)  1872
  10: [24576..26495]:  1176611936..1176613855  2 (4500576..4502495)  1920
  11: [26496..26655]:  1176615040..1176615199  2 (4503680..4503839)   160
  12: [26656..46879]:  hole                                         20224

But in some other WAL I see like this:
# xfs_bmap -v
/mnt/data01/kudu-tserver-wal/wals/508ecdfa8904bdb97a02078a91822af/index.000000000
/mnt/data01/kudu-tserver-wal/wals/508ecdfa89054bdb97a02078a91822af/index.000000000:
 EXT: FILE-OFFSET      BLOCK-RANGE            AG AG-OFFSET        TOTAL
   0: [0..7]:          1758753776..1758753783  3 (586736..586743)     8
   1: [8..46879]:      hole                                       46872

Looks like there actually used only 8 blocks and all other blocks are the
hole.


So looks like I can use formulas with confidence.
Normal case: 8 MB/segment * 80 max segments * 2000 tablets = 1,280,000 MB =
~1.3 TB (+ some minor index overhead)
Worse case: 8 MB/segment * 1 segment * 2000 tablets = 1,280,000 MB = ~16 GB
(+ some minor index overhead)

Right?


ср, 19 июн. 2019 г. в 09:35, Todd Lipcon <todd@cloudera.com>:

> Hi Pavel,
>
> That's not quite expected. For example, on one of our test clusters here,
> we have about 65GB of WALs and about 1GB of index files. If I recall
> correctly, the index files store 8 bytes per WAL entry, so typically a
> couple orders of magnitude smaller than the WALs themselves.
>
> One thing is that the index files are sparse. Any chance you've done
> something like copy the files away and back that might cause them to lose
> their sparseness? If I use du --apparent-size on mine, it's total of about
> 180GB vs the 1GB of actual size.
>
> Also, any chance you're using XFS here? XFS sometimes likes to preallocate
> large amounts of data into files while they're open, and only frees it up
> if disk space is contended. I think you can use 'xfs_bmap' on an index file
> to see the allocation status, which might be interesting.
>
> -Todd
>
> On Tue, Jun 18, 2019 at 11:12 PM Pavel Martynov <mr.xkurt@gmail.com>
> wrote:
>
>> Hi guys!
>>
>> We want to buy SSDs for TServers WALs for our cluster. I'm working on
>> capacity estimation for this SSDs using "Getting Started with Kudu" book,
>> Chapter 4, Write-Ahead Log (
>> https://www.oreilly.com/library/view/getting-started-with/9781491980248/ch04.html
>> <https://www.oreilly.com/library/view/getting-started-with/9781491980248/ch04.html#idm139738927926240>
>> ).
>>
>> NB: we use default Kudu WAL configuration settings.
>>
>> There is a formula for worse-case:
>> 8 MB/segment * 80 max segments * 2000 tablets = 1,280,000 MB = ~1.3 TB
>>
>> So, this formula takes into account only segment files. But in our
>> cluster, I see that every segment file has >= 1 corresponding index files.
>> And every index file actually larger than segment file.
>>
>> Numbers from one of our nodes.
>> WALs count:
>> $ ls /mnt/data01/kudu-tserver-wal/wals/ | wc -l
>> 711
>>
>> Overall WAL size:
>> $ du -d 0 -h /mnt/data01/kudu-tserver-wal/
>> 13G     /mnt/data01/kudu-tserver-wal/
>>
>> Size of all segment files:
>> $ find /mnt/data01/kudu-tserver-wal/ -type f -name 'wal-*' -exec du -ch
>> {} + | grep total$
>> 6.1G    total
>>
>> Size of all index files:
>> $ find /mnt/data01/kudu-tserver-wal/ -type f -name 'index*' -exec du -ch
>> {} + | grep total$
>> 6.5G    total
>>
>> So I have questions.
>>
>> 1. How can I estimate the size of index files?
>> Looks like in our cluster size of index files approximately equal to size
>> segment files.
>>
>> 2. There is some WALs with more than one index files. For example:
>> $ ls -lh
>> /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f/
>> total 296M
>> -rw-r--r-- 1 root root  23M Jun 18 21:31 index.000000108
>> -rw-r--r-- 1 root root  23M Jun 18 21:41 index.000000109
>> -rw-r--r-- 1 root root  23M Jun 18 21:52 index.000000110
>> -rw-r--r-- 1 root root  23M Jun 18 22:10 index.000000111
>> -rw-r--r-- 1 root root  23M Jun 18 22:22 index.000000112
>> -rw-r--r-- 1 root root  23M Jun 18 22:35 index.000000113
>> -rw-r--r-- 1 root root  23M Jun 18 22:48 index.000000114
>> -rw-r--r-- 1 root root  23M Jun 18 23:01 index.000000115
>> -rw-r--r-- 1 root root  23M Jun 18 23:14 index.000000116
>> -rw-r--r-- 1 root root  23M Jun 18 23:27 index.000000117
>> -rw-r--r-- 1 root root  23M Jun 18 23:40 index.000000118
>> -rw-r--r-- 1 root root  23M Jun 18 23:52 index.000000119
>> -rw-r--r-- 1 root root  23M Jun 19 01:13 index.000000120
>> -rw-r--r-- 1 root root 8.0M Jun 19 01:13 wal-000007799
>>
>> Is this a normal situation?
>>
>> 3. Not a question. Please, consider adding documentation about the
>> estimation of WAL storage. Also, I can't found any mentions about index
>> files, except here
>> https://kudu.apache.org/docs/scaling_guide.html#file_descriptors.
>>
>> Thanks!
>>
>> --
>> with best regards, Pavel Martynov
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>


-- 
with best regards, Pavel Martynov

Mime
View raw message