hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Data taking up too much space when put into HBase
Date Thu, 11 Nov 2010 17:15:06 GMT
Oh I see, you are using 4 families. An important thing to know (and
it's not super obvious) is that the regions flush on the total size of
the memstore across all families (there's one memstore per family,
learn more here
http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html).

This is actually a deficiency which will be solved in the context of
https://issues.apache.org/jira/browse/HBASE-3149

Generally, I rarely see any reason to use more than 1 family. You
really have to be in a case where the stored data is very different in
nature and requires specific family-level configurations. Here across
our 100+ tables, only 3-4 have more than one family and I'm sure that
number should be lower.

J-D

On Thu, Nov 11, 2010 at 12:54 AM, Hari Sreekumar
<hsreekumar@clickable.com> wrote:
> Here's the output of lsr on one of the tables:
>
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1102232448
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:33
> /hbase/Webevent/1102232448/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1102232448/Channel
> -rw-r--r--   3 hadoop supergroup   16943616 2010-11-11 13:33
> /hbase/Webevent/1102232448/Channel/7714679806810147132
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1102232448/Customer
> -rw-r--r--   3 hadoop supergroup   19089809 2010-11-11 13:33
> /hbase/Webevent/1102232448/Customer/228422950590673569
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1102232448/Event
> -rw-r--r--   3 hadoop supergroup   96925019 2010-11-11 13:33
> /hbase/Webevent/1102232448/Event/3246797304454611713
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/1102232448/User
> -rw-r--r--   3 hadoop supergroup  176008329 2010-11-11 13:33
> /hbase/Webevent/1102232448/User/6713166405821540696
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/1102232448/http
> -rw-r--r--   3 hadoop supergroup   36644077 2010-11-11 13:34
> /hbase/Webevent/1102232448/http/5528514474393215140
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/1181349092
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:40
> /hbase/Webevent/1181349092/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/1181349092/Channel
> -rw-r--r--   3 hadoop supergroup   14203831 2010-11-11 13:40
> /hbase/Webevent/1181349092/Channel/1711324265142021994
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/1181349092/Customer
> -rw-r--r--   3 hadoop supergroup   14091927 2010-11-11 13:40
> /hbase/Webevent/1181349092/Customer/3269372098573435637
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/1181349092/Event
> -rw-r--r--   3 hadoop supergroup   80842368 2010-11-11 13:40
> /hbase/Webevent/1181349092/Event/1632526964097525926
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:41
> /hbase/Webevent/1181349092/User
> -rw-r--r--   3 hadoop supergroup  146490419 2010-11-11 13:40
> /hbase/Webevent/1181349092/User/723684665063798772
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:41
> /hbase/Webevent/1181349092/http
> -rw-r--r--   3 hadoop supergroup   27612664 2010-11-11 13:41
> /hbase/Webevent/1181349092/http/3591070734425406504
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
> /hbase/Webevent/124990928
> -rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:28
> /hbase/Webevent/124990928/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
> /hbase/Webevent/124990928/Channel
> -rw-r--r--   3 hadoop supergroup   23700865 2010-11-11 13:35
> /hbase/Webevent/124990928/Channel/3488091559288595522
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
> /hbase/Webevent/124990928/Customer
> -rw-r--r--   3 hadoop supergroup   23572454 2010-11-11 13:35
> /hbase/Webevent/124990928/Customer/522070966307001888
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
> /hbase/Webevent/124990928/Event
> -rw-r--r--   3 hadoop supergroup  126857284 2010-11-11 13:35
> /hbase/Webevent/124990928/Event/8659573512216796018
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
> /hbase/Webevent/124990928/User
> -rw-r--r--   3 hadoop supergroup  229590074 2010-11-11 13:36
> /hbase/Webevent/124990928/User/4169913968975354294
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
> /hbase/Webevent/124990928/http
> -rw-r--r--   3 hadoop supergroup   43849622 2010-11-11 13:36
> /hbase/Webevent/124990928/http/798925777717846362
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:22
> /hbase/Webevent/13518424
> -rw-r--r--   3 hadoop supergroup       2316 2010-11-11 13:22
> /hbase/Webevent/13518424/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:24
> /hbase/Webevent/13518424/Channel
> -rw-r--r--   3 hadoop supergroup   11192244 2010-11-11 13:24
> /hbase/Webevent/13518424/Channel/6283534518250465269
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:24
> /hbase/Webevent/13518424/Customer
> -rw-r--r--   3 hadoop supergroup   16335757 2010-11-11 13:24
> /hbase/Webevent/13518424/Customer/8233555538562313638
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
> /hbase/Webevent/13518424/Event
> -rw-r--r--   3 hadoop supergroup   86782869 2010-11-11 13:24
> /hbase/Webevent/13518424/Event/7296313542067955537
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
> /hbase/Webevent/13518424/User
> -rw-r--r--   3 hadoop supergroup  157614762 2010-11-11 13:25
> /hbase/Webevent/13518424/User/5713897981539665344
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
> /hbase/Webevent/13518424/http
> -rw-r--r--   3 hadoop supergroup   31036461 2010-11-11 13:25
> /hbase/Webevent/13518424/http/3276765473089850908
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:22
> /hbase/Webevent/1397796225
> -rw-r--r--   3 hadoop supergroup       2144 2010-11-11 13:22
> /hbase/Webevent/1397796225/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
> /hbase/Webevent/1397796225/Channel
> -rw-r--r--   3 hadoop supergroup    3937460 2010-11-11 13:30
> /hbase/Webevent/1397796225/Channel/3684194843745008101
> -rw-r--r--   3 hadoop supergroup   13426908 2010-11-11 13:27
> /hbase/Webevent/1397796225/Channel/5763776518727398923
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
> /hbase/Webevent/1397796225/Customer
> -rw-r--r--   3 hadoop supergroup    9358001 2010-11-11 13:30
> /hbase/Webevent/1397796225/Customer/2373893879659383981
> -rw-r--r--   3 hadoop supergroup   15152448 2010-11-11 13:23
> /hbase/Webevent/1397796225/Customer/5404281688196690956
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/1397796225/Event
> -rw-r--r--   3 hadoop supergroup   49691275 2010-11-11 13:30
> /hbase/Webevent/1397796225/Event/1611219478516160819
> -rw-r--r--   3 hadoop supergroup   80075191 2010-11-11 13:23
> /hbase/Webevent/1397796225/Event/4491108423840726530
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/1397796225/User
> -rw-r--r--   3 hadoop supergroup  235564578 2010-11-11 13:31
> /hbase/Webevent/1397796225/User/1070607442453415896
> -rw-r--r--   3 hadoop supergroup  145355910 2010-11-11 13:23
> /hbase/Webevent/1397796225/User/6446151707620200218
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
> /hbase/Webevent/1397796225/http
> -rw-r--r--   3 hadoop supergroup   46665707 2010-11-11 13:31
> /hbase/Webevent/1397796225/http/2613117415168100829
> -rw-r--r--   3 hadoop supergroup   28997988 2010-11-11 13:24
> /hbase/Webevent/1397796225/http/7620282531029987336
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
> /hbase/Webevent/1568886745
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:30
> /hbase/Webevent/1568886745/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
> /hbase/Webevent/1568886745/Channel
> -rw-r--r--   3 hadoop supergroup   22384663 2010-11-11 13:32
> /hbase/Webevent/1568886745/Channel/3092028782443043693
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
> /hbase/Webevent/1568886745/Customer
> -rw-r--r--   3 hadoop supergroup   24060024 2010-11-11 13:32
> /hbase/Webevent/1568886745/Customer/2143995643997658656
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
> /hbase/Webevent/1568886745/Event
> -rw-r--r--   3 hadoop supergroup  111172989 2010-11-11 13:32
> /hbase/Webevent/1568886745/Event/606180646892333139
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1568886745/User
> -rw-r--r--   3 hadoop supergroup  201627486 2010-11-11 13:32
> /hbase/Webevent/1568886745/User/1159084185112718235
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1568886745/http
> -rw-r--r--   3 hadoop supergroup   42824881 2010-11-11 13:33
> /hbase/Webevent/1568886745/http/3005498889980823864
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
> /hbase/Webevent/1585185360
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:32
> /hbase/Webevent/1585185360/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1585185360/Channel
> -rw-r--r--   3 hadoop supergroup   13146621 2010-11-11 13:38
> /hbase/Webevent/1585185360/Channel/2384148253824087933
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1585185360/Customer
> -rw-r--r--   3 hadoop supergroup   17772527 2010-11-11 13:38
> /hbase/Webevent/1585185360/Customer/7079893521022823531
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:39
> /hbase/Webevent/1585185360/Event
> -rw-r--r--   3 hadoop supergroup   97860459 2010-11-11 13:38
> /hbase/Webevent/1585185360/Event/4129421247504808018
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:39
> /hbase/Webevent/1585185360/User
> -rw-r--r--   3 hadoop supergroup  177262872 2010-11-11 13:39
> /hbase/Webevent/1585185360/User/5689647586095222756
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:39
> /hbase/Webevent/1585185360/http
> -rw-r--r--   3 hadoop supergroup   38392938 2010-11-11 13:39
> /hbase/Webevent/1585185360/http/1513015171284860625
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/1679169023
> -rw-r--r--   3 hadoop supergroup       1970 2010-11-11 13:31
> /hbase/Webevent/1679169023/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/1679169023/Channel
> -rw-r--r--   3 hadoop supergroup   16691718 2010-11-11 13:37
> /hbase/Webevent/1679169023/Channel/3995013105248642215
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/1679169023/Customer
> -rw-r--r--   3 hadoop supergroup   18627546 2010-11-11 13:37
> /hbase/Webevent/1679169023/Customer/2697135409291299740
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/1679169023/Event
> -rw-r--r--   3 hadoop supergroup   97721412 2010-11-11 13:37
> /hbase/Webevent/1679169023/Event/5517850771377063599
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1679169023/User
> -rw-r--r--   3 hadoop supergroup  177198181 2010-11-11 13:37
> /hbase/Webevent/1679169023/User/1664697801534568988
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1679169023/http
> -rw-r--r--   3 hadoop supergroup   35558386 2010-11-11 13:38
> /hbase/Webevent/1679169023/http/2236900881608337670
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
> /hbase/Webevent/1837902643
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:30
> /hbase/Webevent/1837902643/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1837902643/Channel
> -rw-r--r--   3 hadoop supergroup   12956819 2010-11-11 13:33
> /hbase/Webevent/1837902643/Channel/7551397343290053516
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1837902643/Customer
> -rw-r--r--   3 hadoop supergroup   18017948 2010-11-11 13:33
> /hbase/Webevent/1837902643/Customer/1637842838964675843
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/1837902643/Event
> -rw-r--r--   3 hadoop supergroup   99238886 2010-11-11 13:33
> /hbase/Webevent/1837902643/Event/4961580175946952300
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/1837902643/User
> -rw-r--r--   3 hadoop supergroup  179431668 2010-11-11 13:33
> /hbase/Webevent/1837902643/User/8513763763938668916
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/1837902643/http
> -rw-r--r--   3 hadoop supergroup   35275755 2010-11-11 13:34
> /hbase/Webevent/1837902643/http/1801439100480395261
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
> /hbase/Webevent/1840258192
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:25
> /hbase/Webevent/1840258192/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/1840258192/Channel
> -rw-r--r--   3 hadoop supergroup   15810928 2010-11-11 13:34
> /hbase/Webevent/1840258192/Channel/8758451310929982789
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/1840258192/Customer
> -rw-r--r--   3 hadoop supergroup   16184063 2010-11-11 13:34
> /hbase/Webevent/1840258192/Customer/8209107027540853853
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
> /hbase/Webevent/1840258192/Event
> -rw-r--r--   3 hadoop supergroup   89893065 2010-11-11 13:34
> /hbase/Webevent/1840258192/Event/2507733338503153306
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
> /hbase/Webevent/1840258192/User
> -rw-r--r--   3 hadoop supergroup  162202298 2010-11-11 13:35
> /hbase/Webevent/1840258192/User/3877054643528147835
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
> /hbase/Webevent/1840258192/http
> -rw-r--r--   3 hadoop supergroup   30458950 2010-11-11 13:35
> /hbase/Webevent/1840258192/http/7057895626422451135
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
> /hbase/Webevent/1857066524
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:28
> /hbase/Webevent/1857066524/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
> /hbase/Webevent/1857066524/Channel
> -rw-r--r--   3 hadoop supergroup   17158229 2010-11-11 13:28
> /hbase/Webevent/1857066524/Channel/660294007043817390
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
> /hbase/Webevent/1857066524/Customer
> -rw-r--r--   3 hadoop supergroup   17982120 2010-11-11 13:28
> /hbase/Webevent/1857066524/Customer/8154314358497892797
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
> /hbase/Webevent/1857066524/Event
> -rw-r--r--   3 hadoop supergroup  103807737 2010-11-11 13:28
> /hbase/Webevent/1857066524/Event/8608458148878068560
> -rw-r--r--   3 hadoop supergroup  103807737 2010-11-11 13:36
> /hbase/Webevent/1857066524/Event/8753716512365715611
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:29
> /hbase/Webevent/1857066524/User
> -rw-r--r--   3 hadoop supergroup  188208796 2010-11-11 13:28
> /hbase/Webevent/1857066524/User/5807656088473870598
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:29
> /hbase/Webevent/1857066524/http
> -rw-r--r--   3 hadoop supergroup   35830676 2010-11-11 13:29
> /hbase/Webevent/1857066524/http/4192260931766222885
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/1954991296
> -rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:31
> /hbase/Webevent/1954991296/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1954991296/Channel
> -rw-r--r--   3 hadoop supergroup   14723821 2010-11-11 13:38
> /hbase/Webevent/1954991296/Channel/1271796192395132719
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1954991296/Customer
> -rw-r--r--   3 hadoop supergroup   16998002 2010-11-11 13:38
> /hbase/Webevent/1954991296/Customer/1871613240079217431
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1954991296/Event
> -rw-r--r--   3 hadoop supergroup   90132913 2010-11-11 13:38
> /hbase/Webevent/1954991296/Event/8627908912432238564
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1954991296/User
> -rw-r--r--   3 hadoop supergroup  163362248 2010-11-11 13:38
> /hbase/Webevent/1954991296/User/8343583184278031381
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
> /hbase/Webevent/1954991296/http
> -rw-r--r--   3 hadoop supergroup   37650515 2010-11-11 13:38
> /hbase/Webevent/1954991296/http/783502764043910698
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:16
> /hbase/Webevent/387441199
> -rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:16
> /hbase/Webevent/387441199/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:23
> /hbase/Webevent/387441199/Channel
> -rw-r--r--   3 hadoop supergroup    8751094 2010-11-11 13:22
> /hbase/Webevent/387441199/Channel/6907788666949153760
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:23
> /hbase/Webevent/387441199/Customer
> -rw-r--r--   3 hadoop supergroup   16526400 2010-11-11 13:23
> /hbase/Webevent/387441199/Customer/52924882214004995
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:23
> /hbase/Webevent/387441199/Event
> -rw-r--r--   3 hadoop supergroup   96466783 2010-11-11 13:23
> /hbase/Webevent/387441199/Event/991918398642333797
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:24
> /hbase/Webevent/387441199/User
> -rw-r--r--   3 hadoop supergroup  173755411 2010-11-11 13:23
> /hbase/Webevent/387441199/User/3697716047653972271
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:22
> /hbase/Webevent/387441199/http
> -rw-r--r--   3 hadoop supergroup   29164625 2010-11-11 13:19
> /hbase/Webevent/387441199/http/2172660655272329198
> -rw-r--r--   3 hadoop supergroup    3505176 2010-11-11 13:22
> /hbase/Webevent/387441199/http/9190482934578742068
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
> /hbase/Webevent/480045516
> -rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:30
> /hbase/Webevent/480045516/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/480045516/Channel
> -rw-r--r--   3 hadoop supergroup   14777812 2010-11-11 13:37
> /hbase/Webevent/480045516/Channel/2328066899305806515
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/480045516/Customer
> -rw-r--r--   3 hadoop supergroup   18953627 2010-11-11 13:37
> /hbase/Webevent/480045516/Customer/2078047623290175963
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/480045516/Event
> -rw-r--r--   3 hadoop supergroup  104229664 2010-11-11 13:37
> /hbase/Webevent/480045516/Event/910211247163239598
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/480045516/User
> -rw-r--r--   3 hadoop supergroup  189096799 2010-11-11 13:37
> /hbase/Webevent/480045516/User/5717389634644419119
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
> /hbase/Webevent/480045516/http
> -rw-r--r--   3 hadoop supergroup   36533404 2010-11-11 13:37
> /hbase/Webevent/480045516/http/8604372036650962237
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/601109706
> -rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:40
> /hbase/Webevent/601109706/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/601109706/Channel
> -rw-r--r--   3 hadoop supergroup   14155967 2010-11-11 13:40
> /hbase/Webevent/601109706/Channel/1819667230290028427
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/601109706/Customer
> -rw-r--r--   3 hadoop supergroup   14563111 2010-11-11 13:40
> /hbase/Webevent/601109706/Customer/7336170720169514891
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/601109706/Event
> -rw-r--r--   3 hadoop supergroup   82278013 2010-11-11 13:40
> /hbase/Webevent/601109706/Event/5064894617590864583
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/601109706/User
> -rw-r--r--   3 hadoop supergroup  149299853 2010-11-11 13:40
> /hbase/Webevent/601109706/User/5997879119834564841
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
> /hbase/Webevent/601109706/http
> -rw-r--r--   3 hadoop supergroup   29266049 2010-11-11 13:40
> /hbase/Webevent/601109706/http/3987271255931462679
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
> /hbase/Webevent/666508206
> -rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:25
> /hbase/Webevent/666508206/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/666508206/Channel
> -rw-r--r--   3 hadoop supergroup   22727461 2010-11-11 13:33
> /hbase/Webevent/666508206/Channel/9154587641511700292
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
> /hbase/Webevent/666508206/Customer
> -rw-r--r--   3 hadoop supergroup   23277615 2010-11-11 13:33
> /hbase/Webevent/666508206/Customer/3760018687145755911
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/666508206/Event
> -rw-r--r--   3 hadoop supergroup  111133668 2010-11-11 13:33
> /hbase/Webevent/666508206/Event/3598650053650721687
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/666508206/User
> -rw-r--r--   3 hadoop supergroup  201631388 2010-11-11 13:34
> /hbase/Webevent/666508206/User/3597127170470234124
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/666508206/http
> -rw-r--r--   3 hadoop supergroup   39920111 2010-11-11 13:34
> /hbase/Webevent/666508206/http/1455502897668123089
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
> /hbase/Webevent/717393157
> -rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:32
> /hbase/Webevent/717393157/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/717393157/Channel
> -rw-r--r--   3 hadoop supergroup    7937724 2010-11-11 13:34
> /hbase/Webevent/717393157/Channel/4038125755496042580
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/717393157/Customer
> -rw-r--r--   3 hadoop supergroup   14666396 2010-11-11 13:34
> /hbase/Webevent/717393157/Customer/8406371944316504992
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
> /hbase/Webevent/717393157/Event
> -rw-r--r--   3 hadoop supergroup   85611423 2010-11-11 13:34
> /hbase/Webevent/717393157/Event/127456153926503346
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
> /hbase/Webevent/717393157/User
> -rw-r--r--   3 hadoop supergroup  154335622 2010-11-11 13:34
> /hbase/Webevent/717393157/User/7421172344231467438
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
> /hbase/Webevent/717393157/http
> -rw-r--r--   3 hadoop supergroup   28943243 2010-11-11 13:35
> /hbase/Webevent/717393157/http/7543152081662309456
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
> /hbase/Webevent/902882312
> -rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:30
> /hbase/Webevent/902882312/.regioninfo
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
> /hbase/Webevent/902882312/Channel
> -rw-r--r--   3 hadoop supergroup    9541469 2010-11-11 13:30
> /hbase/Webevent/902882312/Channel/3254461494206070427
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/902882312/Customer
> -rw-r--r--   3 hadoop supergroup   16270772 2010-11-11 13:30
> /hbase/Webevent/902882312/Customer/3583245475353507819
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/902882312/Event
> -rw-r--r--   3 hadoop supergroup   90805116 2010-11-11 13:31
> /hbase/Webevent/902882312/Event/1032140072520109551
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/902882312/User
> -rw-r--r--   3 hadoop supergroup  164990613 2010-11-11 13:31
> /hbase/Webevent/902882312/User/5112158281218703912
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
> /hbase/Webevent/902882312/http
> -rw-r--r--   3 hadoop supergroup   38405659 2010-11-11 13:31
> /hbase/Webevent/902882312/http/5928256232381135445
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:41
> /hbase/Webevent/compaction.dir
> drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
> /hbase/Webevent/compaction.dir/1857066524
> -rw-r--r--   3 hadoop supergroup  153276719 2010-11-11 13:36
> /hbase/Webevent/compaction.dir/1857066524/8008135349377513409
>
> There are many smaller files of sizes < 20 MB which might be actually taking
> up 64*3=192 MB after replication. And even for the larger files, a file of
> 129 MB would use up 3 blocks right? Or is it somehow optimized to minimize
> space usage?
>
> On Wed, Nov 10, 2010 at 11:07 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> Can you pastebin the output of the lsr command on the table's dir?
>>
>> Thx
>>
>> J-D
>>
>> On Tue, Nov 9, 2010 at 10:54 PM, Hari Sreekumar
>> <hsreekumar@clickable.com> wrote:
>> > I checked the "browse filesystem" link in the web interface (50070).
>> HBase
>> > creates a directly named after the table ,and in the directory, there are
>> > files which are 5-6 MB in size, on average. Some are in kbs, and there
>> are
>> > some of 12-13 MB size, but most are around  6 MB. I was thinking these
>> files
>> > are stored in 64 MB blocks, leading to the space usage.
>> >
>> > hari
>> >
>> > On Wed, Nov 10, 2010 at 11:56 AM, Jean-Daniel Cryans <
>> jdcryans@apache.org>wrote:
>> >
>> >> I'm pretty sure that's not how it's reported by the "du" command, but
>> >> I wouldn't expect to see files of 5MB on average. Can you be more
>> >> specific?
>> >>
>> >> J-D
>> >>
>> >> On Tue, Nov 9, 2010 at 9:58 PM, Hari Sreekumar <
>> hsreekumar@clickable.com>
>> >> wrote:
>> >> > Ah, so the bloat is not because of the files being 5-6 MB in size?
>> >> Wouldn't
>> >> > a 6 MB file occupy 64 MB if I set block size as 64 MB?
>> >> >
>> >> > hari
>> >> >
>> >> > On Wed, Nov 10, 2010 at 11:16 AM, Jean-Daniel Cryans <
>> >> jdcryans@apache.org>wrote:
>> >> >
>> >> >> Each value is stored with it's full key e.g. row key + family +
>> >> >> qualifier + timestamp + offsets. You don't give any information
>> >> >> regarding how you stored the data, but if you have large enough
keys
>> >> >> then it should easily explain the bloat.
>> >> >>
>> >> >> J-D
>> >> >>
>> >> >> On Tue, Nov 9, 2010 at 9:21 PM, Hari Sreekumar <
>> >> hsreekumar@clickable.com>
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> >     Data seems to be taking up too much space when I put
into
>> HBase.
>> >> e.g,
>> >> >> I
>> >> >> > have a 2 GB text file which seems to be taking up ~70 GB when
I
>> dump
>> >> into
>> >> >> > HBase. I have block size set to 64 MB and replication=3, which
I
>> think
>> >> is
>> >> >> > the possible reason for this expansion. But if that is the
case,
>> how
>> >> can
>> >> >> I
>> >> >> > prevent it? Decreasing the block size will have a negative
impact
>> on
>> >> >> > performance, so is there a way I can increase the average
size on
>> >> >> > HBase-created  files to be comparable to 64 MB. Right now
they are
>> ~5
>> >> MB
>> >> >> on
>> >> >> > average. Or is this an entirely different thing at work here?
>> >> >> >
>> >> >> > thanks,
>> >> >> > hari
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>

Mime
View raw message