hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Sreekumar <hsreeku...@clickable.com>
Subject Re: Data taking up too much space when put into HBase
Date Thu, 11 Nov 2010 08:54:58 GMT
Here's the output of lsr on one of the tables:

drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1102232448
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:33
/hbase/Webevent/1102232448/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1102232448/Channel
-rw-r--r--   3 hadoop supergroup   16943616 2010-11-11 13:33
/hbase/Webevent/1102232448/Channel/7714679806810147132
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1102232448/Customer
-rw-r--r--   3 hadoop supergroup   19089809 2010-11-11 13:33
/hbase/Webevent/1102232448/Customer/228422950590673569
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1102232448/Event
-rw-r--r--   3 hadoop supergroup   96925019 2010-11-11 13:33
/hbase/Webevent/1102232448/Event/3246797304454611713
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/1102232448/User
-rw-r--r--   3 hadoop supergroup  176008329 2010-11-11 13:33
/hbase/Webevent/1102232448/User/6713166405821540696
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/1102232448/http
-rw-r--r--   3 hadoop supergroup   36644077 2010-11-11 13:34
/hbase/Webevent/1102232448/http/5528514474393215140
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/1181349092
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:40
/hbase/Webevent/1181349092/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/1181349092/Channel
-rw-r--r--   3 hadoop supergroup   14203831 2010-11-11 13:40
/hbase/Webevent/1181349092/Channel/1711324265142021994
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/1181349092/Customer
-rw-r--r--   3 hadoop supergroup   14091927 2010-11-11 13:40
/hbase/Webevent/1181349092/Customer/3269372098573435637
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/1181349092/Event
-rw-r--r--   3 hadoop supergroup   80842368 2010-11-11 13:40
/hbase/Webevent/1181349092/Event/1632526964097525926
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:41
/hbase/Webevent/1181349092/User
-rw-r--r--   3 hadoop supergroup  146490419 2010-11-11 13:40
/hbase/Webevent/1181349092/User/723684665063798772
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:41
/hbase/Webevent/1181349092/http
-rw-r--r--   3 hadoop supergroup   27612664 2010-11-11 13:41
/hbase/Webevent/1181349092/http/3591070734425406504
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
/hbase/Webevent/124990928
-rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:28
/hbase/Webevent/124990928/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
/hbase/Webevent/124990928/Channel
-rw-r--r--   3 hadoop supergroup   23700865 2010-11-11 13:35
/hbase/Webevent/124990928/Channel/3488091559288595522
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
/hbase/Webevent/124990928/Customer
-rw-r--r--   3 hadoop supergroup   23572454 2010-11-11 13:35
/hbase/Webevent/124990928/Customer/522070966307001888
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
/hbase/Webevent/124990928/Event
-rw-r--r--   3 hadoop supergroup  126857284 2010-11-11 13:35
/hbase/Webevent/124990928/Event/8659573512216796018
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
/hbase/Webevent/124990928/User
-rw-r--r--   3 hadoop supergroup  229590074 2010-11-11 13:36
/hbase/Webevent/124990928/User/4169913968975354294
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
/hbase/Webevent/124990928/http
-rw-r--r--   3 hadoop supergroup   43849622 2010-11-11 13:36
/hbase/Webevent/124990928/http/798925777717846362
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:22
/hbase/Webevent/13518424
-rw-r--r--   3 hadoop supergroup       2316 2010-11-11 13:22
/hbase/Webevent/13518424/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:24
/hbase/Webevent/13518424/Channel
-rw-r--r--   3 hadoop supergroup   11192244 2010-11-11 13:24
/hbase/Webevent/13518424/Channel/6283534518250465269
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:24
/hbase/Webevent/13518424/Customer
-rw-r--r--   3 hadoop supergroup   16335757 2010-11-11 13:24
/hbase/Webevent/13518424/Customer/8233555538562313638
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
/hbase/Webevent/13518424/Event
-rw-r--r--   3 hadoop supergroup   86782869 2010-11-11 13:24
/hbase/Webevent/13518424/Event/7296313542067955537
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
/hbase/Webevent/13518424/User
-rw-r--r--   3 hadoop supergroup  157614762 2010-11-11 13:25
/hbase/Webevent/13518424/User/5713897981539665344
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
/hbase/Webevent/13518424/http
-rw-r--r--   3 hadoop supergroup   31036461 2010-11-11 13:25
/hbase/Webevent/13518424/http/3276765473089850908
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:22
/hbase/Webevent/1397796225
-rw-r--r--   3 hadoop supergroup       2144 2010-11-11 13:22
/hbase/Webevent/1397796225/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
/hbase/Webevent/1397796225/Channel
-rw-r--r--   3 hadoop supergroup    3937460 2010-11-11 13:30
/hbase/Webevent/1397796225/Channel/3684194843745008101
-rw-r--r--   3 hadoop supergroup   13426908 2010-11-11 13:27
/hbase/Webevent/1397796225/Channel/5763776518727398923
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
/hbase/Webevent/1397796225/Customer
-rw-r--r--   3 hadoop supergroup    9358001 2010-11-11 13:30
/hbase/Webevent/1397796225/Customer/2373893879659383981
-rw-r--r--   3 hadoop supergroup   15152448 2010-11-11 13:23
/hbase/Webevent/1397796225/Customer/5404281688196690956
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/1397796225/Event
-rw-r--r--   3 hadoop supergroup   49691275 2010-11-11 13:30
/hbase/Webevent/1397796225/Event/1611219478516160819
-rw-r--r--   3 hadoop supergroup   80075191 2010-11-11 13:23
/hbase/Webevent/1397796225/Event/4491108423840726530
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/1397796225/User
-rw-r--r--   3 hadoop supergroup  235564578 2010-11-11 13:31
/hbase/Webevent/1397796225/User/1070607442453415896
-rw-r--r--   3 hadoop supergroup  145355910 2010-11-11 13:23
/hbase/Webevent/1397796225/User/6446151707620200218
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
/hbase/Webevent/1397796225/http
-rw-r--r--   3 hadoop supergroup   46665707 2010-11-11 13:31
/hbase/Webevent/1397796225/http/2613117415168100829
-rw-r--r--   3 hadoop supergroup   28997988 2010-11-11 13:24
/hbase/Webevent/1397796225/http/7620282531029987336
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
/hbase/Webevent/1568886745
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:30
/hbase/Webevent/1568886745/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
/hbase/Webevent/1568886745/Channel
-rw-r--r--   3 hadoop supergroup   22384663 2010-11-11 13:32
/hbase/Webevent/1568886745/Channel/3092028782443043693
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
/hbase/Webevent/1568886745/Customer
-rw-r--r--   3 hadoop supergroup   24060024 2010-11-11 13:32
/hbase/Webevent/1568886745/Customer/2143995643997658656
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
/hbase/Webevent/1568886745/Event
-rw-r--r--   3 hadoop supergroup  111172989 2010-11-11 13:32
/hbase/Webevent/1568886745/Event/606180646892333139
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1568886745/User
-rw-r--r--   3 hadoop supergroup  201627486 2010-11-11 13:32
/hbase/Webevent/1568886745/User/1159084185112718235
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1568886745/http
-rw-r--r--   3 hadoop supergroup   42824881 2010-11-11 13:33
/hbase/Webevent/1568886745/http/3005498889980823864
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
/hbase/Webevent/1585185360
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:32
/hbase/Webevent/1585185360/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1585185360/Channel
-rw-r--r--   3 hadoop supergroup   13146621 2010-11-11 13:38
/hbase/Webevent/1585185360/Channel/2384148253824087933
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1585185360/Customer
-rw-r--r--   3 hadoop supergroup   17772527 2010-11-11 13:38
/hbase/Webevent/1585185360/Customer/7079893521022823531
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:39
/hbase/Webevent/1585185360/Event
-rw-r--r--   3 hadoop supergroup   97860459 2010-11-11 13:38
/hbase/Webevent/1585185360/Event/4129421247504808018
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:39
/hbase/Webevent/1585185360/User
-rw-r--r--   3 hadoop supergroup  177262872 2010-11-11 13:39
/hbase/Webevent/1585185360/User/5689647586095222756
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:39
/hbase/Webevent/1585185360/http
-rw-r--r--   3 hadoop supergroup   38392938 2010-11-11 13:39
/hbase/Webevent/1585185360/http/1513015171284860625
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/1679169023
-rw-r--r--   3 hadoop supergroup       1970 2010-11-11 13:31
/hbase/Webevent/1679169023/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/1679169023/Channel
-rw-r--r--   3 hadoop supergroup   16691718 2010-11-11 13:37
/hbase/Webevent/1679169023/Channel/3995013105248642215
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/1679169023/Customer
-rw-r--r--   3 hadoop supergroup   18627546 2010-11-11 13:37
/hbase/Webevent/1679169023/Customer/2697135409291299740
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/1679169023/Event
-rw-r--r--   3 hadoop supergroup   97721412 2010-11-11 13:37
/hbase/Webevent/1679169023/Event/5517850771377063599
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1679169023/User
-rw-r--r--   3 hadoop supergroup  177198181 2010-11-11 13:37
/hbase/Webevent/1679169023/User/1664697801534568988
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1679169023/http
-rw-r--r--   3 hadoop supergroup   35558386 2010-11-11 13:38
/hbase/Webevent/1679169023/http/2236900881608337670
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
/hbase/Webevent/1837902643
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:30
/hbase/Webevent/1837902643/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1837902643/Channel
-rw-r--r--   3 hadoop supergroup   12956819 2010-11-11 13:33
/hbase/Webevent/1837902643/Channel/7551397343290053516
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1837902643/Customer
-rw-r--r--   3 hadoop supergroup   18017948 2010-11-11 13:33
/hbase/Webevent/1837902643/Customer/1637842838964675843
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/1837902643/Event
-rw-r--r--   3 hadoop supergroup   99238886 2010-11-11 13:33
/hbase/Webevent/1837902643/Event/4961580175946952300
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/1837902643/User
-rw-r--r--   3 hadoop supergroup  179431668 2010-11-11 13:33
/hbase/Webevent/1837902643/User/8513763763938668916
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/1837902643/http
-rw-r--r--   3 hadoop supergroup   35275755 2010-11-11 13:34
/hbase/Webevent/1837902643/http/1801439100480395261
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
/hbase/Webevent/1840258192
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:25
/hbase/Webevent/1840258192/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/1840258192/Channel
-rw-r--r--   3 hadoop supergroup   15810928 2010-11-11 13:34
/hbase/Webevent/1840258192/Channel/8758451310929982789
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/1840258192/Customer
-rw-r--r--   3 hadoop supergroup   16184063 2010-11-11 13:34
/hbase/Webevent/1840258192/Customer/8209107027540853853
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
/hbase/Webevent/1840258192/Event
-rw-r--r--   3 hadoop supergroup   89893065 2010-11-11 13:34
/hbase/Webevent/1840258192/Event/2507733338503153306
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
/hbase/Webevent/1840258192/User
-rw-r--r--   3 hadoop supergroup  162202298 2010-11-11 13:35
/hbase/Webevent/1840258192/User/3877054643528147835
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
/hbase/Webevent/1840258192/http
-rw-r--r--   3 hadoop supergroup   30458950 2010-11-11 13:35
/hbase/Webevent/1840258192/http/7057895626422451135
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
/hbase/Webevent/1857066524
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:28
/hbase/Webevent/1857066524/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
/hbase/Webevent/1857066524/Channel
-rw-r--r--   3 hadoop supergroup   17158229 2010-11-11 13:28
/hbase/Webevent/1857066524/Channel/660294007043817390
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:28
/hbase/Webevent/1857066524/Customer
-rw-r--r--   3 hadoop supergroup   17982120 2010-11-11 13:28
/hbase/Webevent/1857066524/Customer/8154314358497892797
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
/hbase/Webevent/1857066524/Event
-rw-r--r--   3 hadoop supergroup  103807737 2010-11-11 13:28
/hbase/Webevent/1857066524/Event/8608458148878068560
-rw-r--r--   3 hadoop supergroup  103807737 2010-11-11 13:36
/hbase/Webevent/1857066524/Event/8753716512365715611
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:29
/hbase/Webevent/1857066524/User
-rw-r--r--   3 hadoop supergroup  188208796 2010-11-11 13:28
/hbase/Webevent/1857066524/User/5807656088473870598
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:29
/hbase/Webevent/1857066524/http
-rw-r--r--   3 hadoop supergroup   35830676 2010-11-11 13:29
/hbase/Webevent/1857066524/http/4192260931766222885
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/1954991296
-rw-r--r--   3 hadoop supergroup       2318 2010-11-11 13:31
/hbase/Webevent/1954991296/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1954991296/Channel
-rw-r--r--   3 hadoop supergroup   14723821 2010-11-11 13:38
/hbase/Webevent/1954991296/Channel/1271796192395132719
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1954991296/Customer
-rw-r--r--   3 hadoop supergroup   16998002 2010-11-11 13:38
/hbase/Webevent/1954991296/Customer/1871613240079217431
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1954991296/Event
-rw-r--r--   3 hadoop supergroup   90132913 2010-11-11 13:38
/hbase/Webevent/1954991296/Event/8627908912432238564
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1954991296/User
-rw-r--r--   3 hadoop supergroup  163362248 2010-11-11 13:38
/hbase/Webevent/1954991296/User/8343583184278031381
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:38
/hbase/Webevent/1954991296/http
-rw-r--r--   3 hadoop supergroup   37650515 2010-11-11 13:38
/hbase/Webevent/1954991296/http/783502764043910698
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:16
/hbase/Webevent/387441199
-rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:16
/hbase/Webevent/387441199/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:23
/hbase/Webevent/387441199/Channel
-rw-r--r--   3 hadoop supergroup    8751094 2010-11-11 13:22
/hbase/Webevent/387441199/Channel/6907788666949153760
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:23
/hbase/Webevent/387441199/Customer
-rw-r--r--   3 hadoop supergroup   16526400 2010-11-11 13:23
/hbase/Webevent/387441199/Customer/52924882214004995
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:23
/hbase/Webevent/387441199/Event
-rw-r--r--   3 hadoop supergroup   96466783 2010-11-11 13:23
/hbase/Webevent/387441199/Event/991918398642333797
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:24
/hbase/Webevent/387441199/User
-rw-r--r--   3 hadoop supergroup  173755411 2010-11-11 13:23
/hbase/Webevent/387441199/User/3697716047653972271
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:22
/hbase/Webevent/387441199/http
-rw-r--r--   3 hadoop supergroup   29164625 2010-11-11 13:19
/hbase/Webevent/387441199/http/2172660655272329198
-rw-r--r--   3 hadoop supergroup    3505176 2010-11-11 13:22
/hbase/Webevent/387441199/http/9190482934578742068
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
/hbase/Webevent/480045516
-rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:30
/hbase/Webevent/480045516/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/480045516/Channel
-rw-r--r--   3 hadoop supergroup   14777812 2010-11-11 13:37
/hbase/Webevent/480045516/Channel/2328066899305806515
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/480045516/Customer
-rw-r--r--   3 hadoop supergroup   18953627 2010-11-11 13:37
/hbase/Webevent/480045516/Customer/2078047623290175963
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/480045516/Event
-rw-r--r--   3 hadoop supergroup  104229664 2010-11-11 13:37
/hbase/Webevent/480045516/Event/910211247163239598
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/480045516/User
-rw-r--r--   3 hadoop supergroup  189096799 2010-11-11 13:37
/hbase/Webevent/480045516/User/5717389634644419119
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:37
/hbase/Webevent/480045516/http
-rw-r--r--   3 hadoop supergroup   36533404 2010-11-11 13:37
/hbase/Webevent/480045516/http/8604372036650962237
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/601109706
-rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:40
/hbase/Webevent/601109706/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/601109706/Channel
-rw-r--r--   3 hadoop supergroup   14155967 2010-11-11 13:40
/hbase/Webevent/601109706/Channel/1819667230290028427
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/601109706/Customer
-rw-r--r--   3 hadoop supergroup   14563111 2010-11-11 13:40
/hbase/Webevent/601109706/Customer/7336170720169514891
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/601109706/Event
-rw-r--r--   3 hadoop supergroup   82278013 2010-11-11 13:40
/hbase/Webevent/601109706/Event/5064894617590864583
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/601109706/User
-rw-r--r--   3 hadoop supergroup  149299853 2010-11-11 13:40
/hbase/Webevent/601109706/User/5997879119834564841
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:40
/hbase/Webevent/601109706/http
-rw-r--r--   3 hadoop supergroup   29266049 2010-11-11 13:40
/hbase/Webevent/601109706/http/3987271255931462679
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:25
/hbase/Webevent/666508206
-rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:25
/hbase/Webevent/666508206/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/666508206/Channel
-rw-r--r--   3 hadoop supergroup   22727461 2010-11-11 13:33
/hbase/Webevent/666508206/Channel/9154587641511700292
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:33
/hbase/Webevent/666508206/Customer
-rw-r--r--   3 hadoop supergroup   23277615 2010-11-11 13:33
/hbase/Webevent/666508206/Customer/3760018687145755911
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/666508206/Event
-rw-r--r--   3 hadoop supergroup  111133668 2010-11-11 13:33
/hbase/Webevent/666508206/Event/3598650053650721687
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/666508206/User
-rw-r--r--   3 hadoop supergroup  201631388 2010-11-11 13:34
/hbase/Webevent/666508206/User/3597127170470234124
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/666508206/http
-rw-r--r--   3 hadoop supergroup   39920111 2010-11-11 13:34
/hbase/Webevent/666508206/http/1455502897668123089
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:32
/hbase/Webevent/717393157
-rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:32
/hbase/Webevent/717393157/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/717393157/Channel
-rw-r--r--   3 hadoop supergroup    7937724 2010-11-11 13:34
/hbase/Webevent/717393157/Channel/4038125755496042580
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/717393157/Customer
-rw-r--r--   3 hadoop supergroup   14666396 2010-11-11 13:34
/hbase/Webevent/717393157/Customer/8406371944316504992
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:34
/hbase/Webevent/717393157/Event
-rw-r--r--   3 hadoop supergroup   85611423 2010-11-11 13:34
/hbase/Webevent/717393157/Event/127456153926503346
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
/hbase/Webevent/717393157/User
-rw-r--r--   3 hadoop supergroup  154335622 2010-11-11 13:34
/hbase/Webevent/717393157/User/7421172344231467438
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:35
/hbase/Webevent/717393157/http
-rw-r--r--   3 hadoop supergroup   28943243 2010-11-11 13:35
/hbase/Webevent/717393157/http/7543152081662309456
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
/hbase/Webevent/902882312
-rw-r--r--   3 hadoop supergroup       2317 2010-11-11 13:30
/hbase/Webevent/902882312/.regioninfo
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:30
/hbase/Webevent/902882312/Channel
-rw-r--r--   3 hadoop supergroup    9541469 2010-11-11 13:30
/hbase/Webevent/902882312/Channel/3254461494206070427
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/902882312/Customer
-rw-r--r--   3 hadoop supergroup   16270772 2010-11-11 13:30
/hbase/Webevent/902882312/Customer/3583245475353507819
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/902882312/Event
-rw-r--r--   3 hadoop supergroup   90805116 2010-11-11 13:31
/hbase/Webevent/902882312/Event/1032140072520109551
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/902882312/User
-rw-r--r--   3 hadoop supergroup  164990613 2010-11-11 13:31
/hbase/Webevent/902882312/User/5112158281218703912
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:31
/hbase/Webevent/902882312/http
-rw-r--r--   3 hadoop supergroup   38405659 2010-11-11 13:31
/hbase/Webevent/902882312/http/5928256232381135445
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:41
/hbase/Webevent/compaction.dir
drwxr-xr-x   - hadoop supergroup          0 2010-11-11 13:36
/hbase/Webevent/compaction.dir/1857066524
-rw-r--r--   3 hadoop supergroup  153276719 2010-11-11 13:36
/hbase/Webevent/compaction.dir/1857066524/8008135349377513409

There are many smaller files of sizes < 20 MB which might be actually taking
up 64*3=192 MB after replication. And even for the larger files, a file of
129 MB would use up 3 blocks right? Or is it somehow optimized to minimize
space usage?

On Wed, Nov 10, 2010 at 11:07 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> Can you pastebin the output of the lsr command on the table's dir?
>
> Thx
>
> J-D
>
> On Tue, Nov 9, 2010 at 10:54 PM, Hari Sreekumar
> <hsreekumar@clickable.com> wrote:
> > I checked the "browse filesystem" link in the web interface (50070).
> HBase
> > creates a directly named after the table ,and in the directory, there are
> > files which are 5-6 MB in size, on average. Some are in kbs, and there
> are
> > some of 12-13 MB size, but most are around  6 MB. I was thinking these
> files
> > are stored in 64 MB blocks, leading to the space usage.
> >
> > hari
> >
> > On Wed, Nov 10, 2010 at 11:56 AM, Jean-Daniel Cryans <
> jdcryans@apache.org>wrote:
> >
> >> I'm pretty sure that's not how it's reported by the "du" command, but
> >> I wouldn't expect to see files of 5MB on average. Can you be more
> >> specific?
> >>
> >> J-D
> >>
> >> On Tue, Nov 9, 2010 at 9:58 PM, Hari Sreekumar <
> hsreekumar@clickable.com>
> >> wrote:
> >> > Ah, so the bloat is not because of the files being 5-6 MB in size?
> >> Wouldn't
> >> > a 6 MB file occupy 64 MB if I set block size as 64 MB?
> >> >
> >> > hari
> >> >
> >> > On Wed, Nov 10, 2010 at 11:16 AM, Jean-Daniel Cryans <
> >> jdcryans@apache.org>wrote:
> >> >
> >> >> Each value is stored with it's full key e.g. row key + family +
> >> >> qualifier + timestamp + offsets. You don't give any information
> >> >> regarding how you stored the data, but if you have large enough keys
> >> >> then it should easily explain the bloat.
> >> >>
> >> >> J-D
> >> >>
> >> >> On Tue, Nov 9, 2010 at 9:21 PM, Hari Sreekumar <
> >> hsreekumar@clickable.com>
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> >     Data seems to be taking up too much space when I put into
> HBase.
> >> e.g,
> >> >> I
> >> >> > have a 2 GB text file which seems to be taking up ~70 GB when
I
> dump
> >> into
> >> >> > HBase. I have block size set to 64 MB and replication=3, which
I
> think
> >> is
> >> >> > the possible reason for this expansion. But if that is the case,
> how
> >> can
> >> >> I
> >> >> > prevent it? Decreasing the block size will have a negative impact
> on
> >> >> > performance, so is there a way I can increase the average size
on
> >> >> > HBase-created  files to be comparable to 64 MB. Right now they
are
> ~5
> >> MB
> >> >> on
> >> >> > average. Or is this an entirely different thing at work here?
> >> >> >
> >> >> > thanks,
> >> >> > hari
> >> >> >
> >> >>
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message