hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Pi (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4608) HLog Compression
Date Sun, 25 Dec 2011 22:42:31 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175855#comment-13175855
] 

Li Pi commented on HBASE-4608:
------------------------------

Okay. I'm confused.

I disabled compression, went back to trunk, and changed these lines of code in HLogKey

System.out.println("Writing region: " + this.encodedRegionName.hashCode());
Bytes.writeByteArray(out, this.encodedRegionName);
System.out.println("Writing table: " + this.tablename.hashCode());
Bytes.writeByteArray(out, this.tablename);

And

in.readFully(this.encodedRegionName);
System.out.println("Reading region: " + this.encodedRegionName.hashCode());
this.tablename = Bytes.readByteArray(in);
System.out.println("Reading table: " + this.tablename.hashCode());

then I ran test replay after partial flush.

Got this as output

PositionWritten 124
Writing region: 1251181435
Writing table: 446506621
PositionWritten 319
Writing region: 1251181435
Writing table: 446506621
PositionWritten 514
Writing region: 1251181435
Writing table: 446506621
PositionWritten 709
Writing region: 1251181435
Writing table: 446506621
PositionWritten 904
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1099
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1294
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1489
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1684
Writing region: 1251181435
Writing table: 446506621
PositionWritten 1879
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2074
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2289
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2484
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2679
Writing region: 1251181435
Writing table: 446506621
PositionWritten 2874
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3069
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3264
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3459
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3654
Writing region: 1251181435
Writing table: 446506621
PositionWritten 3849
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4044
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4239
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4454
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4649
Writing region: 1251181435
Writing table: 446506621
PositionWritten 4844
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5039
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5234
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5429
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5624
Writing region: 1251181435
Writing table: 446506621
PositionWritten 5819
Writing region: 1251181435
Writing table: 446506621
PositionWritten 124
Writing region: 736259394
Writing table: 510860944
PositionWritten 319
Writing region: 1336786910
Writing table: 403681456
PositionWritten 514
Writing region: 1336786910
Writing table: 403681456
PositionWritten 709
Writing region: 1336786910
Writing table: 403681456
PositionWritten 904
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1099
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1294
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1489
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1684
Writing region: 1336786910
Writing table: 403681456
PositionWritten 1879
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2074
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2289
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2484
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2679
Writing region: 1336786910
Writing table: 403681456
PositionWritten 2874
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3069
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3264
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3459
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3654
Writing region: 1336786910
Writing table: 403681456
PositionWritten 3849
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4044
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4239
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4454
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4649
Writing region: 1336786910
Writing table: 403681456
PositionWritten 4844
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5039
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5234
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5429
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5624
Writing region: 1336786910
Writing table: 403681456
PositionWritten 5819
Writing region: 1336786910
Writing table: 403681456
PositionWritten 6014
Writing region: 1336786910
Writing table: 403681456

followed by


PositionRead 124
Reading region: 1037916733
Reading table: 256866950
PositionRead 319
Reading region: 720698180
Reading table: 966542180
PositionRead 514
Reading region: 1108113352
Reading table: 1082920280
PositionRead 709
Reading region: 717237635
Reading table: 787220834
PositionRead 904
Reading region: 173807871
Reading table: 611127977
PositionRead 1099
Reading region: 1961109485
Reading table: 788100239
PositionRead 1294
Reading region: 2069065824
Reading table: 586608097
PositionRead 1489
Reading region: 24862902
Reading table: 1258966396
PositionRead 1684
Reading region: 291843681
Reading table: 164096819
PositionRead 1879
Reading region: 606234185
Reading table: 1315525927
PositionRead 2074
Reading region: 1700109224
Reading table: 1465804433
PositionRead 2289
Reading region: 1990190694
Reading table: 2077192033
PositionRead 2484
Reading region: 1872332999
Reading table: 1222834702
PositionRead 2679
Reading region: 764334724
Reading table: 2074013561
PositionRead 2874
Reading region: 2138845270
Reading table: 843685757
PositionRead 3069
Reading region: 2139480405
Reading table: 780981467
PositionRead 3264
Reading region: 535465405
Reading table: 1610580905
PositionRead 3459
Reading region: 1899900
Reading table: 1866848242
PositionRead 3654
Reading region: 1382320624
Reading table: 1184634322
PositionRead 3849
Reading region: 828158517
Reading table: 1018679012
PositionRead 4044
Reading region: 1198520800
Reading table: 142476740
PositionRead 4239
Reading region: 162302775
Reading table: 518507735
PositionRead 4454
Reading region: 70862619
Reading table: 1282097095
PositionRead 4649
Reading region: 354961667
Reading table: 131165903
PositionRead 4844
Reading region: 1187109899
Reading table: 1632991863
PositionRead 5039
Reading region: 853232781
Reading table: 1535039248
PositionRead 5234
Reading region: 1683589725
Reading table: 847975203
PositionRead 5429
Reading region: 1217755329
Reading table: 1294658593
PositionRead 5624
Reading region: 1022661147
Reading table: 1554270688
PositionRead 5819
Reading region: 636371108
Reading table: 1020650096
PositionRead 6014
Reading region: 2114274883
Reading table: 206051672
PositionRead 6176


It doesn't seem like we're reading what we wrote.
                
> HLog Compression
> ----------------
>
>                 Key: HBASE-4608
>                 URL: https://issues.apache.org/jira/browse/HBASE-4608
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Li Pi
>            Assignee: Li Pi
>         Attachments: 4608v1.txt
>
>
> The current bottleneck to HBase write speed is replicating the WAL appends across different
datanodes. We can speed up this process by compressing the HLog. Current plan involves using
a dictionary to compress table name, region id, cf name, and possibly other bits of repeated
data. Also, HLog format may be changed in other ways to produce a smaller HLog.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message