hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars George <lars.geo...@gmail.com>
Subject HLog sync() questions
Date Fri, 29 Jan 2010 13:17:55 GMT
Hi JD,

I have more question in regards to HLog and more concretely the
underlaying SequenceFileLogWriter. This is what is called from HLog
(assuming that is current, you said you changed things to hflush() in
the replication context?):

  public void sync() throws IOException {
    if (this.writer_out != null) {

The first sync calls SequenceFile.Writer.sync() which is not at all
what we want, i.e. the hflush or flush/sync in general but setting a
sync marker in the file. What purpose does that have and why is that
needed here?

The second call is the deprecated FSDataOutputStream.sync() which does
the actual hflush() internally. Just wondering what the sync is for

Looking at the old 0.20 call

  public void sync() throws IOException {
    lastLogFlushTime = System.currentTimeMillis();
    if (this.append && syncfs != null) {
      try {
        this.syncfs.invoke(this.writer, NO_ARGS);
      } catch (Exception e) {
        throw new IOException("Reflection", e);
    } else {
    syncTime += System.currentTimeMillis() - lastLogFlushTime;

That is calling thew actual syncfs aka hflush if available otherwise
also the above sync() on SequenceFile.Writer, which did and does this
(as noted above):

    /** create a sync point */
    public void sync() throws IOException {
      if (sync != null && lastSyncPos != out.getPos()) {
        out.writeInt(SYNC_ESCAPE);                // mark the start of the sync
        out.write(sync);                          // write sync
        lastSyncPos = out.getPos();               // update lastSyncPos

Well, what that does I can see in the
SequenceFile.Reader.readRecordLength(). But that certainly does no OS
sync (or fsync or syncfs) nor hflush nor even a stream flush.

Why are we calling it? It is called internally anyways after every few
hundred bytes it says here

    synchronized void checkAndWriteSync() throws IOException {
      if (sync != null &&
          out.getPos() >= lastSyncPos+SYNC_INTERVAL) { // time to emit sync

Just when a KV is say 10KB then it is basically in between nearly
every KV (unless the HLogKey + KeyValue is minute).

So overall I think we should *not* call writer.sync()  unless it is
Syncable and then call hflush() instead?


View raw message