spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zsxwing <>
Subject [GitHub] spark pull request: Add a file sink log to support versioning and ...
Date Fri, 15 Apr 2016 23:24:19 GMT
GitHub user zsxwing opened a pull request:

    Add a file sink log to support versioning and compaction

    ## What changes were proposed in this pull request?
    This PR adds a special log for FileStreamSink for two purposes:
    - Versioning. A future Spark version should be able to read the metadata of an old FileStreamSink.
    - Compaction. As reading from many small files is usually pretty slow, we should compact
small metadata files into big files.
    FileStreamSinkLog has a new log format instead of Java serialization format. It will write
one log file for each batch. The first line of the log file is the version number, and there
are multiple JSON lines following. Each JSON line is a JSON format of FileLog.
    FileStreamSinkLog will compact log files every "spark.sql.sink.file.log.compactLen" batches
into a big file. When doing a compact, it will read all history logs and merge them with the
new batch. During the compaction, it will also delete the files that are deleted (marked by
FileLog.action). When the reader uses allLogs to list all files, this method only returns
the visible files (drops the deleted files).
    ## How was this patch tested?

You can merge this pull request into a Git repository by running:

    $ git pull sink-log

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #12435
commit 29e3088131beff4b36841a3d6544d399b0bfa0c2
Author: Shixiong Zhu <>
Date:   2016-04-15T19:28:46Z

    Add a file sink log to support versioning and compaction


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message