brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ahgittin <>
Subject [GitHub] brooklyn-server issue #791: Sync file system on temp file before moving it.
Date Thu, 07 Sep 2017 23:20:43 GMT
Github user ahgittin commented on the issue:
    @geomacy @aledsage I hadn't realized this either; javadocs and online searches suggest
the `fd.sync()` may be needed.
    i'm curious whether this really is the issue -- at least on Mac -- as i've done some testing
and the `sync` seems to be very fast whereas `write` is what takes the time.  this is true
even under contention:  `write`s contend with each other (8 files written concurrently take
8+ times as long as writing 1 file; if the OS cache is in play i'd expect CPU contention which
with 4 cores we'd expect 8 files to take twice as long, which is what happens if we do a `System.arraycopy`
instead of a `write`) and if there's background disk activity it also slows down substantially.
 numbers and code below.
    worth having this i agree, based on what i've read, though i have one worry -- if `sync`
is done _after_ the `close`, i always get the `SyncFailedException`; and it looks like the
code in this PR is synching after close.  we might want to understand that before merging?
    i also came across the code at
:  here to get sync they included some custom JNI calls.  this makes me wonder whether --
although things acted nicely in my tests -- there aren't actually any strong guarantees with
java+mac, and you need that type of low-level library?
    although another part of me wonders if
was in fact caused by something completely different (disk corruption?).
    one thread:
    t0 took 1.390 seconds to write 1,048,576,000 bytes to file
    t0 took 0.010 seconds to sync
    t0 took 0.000 seconds to close
    8 threads:
    t7 took 13.180 seconds to write 1,048,576,000 bytes to file
    t7 took 0.099 seconds to sync
    t7 took 0.000 seconds to close
    public class FileWrite {
        public static void main(String[] args) throws Exception {
            int NUM_THREADS = 8;
            for (int x=0; x<NUM_THREADS; x++) {
                final String id = "t"+x;
                new Thread() {
                    public void run() {
                        try {
                        } catch (Exception e) {
        public static void write(String id) throws Exception {
            Stopwatch t = Stopwatch.createStarted();
            File file = File.createTempFile("deleteme", ".dat"); // put your file here.
            System.out.printf("%s took %.3f seconds to create %s%n", id, t.elapsed(TimeUnit.NANOSECONDS)
/ 1e9, file.getAbsolutePath()); t.reset(); t.start();
            FileOutputStream fos = new FileOutputStream(file);
            byte[] bytes = new byte[32 * 1024];
            // byte[] bytes2 = new byte[32 * 1024];
            for (long l = 0; l < 1000 * 1024 * 1024; l += bytes.length) { 
                // System.arraycopy(bytes, 0, bytes2, 0, 32 * 1024);
            System.out.printf("%s took %.3f seconds to write %,d bytes to file%n", id, t.elapsed(TimeUnit.NANOSECONDS)
/ 1e9, file.length()); t.reset(); t.start();
            System.out.printf("%s took %.3f seconds to sync%n", id, t.elapsed(TimeUnit.NANOSECONDS)
/ 1e9); t.reset(); t.start();
            System.out.printf("%s took %.3f seconds to close%n", id, t.elapsed(TimeUnit.NANOSECONDS)
/ 1e9); t.reset(); t.start();
            File file2 = File.createTempFile("deleteme", ".dat"); // put your file here.
            java.nio.file.Files.move(file.toPath(), file2.toPath(), StandardCopyOption.ATOMIC_MOVE);
            System.out.printf("%s took %.3f seconds to move to %s%n", id, t.elapsed(TimeUnit.NANOSECONDS)
/ 1e9, file2.getAbsolutePath()); t.reset(); t.start();


View raw message