ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrey Gura (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-6339) WAL: Avoid closed by interruption exception when user thread is interrupted
Date Thu, 07 Dec 2017 12:50:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281803#comment-16281803
] 

Andrey Gura commented on IGNITE-6339:
-------------------------------------

As solution the following are implemented:

- Ring buffer (see {{SegmentedRingByteBuffer}}) that reserve segment (buffer slice) for thread
that want write WAL records. Thread serializes WAL records to the segment and calculate CRC
if enabled. As result all threads serialize record in parallel manner to the ring buffer.
- Dedicated single thread that handles fsync requests and flushes data from the ring buffer
to the file channel. This thread can't be interrupted via public API in distinction of current
implementation. Threads that initiate fsync will be parked until data will be flushed to disk.


This solution solves problem with interrupts and improve WAL write performance for most cases
in {{LOG_ONLY}} WAL mode. But this change leads to performance drop for cases when 
local thread writes data to the local node. It is valid for cases with 1-4 threads. For larger
amount of thread we have better performance.

In order to improve performance for case mentioned above we tried mapped file. This solution
works very well and has better performance for any amount of threads. But {{MappedByteBuffer}}
don't have method for partial fsync of data and for naive fsync ({{#force}} method) has performance
drop for {{DEFAULT}} WAL mode.

So we have three implementations: master, single writer and mmap. 

See rough benchmarks results below (throughput ops/sec):

*BACKGROUND WAL mode*

| Threads | master | single writer | mmap |
| 1 | 70-80 K | 64-65 K | 70-80 K |
| 2 | 127-128 K | 100-110 K | 127-128 K |
| 4 | 190-200 K | 170-180 K | 190-200 K |
| 8 | 220-230K | 220-230 K | 220-230 K |

For {{BACKGROUND}} mode throughput values are comparable for all implementations.

*LOG_ONLY WAL mode*

| Threads | master | single writer | mmap |
| 1 | 60-65 K | 36-37 K | 70-80 K |
| 2 | 100-105 K | 60-70 K | 124-125 K |
| 4 | 130-140 K | 100-110 K | 190-200 K |
| 8 | 50-60 K | 150-160 K | 210-220 K |

For {{LOG_ONLY}} mode single writer performs better than master for larger amount of threads.
But mmap is the best.

*DEFAULT WAL mode*

| Threads | master | single writer | mmap |
| 1 | 1 K | 1 K | 1 K |
| 2 | 1-2 K | 1-2 K | 1 K |
| 4 | 3-4 K | 3-4 K | 1 K |
| 8 | 5-6 K | 5-6 K | 1 K |

For {{DEFAULT}} mode we have performance drop with mmap. But it seems that partial fsync should
solve it. Any way changes related with this issue allow switch between mmap and signle writer
solution using system property.

*Note*: single writer and mmap still use {{fileIO.close()}} call that is interruptible. So
"closed by interruption" exception still has a chance to be thrown. This problem is still
in TODO's that should be fixed.

> WAL: Avoid closed by interruption exception when user thread is interrupted
> ---------------------------------------------------------------------------
>
>                 Key: IGNITE-6339
>                 URL: https://issues.apache.org/jira/browse/IGNITE-6339
>             Project: Ignite
>          Issue Type: Bug
>          Components: persistence
>            Reporter: Andrey Gura
>            Assignee: Andrey Gura
>            Priority: Blocker
>              Labels: important
>             Fix For: 2.4
>
>
> We should have a separate writer thread for WAL that will write completed serialized
chunks of data. This will allow us avoid closed by interruption exception when user thread
is interrupted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message