cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arsen Vladimirskiy <arsenv...@outlook.com>
Subject Write perf better when commitlog is on ext4 vs xfs
Date Mon, 17 Jun 2019 02:35:53 GMT
Hi,

On CentOS 7.5 (3.10.0-957.21.2.el7.x86_64), I am seeing better write performance when commitlog
is using "ext4" versus "xfs" filesystem. It is the same local/ephemeral disk within an Azure
VM and the only change I am making is switching the filesystem on the disk from "ext4" to
"xfs".

Is this expected? Any ideas on how to troubleshoot and tune the filesystem (or Linux or Cassandra)
so that xfs performs same or better than ext4?

Details below.

OpenJDK 1.8.0 (same with Oracle JDK 1.8.0)
Cassandra version: 3.11.4

cassandra.yaml has this default config:
 commitlog_sync: periodic
 commitlog_sync_period_in_ms: 10000
 commitlog_segment_size_in_mb: 32

I am testing write performance using the following cassandra-stress command:

cqlsh --execute "DROP KEYSPACE IF EXISTS keyspace1;" 10.0.0.4
cassandra-stress write no-warmup n=2000000 cl=LOCAL_QUORUM \
-rate threads=32 throttle=0/s \
-pop seq="1..10000000" -col size=FIXED\(818\) \
-node "10.0.0.4" \
-schema keyspace=keyspace1 \
-log interval=1 -mode native cql3 connectionsPerHost=128

ext4 mount and ops/s look like this:

/dev/sdb1 on /mnt/resource type ext4 (rw,relatime,seclabel,data=ordered)

Connected to cluster: avcassandra, max pending requests per connection 128, max connections
per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type       total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
total,           749,     749,     749,     749,    14.4,     8.4,    43.5,    79.7,   107.0,
  131.9,    1.0,  0.00000,      0,      0,       0,       0,       0,       0
total,          4776,    4027,    4027,    4027,     7.9,     4.9,    24.6,    44.8,    65.4,
   76.7,    2.0,  0.47357,      0,      0,       0,       0,       0,       0
total,         11129,    6353,    6353,    6353,     5.0,     2.5,    16.4,    26.6,    40.8,
   46.5,    3.0,  0.34745,      0,      0,       0,       0,       0,       0
total,         17547,    6418,    6418,    6418,     4.9,     2.2,    16.4,    31.8,    74.1,
   83.6,    4.0,  0.25930,      0,      0,       0,       0,       0,       0
total,         28396,   10849,   10849,   10849,     3.0,     1.5,    11.4,    21.2,    37.9,
   49.0,    5.0,  0.25513,      0,      0,       0,       0,       0,       0
total,         40808,   12412,   12412,   12412,     2.5,     1.5,     9.0,    19.5,    33.8,
   68.0,    6.0,  0.23341,      0,      0,       0,       0,       0,       0
total,         53080,   12272,   12272,   12272,     2.6,     1.3,    10.0,    19.0,    60.8,
   73.5,    7.0,  0.20377,      0,      0,       0,       0,       0,       0
total,         67025,   13945,   13945,   13945,     2.3,     1.3,     9.1,    17.7,    30.5,
   40.8,    8.0,  0.18276,      0,      0,       0,       0,       0,       0
total,         86886,   19861,   19861,   19861,     1.6,     1.2,     2.9,     9.3,    42.7,
   56.5,    9.0,  0.18765,      0,      0,       0,       0,       0,       0
total,        106184,   19298,   19298,   19298,     1.6,     1.2,     3.1,    10.8,    57.8,
   61.5,   10.0,  0.17614,      0,      0,       0,       0,       0,       0
total,        128877,   22693,   22693,   22693,     1.4,     1.2,     2.6,     4.1,    14.7,
   30.2,   11.0,  0.16866,      0,      0,       0,       0,       0,       0
total,        148331,   19454,   19454,   19454,     1.6,     1.3,     3.0,     5.7,    45.3,
   47.7,   12.0,  0.15466,      0,      0,       0,       0,       0,       0
total,        156974,    8643,    8643,    8643,     1.6,     1.2,     3.3,    10.2,    24.6,
   41.2,   13.0,  0.14813,      0,      0,       0,       0,       0,       0
total,        170654,   13680,   13680,   13680,     3.6,     1.3,     3.1,     9.2,   890.8,
  896.5,   14.0,  0.39489,      0,      0,       0,       0,       0,       0
total,        187715,   17061,   17061,   17061,     1.9,     1.2,     4.5,    13.0,    27.2,
   44.2,   15.0,  0.37273,      0,      0,       0,       0,       0,       0
total,        205718,   18003,   18003,   18003,     1.7,     1.3,     3.7,    11.3,    32.8,
   53.4,   16.0,  0.35161,      0,      0,       0,       0,       0,       0
total,        212945,    7227,    7227,    7227,     1.8,     1.2,     3.2,    11.2,    71.3,
   88.5,   17.0,  0.34545,      0,      0,       0,       0,       0,       0
total,        220825,    7880,    7880,    7880,     6.4,     1.2,     3.1,    11.1,  1200.6,
 1209.0,   18.0,  0.42966,      0,      0,       0,       0,       0,       0
total,        241273,   20448,   20448,   20448,     1.5,     1.2,     3.1,     7.3,    23.0,
   60.4,   19.0,  0.40131,      0,      0,       0,       0,       0,       0
total,        261889,   20616,   20616,   20616,     1.5,     1.2,     2.9,     8.0,    25.0,
   33.5,   20.0,  0.37691,      0,      0,       0,       0,       0,       0
total,        282464,   20575,   20575,   20575,     1.5,     1.2,     2.7,     5.2,    68.6,
   72.7,   21.0,  0.35456,      0,      0,       0,       0,       0,       0
total,        305429,   22965,   22965,   22965,     1.4,     1.2,     2.5,     3.8,    17.2,
   23.8,   22.0,  0.33370,      0,      0,       0,       0,       0,       0
total,        326504,   21075,   21075,   21075,     1.5,     1.2,     2.9,     5.6,    17.9,
   31.5,   23.0,  0.31637,      0,      0,       0,       0,       0,       0
total,        342653,   16149,   16149,   16149,     1.9,     1.3,     3.8,    13.2,   107.0,
  124.4,   24.0,  0.30293,      0,      0,       0,       0,       0,       0
total,        360972,   18319,   18319,   18319,     1.7,     1.2,     3.8,    11.5,    36.1,
   63.7,   25.0,  0.28988,      0,      0,       0,       0,       0,       0
total,        379092,   18120,   18120,   18120,     1.7,     1.2,     2.9,     8.2,   133.5,
  161.1,   26.0,  0.27680,      0,      0,       0,       0,       0,       0
total,        400379,   21287,   21287,   21287,     1.5,     1.2,     2.9,     6.4,    30.8,
   43.2,   27.0,  0.26454,      0,      0,       0,       0,       0,       0
total,        422330,   21951,   21951,   21951,     1.4,     1.2,     2.6,     4.3,    21.9,
   35.3,   28.0,  0.25311,      0,      0,       0,       0,       0,       0
total,        442491,   20161,   20161,   20161,     1.4,     1.2,     2.7,     5.0,    21.6,
   41.5,   29.0,  0.24331,      0,      0,       0,       0,       0,       0
total,        456035,   13544,   13544,   13544,    11.9,     1.3,     3.4,     9.7,  4366.3,
 4366.3,   34.0,  0.25639,      0,      0,       0,       0,       0,       0
total,        458788,    2753,    2753,    2753,     1.5,     1.3,     2.9,     4.2,    32.8,
   40.9,   35.0,  0.25612,      0,      0,       0,       0,       0,       0
total,        468346,    9558,    9558,    9558,    46.4,     1.4,     7.1,    15.6, 13203.7,
13228.8,   48.0,  0.27131,      0,      0,       0,       0,       0,       0
total,        489463,   21117,   21117,   21117,     1.5,     1.2,     2.8,     8.5,    19.7,
   32.8,   49.0,  0.26143,      0,      0,       0,       0,       0,       0
total,        510484,   21021,   21021,   21021,     1.5,     1.2,     2.9,     5.7,    20.2,
   28.0,   50.0,  0.25232,      0,      0,       0,       0,       0,       0
total,        532895,   22411,   22411,   22411,     1.4,     1.2,     2.5,     4.8,    28.4,
   40.7,   51.0,  0.24335,      0,      0,       0,       0,       0,       0
total,        553656,   20761,   20761,   20761,     1.5,     1.2,     3.0,     9.0,    18.2,
   33.0,   52.0,  0.23551,      0,      0,       0,       0,       0,       0

xfs mount and ops/s look like this:

/dev/sdb1 on /mnt/resource type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Connected to cluster: avcassandra, max pending requests per connection 128, max connections
per host 128
Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running WRITE with 32 threads for 2000000 iteration
Failed to connect over JMX; not collecting these stats
type       total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
total,          1550,    1550,    1550,    1550,    14.9,    12.2,    37.8,    61.5,    75.6,
   82.2,    1.0,  0.00000,      0,      0,       0,       0,       0,       0
total,          5742,    4192,    4192,    4192,     7.6,     4.6,    23.4,    36.7,    63.0,
   78.8,    2.0,  0.32434,      0,      0,       0,       0,       0,       0
total,         10026,    4284,    4284,    4284,     7.4,     5.1,    20.4,    33.5,    95.2,
  112.1,    3.0,  0.22200,      0,      0,       0,       0,       0,       0
total,         15802,    5776,    5776,    5776,     5.5,     3.2,    17.0,    32.8,    50.5,
   62.6,    4.0,  0.18920,      0,      0,       0,       0,       0,       0
total,         23728,    7926,    7926,    7926,     4.0,     2.6,    11.6,    23.2,    54.7,
   90.6,    5.0,  0.19722,      0,      0,       0,       0,       0,       0
total,         32143,    8415,    8415,    8415,     3.7,     2.5,    11.8,    20.4,    40.8,
   48.1,    6.0,  0.17557,      0,      0,       0,       0,       0,       0
total,         41506,    9363,    9363,    9363,     3.4,     2.0,    11.0,    26.1,    72.2,
   84.2,    7.0,  0.16341,      0,      0,       0,       0,       0,       0
total,         52696,   11190,   11190,   11190,     2.8,     2.6,     5.0,    10.8,    20.1,
   32.3,    8.0,  0.15527,      0,      0,       0,       0,       0,       0
total,         63707,   11011,   11011,   11011,     2.9,     2.6,     4.6,    11.2,    20.7,
   62.6,    9.0,  0.14386,      0,      0,       0,       0,       0,       0
total,         74246,   10539,   10539,   10539,     3.0,     2.7,     5.2,    11.9,    41.4,
   46.3,   10.0,  0.13055,      0,      0,       0,       0,       0,       0
total,         84693,   10447,   10447,   10447,     3.0,     2.8,     5.0,     7.2,    13.6,
   30.0,   11.0,  0.11879,      0,      0,       0,       0,       0,       0
total,         95304,   10611,   10611,   10611,     3.0,     2.8,     5.0,     7.2,    14.6,
   32.6,   12.0,  0.10912,      0,      0,       0,       0,       0,       0
total,        105865,   10561,   10561,   10561,     3.0,     2.7,     4.8,    10.4,    53.1,
   60.6,   13.0,  0.10111,      0,      0,       0,       0,       0,       0
total,        116431,   10566,   10566,   10566,     3.0,     2.8,     5.0,     8.9,    14.8,
   25.2,   14.0,  0.09380,      0,      0,       0,       0,       0,       0
total,        127856,   11425,   11425,   11425,     2.8,     2.6,     4.5,     6.8,    16.6,
   24.4,   15.0,  0.08816,      0,      0,       0,       0,       0,       0
total,        139136,   11280,   11280,   11280,     2.8,     2.6,     4.4,     6.5,    17.6,
   20.6,   16.0,  0.08292,      0,      0,       0,       0,       0,       0
total,        149943,   10807,   10807,   10807,     2.9,     2.7,     4.5,     6.1,    50.0,
   52.2,   17.0,  0.07817,      0,      0,       0,       0,       0,       0
total,        161338,   11395,   11395,   11395,     2.8,     2.6,     4.8,    10.1,    18.4,
   30.0,   18.0,  0.07411,      0,      0,       0,       0,       0,       0
total,        172127,   10789,   10789,   10789,     2.9,     2.6,     5.3,    12.3,    41.9,
   53.9,   19.0,  0.07025,      0,      0,       0,       0,       0,       0
total,        182078,    9951,    9951,    9951,     3.2,     2.1,    10.9,    20.9,    36.3,
   80.7,   20.0,  0.06662,      0,      0,       0,       0,       0,       0
total,        193140,   11062,   11062,   11062,     2.9,     2.5,     5.3,    12.0,    29.0,
   44.7,   21.0,  0.06354,      0,      0,       0,       0,       0,       0
total,        203536,   10396,   10396,   10396,     3.1,     2.6,     5.2,    11.7,    60.6,
   68.7,   22.0,  0.06061,      0,      0,       0,       0,       0,       0
total,        214575,   11039,   11039,   11039,     2.9,     2.6,     5.0,    10.1,    22.5,
   36.2,   23.0,  0.05801,      0,      0,       0,       0,       0,       0
total,        225072,   10497,   10497,   10497,     3.0,     2.6,     5.6,    13.2,    29.2,
   48.2,   24.0,  0.05553,      0,      0,       0,       0,       0,       0
total,        235155,   10083,   10083,   10083,     3.1,     2.7,     4.8,     9.9,    89.3,
  102.6,   25.0,  0.05329,      0,      0,       0,       0,       0,       0
total,        245257,   10102,   10102,   10102,     3.1,     2.6,     5.2,    11.4,    65.6,
   67.3,   26.0,  0.05118,      0,      0,       0,       0,       0,       0
total,        256802,   11545,   11545,   11545,     2.7,     2.6,     4.5,     8.2,    18.6,
   36.6,   27.0,  0.04946,      0,      0,       0,       0,       0,       0
total,        267959,   11157,   11157,   11157,     2.8,     2.6,     4.6,     9.2,    20.9,
   69.1,   28.0,  0.04784,      0,      0,       0,       0,       0,       0
total,        279085,   11126,   11126,   11126,     2.8,     2.6,     4.6,     8.2,    16.0,
   30.8,   29.0,  0.04620,      0,      0,       0,       0,       0,       0
total,        289773,   10688,   10688,   10688,     3.0,     2.7,     4.9,     7.9,    19.0,
   35.5,   30.0,  0.04463,      0,      0,       0,       0,       0,       0
total,        299929,   10156,   10156,   10156,     3.1,     2.7,     4.7,    10.8,    87.3,
   90.8,   31.0,  0.04316,      0,      0,       0,       0,       0,       0
total,        310716,   10787,   10787,   10787,     2.9,     2.7,     5.0,     8.9,    16.3,
   29.2,   32.0,  0.04179,      0,      0,       0,       0,       0,       0
total,        320931,   10215,   10215,   10215,     2.8,     2.6,     4.5,     7.2,    18.4,
   25.5,   33.0,  0.04050,      0,      0,       0,       0,       0,       0
total,        322021,    1090,    1090,    1090,    31.9,     3.1,    11.0,   974.1,   984.6,
  985.1,   34.0,  0.15458,      0,      0,       0,       0,       0,       0
total,        333309,   11288,   11288,   11288,     2.8,     2.6,     4.7,     9.9,    21.0,
   25.4,   35.0,  0.15035,      0,      0,       0,       0,       0,       0
total,        344239,   10930,   10930,   10930,     2.9,     2.6,     5.0,     9.9,    39.6,
   54.8,   36.0,  0.14636,      0,      0,       0,       0,       0,       0
total,        355560,   11321,   11321,   11321,     2.8,     2.6,     4.7,     9.3,    22.8,
   41.4,   37.0,  0.14249,      0,      0,       0,       0,       0,       0
total,        364990,    9430,    9430,    9430,     3.4,     2.6,     5.0,    12.3,   141.3,
  158.2,   38.0,  0.13902,      0,      0,       0,       0,       0,       0
total,        376297,   11307,   11307,   11307,     2.8,     2.6,     4.5,     8.6,    21.8,
   24.0,   39.0,  0.13559,      0,      0,       0,       0,       0,       0
total,        387651,   11354,   11354,   11354,     2.8,     2.5,     4.7,     8.9,    20.3,
   26.4,   40.0,  0.13230,      0,      0,       0,       0,       0,       0
total,        394128,    6477,    6477,    6477,     2.7,     2.6,     4.3,     5.8,    12.0,
   14.0,   41.0,  0.13099,      0,      0,       0,       0,       0,       0
total,        396263,    2135,    2135,    2135,    21.6,     2.7,     4.7,  1256.2,  1258.3,
 1259.3,   42.0,  0.13955,      0,      0,       0,       0,       0,       0
total,        407132,   10869,   10869,   10869,     2.9,     2.6,     4.7,    10.1,    18.8,
   24.2,   43.0,  0.13638,      0,      0,       0,       0,       0,       0
total,        418276,   11144,   11144,   11144,     2.8,     2.6,     4.6,     9.0,    15.6,
   26.1,   44.0,  0.13327,      0,      0,       0,       0,       0,       0
total,        429779,   11503,   11503,   11503,     2.8,     2.6,     4.3,     5.6,    13.4,
   22.0,   45.0,  0.13022,      0,      0,       0,       0,       0,       0
total,        439888,   10109,   10109,   10109,     3.1,     2.6,     4.5,     9.7,   106.7,
  108.4,   46.0,  0.12741,      0,      0,       0,       0,       0,       0
total,        450854,   10966,   10966,   10966,     2.9,     2.6,     4.7,     8.5,    21.0,
   40.6,   47.0,  0.12470,      0,      0,       0,       0,       0,       0
total,        462404,   11550,   11550,   11550,     2.7,     2.6,     4.3,     5.9,    11.4,
   19.3,   48.0,  0.12202,      0,      0,       0,       0,       0,       0
total,        472649,   10245,   10245,   10245,     2.7,     2.5,     4.3,     8.8,    13.8,
   43.7,   49.0,  0.11969,      0,      0,       0,       0,       0,       0
total,        476297,    3648,    3648,    3648,     9.8,     2.7,     4.3,    12.1,   804.3,
  805.3,   50.0,  0.11649,      0,      0,       0,       0,       0,       0
total,        487422,   11125,   11125,   11125,     2.8,     2.7,     4.5,     6.5,    16.0,
   32.0,   51.0,  0.11421,      0,      0,       0,       0,       0,       0
total,        498707,   11285,   11285,   11285,     2.8,     2.6,     4.4,     8.9,    22.1,
   32.6,   52.0,  0.11200,      0,      0,       0,       0,       0,       0
total,        509903,   11196,   11196,   11196,     2.8,     2.6,     4.5,     9.0,    16.5,
   33.3,   53.0,  0.10988,      0,      0,       0,       0,       0,       0


Mime
View raw message