cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ala' Alkhaldi (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-7316) Windows feature parity - lock JVM in memory to prevent swapping
Date Mon, 07 Jul 2014 19:19:35 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054039#comment-14054039
] 

Ala' Alkhaldi edited comment on CASSANDRA-7316 at 7/7/14 7:18 PM:
------------------------------------------------------------------

I compared C* performance using stress test while using large memory pages vs small pages.
After running several tests the results were similar (within test error in my opinion). Here
is a sample of the results
The tests were run on AWS m1.large Windows server 2008 R2 instance. The stress test client
is run from a separate instance.

{code:title=Small Pages}
cassandra-stress mixed ratio(read=1,write=1) -node 10.168.93.204 -mode native cql3
            id, ops       ,    op/s,   key/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr
  4 threadCount, 3800      ,     119,     119,    33.5,    31.2,    50.2,    77.7,   112.1,
  122.1,   32.0,  0.00970
  8 threadCount, 8300      ,     253,     253,    31.1,    31.1,    47.6,    67.2,    83.6,
   94.1,   32.8,  0.02082
 16 threadCount, 233800    ,    1123,    1123,    14.2,    14.8,    32.5,    64.6,   142.2,
  426.5,  208.2,  0.02105
 24 threadCount, 101400    ,    1736,    1736,    13.7,    14.3,    29.4,    73.2,   144.7,
  632.4,   58.4,  0.02155
 36 threadCount, 81700     ,    2512,    2512,    14.2,     9.7,    37.6,    94.8,   228.7,
  405.6,   32.5,  0.01771
 54 threadCount, 99550     ,    2944,    2944,    18.1,    12.9,    57.8,   120.5,   274.0,
  823.6,   33.8,  0.01549
 81 threadCount, 135500    ,    2564,    2564,    31.3,    27.3,    77.8,   141.8,   373.9,
  796.8,   52.9,  0.02762
121 threadCount, 98200     ,    2611,    2611,    45.7,    31.3,   115.5,   236.8,  2093.5,
 2422.0,   37.6,  0.01753
181 threadCount, 123800    ,    3029,    3029,    58.6,    37.2,   165.9,   355.1,   643.0,
 1141.0,   40.9,  0.02268
271 threadCount, 339650    ,    3035,    3035,    88.6,    44.3,   225.0,   640.5, 14452.1,
15317.0,  111.9,  0.02556
406 threadCount, 362950    ,    4012,    4012,    99.8,    56.6,   263.8,   859.2,  2570.8,
 5657.2,   90.5,  0.03540
609 threadCount, 368000    ,    4076,    4076,   146.1,    73.2,   500.7,  1268.5,  2478.4,
 8818.8,   90.3,  0.03571
913 threadCount, 798700    ,    4079,    4079,   219.0,    78.2,   922.8,  2056.4, 10413.4,
17735.8,  195.8,  0.03285
{code}

{code:title=Large Pages}
cassandra-stress mixed ratio(read=1,write=1) -node 10.168.93.204 -mode native cql3       
   
             id, ops       ,    op/s,   key/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr
  4 threadCount, 4200      ,     132,     132,    30.1,    31.1,    32.7,    53.4,    78.8,
   84.0,   31.8,  0.00832
  8 threadCount, 62700     ,     320,     320,    25.0,    30.9,    33.1,    54.2,    79.7,
  867.6,  196.2,  0.02037
 16 threadCount, 161850    ,    1210,    1210,    13.1,     8.8,    33.0,    68.6,   126.8,
  398.2,  133.8,  0.02185
 24 threadCount, 67000     ,    1704,    1704,    13.9,    15.0,    26.2,    64.7,   137.6,
  377.4,   39.3,  0.02431
 36 threadCount, 81000     ,    2118,    2118,    16.9,    11.6,    38.1,    87.1,   296.8,
 5627.4,   38.2,  0.01901
 54 threadCount, 92350     ,    2835,    2835,    18.6,    15.7,    32.5,    76.1,   193.1,
  360.8,   32.6,  0.03706
 81 threadCount, 97700     ,    2795,    2795,    28.6,    25.9,    68.4,   1380,   414.9,
  677.2,   35.0,  0.01987
121 threadCount, 125950    ,    2647,    2647,    45.0,    30.8,   129.4,   227.2,   508.0,
 1110.2,   47.6,  0.02548
181 threadCount, 139700    ,    3042,    3042,    58.5,    37.5,   161.0,   280.4,   698.6,
 1622.6,   45.9,  0.02177
271 threadCount, 203350    ,    3512,    3512,    76.0,    41.6,   189.7,   580.3,  1807.0,
 3253.5,   57.9,  0.03517
406 threadCount, 683650    ,    3701,    3701,   108.9,    59.0,   326.8,   784.8,  1762.7,
10646.9,  184.7,  0.02278
609 threadCount, 560800    ,    4319,    4319,   138.7,    69.1,   496.5,  1356.9,  3223.8,
16892.8,  129.8,  0.03182
913 threadCount, 800200    ,    4503,    4503,   199.9,    88.1,   843.7,  2151.5,  4249.6,
10486.0,  177.7,  0.02249
{code}

The question of whether the large pages feature is actually used or not follows from this
results. How can I verify that windows is assigning large pages to the JVM. Unfortunately,
I can't find away to achieve that. Below is a description of the setup and trials I have done:

*Environment Setup*
I am using 64-bit Windows 7 professional edition for my tests. "Lock pages in memory" Windows
security policy is enabled as described in [Java Support for Large Memory Pages|http://www.oracle.com/technetwork/java/javase/tech/largememory-jsp-137182.html].
I also verified that Large Pages feature is enabled by running java version command as the
following:
{code}
    java -XX:+UseLargePages -version
{code}
Where I get the following results, which dedicate that large pages feature is enabled:
{code}
    java version "1.7.0_60"
    Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
    Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
{code}

To test large pages usage I consumed all the memory available for the heap using this sample
Java program from [this video|https://www.youtube.com/watch?v=6dLAo1LGHrk] in all my trials
to consume all the memory available for the java heap:
{code}
    public class InfiniteStringHashmap {
    public static void main(String[] args) throws Exception{
        Map<Long, String> map = new HashMap<Long, String>();
        for(long i=1; true; i++){
            StringBuilder sb = new StringBuilder();
            for(long j=0;j<i;j++) sb.append(" ");
            map.put(i, sb.toString());
            if(i % 1000 == 0){
                System.out.print(".");
                Thread.sleep(1000);
            }
        }
      }
    }
{code}

I ran this sample program using the following command (with a fixed heap size of 512m for
example):
{code}
    java -Xms512m -Xmx512m -XX:+UseLargePages InfiniteStringHashmap
{code}
I also tried other heap sizes as small as 12MB and as large as 10GB. Please note that I run
my test application immediately after restarting the machine to make sure my free RAM memory
is not fragmented.

*Failed trials to monitor the memory*
In order to verify whether Large Memory Pages are used or not, I tried:
# [RamMap tool|http://technet.microsoft.com/en-us/sysinternals/ff700229.aspx] from Windows
SystInternals which has a specific field for Large Pages. No matter how I change the heap
size, it does not show usage of Large Memory pages. To verify it, I tried [Creating a File
Mapping Using Large Pages|http://msdn.microsoft.com/en-us/library/windows/desktop/aa366543%28v=vs.85%29.aspx]
Windows API example from the MSDN to narrow down the possibilities. It worked perfectly fine
(it print a 2MB page size from inside the code). RamMap tool does not show anything.
# Printing the page size used by the Java process using _sun.misc.Unsafe_ [pageSize()|http://www.docjar.com/docs/api/sun/misc/Unsafe.html#pageSize]
as suggested [here|http://stackoverflow.com/questions/19047584/getting-virtual-memory-page-size-by-java-code].
It prints 4096 (the default page size on Windows) all the time. To verify it, I tried Large
Pages on Linux version of the Hotspot JVM as described in [this video|https://www.youtube.com/watch?v=6dLAo1LGHrk],
and it worked. However, printing the page size didn't work (it prints 4096 all the time).
# [Vadump|http://www.softpedia.com/get/System/System-Miscellaneous/Vadump.shtml] tool that
shows virtual memory statistics about a specific process. The "-o" option is supposed to show
the types of VM used by a process, the number of used pages, and the total size of occupied
by each type which can be used to deduce whether large pages are used. Unfortunately, this
command fails with error code 24 when I set the JVM UseLargePages option.
# [VmMap|http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx] tool does not update
the looked memory column.
# Simple monitoring tools like TaskManager and Perfmon do not provide detailed information
about Large pages.
# Java Mission Controller: Doesn't provide page size info.
# Process Explorer: same as above

Do you know of a way to measure the page size of a process or monitor the usage of large memory
pages in Windows? one of the suggestions is to monitor system calls for made by the JVM.

Please note that even if large memory pages is possible on Windows, the JVM can't use large
memory pages partially (it tries to assign the entire heap to large memory pages. otherwise
it revert to using small pages). In addition, Windows will not assign the JVM large memory
pages if the memory is very fragmented.


was (Author: ala.alkhaldi):
I compared C* performance using stress test while using large memory pages vs small pages.
After running several tests the results were similar (within test error in my opinion). Here
is a sample of the results
The tests were run on AWS m1.large Windows server 2008 R2 instance. The stress test client
is run from a separate instance.

{code:title=Small Pages}
cassandra-stress mixed ratio(read=1,write=1) -node 10.168.93.204 -mode native cql3
            id, ops       ,    op/s,   key/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr
  4 threadCount, 3800      ,     119,     119,    33.5,    31.2,    50.2,    77.7,   112.1,
  122.1,   32.0,  0.00970
  8 threadCount, 8300      ,     253,     253,    31.1,    31.1,    47.6,    67.2,    83.6,
   94.1,   32.8,  0.02082
 16 threadCount, 233800    ,    1123,    1123,    14.2,    14.8,    32.5,    64.6,   142.2,
  426.5,  208.2,  0.02105
 24 threadCount, 101400    ,    1736,    1736,    13.7,    14.3,    29.4,    73.2,   144.7,
  632.4,   58.4,  0.02155
 36 threadCount, 81700     ,    2512,    2512,    14.2,     9.7,    37.6,    94.8,   228.7,
  405.6,   32.5,  0.01771
 54 threadCount, 99550     ,    2944,    2944,    18.1,    12.9,    57.8,   120.5,   274.0,
  823.6,   33.8,  0.01549
 81 threadCount, 135500    ,    2564,    2564,    31.3,    27.3,    77.8,   141.8,   373.9,
  796.8,   52.9,  0.02762
121 threadCount, 98200     ,    2611,    2611,    45.7,    31.3,   115.5,   236.8,  2093.5,
 2422.0,   37.6,  0.01753
181 threadCount, 123800    ,    3029,    3029,    58.6,    37.2,   165.9,   355.1,   643.0,
 1141.0,   40.9,  0.02268
271 threadCount, 339650    ,    3035,    3035,    88.6,    44.3,   225.0,   640.5, 14452.1,
15317.0,  111.9,  0.02556
406 threadCount, 362950    ,    4012,    4012,    99.8,    56.6,   263.8,   859.2,  2570.8,
 5657.2,   90.5,  0.03540
609 threadCount, 368000    ,    4076,    4076,   146.1,    73.2,   500.7,  1268.5,  2478.4,
 8818.8,   90.3,  0.03571
913 threadCount, 798700    ,    4079,    4079,   219.0,    78.2,   922.8,  2056.4, 10413.4,
17735.8,  195.8,  0.03285
{code}

{code:title=Large Pages}
cassandra-stress mixed ratio(read=1,write=1) -node 10.168.93.204 -mode native cql3       
   
             id, ops       ,    op/s,   key/s,    mean,     med,     .95,     .99,    .999,
    max,   time,   stderr
  4 threadCount, 4200      ,     132,     132,    30.1,    31.1,    32.7,    53.4,    78.8,
   84.0,   31.8,  0.00832
  8 threadCount, 62700     ,     320,     320,    25.0,    30.9,    33.1,    54.2,    79.7,
  867.6,  196.2,  0.02037
 16 threadCount, 161850    ,    1210,    1210,    13.1,     8.8,    33.0,    68.6,   126.8,
  398.2,  133.8,  0.02185
 24 threadCount, 67000     ,    1704,    1704,    13.9,    15.0,    26.2,    64.7,   137.6,
  377.4,   39.3,  0.02431
 36 threadCount, 81000     ,    2118,    2118,    16.9,    11.6,    38.1,    87.1,   296.8,
 5627.4,   38.2,  0.01901
 54 threadCount, 92350     ,    2835,    2835,    18.6,    15.7,    32.5,    76.1,   193.1,
  360.8,   32.6,  0.03706
 81 threadCount, 97700     ,    2795,    2795,    28.6,    25.9,    68.4,   1380,   414.9,
  677.2,   35.0,  0.01987
121 threadCount, 125950    ,    2647,    2647,    45.0,    30.8,   129.4,   227.2,   508.0,
 1110.2,   47.6,  0.02548
181 threadCount, 139700    ,    3042,    3042,    58.5,    37.5,   161.0,   280.4,   698.6,
 1622.6,   45.9,  0.02177
271 threadCount, 203350    ,    3512,    3512,    76.0,    41.6,   189.7,   580.3,  1807.0,
 3253.5,   57.9,  0.03517
406 threadCount, 683650    ,    3701,    3701,   108.9,    59.0,   326.8,   784.8,  1762.7,
10646.9,  184.7,  0.02278
609 threadCount, 560800    ,    4319,    4319,   138.7,    69.1,   496.5,  1356.9,  3223.8,
16892.8,  129.8,  0.03182
913 threadCount, 800200    ,    4503,    4503,   199.9,    88.1,   843.7,  2151.5,  4249.6,
10486.0,  177.7,  0.02249
{code}

The question of whether the large pages feature is actually used or not follows from this
results. How can I verify that windows is assigning large pages to the JVM. Unfortunately,
I can't find away to achieve that. Below is a description of the setup and trials I have done:

*Environment Setup*
I am using 64-bit Windows 7 professional edition for my tests. "Lock pages in memory" Windows
security policy is enabled as described in [Java Support for Large Memory Pages|http://www.oracle.com/technetwork/java/javase/tech/largememory-jsp-137182.html].
I also verified that Large Pages feature is enabled by running java version command as the
following:
{code}
    java -XX:+UseLargePages -version
{code}
Where I get the following results, which dedicate that large pages feature is enabled:
{code}
    java version "1.7.0_60"
    Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
    Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
{code}

To test large pages usage I consumed all the memory available for the heap using this sample
Java program from [this video|https://www.youtube.com/watch?v=6dLAo1LGHrk] in all my trials
to consume all the memory available for the java heap:
{code}
    public class InfiniteStringHashmap {
    public static void main(String[] args) throws Exception{
        Map<Long, String> map = new HashMap<Long, String>();
        for(long i=1; true; i++){
            StringBuilder sb = new StringBuilder();
            for(long j=0;j<i;j++) sb.append(" ");
            map.put(i, sb.toString());
            if(i % 1000 == 0){
                System.out.print(".");
                Thread.sleep(1000);
            }
        }
      }
    }
{code}

I ran this sample program using the following command (with a fixed heap size of 512m for
example):
{code}
    java -Xms512m -Xmx512m -XX:+UseLargePages InfiniteStringHashmap
{code}
I also tried other heap sizes as small as 12MB and as large as 10GB. Please note that I run
my test application immediately after restarting the machine to make sure my free RAM memory
is not fragmented.

*Failed trials to monitor the memory*
In order to verify whether Large Memory Pages are used or not, I tried:
# [RamMap tool|http://technet.microsoft.com/en-us/sysinternals/ff700229.aspx] from Windows
SystInternals which has a specific field for Large Pages. No matter how I change the heap
size, it does not show usage of Large Memory pages. To verify it, I tried [Creating a File
Mapping Using Large Pages|http://msdn.microsoft.com/en-us/library/windows/desktop/aa366543%28v=vs.85%29.aspx]
Windows API example from the MSDN to narrow down the possibilities. It worked perfectly fine
(it print a 2MB page size from inside the code). RamMap tool does not show anything.
# Printing the page size used by the Java process using _sun.misc.Unsafe_ [pageSize()|http://www.docjar.com/docs/api/sun/misc/Unsafe.html#pageSize]
as suggested [here|http://stackoverflow.com/questions/19047584/getting-virtual-memory-page-size-by-java-code].
It prints 4096 (the default page size on Windows) all the time. To verify it, I tried Large
Pages on Linux version of the Hotspot JVM as described in [this video|https://www.youtube.com/watch?v=6dLAo1LGHrk],
and it worked. However, printing the page size didn't work (it prints 4096 all the time).
# [Vadump|http://www.softpedia.com/get/System/System-Miscellaneous/Vadump.shtml] tool that
shows virtual memory statistics about a specific process. The "-o" option is supposed to show
the types of VM used by a process, the number of used pages, and the total size of occupied
by each type which can be used to deduce whether large pages are used. Unfortunately, this
command fails with error code 24 when I set the JVM UseLargePages option.
# [VmMap|http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx] tool does not update
the looked memory column.
# Simple monitoring tools like TaskManager and Perfmon do not provide detailed information
about Large pages.
# Java Mission Controller: Doesn't provide page size info.
# Process Explorer: same as above

Do you know of a way to measure the page size of a process or monitor the usage of large memory
pages in Windows?

> Windows feature parity - lock JVM in memory to prevent swapping
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-7316
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7316
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Joshua McKenzie
>            Assignee: Ala' Alkhaldi
>            Priority: Minor
>              Labels: Windows, perfomance
>             Fix For: 3.0
>
>
> Similar to mlockall() in CLibrary.java for linux, it would be nice to lock the virtual
address space on Windows to prevent page faults.
> One option: Reference API:  http://msdn.microsoft.com/en-us/library/windows/desktop/aa366895(v=vs.85).aspx



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message