geode-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kirk Lund (Jira)" <j...@apache.org>
Subject [jira] [Updated] (GEODE-7120) Adjust pipeline values to avoid timeouts and out of memory failures
Date Fri, 23 Aug 2019 17:54:00 GMT

     [ https://issues.apache.org/jira/browse/GEODE-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kirk Lund updated GEODE-7120:
-----------------------------
    Description: 
Adjust pipeline values to avoid timeouts and out of memory failures:
* Increase distributedTest RAM memory from 180 to 250
* Increase rsync timeout from 5m to 10m
* Increase acceptanceTest timeout from 45m to 1h

The distributedTest RAM needs to be increased to avoid Jetty/Tomcat out of memory failures
when forking new processes:
{noformat}
org.apache.geode.session.tests.Jetty9CachingClientServerTest > containersShouldHavePersistentSessionData
FAILED
    java.lang.RuntimeException: Something very bad happened when trying to start container
JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05_<unknown>

        Caused by:
        java.lang.RuntimeException: Something very bad happened to this container when starting.
Check the cargo_logs folder for container logs.

            Caused by:
            java.io.IOException: Unable to run modify_war script, command: [/tmp/geode_container_install17845041006471328987/cargo_modules/Apache_Geode_Modules-1.11.0-SNAPSHOT-AppServer/bin/modify_war,
-J, -Xmx2096m, -w, /home/geode/geode/geode-assembly/build/distributedTest254/../../../extensions/session-testing-war/build/libs/session-testing-war.war,
-t, client-server, -o, /tmp/geode_container_install17845041006471328987/cargo_wars/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f053692095078744488223.war,
-p, gemfire.cache.enable_local_cache=true, -p, gemfire.property.log-file=/home/geode/geode/geode-assembly/build/distributedTest254/cargo_logs/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05/gemfire.log,
-p, gemfire.property.cache-xml-file=/home/geode/geode/geode-assembly/build/distributedTest254/cargo_logs/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05/cache-client.xml]
            log file: 
            ERROR: Error updating web.xml
            ng: INFO: os::commit_memory(0x000000077d000000, 2147483648, 0) failed; error='Not
enough space' (errno=12)
{noformat}

The actual failure from OpenJDK is:
{noformat}
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f74a4ba4000, 65536, 1) failed;
error='Not enough space' (errno=12) 
[thread 26510 also had an error]
{noformat}

The rsync timeout needs to be increased to avoid these timeouts:
{noformat}
BUILD SUCCESSFUL in 12s
1 actionable task: 1 up-to-date
+ rsync -e 'ssh -i instance-data/sshkey -o ConnectionAttempts=60 -o StrictHostKeyChecking=no'
-ah geode@10.0.0.116:geode /tmp/build/1a3d1be6/geode-results/
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(642) [generator=3.1.3]
rsync error: received SIGUSR1 (code 19) at main.c(1440) [receiver=3.1.3]
rsync: [generator] write error: Broken pipe (32)

timeout exceeded
{noformat}

The acceptanceTest timeout needs to be increased from 45m to 1h:
{noformat}
> Task :geode-connectors:acceptanceTest
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.geode.internal.size.ObjectTraverser (file:/home/geode/geode/geode-core/build/libs/geode-core-1.11.0-SNAPSHOT.jar)
to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.size.ObjectTraverser
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access
operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.geode.internal.size.ObjectTraverser (file:/home/geode/geode/geode-core/build/libs/geode-core-1.11.0-SNAPSHOT.jar)
to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.size.ObjectTraverser
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access
operations
WARNING: All illegal access operations will be denied in a future release

timeout exceeded
{noformat}

  was:
Adjust pipeline values to avoid timeouts and out of memory failures:
* Increase rsync timeout from 5m to 10m
* Increase distributedTest RAM memory from 180 to 250
* Increase acceptanceTest timeout from 45m to 1h


> Adjust pipeline values to avoid timeouts and out of memory failures
> -------------------------------------------------------------------
>
>                 Key: GEODE-7120
>                 URL: https://issues.apache.org/jira/browse/GEODE-7120
>             Project: Geode
>          Issue Type: Wish
>          Components: ci
>            Reporter: Kirk Lund
>            Priority: Major
>
> Adjust pipeline values to avoid timeouts and out of memory failures:
> * Increase distributedTest RAM memory from 180 to 250
> * Increase rsync timeout from 5m to 10m
> * Increase acceptanceTest timeout from 45m to 1h
> The distributedTest RAM needs to be increased to avoid Jetty/Tomcat out of memory failures
when forking new processes:
> {noformat}
> org.apache.geode.session.tests.Jetty9CachingClientServerTest > containersShouldHavePersistentSessionData
FAILED
>     java.lang.RuntimeException: Something very bad happened when trying to start container
JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05_<unknown>
>         Caused by:
>         java.lang.RuntimeException: Something very bad happened to this container when
starting. Check the cargo_logs folder for container logs.
>             Caused by:
>             java.io.IOException: Unable to run modify_war script, command: [/tmp/geode_container_install17845041006471328987/cargo_modules/Apache_Geode_Modules-1.11.0-SNAPSHOT-AppServer/bin/modify_war,
-J, -Xmx2096m, -w, /home/geode/geode/geode-assembly/build/distributedTest254/../../../extensions/session-testing-war/build/libs/session-testing-war.war,
-t, client-server, -o, /tmp/geode_container_install17845041006471328987/cargo_wars/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f053692095078744488223.war,
-p, gemfire.cache.enable_local_cache=true, -p, gemfire.property.log-file=/home/geode/geode/geode-assembly/build/distributedTest254/cargo_logs/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05/gemfire.log,
-p, gemfire.property.cache-xml-file=/home/geode/geode/geode-assembly/build/distributedTest254/cargo_logs/JETTY9_client-server_containersShouldHavePersistentSessionData_0_a6ebd229-072b-47db-a9bf-ca3713175f05/cache-client.xml]
>             log file: 
>             ERROR: Error updating web.xml
>             ng: INFO: os::commit_memory(0x000000077d000000, 2147483648, 0) failed; error='Not
enough space' (errno=12)
> {noformat}
> The actual failure from OpenJDK is:
> {noformat}
> OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f74a4ba4000, 65536,
1) failed; error='Not enough space' (errno=12) 
> [thread 26510 also had an error]
> {noformat}
> The rsync timeout needs to be increased to avoid these timeouts:
> {noformat}
> BUILD SUCCESSFUL in 12s
> 1 actionable task: 1 up-to-date
> + rsync -e 'ssh -i instance-data/sshkey -o ConnectionAttempts=60 -o StrictHostKeyChecking=no'
-ah geode@10.0.0.116:geode /tmp/build/1a3d1be6/geode-results/
> rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(642) [generator=3.1.3]
> rsync error: received SIGUSR1 (code 19) at main.c(1440) [receiver=3.1.3]
> rsync: [generator] write error: Broken pipe (32)
> timeout exceeded
> {noformat}
> The acceptanceTest timeout needs to be increased from 45m to 1h:
> {noformat}
> > Task :geode-connectors:acceptanceTest
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by org.apache.geode.internal.size.ObjectTraverser
(file:/home/geode/geode/geode-core/build/libs/geode-core-1.11.0-SNAPSHOT.jar) to field java.lang.String.value
> WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.size.ObjectTraverser
> WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access
operations
> WARNING: All illegal access operations will be denied in a future release
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by org.apache.geode.internal.size.ObjectTraverser
(file:/home/geode/geode/geode-core/build/libs/geode-core-1.11.0-SNAPSHOT.jar) to field java.lang.String.value
> WARNING: Please consider reporting this to the maintainers of org.apache.geode.internal.size.ObjectTraverser
> WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access
operations
> WARNING: All illegal access operations will be denied in a future release
> timeout exceeded
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message