cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-13461) Update circle.yml to run dtests and utests in parallel across containers
Date Tue, 18 Apr 2017 19:22:41 GMT
Ariel Weisberg created CASSANDRA-13461:
------------------------------------------

             Summary: Update circle.yml to run dtests and utests in parallel across containers
                 Key: CASSANDRA-13461
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13461
             Project: Cassandra
          Issue Type: Improvement
            Reporter: Ariel Weisberg
            Assignee: Ariel Weisberg


I have a circle.yml that parallelizes the dtests and utests over the 4 free available containers.
It can be tweaked to support however many containers are available.

The unit tests pass normally. The dtests run mostly normally. There are 10 or so tests that
fail on trunk, but 30 that fail when run in CircleCI. It's still better than not running the
dtests IMO. I am currently working on figuring out why the test failures don't match.

{noformat}
version: 2
jobs:
  build:
    resource_class: xlarge
    working_directory: ~/
    parallelism: 4
    docker:
      - image: ubuntu:xenial-20170410
    steps:
      - run:
          name: apt-get install packages
          command: |
            echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee
/etc/apt/sources.list.d/webupd8team-java.list
            echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" |
tee -a /etc/apt/sources.list.d/webupd8team-java.list
            apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
            echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true |
/usr/bin/debconf-set-selections
            apt-get update
            apt-get install -y git-core npm python python-pip python-dev ant ant-optional
oracle-java8-installer net-tools
            ln -s /usr/bin/nodejs /usr/bin/node || true
      - run:
          name: Log environment information
          command: |
              echo '*** id ***'
              id
              echo '*** cat /proc/cpuinfo ***'
              cat /proc/cpuinfo
              echo '*** free -m ***'
              free -m
              echo '*** df -m ***'
              df -m
              echo '*** ifconfig -a ***'
              ifconfig -a
              echo '*** uname -a ***'
              uname -a
              echo '*** mount ***'
              mount
              echo '*** env ***'
              env
      - run:
          name: Clone git repos
          command: |
            git clone --single-branch --depth 1 https://github.com/riptano/cassandra-dtest
~/cassandra-dtest
            git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH git://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git
~/cassandra
      - run:
          name: Install junit-merge
          command: npm install -g junit-merge
      - run:
          name: Install virtualenv
          command: pip install virtualenv 
      - run:
          name: Configure virtualenv and python dependencies
          command: |
            virtualenv --python=python2 --no-site-packages venv
            source venv/bin/activate
            export CASS_DRIVER_NO_EXTENSIONS=true
            export CASS_DRIVER_NO_CYTHON=true
            pip install -r ~/cassandra-dtest/requirements.txt
            pip freeze
      - run:
          name: Build Cassandra
          command: |
            cd ~/cassandra
            # Loop to prevent failure due to maven-ant-tasks not downloading a jar..
            for x in $(seq 1 3); do
                ant clean jar
                RETURN="$?"
                if [ "${RETURN}" -eq "0" ]; then
                    break
                fi
            done
            # Exit, if we didn't build successfully
            if [ "${RETURN}" -ne "0" ]; then
                echo "Build failed with exit code: ${RETURN}"
                exit ${RETURN}
            fi
          no_output_timeout: 20m
      - run:
          name: Determine tests to run
          no_output_timeout: 10m
          command: |
            #"$HOME/cassandra-dtest/**/*.py"
            echo `circleci tests glob "$HOME/cassandra/test/unit/**" "$HOME/cassandra-dtest/**/*.py"
| grep -v upgrade_tests | grep -v "cassandra-thrift" | grep -v "thrift_bindings" | grep -v
"tools" | grep -v "dtest.py" | grep -v "$HOME/cassandra-dtest/bin" | grep -v "plugins" | circleci
tests split --split-by=timings --timings-type=filename` > /tmp/tests.txt
            echo "***processed tests***"
            cat /tmp/tests.txt | sed -e 's/\s\+/\n/g' > /tmp/processed_tests.txt
            cat /tmp/processed_tests.txt
            echo "***java tests***"
            cat /tmp/processed_tests.txt | grep "\.java$" > /tmp/java_tests.txt
            cat /tmp/java_tests.txt
            cat /tmp/java_tests.txt | cut -c 27-1000000 | grep "Test\.java$" > /tmp/java_tests_final.txt
            echo "***final java tests***"
            cat /tmp/java_tests_final.txt
            echo "***python tests***"
            cat /tmp/processed_tests.txt | grep "\.py$" > /tmp/python_tests.txt
            cat /tmp/python_tests.txt
            echo "***final python tests***"
            cat /tmp/python_tests.txt | tr '\n' ' ' > /tmp/python_tests_final.txt
            cat /tmp/python_tests_final.txt
      - run:
         name: Run unit tests
         command: |
            cd ~/cassandra
            ant testclasslist -Dtest.classlistfile=/tmp/java_tests_final.txt || true
         no_output_timeout: 15m
      - run:
          name: Run dtests
          no_output_timeout: 70m
          command: |
            source ~/venv/bin/activate
            cd ~/cassandra-dtest
            mkdir -p /tmp/dtest
            export allow_root="yes"
            export CASSANDRA_DIR=$HOME/cassandra
            export PYTHONIOENCODING="utf-8"
            export PYTHONUNBUFFERED=true
            export CASS_DRIVER_NO_EXTENSIONS=true
            export CASS_DRIVER_NO_CYTHON=true
            export CCM_MAX_HEAP_SIZE="2048M"
            export CCM_HEAP_NEWSIZE="200M"
            export NUM_TOKENS="32"
            ./run_dtests.py --vnodes true --nose-options="--verbosity=3 --with-xunit --nocapture
--attr=!resource-intensive" `cat /tmp/python_tests_final.txt`| tee -a /tmp/dtest/stdout.txt
|| true
      - run:
          name: Collect and merge test results
          command: |
            rm -f $HOME/cassandra-dtest/logs/last || true
            mkdir -p /tmp/results/junit || true
            junit-merge ~/cassandra/build/test/output/*.xml ~/cassandra-dtest/nosetests.xml
|| true
            cp  merged-test-results.xml /tmp/results/junit || true
#            junit-merge ~/cassandra/build/test/output/*.xml ~/cassandra-dtest/nosetests.xml
      - store_test_results:
          path: /tmp/results
      - store_artifacts:
          path: ~/cassandra/build/test/output
          destination: junitxml
      - store_artifacts:
          path: ~/cassandra/build/test/logs
          destination: logs
      - store_artifacts:
          path: /tmp/dtest
          destination: dtest
      - store_artifacts:
          path: ~/cassandra-dtest/logs
          destination: dtest_logs
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message