hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Jenkins jobs leave junk around
Date Wed, 13 Aug 2014 01:06:51 GMT
Apache build uses the following script to capture jstack of the hanging
test:

  ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
  if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
    #It seems sometimes the tests are not dying immediately. Let's give
them 10s
    echo "Suspicious java process found - waiting 10s to see if there are
just slow to stop"
    sleep 10
    ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
    if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
      echo "There are $ZOMBIE_TESTS_COUNT zombie tests, they should have
been killed by surefire but survived"
      echo "************ BEGIN zombies jstack extract"
      ZB_STACK=`jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1
jstack | grep ".test" | grep "\.java"`
      jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack
      echo "************ END  zombies jstack extract"
      JIRA_COMMENT="$JIRA_COMMENT

     {color:red}-1 core zombie tests{color}.  There are
${ZOMBIE_TESTS_COUNT} zombie test(s): ${ZB_STACK}"
      BAD=1
      jps | grep surefirebooter | cut -d ' ' -f 1 | xargs kill -9
    else
      echo "We're ok: there is no zombie test, but some tests took some
time to stop"
    fi
  else
    echo "We're ok: there is no zombie test"
  fi


On Tue, Aug 12, 2014 at 6:02 PM, Mikhail Antonov <olorinbant@gmail.com>
wrote:

> I assume that as a quick-fix solution it's safe to kill those
> processes in the post-build jenkins steps.
>
> -Mikhail
>
> 2014-08-12 18:00 GMT-07:00, Ted Yu <yuzhihong@gmail.com>:
> > Apache Jenkins sets surefire.secondPartThreadCount at 2 - so your build
> > should take shorter than Apache trunk build.
> >
> > Cheers
> >
> >
> > On Tue, Aug 12, 2014 at 5:30 PM, Alex Newman <posix4e@gmail.com> wrote:
> >
> >> So I am trying to setup an internal jenkins. I have noticed, once
> >> builds complete (or fail), sometimes we have processes that stay
> >> around
> >>
> >> 20242 ?        Sl     3:27 /usr/java/jdk1.7.0_55/jre/bin/java
> >> -enableassertions -XX:MaxDirectMemorySize=1G -Xmx1900m
> >> -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> >> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
> >>
> >>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefirebooter1201010766180838953.jar
> >>
> >>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire6309574441534864799tmp
> >>
> >>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire_2192766844117343368134tmp
> >>
> >>
> >> Our pre build maven step uses
> >> -DcompileSource=1.7 clean install -DskipTests to prepare the directory
> >>
> >> Then we run
> >> -PrunAllTests -DcompileSource=1.7
> >> -Dmaven.test.redirectTestOutputToFile=true install
> >> -Dsurefire.secondPartThreadCount=5 -Dit.test=noItTest
> >>
> >> I am curious if this is a common problem. Any tricks we can do to speed
> >> this up?
> >>
> >> -Alex Newman
> >>
> >
>
>
> --
> Thanks,
> Michael Antonov
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message