hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@apache.org>
Subject Re: Fwd: ACTION REQUIRED: disk space on jenkins master nearly full
Date Tue, 11 Jun 2019 01:15:26 GMT
We used to have a build step that compressed our logs for us. I don't think
Jenkins can read the test results if we do the xml files from surefire, so
I'm not sure how much space we can save. That's where I'd start though.

On Mon, Jun 10, 2019, 19:46 张铎(Duo Zhang) <palomino219@gmail.com> wrote:

> Does surefire have some options to truncate the test output if it is too
> large? Or jenkins has some options to truncate or compress a file when
> archiving?
>
> Josh Elser <elserj@apache.org> 于2019年6月11日周二 上午8:40写道:
>
> > Just a cursory glance at some build artifacts showed just test output
> > which sometimes extended into the multiple megabytes.
> >
> > So everyone else knows, I just chatted with ChrisL in Slack and he
> > confirmed that our disk utilization is down already (after HBASE-22563).
> > He thanked us for the quick response.
> >
> > We should keep pulling on this thread now that we're looking at it :)
> >
> > On 6/10/19 8:36 PM, 张铎(Duo Zhang) wrote:
> > > Oh, it is the build artifacts, not the jars...
> > >
> > > Most of our build artifacts are build logs, but maybe the problem is
> that
> > > some of the logs are very large if the test hangs...
> > >
> > > 张铎(Duo Zhang) <palomino219@gmail.com> 于2019年6月11日周二 上午8:16写道:
> > >
> > >> For flakey we just need the commit id in the console output then we
> can
> > >> build the artifacts locally. +1 on removing artifacts caching.
> > >>
> > >> Josh Elser <elserj@apache.org> 于2019年6月11日周二 上午7:50写道:
> > >>
> > >>> Sure, Misty. No arguments here.
> > >>>
> > >>> I think that might be a bigger untangling. Maybe Peter or Busbey know
> > >>> better about how these could be de-coupled (e.g. I think flakies
> > >>> actually look back at old artifacts), but I'm not sure off the top
of
> > my
> > >>> head. I was just going for a quick fix to keep Infra from doing
> > >>> something super-destructive.
> > >>>
> > >>> For context, I've dropped them a note in Slack to make sure what I'm
> > >>> doing is having a positive effect.
> > >>>
> > >>> On 6/10/19 7:34 PM, Misty Linville wrote:
> > >>>> Keeping artifacts and keeping build logs are two separate things.
I
> > >>> don’t
> > >>>> see a need to keep any artifacts past the most recent green and
most
> > >>> recent
> > >>>> red builds. Alternately if we need the artifacts let’s have Jenkins
> > put
> > >>>> them somewhere rather than keeping them there. You can get back
to
> > >>> whatever
> > >>>> hash you need within git to reproduce a build problem.
> > >>>>
> > >>>> On Mon, Jun 10, 2019 at 2:26 PM Josh Elser <elserj@apache.org>
> wrote:
> > >>>>
> > >>>>> https://issues.apache.org/jira/browse/HBASE-22563 for a quick
> > bandaid
> > >>> (I
> > >>>>> hope).
> > >>>>>
> > >>>>> On 6/10/19 4:31 PM, Josh Elser wrote:
> > >>>>>> Eyes on.
> > >>>>>>
> > >>>>>> Looking at master, we already have the linked configuration,
set
> to
> > >>>>>> retain 30 builds.
> > >>>>>>
> > >>>>>> We have some extra branches which we can lop off (branch-1.2,
> > >>>>>> branch-2.0, maybe some feature branches too). A quick fix
might be
> > to
> > >>>>>> just pull back that 30 to 10.
> > >>>>>>
> > >>>>>> Largely figuring out how this stuff works now, give me
a shout in
> > >>> Slack
> > >>>>>> if anyone else has cycles.
> > >>>>>>
> > >>>>>> On 6/10/19 2:34 PM, Peter Somogyi wrote:
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> HBase jobs are using more than 400GB based on this
list.
> > >>>>>>> Could someone take a look at the job configurations
today?
> > >>> Otherwise, I
> > >>>>>>> will look into it tomorrow morning.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Peter
> > >>>>>>>
> > >>>>>>> ---------- Forwarded message ---------
> > >>>>>>> From: Chris Lambertus <cml@apache.org>
> > >>>>>>> Date: Mon, Jun 10, 2019 at 7:57 PM
> > >>>>>>> Subject: ACTION REQUIRED: disk space on jenkins master
nearly
> full
> > >>>>>>> To: <builds@apache.org>
> > >>>>>>> Cc: <dev@mesos.apache.org>, <dev@pulsar.apache.org>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Hello,
> > >>>>>>>
> > >>>>>>> The jenkins master is nearly full.
> > >>>>>>>
> > >>>>>>> The workspaces listed below need significant size reduction
> within
> > 24
> > >>>>>>> hours
> > >>>>>>> or Infra will need to perform some manual pruning of
old builds
> to
> > >>>>>>> keep the
> > >>>>>>> jenkins system running. The Mesos “Packaging” job
also needs to
> be
> > >>>>>>> corrected to include the project name (mesos-packaging)
please.
> > >>>>>>>
> > >>>>>>> It appears that the typical ‘Discard Old Builds’
checkbox in the
> > job
> > >>>>>>> configuration may not be working for multibranch pipeline
jobs.
> > >>> Please
> > >>>>>>> refer to these articles for information on discarding
builds in
> > >>>>>>> multibranch
> > >>>>>>> jobs:
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> >
> https://support.cloudbees.com/hc/en-us/articles/115000237071-How-do-I-set-discard-old-builds-for-a-Multi-Branch-Pipeline-Job-
> > >>>>>>>
> > >>>>>>> https://issues.jenkins-ci.org/browse/JENKINS-35642
> > >>>>>>>
> > >>>>>
> > >>>
> >
> https://issues.jenkins-ci.org/browse/JENKINS-34738?focusedCommentId=263489&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-263489
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> NB: I have not fully vetted the above information,
I just notice
> > that
> > >>>>>>> many
> > >>>>>>> of these jobs have ‘Discard old builds’ checked,
but it is
> clearly
> > >>> not
> > >>>>>>> working.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> If you are unable to reduce your disk usage beyond
what is
> listed,
> > >>>>> please
> > >>>>>>> let me know what the reasons are and we’ll see if
we can find a
> > >>>>> solution.
> > >>>>>>> If you believe you’ve configured your job properly
and the space
> > >>> usage
> > >>>>> is
> > >>>>>>> more than you expect, please comment here and we’ll
take a look
> at
> > >>> what
> > >>>>>>> might be going on.
> > >>>>>>>
> > >>>>>>> I cut this list off arbitrarily at 40GB workspaces
and larger.
> > There
> > >>> are
> > >>>>>>> many which are between 20 and 30GB which also need
to be
> addressed,
> > >>> but
> > >>>>>>> these are the current top contributors to the disk
space
> situation.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> 594G    Packaging
> > >>>>>>> 425G    pulsar-website-build
> > >>>>>>> 274G    pulsar-master
> > >>>>>>> 195G    hadoop-multibranch
> > >>>>>>> 173G    HBase Nightly
> > >>>>>>> 138G    HBase-Flaky-Tests
> > >>>>>>> 119G    netbeans-release
> > >>>>>>> 108G    Any23-trunk
> > >>>>>>> 101G    netbeans-linux-experiment
> > >>>>>>> 96G     Jackrabbit-Oak-Windows
> > >>>>>>> 94G     HBase-Find-Flaky-Tests
> > >>>>>>> 88G     PreCommit-ZOOKEEPER-github-pr-build
> > >>>>>>> 74G     netbeans-windows
> > >>>>>>> 71G     stanbol-0.12
> > >>>>>>> 68G     Sling
> > >>>>>>> 63G     Atlas-master-NoTests
> > >>>>>>> 48G     FlexJS Framework (maven)
> > >>>>>>> 45G     HBase-PreCommit-GitHub-PR
> > >>>>>>> 42G     pulsar-pull-request
> > >>>>>>> 40G     Atlas-1.0-NoTests
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Chris
> > >>>>>>> ASF Infra
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message