accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: [VOTE] Accumulo 1.8.0-rc2
Date Fri, 19 Aug 2016 19:22:11 GMT
Did my tweaks fix the count?  Short answer is not yet.

After giving mappers and reducers 2048m, setting mapreduce.task.io.sort.mb
to 1024 and reducing the number of mappers and reducers from 264 to 140, I
reran with agitation yesterday.  Josh had a good run without agitation and
I had removed the ci table already.  On the run-verify, enough reducers ran
out of memory that the job failed.  I have not gotten this work yet, was
hoping to talk to Keith when gets back.

On Fri, Aug 19, 2016 at 3:13 PM, Mike Drob <mdrob@mdrob.com> wrote:

> Did this fix your count discrepancy?
>
> On Thu, Aug 18, 2016, 10:53 AM Michael Wall <mjwall@gmail.com> wrote:
>
> > Ok, the run-verify MR took over 5 hours.  I think muchos needs some
> tweaks
> > in yarn settings.
> >
> > On Thu, Aug 18, 2016 at 11:50 AM, Josh Elser <josh.elser@gmail.com>
> wrote:
> >
> > > Uhh, woah. That's a rather big discrepancy. Yes, the number of
> > > referenced+unreferenced should be equal to the number of entries in the
> > > table (exactly, IIRC).
> > >
> > >
> > > Michael Wall wrote:
> > >
> > >> The accumulo monitor showed 22.86 B entries in the ci table.  Should
> > that
> > >> match the verify output?
> > >>
> > >> On Thu, Aug 18, 2016 at 11:38 AM, Josh Elser<josh.elser@gmail.com>
> > >> wrote:
> > >>
> > >> I didn't time it closely, but ingest ran for about 24hrs.
> > >>>
> > >>> 3B entries seems low for 8nodes, but I'm sure it's fine. I also had
> to
> > >>> knock back the hdfs replication for the ci table from 3 to 2 after
> > about
> > >>> 12
> > >>> hrs because I was running out of space. That probably also helped my
> > run
> > >>> go
> > >>> faster :)
> > >>>
> > >>>
> > >>> Michael Wall wrote:
> > >>>
> > >>> Cool, thanks Josh.  How long did you run?  I ran 24 hours with 11
> nodes
> > >>>> (8
> > >>>> tservers) on EC2 without agitation and got
> > >>>>
> > >>>>                   REFERENCED=3118473398
> > >>>>                   UNREFERENCED=8002310
> > >>>>
> > >>>> I started again with agitation last night and will check tonight.
> > >>>>
> > >>>>
> > >>>> On Thu, Aug 18, 2016 at 11:00 AM, Josh Elser<elserj@apache.org>
> > >>>>  wrote:
> > >>>>
> > >>>> My verification on 5tservers with CI just finished successfully
> > without
> > >>>>
> > >>>>> agitation:
> > >>>>>
> > >>>>>           org.apache.accumulo.test.continuous.ContinuousVerify$
> Counts
> > >>>>>                   REFERENCED=7788662607
> > >>>>>                   UNREFERENCED=2000310
> > >>>>>
> > >>>>> I will be starting a run with agitation now.
> > >>>>>
> > >>>>>
> > >>>>> Josh Elser wrote:
> > >>>>>
> > >>>>> I retract my -1 (but not yet voting +1 though). It seems like
when
> > >>>>> fixed
> > >>>>>
> > >>>>>> ports are used and the tserver comes back online, the master
will
> > >>>>>> resume
> > >>>>>> balancing. Not a blocker then.
> > >>>>>>
> > >>>>>> Josh Elser wrote:
> > >>>>>>
> > >>>>>> -1 due to https://issues.apache.org/jira/browse/ACCUMULO-4410.
> This
> > >>>>>>
> > >>>>>>> appears to be a regression to me. It would stink for
OPs as there
> > is
> > >>>>>>> no
> > >>>>>>> way to gracefully stop a TabletServer without also
having to
> > restart
> > >>>>>>> the
> > >>>>>>> Master each time.
> > >>>>>>>
> > >>>>>>> I still need to go through the rest of the diligence
on rc2 (for
> > >>>>>>> licensing stuff) as well as complete the verify.
> > >>>>>>>
> > >>>>>>> Michael Wall wrote:
> > >>>>>>>
> > >>>>>>> Accumulo Developers,
> > >>>>>>>
> > >>>>>>>> Please consider the following candidate for Accumulo
1.8.0.
> > >>>>>>>>
> > >>>>>>>> Git Commit:
> > >>>>>>>> e5694cfa5ec3861e6f8f764e2b8477716e3072b9
> > >>>>>>>> Branch:
> > >>>>>>>> 1.8.0-rc2
> > >>>>>>>>
> > >>>>>>>> If this vote passes, a gpg-signed tag will be created
using:
> > >>>>>>>> git tag -f -m 'Apache Accumulo 1.8.0' -s rel/1.8.0
> > >>>>>>>> e5694cfa5ec3861e6f8f764e2b8477716e3072b9
> > >>>>>>>>
> > >>>>>>>> Staging repo:
> > >>>>>>>> https://repository.apache.org/content/repositories/orgapache
> > >>>>>>>> accumulo-1055
> > >>>>>>>>
> > >>>>>>>> Source (official release artifact):
> > >>>>>>>> https://repository.apache.org/content/repositories/orgapache
> > >>>>>>>> accumulo-1055/org/apache/accumulo/accumulo/1.8.0/
> > >>>>>>>> accumulo-1.8.0-src.tar.gz
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Binary:
> > >>>>>>>> https://repository.apache.org/content/repositories/orgapache
> > >>>>>>>> accumulo-1055/org/apache/accumulo/accumulo/1.8.0/
> > >>>>>>>> accumulo-1.8.0-bin.tar.gz
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> (Append ".sha1", ".md5", or ".asc" to download
the
> signature/hash
> > >>>>>>>> for
> > >>>>>>>> a
> > >>>>>>>> given artifact.)
> > >>>>>>>>
> > >>>>>>>> All artifacts were built and staged with:
> > >>>>>>>> mvn release:prepare&&   mvn release:perform
> > >>>>>>>>
> > >>>>>>>> Signing keys are available at https://www.apache.org/dist/ac
> > >>>>>>>> cumulo/KEYS
> > >>>>>>>> (Expected fingerprint: 3E38C02F85303BAB37EE58C2F899DB
> F0B8C3D406)
> > >>>>>>>>
> > >>>>>>>> Release notes (in progress) can be found at:
> > >>>>>>>> https://accumulo.apache.org/release_notes/1.8.0
> > >>>>>>>>
> > >>>>>>>> Please vote one of:
> > >>>>>>>> [ ] +1 - I have verified and accept...
> > >>>>>>>> [ ] +0 - I have reservations, but not strong enough
to vote
> > >>>>>>>> against...
> > >>>>>>>> [ ] -1 - Because..., I do not accept...
> > >>>>>>>> ... these artifacts as the 1.8.0 release of Apache
Accumulo.
> > >>>>>>>>
> > >>>>>>>> This vote will end on Fri Aug 19 16:30:00 UTC 2016
> > >>>>>>>> (Fri Aug 19 12:30:00 EDT 2016 / Fri Aug 19 09:30:00
PDT 2016)
> > >>>>>>>>
> > >>>>>>>> Thanks!
> > >>>>>>>>
> > >>>>>>>> P.S. Hint: download the whole staging repo with
> > >>>>>>>> wget -erobots=off -r -l inf -np -nH \
> > >>>>>>>>
> > >>>>>>>> https://repository.apache.org/content/repositories/orgapache
> > >>>>>>>> accumulo-1055/
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> # note the trailing slash is needed
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message