accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: [VOTE] Accumulo 1.8.0-rc2
Date Fri, 19 Aug 2016 21:08:00 GMT
24hr CI with agitation also just finished successfully for me:

         org.apache.accumulo.test.continuous.ContinuousVerify$Counts
                 REFERENCED=6598144114
                 UNREFERENCED=2000263

Michael Wall wrote:
> Did my tweaks fix the count?  Short answer is not yet.
>
> After giving mappers and reducers 2048m, setting mapreduce.task.io.sort.mb
> to 1024 and reducing the number of mappers and reducers from 264 to 140, I
> reran with agitation yesterday.  Josh had a good run without agitation and
> I had removed the ci table already.  On the run-verify, enough reducers ran
> out of memory that the job failed.  I have not gotten this work yet, was
> hoping to talk to Keith when gets back.
>
> On Fri, Aug 19, 2016 at 3:13 PM, Mike Drob<mdrob@mdrob.com>  wrote:
>
>> Did this fix your count discrepancy?
>>
>> On Thu, Aug 18, 2016, 10:53 AM Michael Wall<mjwall@gmail.com>  wrote:
>>
>>> Ok, the run-verify MR took over 5 hours.  I think muchos needs some
>> tweaks
>>> in yarn settings.
>>>
>>> On Thu, Aug 18, 2016 at 11:50 AM, Josh Elser<josh.elser@gmail.com>
>> wrote:
>>>> Uhh, woah. That's a rather big discrepancy. Yes, the number of
>>>> referenced+unreferenced should be equal to the number of entries in the
>>>> table (exactly, IIRC).
>>>>
>>>>
>>>> Michael Wall wrote:
>>>>
>>>>> The accumulo monitor showed 22.86 B entries in the ci table.  Should
>>> that
>>>>> match the verify output?
>>>>>
>>>>> On Thu, Aug 18, 2016 at 11:38 AM, Josh Elser<josh.elser@gmail.com>
>>>>> wrote:
>>>>>
>>>>> I didn't time it closely, but ingest ran for about 24hrs.
>>>>>> 3B entries seems low for 8nodes, but I'm sure it's fine. I also had
>> to
>>>>>> knock back the hdfs replication for the ci table from 3 to 2 after
>>> about
>>>>>> 12
>>>>>> hrs because I was running out of space. That probably also helped
my
>>> run
>>>>>> go
>>>>>> faster :)
>>>>>>
>>>>>>
>>>>>> Michael Wall wrote:
>>>>>>
>>>>>> Cool, thanks Josh.  How long did you run?  I ran 24 hours with 11
>> nodes
>>>>>>> (8
>>>>>>> tservers) on EC2 without agitation and got
>>>>>>>
>>>>>>>                    REFERENCED=3118473398
>>>>>>>                    UNREFERENCED=8002310
>>>>>>>
>>>>>>> I started again with agitation last night and will check tonight.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 18, 2016 at 11:00 AM, Josh Elser<elserj@apache.org>
>>>>>>>   wrote:
>>>>>>>
>>>>>>> My verification on 5tservers with CI just finished successfully
>>> without
>>>>>>>> agitation:
>>>>>>>>
>>>>>>>>            org.apache.accumulo.test.continuous.ContinuousVerify$
>> Counts
>>>>>>>>                    REFERENCED=7788662607
>>>>>>>>                    UNREFERENCED=2000310
>>>>>>>>
>>>>>>>> I will be starting a run with agitation now.
>>>>>>>>
>>>>>>>>
>>>>>>>> Josh Elser wrote:
>>>>>>>>
>>>>>>>> I retract my -1 (but not yet voting +1 though). It seems
like when
>>>>>>>> fixed
>>>>>>>>
>>>>>>>>> ports are used and the tserver comes back online, the
master will
>>>>>>>>> resume
>>>>>>>>> balancing. Not a blocker then.
>>>>>>>>>
>>>>>>>>> Josh Elser wrote:
>>>>>>>>>
>>>>>>>>> -1 due to https://issues.apache.org/jira/browse/ACCUMULO-4410.
>> This
>>>>>>>>>> appears to be a regression to me. It would stink
for OPs as there
>>> is
>>>>>>>>>> no
>>>>>>>>>> way to gracefully stop a TabletServer without also
having to
>>> restart
>>>>>>>>>> the
>>>>>>>>>> Master each time.
>>>>>>>>>>
>>>>>>>>>> I still need to go through the rest of the diligence
on rc2 (for
>>>>>>>>>> licensing stuff) as well as complete the verify.
>>>>>>>>>>
>>>>>>>>>> Michael Wall wrote:
>>>>>>>>>>
>>>>>>>>>> Accumulo Developers,
>>>>>>>>>>
>>>>>>>>>>> Please consider the following candidate for Accumulo
1.8.0.
>>>>>>>>>>>
>>>>>>>>>>> Git Commit:
>>>>>>>>>>> e5694cfa5ec3861e6f8f764e2b8477716e3072b9
>>>>>>>>>>> Branch:
>>>>>>>>>>> 1.8.0-rc2
>>>>>>>>>>>
>>>>>>>>>>> If this vote passes, a gpg-signed tag will be
created using:
>>>>>>>>>>> git tag -f -m 'Apache Accumulo 1.8.0' -s rel/1.8.0
>>>>>>>>>>> e5694cfa5ec3861e6f8f764e2b8477716e3072b9
>>>>>>>>>>>
>>>>>>>>>>> Staging repo:
>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapache
>>>>>>>>>>> accumulo-1055
>>>>>>>>>>>
>>>>>>>>>>> Source (official release artifact):
>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapache
>>>>>>>>>>> accumulo-1055/org/apache/accumulo/accumulo/1.8.0/
>>>>>>>>>>> accumulo-1.8.0-src.tar.gz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Binary:
>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapache
>>>>>>>>>>> accumulo-1055/org/apache/accumulo/accumulo/1.8.0/
>>>>>>>>>>> accumulo-1.8.0-bin.tar.gz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> (Append ".sha1", ".md5", or ".asc" to download
the
>> signature/hash
>>>>>>>>>>> for
>>>>>>>>>>> a
>>>>>>>>>>> given artifact.)
>>>>>>>>>>>
>>>>>>>>>>> All artifacts were built and staged with:
>>>>>>>>>>> mvn release:prepare&&    mvn release:perform
>>>>>>>>>>>
>>>>>>>>>>> Signing keys are available at https://www.apache.org/dist/ac
>>>>>>>>>>> cumulo/KEYS
>>>>>>>>>>> (Expected fingerprint: 3E38C02F85303BAB37EE58C2F899DB
>> F0B8C3D406)
>>>>>>>>>>> Release notes (in progress) can be found at:
>>>>>>>>>>> https://accumulo.apache.org/release_notes/1.8.0
>>>>>>>>>>>
>>>>>>>>>>> Please vote one of:
>>>>>>>>>>> [ ] +1 - I have verified and accept...
>>>>>>>>>>> [ ] +0 - I have reservations, but not strong
enough to vote
>>>>>>>>>>> against...
>>>>>>>>>>> [ ] -1 - Because..., I do not accept...
>>>>>>>>>>> ... these artifacts as the 1.8.0 release of Apache
Accumulo.
>>>>>>>>>>>
>>>>>>>>>>> This vote will end on Fri Aug 19 16:30:00 UTC
2016
>>>>>>>>>>> (Fri Aug 19 12:30:00 EDT 2016 / Fri Aug 19 09:30:00
PDT 2016)
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> P.S. Hint: download the whole staging repo with
>>>>>>>>>>> wget -erobots=off -r -l inf -np -nH \
>>>>>>>>>>>
>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapache
>>>>>>>>>>> accumulo-1055/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # note the trailing slash is needed
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>

Mime
View raw message