hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HBASE-13161) ITBLL needs love; verify spins w/o end at 10B rows
Date Mon, 31 Aug 2015 04:02:46 GMT

     [ https://issues.apache.org/jira/browse/HBASE-13161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack resolved HBASE-13161.
---------------------------
       Resolution: Fixed
         Assignee: stack
    Fix Version/s: 1.2.0

Resolving. Did an ITBLL 10B run w/ monkeys. Passed. See below. The change to use securerandom
seems to have gotten us over the observed hang.

Running to 10B trying to save all hfiles and WALs made in case I find an issue so I can dig
in if verify fails is beyond the disk capacity I have on my small cluster; I had to remove
a load of archived material to run verify.

Verify took 18 hours.

Will come back to this to go larger when I have more capacity. Need to also make verify run
faster.

{code}
15/08/29 14:58:54 INFO mapreduce.Job: Job job_1440806964191_0001 completed successfully
15/08/29 14:58:54 INFO mapreduce.Job: Counters: 51
        File System Counters
                FILE: Number of bytes read=3119022047178
                FILE: Number of bytes written=3814321075643
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=2484
                HDFS: Number of bytes written=480
                HDFS: Number of read operations=55
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=10
        Job Counters
                Launched map tasks=20
                Launched reduce tasks=5
                Data-local map tasks=19
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=405431651
                Total time spent by all reduces in occupied slots (ms)=463831492
                Total time spent by all map tasks (ms)=405431651
                Total time spent by all reduce tasks (ms)=231915746
                Total vcore-seconds taken by all map tasks=405431651
                Total vcore-seconds taken by all reduce tasks=231915746
                Total megabyte-seconds taken by all map tasks=1621726604000
                Total megabyte-seconds taken by all reduce tasks=1855325968000
        Map-Reduce Framework
                Map input records=10000000000
                Map output records=20000000000
                Map output bytes=650000000000
                Map output materialized bytes=690000000600
                Input split bytes=2484
                Combine input records=0
                Combine output records=0
                Reduce input groups=10000000000
                Reduce shuffle bytes=690000000600
                Reduce input records=20000000000
                Reduce output records=0
                Spilled Records=110403900880
                Shuffled Maps =100
                Failed Shuffles=0
                Merged Map outputs=100
                GC time elapsed (ms)=2193043
                CPU time spent (ms)=390646120
                Physical memory (bytes) snapshot=23430848512
                Virtual memory (bytes) snapshot=183185506304
                Total committed heap usage (bytes)=21181235200
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify$Counts
                REFERENCED=10000000000
        File Input Format Counters
                Bytes Read=0
        File Output Format Counters
                Bytes Written=480
{code}

> ITBLL needs love; verify spins w/o end at 10B rows
> --------------------------------------------------
>
>                 Key: HBASE-13161
>                 URL: https://issues.apache.org/jira/browse/HBASE-13161
>             Project: HBase
>          Issue Type: Bug
>          Components: integration tests
>            Reporter: stack
>            Assignee: stack
>             Fix For: 1.2.0
>
>
> Making a note of my recent experience with ITBLL; here is stuff that needs fixing.
> + I was able to run generate for about 24 hours with monkey going and loaded 10B rows.
When I ran the verify (a few times), it got stuck at about 90odd percent and never made progress
beyond that (this was about 20 hours in IIRC). I spent no time trying to figure why (ran out
of cluster and time).
> + Verify at least takes too long. Need to make it run faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message