hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14696) Support setting allowPartialResults in mapreduce Mappers
Date Tue, 27 Oct 2015 08:11:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975935#comment-14975935
] 

Hadoop QA commented on HBASE-14696:
-----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12768890/14696-branch-1-v2.txt
  against branch-1 branch at commit 496d20cfca5a30bc72a29e4ef893424964f9fa91.
  ATTACHMENT ID: 12768890

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions
(2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 protoc{color}.  The applied patch does not increase the total number of
protoc compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 checkstyle{color}.  The applied patch does not increase the total number
of checkstyle errors

    {color:green}+1 findbugs{color}.  The patch does not introduce any  new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

     {color:red}-1 core tests{color}.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
                  org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat
                  org.apache.hadoop.hbase.mapreduce.TestHashTable
                  org.apache.hadoop.hbase.mapreduce.TestImportTSVWithTTLs
                  org.apache.hadoop.hbase.mapreduce.TestCellCounter
                  org.apache.hadoop.hbase.mapreduce.TestCopyTable
                  org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
                  org.apache.hadoop.hbase.mapreduce.TestWALPlayer
                  org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
                  org.apache.hadoop.hbase.mapreduce.TestRowCounter
                  org.apache.hadoop.hbase.mapreduce.TestImportTsv
                  org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
                  org.apache.hadoop.hbase.mapred.TestTableMapReduce
                  org.apache.hadoop.hbase.mapreduce.TestImportExport
                  org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
                  org.apache.hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat
                  org.apache.hadoop.hbase.mapreduce.TestSyncTable
                  org.apache.hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
                  org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16241//testReport/
Release Findbugs (version 2.0.3) 	warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16241//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16241//artifact/patchprocess/checkstyle-aggregate.html

  Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16241//console

This message is automatically generated.

> Support setting allowPartialResults in mapreduce Mappers
> --------------------------------------------------------
>
>                 Key: HBASE-14696
>                 URL: https://issues.apache.org/jira/browse/HBASE-14696
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>    Affects Versions: 2.0.0, 1.1.0
>            Reporter: Mindaugas Kairys
>            Assignee: Ted Yu
>             Fix For: 2.0.0, 1.3.0
>
>         Attachments: 14696-branch-1-v1.txt, 14696-branch-1-v2.txt, 14696-v1.txt, 14696-v2.txt
>
>
> It is currently impossible to get partial results in mapreduce mapper jobs.
> When setting setAllowPartialResults(true) for scan jobs, they still fail with OOME on
large rows.
> The reason is that Scan field allowPartialResults is lost during job creation:
>   1. User creates a Job and sets a scan object via TableMapReduceUtil.initTableMapperJob(table_name,
scanObj,...) -> which puts a result of TableMapReduceUtil.convertScanToString(scanObj)
to the job config.
>   2. When the job starts - method TableInputFormat.setConfig retrieves a scan string
from config and converts it to Scan object by calling TableMapReduceUtil.convertStringToScan
- which results in a Scan object with a field allowPartialResults always set to false.
> I have tried to experiment and modify a TableInputFormat method setConfig() by forcing
all scans to allow partial results and after this all jobs succeeded with no more OOME and
I also noticed that mappers began to get partial results (Result.isPartial()).
> My use case is very simple - I just have large rows and expect a mapper to get them partially
- to get same rowid several times with different key/value records.
> This would allow me not to worry about implementing my own result partitioning solution,
which i would encounter in case the big amount of result key values could be transparently
returned for a single large row.
> And from the other side - if a Scan object can return several records for the same rowid
(partial results), perhaps the mapper should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message