hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14849) Add option to set block cache to false on SparkSQL executions
Date Wed, 16 Dec 2015 04:26:46 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059468#comment-15059468

Hadoop QA commented on HBASE-14849:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against master branch at commit 60d33ce34191533bb858852584bd9bddfeb16a23.
  ATTACHMENT ID: 12777865

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 5 new or modified

    {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions
(2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 protoc{color}.  The applied patch does not increase the total number of
protoc compiler warnings.

    {color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 warning messages.

    {color:green}+1 checkstyle{color}. The applied patch does not generate new checkstyle

    {color:green}+1 findbugs{color}.  The patch does not introduce any  new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 lineLengths{color}.  The patch introduces the following lines longer than
    +        sqlContext.sparkContext.getConf.getInt(HBaseSparkConf.CACHE_SIZE, HBaseSparkConf.defaultCachingSize))
+      scan.setCaching(sqlContext.sparkContext.getConf.getInt(HBaseSparkConf.BATCH_NUM, HBaseSparkConf.defaultBatchNum))
+      sparkConf.getBoolean(HBaseSparkConf.BLOCK_CACHE_ENABLE, HBaseSparkConf.defaultBlockCacheEnable))
+      Map("cacheSize" -> "100", "batchNum" -> "100", "blockCacheingEnable" -> "true",
"rowNum" -> "10"))
+      Map("cacheSize" -> "1000", "batchNum" -> "100", "blockCacheingEnable" -> "true",
"rowNum" -> "10"))
+      Map("cacheSize" -> "100", "batchNum" -> "1000", "blockCacheingEnable" -> "true",
"rowNum" -> "10"))
+      Map("cacheSize" -> "100", "batchNum" -> "100", "blockCacheingEnable" -> "false",
"rowNum" -> "10"))
+case class DummyScan(cacheSize: Int, batchNum: Int, blockCachingEnable: Boolean, rowNum:
Int)(@transient val sqlContext: SQLContext)
+  override def buildScan(): RDD[Row] = sqlContext.sparkContext.parallelize(0 until rowNum).map(Row(_)).map{
+        sparkConf.getInt(HBaseSparkConf.CACHE_SIZE, HBaseSparkConf.defaultCachingSize) !=
cacheSize ||

    {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

    {color:green}+1 zombies{color}. No zombie tests found running at the end of the build.

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/16870//testReport/
Release Findbugs (version 2.0.3) 	warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16870//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/16870//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/16870//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/16870//console

This message is automatically generated.

> Add option to set block cache to false on SparkSQL executions
> -------------------------------------------------------------
>                 Key: HBASE-14849
>                 URL: https://issues.apache.org/jira/browse/HBASE-14849
>             Project: HBase
>          Issue Type: New Feature
>          Components: spark
>            Reporter: Ted Malaska
>            Assignee: Zhan Zhang
>         Attachments: HBASE-14849.patch
> I was working at a client with a ported down version of the Spark module for HBase and
realized we didn't add an option to turn of block cache for the scans.  
> At the client I just disabled all caching with Spark SQL, this is an easy but very impactful
> The fix for this patch will make this configurable

This message was sent by Atlassian JIRA

View raw message