hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18175) Add hbase-spark integration test into hbase-spark-it
Date Thu, 29 Jun 2017 17:18:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068644#comment-16068644
] 

Sean Busbey commented on HBASE-18175:
-------------------------------------

I'm set to push this once my local test passes. I'm confident the changes from v4 to v6 aren't
going to show any QA surprises (I'll keep an eye for the response later tonight and revert
if needed)

As a nit, I'm removing this line before commit:

{code}
diff --git a/hbase-spark/hbase-spark-it/src/test/java/org/apache/hadoop/hbase/spark/IntegrationTestSparkBulkLoad.java
b/hbase-spark/hbase-spark-it/src/test/java/org/apache/hadoop/hbase/spark/IntegrationTestSparkBulkLoad.java
index c3d82ad..2d84914 100644
--- a/hbase-spark/hbase-spark-it/src/test/java/org/apache/hadoop/hbase/spark/IntegrationTestSparkBulkLoad.java
+++ b/hbase-spark/hbase-spark-it/src/test/java/org/apache/hadoop/hbase/spark/IntegrationTestSparkBulkLoad.java
@@ -275,7 +275,6 @@ public class IntegrationTestSparkBulkLoad extends IntegrationTestBase
{
     String jobName = IntegrationTestSparkBulkLoad.class.getSimpleName() + "_check" + EnvironmentEdgeManager.currentTime();
 
     SparkConf sparkConf = new SparkConf().setAppName(jobName).setMaster("local");
-    //sparkConf.set("spark.driver.allowMultipleContexts", "true");
     Configuration hbaseConf = new Configuration(getConf());
     JavaSparkContext jsc = new JavaSparkContext(sparkConf);
     JavaHBaseContext hbaseContext = new JavaHBaseContext(jsc, hbaseConf);
{code}

> Add hbase-spark integration test into hbase-spark-it
> ----------------------------------------------------
>
>                 Key: HBASE-18175
>                 URL: https://issues.apache.org/jira/browse/HBASE-18175
>             Project: HBase
>          Issue Type: Test
>          Components: spark
>            Reporter: Yi Liang
>            Assignee: Yi Liang
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: hbase-18175-master-v2.patch, hbase-18175-master-v3.patch, hbase-18175-master-v4.patch,
hbase-18175-master-v5.patch, hbase-18175-master-v6.patch, hbase-18175-v1.patch
>
>
> After HBASE-17574, all test under hbase-spark are regarded as unit test, and this jira
will add integration test of hbase-spark into hbase-it.  This patch run same tests as mapreduce.IntegrationTestBulkLoad,
just change mapreduce to spark.  
> test in Maven:
> mvn verify -Dit.test=IntegrationTestSparkBulkLoad
> test on cluster:
> spark-submit --class org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar
-Dhbase.spark.bulkload.chainlength=500000 -m slowDeterministic



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message