hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
Date Tue, 14 Jul 2015 01:07:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625643#comment-14625643
] 

Hadoop QA commented on HBASE-13992:
-----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12745137/HBASE-13992.patch
  against master branch at commit a3d30892b41f604ab5a62d4f612fa7c230267dfe.
  ATTACHMENT ID: 12745137

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 7 new or modified
tests.

    {color:green}+1 hadoop versions{color}. The patch compiles with all supported hadoop versions
(2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

    {color:red}-1 javac{color}.  The applied patch generated 42 javac compiler warnings (more
than the master's current 20 warnings).

    {color:green}+1 protoc{color}.  The applied patch does not increase the total number of
protoc compiler warnings.

    {color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 4 warning messages.

                {color:red}-1 checkstyle{color}.  The applied patch generated 1890 checkstyle
errors (more than the master's current 1873 errors).

    {color:green}+1 findbugs{color}.  The patch does not introduce any  new Findbugs (version
2.0.3) warnings.

    {color:red}-1 release audit{color}.  The applied patch generated 1 release audit warnings
(more than the master's current 0 warnings).

    {color:red}-1 lineLengths{color}.  The patch introduces the following lines longer than
100:
    +         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+                            <argLine>-Xmx1536m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m</argLine>
+  public static class CustomFunction implements VoidFunction<Tuple2<Iterator<byte[]>,
HConnection>> {
+              .println("JavaHBaseBulkPutExample  {master} {host} {post} {tableName} {columnFamily}");
+    JavaReceiverInputDStream<String> javaDstream = jssc.socketTextStream(host, Integer.parseInt(port));
+  private def bulkMutation[T](rdd: RDD[T], tableName: TableName, f: (T) => Mutation, batchSize:
Integer) {
+  def hbaseRDD[U: ClassTag](tableName: TableName, scan: Scan, f: ((ImmutableBytesWritable,
Result)) => U): RDD[U] = {
+    TableMapReduceUtil.initTableMapperJob(tableName, scan, classOf[IdentityTableMapper],
null, null, job)
+          list.add((CellUtil.cloneFamily(cell), CellUtil.cloneQualifier(cell), CellUtil.cloneValue(cell)))
+                                        configBroadcast: Broadcast[SerializableWritable[Configuration]],

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

     {color:red}-1 core tests{color}.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.rest.client.TestRemoteTable

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/14762//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14762//artifact/patchprocess/patchReleaseAuditWarnings.txt
Release Findbugs (version 2.0.3) 	warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14762//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/14762//artifact/patchprocess/checkstyle-aggregate.html

                Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/14762//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/14762//console

This message is automatically generated.

> Integrate SparkOnHBase into HBase
> ---------------------------------
>
>                 Key: HBASE-13992
>                 URL: https://issues.apache.org/jira/browse/HBASE-13992
>             Project: HBase
>          Issue Type: Bug
>          Components: spark
>            Reporter: Ted Malaska
>            Assignee: Ted Malaska
>             Fix For: 2.0.0
>
>         Attachments: HBASE-13992.patch
>
>
> This Jira is to ask if SparkOnHBase can find a home in side HBase core.
> Here is the github: 
> https://github.com/cloudera-labs/SparkOnHBase
> I am the core author of this project and the license is Apache 2.0
> A blog explaining this project is here
> http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/
> A spark Streaming example is here
> http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/
> A real customer using this in produce is blogged here
> http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/
> Please debate and let me know what I can do to make this happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message