hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
Date Thu, 23 Jul 2015 18:08:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639290#comment-14639290
] 

Sean Busbey commented on HBASE-13992:
-------------------------------------

I'm trying to do a final build to make sure I've got the patch applied correctly, and the
new integration tests are failing.

{code}
[INFO] --- scalatest-maven-plugin:1.0:test (integration-test) @ hbase-spark ---
WARNING: -c has been deprecated and will be reused for a different (but still very cool) purpose
in ScalaTest 2.0. Please change all uses of -c to -P.
Discovery starting.
Discovery completed in 284 milliseconds.
Run starting. Expected test count is: 11
HBaseDStreamFunctionsSuite:
HBaseContextSuite:
HBaseRDDFunctionsSuite:
2015-07-23 10:50:56.702 java[97585:6403] Unable to load realm info from SCDynamicStore
*** RUN ABORTED ***
  java.util.concurrent.ExecutionException: java.io.IOException: Shutting down
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:188)
  at org.scalatest.tools.ConcurrentDistributor.waitUntilDone(ConcurrentDistributor.scala:52)
  at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2549)
  at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
  at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
  at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
  at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
  at org.scalatest.tools.Runner$.main(Runner.scala:860)
  at org.scalatest.tools.Runner.main(Runner.scala)
  ...
  Cause: java.io.IOException: Shutting down
  at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:232)
  at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:94)
  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1040)
  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1006)
  at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.beforeAll(HBaseDStreamFunctionsSuite.scala:44)
  at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
  at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.beforeAll(HBaseDStreamFunctionsSuite.scala:30)
  at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
  at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.run(HBaseDStreamFunctionsSuite.scala:30)
  at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
  ...
  Cause: java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterAddress
already in use
  at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143)
  at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:218)
  at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:154)
  at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:214)
  at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:94)
  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1040)
  at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1006)
  at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.beforeAll(HBaseDStreamFunctionsSuite.scala:44)
  at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
  at org.apache.hadoop.hbase.spark.HBaseDStreamFunctionsSuite.beforeAll(HBaseDStreamFunctionsSuite.scala:30)
  ...
  Cause: java.net.BindException: Port in use: 0.0.0.0:16010
  at org.apache.hadoop.hbase.http.HttpServer.openListeners(HttpServer.java:1013)
  at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:949)
  at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:91)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1789)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:604)
  at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:363)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  ...
  Cause: java.net.BindException: Address already in use
  at sun.nio.ch.Net.bind0(Native Method)
  at sun.nio.ch.Net.bind(Net.java:444)
  at sun.nio.ch.Net.bind(Net.java:436)
  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
  at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
  at org.apache.hadoop.hbase.http.HttpServer.openListeners(HttpServer.java:1008)
  at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:949)
  at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:91)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1789)
  ...
{code}

I'm running {{mvn -DskipTests install}} at the top level and then {{mvn -Dtest=NoUnitTests
clean verify}} in the spark module.

> Integrate SparkOnHBase into HBase
> ---------------------------------
>
>                 Key: HBASE-13992
>                 URL: https://issues.apache.org/jira/browse/HBASE-13992
>             Project: HBase
>          Issue Type: New Feature
>          Components: spark
>            Reporter: Ted Malaska
>            Assignee: Ted Malaska
>             Fix For: 2.0.0
>
>         Attachments: HBASE-13992.5.patch, HBASE-13992.6.patch, HBASE-13992.7.patch, HBASE-13992.8.patch,
HBASE-13992.9.patch, HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4, HBASE-13992.patch.5
>
>
> This Jira is to ask if SparkOnHBase can find a home in side HBase core.
> Here is the github: 
> https://github.com/cloudera-labs/SparkOnHBase
> I am the core author of this project and the license is Apache 2.0
> A blog explaining this project is here
> http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/
> A spark Streaming example is here
> http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/
> A real customer using this in produce is blogged here
> http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/
> Please debate and let me know what I can do to make this happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message