spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xin Ren (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing
Date Tue, 28 Jun 2016 16:45:57 GMT

    [ https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353352#comment-15353352
] 

Xin Ren commented on SPARK-16233:
---------------------------------

Actually I was just following the docs here https://github.com/keypointt/spark/tree/master/R#examples-unit-tests

Maybe we should update the docs here to point it out that "-Phive" could be needed?

{code}
build/mvn -DskipTests -Psparkr package
{code}
{code}
You can also run the unit tests for SparkR by running. You need to install the testthat package
first:

R -e 'install.packages("testthat", repos="http://cran.us.r-project.org")'
./R/run-tests.sh
{code}

> test_sparkSQL.R is failing
> --------------------------
>
>                 Key: SPARK-16233
>                 URL: https://issues.apache.org/jira/browse/SPARK-16233
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR, Tests
>    Affects Versions: 2.0.0
>            Reporter: Xin Ren
>            Priority: Minor
>
> By running 
> {code}
> ./R/run-tests.sh 
> {code}
> Getting error:
> {code}
> xin:spark xr$ ./R/run-tests.sh
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
>     describe
> The following objects are masked from ‘package:stats’:
>     cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
>     as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
>     rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...........
> functions on binary files: ....
> broadcast variables: ..
> functions in client.R: .....
> test functions in sparkR.R: .....Re-using existing Spark Context. Call sparkR.session.stop()
or restart R to create a new Spark Context
> ....Re-using existing Spark Context. Call sparkR.session.stop() or restart R to create
a new Spark Context
> ...........
> include an external JAR in SparkContext: Warning: Ignoring non-spark config property:
SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .........................SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
70B for [label] BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE, BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
87B for [terms, list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
30B for [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings: [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
90B for [labels, list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings: [PLAIN,
RLE]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 92
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
61B for [vectorCol] BINARY: 1 values, 18B raw, 20B comp, 1 pages, encodings: [PLAIN, RLE,
BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
126B for [prefixesToRewrite, key_value, key] BINARY: 2 values, 61B raw, 61B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
58B for [prefixesToRewrite, key_value, value] BINARY: 2 values, 15B raw, 17B comp, 1 pages,
encodings: [RLE, PLAIN_DICTIONARY], dic { 1 entries, 12B raw, 1B comp}
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 54
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
122B for [columnsToPrune, list, element] BINARY: 2 values, 59B raw, 59B comp, 1 pages, encodings:
[PLAIN, RLE]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 56
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
51B for [intercept] DOUBLE: 1 values, 8B raw, 10B comp, 1 pages, encodings: [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
45B for [coefficients, type] INT32: 1 values, 10B raw, 12B comp, 1 pages, encodings: [PLAIN,
RLE, BIT_PACKED]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
30B for [coefficients, size] INT32: 1 values, 7B raw, 9B comp, 1 pages, encodings: [PLAIN,
RLE, BIT_PACKED]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
36B for [coefficients, indices, list, element] INT32: 1 values, 13B raw, 15B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
79B for [coefficients, values, list, element] DOUBLE: 3 values, 37B raw, 38B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer ver.................................................................W..........
> parallelize() and collect(): .............................
> ...................................................................................................................................................................................................................................................................
> SerDe functionality: ...................
> partitionBy, groupByKey, reduceByKey etc.: ....................
> SparkSQL functions: .........................................................S................................................................................................................................................................................................................................................................S......................................................................................................................................................................1.....................................S
                                                       .....................
> tests RDD function take(): ................
> the textFile() function: .............
> functions in utils.R: ....................................
> Windows-specific tests: S
> Skipped ------------------------------------------------------------------------
> 1. create DataFrame from RDD (@test_sparkSQL.R#200) - Hive is not build with SparkSQL,
skipped
> 2. test HiveContext (@test_sparkSQL.R#1003) - Hive is not build with SparkSQL, skipped
> 3. enableHiveSupport on SparkSession (@test_sparkSQL.R#2395) - Hive is not build with
SparkSQL, skipped
> 4. sparkJars tag in SparkContext (@test_Windows.R#21) - This test is only for Windows,
skipped
> Warnings -----------------------------------------------------------------------
> 1. spark.naiveBayes (@test_mllib.R#390) - `not()` is deprecated.
> Failed -------------------------------------------------------------------------
> 1. Error: read/write ORC files (@test_sparkSQL.R#1705) -------------------------
> org.apache.spark.sql.AnalysisException: The ORC data source must be used with Hive support
enabled;
> 	at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:137)
> 	at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:78)
> 	at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:78)
> 	at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:414)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:194)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:483)
> 	at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141)
> 	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86)
> 	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38)
> 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> 1: write.df(df, orcPath, "orc", mode = "overwrite") at /Users/xin/workspace/spark/R/lib/SparkR/tests/testthat/test_sparkSQL.R:1705
> 2: write.df(df, orcPath, "orc", mode = "overwrite")
> 3: .local(df, path, ...)
> 4: callJMethod(write, "save", path)
> 5: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
> 6: stop(readString(conn))
> DONE ===========================================================================
> Error: Test failures
> Execution halted
> Warning: Ignoring non-spark config property: SPARK_SCALA_VERSION=2.11
> Loading required package: methods
> Attaching package: ‘SparkR’
> The following object is masked from ‘package:testthat’:
>     describe
> The following objects are masked from ‘package:stats’:
>     cov, filter, lag, na.omit, predict, sd, var, window
> The following objects are masked from ‘package:base’:
>     as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
>     rank, rbind, sample, startsWith, subset, summary, transform, union
> binary functions: ...........
> functions on binary files: ....
> broadcast variables: ..
> functions in client.R: .....
> test functions in sparkR.R: .....Re-using existing Spark Context. Call sparkR.session.stop()
or restart R to create a new Spark Context
> ....Re-using existing Spark Context. Call sparkR.session.stop() or restart R to create
a new Spark Context
> ...........
> include an external JAR in SparkContext: Warning: Ignoring non-spark config property:
SPARK_SCALA_VERSION=2.11
> ..
> include R packages:
> MLlib functions: .........................SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> .27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 65,622
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
70B for [label] BINARY: 1 values, 21B raw, 23B comp, 1 pages, encodings: [PLAIN, RLE, BIT_PACKED]
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
87B for [terms, list, element, list, element] BINARY: 2 values, 42B raw, 43B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:25 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
30B for [hasIntercept] BOOLEAN: 1 values, 1B raw, 3B comp, 1 pages, encodings: [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 49
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
90B for [labels, list, element] BINARY: 3 values, 50B raw, 50B comp, 1 pages, encodings: [PLAIN,
RLE]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 92
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
61B for [vectorCol] BINARY: 1 values, 18B raw, 20B comp, 1 pages, encodings: [PLAIN, RLE,
BIT_PACKED]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
126B for [prefixesToRewrite, key_value, key] BINARY: 2 values, 61B raw, 61B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:26 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
58B for [prefixesToRewrite, key_value, value] BINARY: 2 values, 15B raw, 17B comp, 1 pages,
encodings: [RLE, PLAIN_DICTIONARY], dic { 1 entries, 12B raw, 1B comp}
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 54
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
122B for [columnsToPrune, list, element] BINARY: 2 values, 59B raw, 59B comp, 1 pages, encodings:
[PLAIN, RLE]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer version
is: PARQUET_1_0
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Maximum row
group padding size is 0 bytes
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.InternalParquetRecordWriter: Flushing
mem columnStore to file. allocated memory: 56
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
51B for [intercept] DOUBLE: 1 values, 8B raw, 10B comp, 1 pages, encodings: [PLAIN, BIT_PACKED]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
45B for [coefficients, type] INT32: 1 values, 10B raw, 12B comp, 1 pages, encodings: [PLAIN,
RLE, BIT_PACKED]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
30B for [coefficients, size] INT32: 1 values, 7B raw, 9B comp, 1 pages, encodings: [PLAIN,
RLE, BIT_PACKED]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
36B for [coefficients, indices, list, element] INT32: 1 values, 13B raw, 15B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written
79B for [coefficients, values, list, element] DOUBLE: 3 values, 37B raw, 38B comp, 1 pages,
encodings: [PLAIN, RLE]
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression:
SNAPPY
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet block
size to 134217728
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet page
size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Parquet dictionary
page size to 1048576
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Dictionary
is on
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Validation
is off
> 27-Jun-2016 1:51:27 PM INFO: org.apache.parquet.hadoop.ParquetOutputFormat: Writer ver.................................................................W..........
> parallelize() and collect(): .............................
> ...................................................................................................................................................................................................................................................................
> SerDe functionality: ...................
> partitionBy, groupByKey, reduceByKey etc.: ....................
> SparkSQL functions: .........................................................S................................................................................................................................................................................................................................................................S......................................................................................................................................................................1.....................................S
                                                       .....................
> tests RDD function take(): ................
> the textFile() function: .............
> functions in utils.R: ....................................
> Windows-specific tests: S
> Skipped ------------------------------------------------------------------------
> 1. create DataFrame from RDD (@test_sparkSQL.R#200) - Hive is not build with SparkSQL,
skipped
> 2. test HiveContext (@test_sparkSQL.R#1003) - Hive is not build with SparkSQL, skipped
> 3. enableHiveSupport on SparkSession (@test_sparkSQL.R#2395) - Hive is not build with
SparkSQL, skipped
> 4. sparkJars tag in SparkContext (@test_Windows.R#21) - This test is only for Windows,
skipped
> Warnings -----------------------------------------------------------------------
> 1. spark.naiveBayes (@test_mllib.R#390) - `not()` is deprecated.
> Failed -------------------------------------------------------------------------
> 1. Error: read/write ORC files (@test_sparkSQL.R#1705) -------------------------
> org.apache.spark.sql.AnalysisException: The ORC data source must be used with Hive support
enabled;
> 	at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:137)
> 	at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:78)
> 	at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:78)
> 	at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:414)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
> 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:194)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:483)
> 	at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141)
> 	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86)
> 	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38)
> 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> 1: write.df(df, orcPath, "orc", mode = "overwrite") at /Users/xin/workspace/spark/R/lib/SparkR/tests/testthat/test_sparkSQL.R:1705
> 2: write.df(df, orcPath, "orc", mode = "overwrite")
> 3: .local(df, path, ...)
> 4: callJMethod(write, "save", path)
> 5: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
> 6: stop(readString(conn))
> DONE ===========================================================================
> Error: Test failures
> Execution halted
> Had test failures; see logs.{code}
> Cause: most probably these tests are using 'createDataFrame(sqlContext...)' which is
deprecated. Should update tests method invocations. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message