hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Xiang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-8352) Enable windowing.q for spark
Date Sun, 05 Oct 2014 19:09:33 GMT

     [ https://issues.apache.org/jira/browse/HIVE-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jimmy Xiang updated HIVE-8352:
------------------------------
    Status: Patch Available  (was: Open)

Attached a patch that enabled windowing.q for spark. The test output is about the same as
that of MapReduce, except some counter related exceptions:
{noformat}
[Error 30017]: Skipping stats aggregation by error org.apache.hadoop.h    <
[Error 30017]: Skipping stats aggregation by error org.apache.hadoop.h    <
[Error 30017]: Skipping stats aggregation by error org.apache.hadoop.h    <
{noformat}

> Enable windowing.q for spark
> ----------------------------
>
>                 Key: HIVE-8352
>                 URL: https://issues.apache.org/jira/browse/HIVE-8352
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Brock Noland
>            Assignee: Jimmy Xiang
>            Priority: Minor
>         Attachments: hive-8385.patch
>
>
> We should enable windowing.q for basic windowing coverage. After checking out the spark
branch, we would build:
> {noformat}
> $ mvn clean install -DskipTests -Phadoop-2
> $ cd itests/
> $ mvn clean install -DskipTests -Phadoop-2
> {noformat}
> Then generate the windowing.q.out file:
> {noformat}
> $ cd qtest-spark/
> $ mvn test -Dtest=TestSparkCliDriver -Dqfile=windowing.q -Phadoop-2 -Dtest.output.overwrite=true
> {noformat}
> Compare the output against MapReduce:
> {noformat}
> $ diff -y -W 150 ../../ql/src/test/results/clientpositive/spark/windowing.q.out ../../ql/src/test/results/clientpositive/windowing.q.out|
less
> {noformat}
> And if everything looks good, add it to {{spark.query.files}} in {{./itests/src/test/resources/testconfiguration.properties}}
> then submit the patch including the .q file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message