Return-Path: X-Original-To: apmail-asterixdb-commits-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CF30517B5F for ; Fri, 24 Apr 2015 18:43:55 +0000 (UTC) Received: (qmail 14327 invoked by uid 500); 24 Apr 2015 18:43:55 -0000 Delivered-To: apmail-asterixdb-commits-archive@asterixdb.apache.org Received: (qmail 14288 invoked by uid 500); 24 Apr 2015 18:43:55 -0000 Mailing-List: contact commits-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.incubator.apache.org Delivered-To: mailing list commits@asterixdb.incubator.apache.org Received: (qmail 14245 invoked by uid 99); 24 Apr 2015 18:43:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Apr 2015 18:43:55 +0000 X-ASF-Spam-Status: No, hits=0.0 required=5.0 tests=SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of root@apache.org designates 54.191.145.13 as permitted sender) Received: from [54.191.145.13] (HELO mx1-us-west.apache.org) (54.191.145.13) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Apr 2015 18:43:49 +0000 Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with SMTP id A008025FCB for ; Fri, 24 Apr 2015 18:42:28 +0000 (UTC) Received: (qmail 11515 invoked by uid 99); 24 Apr 2015 18:42:28 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Apr 2015 18:42:28 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 7ACAAE17DC; Fri, 24 Apr 2015 18:42:28 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: imaxon@apache.org To: commits@asterixdb.incubator.apache.org Date: Fri, 24 Apr 2015 18:43:20 -0000 Message-Id: In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [54/58] [abbrv] incubator-asterixdb git commit: Several CSV, API, HTTP API, and Web interface improvements. X-Virus-Checked: Checked by ClamAV on apache.org Several CSV, API, HTTP API, and Web interface improvements. - APIFramework: internal refactoring to consolidate output PrintWriter, OutputFormat, and all output flags into SessionConfig - APIFramework: "HTML" is now a flag, rather than an OutputFormat - HTTP API: Output format can be select via query parameter in addition to HTTP Accept header - CSV: default output is now without header, to improve roundtripping - CSV: header can be requested via Accept header or "header" query parameter - Web interface: Added ability to select output format (JSON, CSV or ADM) Change-Id: I91398bd30dbd6f3b1f69eb51fbf201010d0e5d93 Reviewed-on: http://fulliautomatix.ics.uci.edu:8443/242 Reviewed-by: Chris Hillery Tested-by: Chris Hillery Project: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/commit/833774e0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/tree/833774e0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/diff/833774e0 Branch: refs/heads/master Commit: 833774e02f4164c54eae9c7fd954af8933e861af Parents: a5c92b3 Author: Chris Hillery Authored: Wed Mar 18 23:25:04 2015 -0700 Committer: Chris Hillery Committed: Wed Apr 8 00:46:39 2015 -0700 ---------------------------------------------------------------------- .../ics/asterix/api/common/APIFramework.java | 183 ++++++---------- .../ics/asterix/api/common/SessionConfig.java | 212 ++++++++++++++----- .../asterix/api/http/servlet/APIServlet.java | 32 ++- .../api/http/servlet/QueryResultAPIServlet.java | 32 +-- .../api/http/servlet/RESTAPIServlet.java | 94 +++++--- .../ics/asterix/api/java/AsterixJavaClient.java | 18 +- .../asterix/aql/translator/AqlTranslator.java | 34 ++- .../bootstrap/FeedLifecycleListener.java | 8 +- .../edu/uci/ics/asterix/result/ResultUtils.java | 62 +++--- .../src/main/resources/webui/querytemplate.html | 22 +- .../csv/basic-types-header/basic-types.1.csv | 2 + .../results/csv/basic-types/basic-types.1.csv | 1 - .../src/test/resources/runtimets/testsuite.xml | 5 + .../uci/ics/asterix/test/aql/TestsUtils.java | 12 +- asterix-doc/src/site/markdown/csv.md | 54 +++-- .../testframework/context/TestCaseContext.java | 5 +- .../src/main/resources/Catalog.xsd | 1 + 17 files changed, 465 insertions(+), 312 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/APIFramework.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/APIFramework.java b/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/APIFramework.java index 1ee9a3e..84a01da 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/APIFramework.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/APIFramework.java @@ -153,43 +153,26 @@ public class APIFramework { } - /** - * Used to select the output from the various servlets. Note: "HTML" is - * primarily intended for use by the built-in web interface. It produces - * ADM output with various HTML wrappers. - */ - public enum OutputFormat { - ADM, - HTML, - JSON, - CSV - } - public static Pair reWriteQuery(List declaredFunctions, - AqlMetadataProvider metadataProvider, Query q, SessionConfig pc, PrintWriter out, OutputFormat pdf) + AqlMetadataProvider metadataProvider, Query q, SessionConfig conf) throws AsterixException { - if (!pc.isPrintPhysicalOpsOnly() && pc.isPrintExprParam()) { - out.println(); - switch (pdf) { - case HTML: { - out.println("

Expression tree:

"); - out.println("
");
-                    break;
-                }
-                default: {
-                    out.println("----------Expression tree:");
-                    break;
-                }
+        if (conf.is(SessionConfig.FORMAT_ONLY_PHYSICAL_OPS) && conf.is(SessionConfig.OOB_EXPR_TREE)) {
+            conf.out().println();
+
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
Expression tree:
");
+                conf.out().println("
");
+            } else {
+                conf.out().println("----------Expression tree:");
             }
+
             if (q != null) {
-                q.accept(new AQLPrintVisitor(out), 0);
+                q.accept(new AQLPrintVisitor(conf.out()), 0);
             }
-            switch (pdf) {
-                case HTML: {
-                    out.println("
");
-                    break;
-                }
+
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
"); } } AqlRewriter rw = new AqlRewriter(declaredFunctions, q, metadataProvider); @@ -200,35 +183,26 @@ public class APIFramework { public static JobSpecification compileQuery(List declaredFunctions, AqlMetadataProvider queryMetadataProvider, Query rwQ, int varCounter, String outputDatasetName, - SessionConfig pc, PrintWriter out, OutputFormat pdf, ICompiledDmlStatement statement) + SessionConfig conf, ICompiledDmlStatement statement) throws AsterixException, AlgebricksException, JSONException, RemoteException, ACIDException { - if (!pc.isPrintPhysicalOpsOnly() && pc.isPrintRewrittenExprParam()) { - out.println(); + if (conf.is(SessionConfig.FORMAT_ONLY_PHYSICAL_OPS) && conf.is(SessionConfig.OOB_REWRITTEN_EXPR_TREE)) { + conf.out().println(); - switch (pdf) { - case HTML: { - out.println("

Rewritten expression tree:

"); - out.println("
");
-                    break;
-                }
-                default: {
-                    out.println("----------Rewritten expression:");
-                    break;
-                }
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
Rewritten expression tree:
");
+                conf.out().println("
");
+            } else {
+                conf.out().println("----------Rewritten expression:");
             }
 
             if (rwQ != null) {
-                rwQ.accept(new AQLPrintVisitor(out), 0);
+                rwQ.accept(new AQLPrintVisitor(conf.out()), 0);
             }
 
-            switch (pdf) {
-                case HTML: {
-                    out.println("
");
-                    break;
-                }
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
"); } - } edu.uci.ics.asterix.common.transactions.JobId asterixJobId = JobIdFactory.generateJobId(); @@ -246,31 +220,24 @@ public class APIFramework { boolean isWriteTransaction = queryMetadataProvider.isWriteTransaction(); LogicalOperatorPrettyPrintVisitor pvisitor = new LogicalOperatorPrettyPrintVisitor(); - if (!pc.isPrintPhysicalOpsOnly() && pc.isPrintLogicalPlanParam()) { - - switch (pdf) { - case HTML: { - out.println("

Logical plan:

"); - out.println("
");
-                    break;
-                }
-                default: {
-                    out.println("----------Logical plan:");
-                    break;
-                }
+        if (conf.is(SessionConfig.FORMAT_ONLY_PHYSICAL_OPS) && conf.is(SessionConfig.OOB_LOGICAL_PLAN)) {
+            conf.out().println();
+
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
Logical plan:
");
+                conf.out().println("
");
+            } else {
+                conf.out().println("----------Logical plan:");
             }
 
             if (rwQ != null || statement.getKind() == Kind.LOAD) {
                 StringBuilder buffer = new StringBuilder();
                 PlanPrettyPrinter.printPlan(plan, buffer, pvisitor, 0);
-                out.print(buffer);
+                conf.out().print(buffer);
             }
 
-            switch (pdf) {
-                case HTML: {
-                    out.println("
");
-                    break;
-                }
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
"); } } @@ -305,45 +272,39 @@ public class APIFramework { builder.setNullableTypeComputer(AqlNullableTypeComputer.INSTANCE); ICompiler compiler = compilerFactory.createCompiler(plan, queryMetadataProvider, t.getVarCounter()); - if (pc.isOptimize()) { + if (conf.isOptimize()) { compiler.optimize(); //plot optimized logical plan if (plot) PlanPlotter.printOptimizedLogicalPlan(plan); - if (pc.isPrintOptimizedLogicalPlanParam()) { - if (pc.isPrintPhysicalOpsOnly()) { + if (conf.is(SessionConfig.OOB_OPTIMIZED_LOGICAL_PLAN)) { + if (conf.is(SessionConfig.FORMAT_ONLY_PHYSICAL_OPS)) { // For Optimizer tests. StringBuilder buffer = new StringBuilder(); PlanPrettyPrinter.printPhysicalOps(plan, buffer, 0); - out.print(buffer); + conf.out().print(buffer); } else { - switch (pdf) { - case HTML: { - out.println("

Optimized logical plan:

"); - out.println("
");
-                            break;
-                        }
-                        default: {
-                            out.println("----------Optimized logical plan:");
-                            break;
-                        }
+                    if (conf.is(SessionConfig.FORMAT_HTML)) {
+                        conf.out().println("
Optimized logical plan:
");
+                        conf.out().println("
");
+                    } else {
+                        conf.out().println("----------Optimized logical plan:");
                     }
+
                     if (rwQ != null || statement.getKind() == Kind.LOAD) {
                         StringBuilder buffer = new StringBuilder();
                         PlanPrettyPrinter.printPlan(plan, buffer, pvisitor, 0);
-                        out.print(buffer);
+                        conf.out().print(buffer);
                     }
-                    switch (pdf) {
-                        case HTML: {
-                            out.println("
");
-                            break;
-                        }
+
+                    if (conf.is(SessionConfig.FORMAT_HTML)) {
+                        conf.out().println("
"); } } } } - if (!pc.isGenerateJobSpec()) { + if (!conf.isGenerateJobSpec()) { return null; } @@ -359,16 +320,18 @@ public class APIFramework { builder.setNullWriterFactory(format.getNullWriterFactory()); builder.setPredicateEvaluatorFactoryProvider(format.getPredicateEvaluatorFactoryProvider()); - switch (pdf) { + switch (conf.fmt()) { case JSON: builder.setPrinterProvider(format.getJSONPrinterFactoryProvider()); break; case CSV: builder.setPrinterProvider(format.getCSVPrinterFactoryProvider()); break; - default: + case ADM: builder.setPrinterProvider(format.getPrinterFactoryProvider()); break; + default: + throw new RuntimeException("Unexpected OutputFormat!"); } builder.setSerializerDeserializerProvider(format.getSerdeProvider()); @@ -379,34 +342,28 @@ public class APIFramework { isWriteTransaction); JobSpecification spec = compiler.createJob(AsterixAppContextInfo.getInstance(), jobEventListenerFactory); - if (pc.isPrintJob()) { - switch (pdf) { - case HTML: { - out.println("

Hyracks job:

"); - out.println("
");
-                    break;
-                }
-                default: {
-                    out.println("----------Hyracks job:");
-                    break;
-                }
+        if (conf.is(SessionConfig.OOB_HYRACKS_JOB)) {
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
Hyracks job:
");
+                conf.out().println("
");
+            } else {
+                conf.out().println("----------Hyracks job:");
             }
+
             if (rwQ != null) {
-                out.println(spec.toJSON().toString(1));
-                out.println(spec.getUserConstraints());
+                conf.out().println(spec.toJSON().toString(1));
+                conf.out().println(spec.getUserConstraints());
             }
-            switch (pdf) {
-                case HTML: {
-                    out.println("
");
-                    break;
-                }
+
+            if (conf.is(SessionConfig.FORMAT_HTML)) {
+                conf.out().println("
"); } } return spec; } - public static void executeJobArray(IHyracksClientConnection hcc, JobSpecification[] specs, PrintWriter out, - OutputFormat pdf) throws Exception { + public static void executeJobArray(IHyracksClientConnection hcc, JobSpecification[] specs, PrintWriter out) + throws Exception { for (int i = 0; i < specs.length; i++) { specs[i].setMaxReattempts(0); JobId jobId = hcc.startJob(specs[i]); @@ -419,7 +376,7 @@ public class APIFramework { } - public static void executeJobArray(IHyracksClientConnection hcc, Job[] jobs, PrintWriter out, OutputFormat pdf) + public static void executeJobArray(IHyracksClientConnection hcc, Job[] jobs, PrintWriter out) throws Exception { for (int i = 0; i < jobs.length; i++) { jobs[i].getJobSpec().setMaxReattempts(0); http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/SessionConfig.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/SessionConfig.java b/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/SessionConfig.java index a2f27b4..a82558f 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/SessionConfig.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/api/common/SessionConfig.java @@ -14,80 +14,188 @@ */ package edu.uci.ics.asterix.api.common; +import java.io.PrintWriter; +import java.util.HashMap; +import java.util.Map; + +/** + * SessionConfig captures several different parameters for controlling + * the execution of an APIFramework call. + *
  • It specifies how the execution will proceed (for instance, + * whether to optimize, or whether to execute at all). + *
  • It allows you specify where the primary execution output will + * be sent. + *
  • It also allows you to request additional output for optional + * out-of-band data about the execution (query plan, etc). + *
  • It allows you to specify the output format for the primary + * execution output - JSON, CSV, etc. + *
  • It allows you to specify output format-specific parameters. + */ + public class SessionConfig { - private final boolean optimize; - private final boolean printExprParam; - private final boolean printRewrittenExprParam; - private final boolean printLogicalPlanParam; - private final boolean printOptimizedLogicalPlanParam; - private final boolean printPhysicalOpsOnly; + /** + * Used to specify the output format for the primary execution. + */ + public enum OutputFormat { + ADM, + JSON, + CSV + }; + + /** + * Produce out-of-band output for Hyracks Job. + */ + public static final String OOB_HYRACKS_JOB = "oob-hyracks-job"; + + /** + * Produce out-of-band output for Expression Tree. + */ + public static final String OOB_EXPR_TREE = "oob-expr-tree"; + + /** + * Produce out-of-band output for Rewritten Expression Tree. + */ + public static final String OOB_REWRITTEN_EXPR_TREE = "oob-rewritten-expr-tree"; + + /** + * Produce out-of-band output for Logical Plan. + */ + public static final String OOB_LOGICAL_PLAN = "oob-logical-plan"; + + /** + * Produce out-of-band output for Optimized Logical Plan. + */ + public static final String OOB_OPTIMIZED_LOGICAL_PLAN = "oob-optimized-logical-plan"; + + /** + * Format flag: print only physical ops (for optimizer tests). + */ + public static final String FORMAT_ONLY_PHYSICAL_OPS = "format-only-physical-ops"; + + /** + * Format flag: wrap out-of-band data in HTML. + */ + public static final String FORMAT_HTML = "format-html"; + + /** + * Format flag: print CSV header line. + */ + public static final String FORMAT_CSV_HEADER = "format-csv-header"; + + // Standard execution flags. private final boolean executeQuery; private final boolean generateJobSpec; - private final boolean printJob; - - /** - * Note: the various "print" options below will cause additional output - * to be generated when invoking servlet functions. This output is NOT - * guaranteed to match the OutputFormat (JSON, ADM...) except when the - * OutputFormat is "HTML". This is primarily for use by the built-in - * web interface (APIServlet). - * @param optimize - * @param printExprParam - * @param printRewrittenExprParam - * @param printLogicalPlanParam - * @param printOptimizedLogicalPlanParam - * @param printPhysicalOpsOnly - * @param executeQuery - * @param generateJobSpec - * @param printJob - */ - public SessionConfig(boolean optimize, boolean printExprParam, boolean printRewrittenExprParam, - boolean printLogicalPlanParam, boolean printOptimizedLogicalPlanParam, boolean printPhysicalOpsOnly, - boolean executeQuery, boolean generateJobSpec, boolean printJob) { - this.optimize = optimize; - this.printExprParam = printExprParam; - this.printRewrittenExprParam = printRewrittenExprParam; - this.printLogicalPlanParam = printLogicalPlanParam; - this.printOptimizedLogicalPlanParam = printOptimizedLogicalPlanParam; - this.printPhysicalOpsOnly = printPhysicalOpsOnly; - this.executeQuery = executeQuery; - this.generateJobSpec = generateJobSpec; - this.printJob = printJob; - } + private final boolean optimize; - public boolean isPrintExprParam() { - return printExprParam; - } + // Output path for primary execution. + private final PrintWriter out; - public boolean isPrintRewrittenExprParam() { - return printRewrittenExprParam; - } + // Output format. + private final OutputFormat fmt; + + // Flags. + private final Map flags; - public boolean isPrintLogicalPlanParam() { - return printLogicalPlanParam; + /** + * Create a SessionConfig object with all default values: + * + * - All format flags set to "false". + * - All out-of-band outputs set to "null". + * - "Optimize" set to "true". + * - "Execute Query" set to "true". + * - "Generate Job Spec" set to "true". + * @param out PrintWriter for execution output. + * @param fmt Output format for execution output. + */ + public SessionConfig(PrintWriter out, OutputFormat fmt) { + this(out, fmt, true, true, true); } - public boolean isPrintOptimizedLogicalPlanParam() { - return printOptimizedLogicalPlanParam; + /** + * Create a SessionConfig object with all optional values set to defaults: + * + * - All format flags set to "false". + * - All out-of-band outputs set to "false". + * @param out PrintWriter for execution output. + * @param fmt Output format for execution output. + * @param optimize Whether to optimize the execution. + * @param executeQuery Whether to execute the query or not. + * @param generateJobSpec Whether to generate the Hyracks job specification (if + * false, job cannot be executed). + */ + public SessionConfig(PrintWriter out, OutputFormat fmt, boolean optimize, boolean executeQuery, boolean generateJobSpec) { + this.out = out; + this.fmt = fmt; + this.optimize = optimize; + this.executeQuery = executeQuery; + this.generateJobSpec = generateJobSpec; + this.flags = new HashMap(); } - public boolean isPrintJob() { - return printJob; + /** + * Retrieve the PrintWriter to produce output to. + */ + public PrintWriter out() { + return this.out; } - public boolean isPrintPhysicalOpsOnly() { - return printPhysicalOpsOnly; + /** + * Retrieve the OutputFormat for this execution. + */ + public OutputFormat fmt() { + return this.fmt; } + /** + * Retrieve the value of the "execute query" flag. + */ public boolean isExecuteQuery() { return executeQuery; } + /** + * Retrieve the value of the "optimize" flag. + */ public boolean isOptimize() { return optimize; } + /** + * Retrieve the value of the "generate job spec" flag. + */ public boolean isGenerateJobSpec() { return generateJobSpec; } -} \ No newline at end of file + + /** + * Specify all out-of-band settings at once. For convenience of older code. + */ + public void setOOBData(boolean expr_tree, boolean rewritten_expr_tree, + boolean logical_plan, boolean optimized_logical_plan, + boolean hyracks_job) { + this.set(OOB_EXPR_TREE, expr_tree); + this.set(OOB_REWRITTEN_EXPR_TREE, rewritten_expr_tree); + this.set(OOB_LOGICAL_PLAN, logical_plan); + this.set(OOB_OPTIMIZED_LOGICAL_PLAN, optimized_logical_plan); + this.set(OOB_HYRACKS_JOB, hyracks_job); + } + + /** + * Specify a flag. + * @param flag One of the OOB_ or FORMAT_ constants from this class. + * @param value Value for the flag (all flags default to "false"). + */ + public void set(String flag, boolean value) { + flags.put(flag, Boolean.valueOf(value)); + } + + /** + * Retrieve the setting of a format-specific flag. + * @param flag One of the FORMAT_ constants from this class. + * @returns true or false (all flags default to "false"). + */ + public boolean is(String flag) { + Boolean value = flags.get(flag); + return value == null ? false : value.booleanValue(); + } +} http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/APIServlet.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/APIServlet.java b/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/APIServlet.java index be88a1e..4984741 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/APIServlet.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/APIServlet.java @@ -30,8 +30,8 @@ import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; -import edu.uci.ics.asterix.api.common.APIFramework.OutputFormat; import edu.uci.ics.asterix.api.common.SessionConfig; +import edu.uci.ics.asterix.api.common.SessionConfig.OutputFormat; import edu.uci.ics.asterix.aql.base.Statement; import edu.uci.ics.asterix.aql.parser.AQLParser; import edu.uci.ics.asterix.aql.parser.ParseException; @@ -54,12 +54,23 @@ public class APIServlet extends HttpServlet { @Override public void doPost(HttpServletRequest request, HttpServletResponse response) throws IOException { - OutputFormat format = OutputFormat.HTML; - if (request.getContentType().equals("application/json")) { - format = OutputFormat.JSON; - } else if (request.getContentType().equals("text/plain")) { + OutputFormat format; + boolean csv_and_header = false; + String output = request.getParameter("output-format"); + if (output.equals("ADM")) { format = OutputFormat.ADM; } + else if (output.equals("CSV")) { + format = OutputFormat.CSV; + } + else if (output.equals("CSV-Header")) { + format = OutputFormat.CSV; + csv_and_header = true; + } + else { + // Default output format + format = OutputFormat.JSON; + } String query = request.getParameter("query"); String printExprParam = request.getParameter("print-expr-tree"); @@ -87,11 +98,14 @@ public class APIServlet extends HttpServlet { } AQLParser parser = new AQLParser(query); List aqlStatements = parser.parse(); - SessionConfig sessionConfig = new SessionConfig(true, isSet(printExprParam), - isSet(printRewrittenExprParam), isSet(printLogicalPlanParam), - isSet(printOptimizedLogicalPlanParam), false, isSet(executeQuery), true, isSet(printJob)); + SessionConfig sessionConfig = new SessionConfig(out, format, true, isSet(executeQuery), true); + sessionConfig.set(SessionConfig.FORMAT_HTML, true); + sessionConfig.set(SessionConfig.FORMAT_CSV_HEADER, csv_and_header); + sessionConfig.setOOBData(isSet(printExprParam), isSet(printRewrittenExprParam), + isSet(printLogicalPlanParam), isSet(printOptimizedLogicalPlanParam), + isSet(printJob)); MetadataManager.INSTANCE.init(); - AqlTranslator aqlTranslator = new AqlTranslator(aqlStatements, out, sessionConfig, format); + AqlTranslator aqlTranslator = new AqlTranslator(aqlStatements, sessionConfig); double duration = 0; long startTime = System.currentTimeMillis(); aqlTranslator.compileAndExecute(hcc, hds, AqlTranslator.ResultDelivery.SYNC); http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/QueryResultAPIServlet.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/QueryResultAPIServlet.java b/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/QueryResultAPIServlet.java index 5ccbfc8..3f104fe 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/QueryResultAPIServlet.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/QueryResultAPIServlet.java @@ -26,6 +26,8 @@ import org.json.JSONArray; import org.json.JSONObject; import edu.uci.ics.asterix.api.common.APIFramework; +import edu.uci.ics.asterix.api.common.SessionConfig; +import edu.uci.ics.asterix.api.common.SessionConfig.OutputFormat; import edu.uci.ics.asterix.result.ResultReader; import edu.uci.ics.asterix.result.ResultUtils; import edu.uci.ics.hyracks.api.client.HyracksConnection; @@ -78,30 +80,14 @@ public class QueryResultAPIServlet extends HttpServlet { ResultReader resultReader = new ResultReader(hcc, hds); resultReader.open(jobId, rsId); - APIFramework.OutputFormat format; - // QQQ This code is duplicated from RESTAPIServlet, and is - // erroneous anyway. The output format is determined by - // the initial query and cannot be modified here, so we need - // to find a way to send the same OutputFormat value here as - // was originally determined there. Need to save this value on + // QQQ The output format is determined by the initial + // query and cannot be modified here, so calling back to + // initResponse() is really an error. We need to find a + // way to send the same OutputFormat value here as was + // originally determined there. Need to save this value on // some object that we can obtain here. - String accept = request.getHeader("Accept"); - if ((accept == null) || (accept.contains("application/x-adm"))) { - format = APIFramework.OutputFormat.ADM; - response.setContentType("application/x-adm"); - } else if (accept.contains("text/html")) { - format = APIFramework.OutputFormat.HTML; - response.setContentType("text/html"); - } else if (accept.contains("text/csv")) { - format = APIFramework.OutputFormat.CSV; - response.setContentType("text/csv; header=present"); - } else { - // JSON output is the default; most generally useful for a - // programmatic HTTP API - format = APIFramework.OutputFormat.JSON; - response.setContentType("application/json"); - } - ResultUtils.displayResults(resultReader, out, format); + SessionConfig sessionConfig = RESTAPIServlet.initResponse(request, response); + ResultUtils.displayResults(resultReader, sessionConfig); } catch (Exception e) { out.println(e.getMessage()); http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/RESTAPIServlet.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/RESTAPIServlet.java b/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/RESTAPIServlet.java index e783741..4e8427d 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/RESTAPIServlet.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/api/http/servlet/RESTAPIServlet.java @@ -31,8 +31,8 @@ import org.apache.commons.io.IOUtils; import org.json.JSONObject; import edu.uci.ics.asterix.api.common.APIFramework; -import edu.uci.ics.asterix.api.common.APIFramework.OutputFormat; import edu.uci.ics.asterix.api.common.SessionConfig; +import edu.uci.ics.asterix.api.common.SessionConfig.OutputFormat; import edu.uci.ics.asterix.aql.base.Statement; import edu.uci.ics.asterix.aql.base.Statement.Kind; import edu.uci.ics.asterix.aql.parser.AQLParser; @@ -55,6 +55,67 @@ abstract class RESTAPIServlet extends HttpServlet { private static final String HYRACKS_DATASET_ATTR = "edu.uci.ics.asterix.HYRACKS_DATASET"; + /** + * Initialize the Content-Type of the response, and construct a + * SessionConfig with the appropriate output writer and output-format + * based on the Accept: header and other servlet parameters. + */ + static SessionConfig initResponse(HttpServletRequest request, HttpServletResponse response) + throws IOException { + response.setCharacterEncoding("utf-8"); + + // JSON output is the default; most generally useful for a + // programmatic HTTP API + OutputFormat format = OutputFormat.JSON; + + // First check the "output" servlet parameter. + String output = request.getParameter("output"); + String accept = request.getHeader("Accept"); + if (output != null) { + if (output.equals("CSV")) { + format = OutputFormat.CSV; + } + else if (output.equals("ADM")) { + format = OutputFormat.ADM; + } + } + else { + // Second check the Accept: HTTP header. + if (accept != null) { + if (accept.contains("application/x-adm")) { + format = OutputFormat.ADM; + } else if (accept.contains("text/csv")) { + format = OutputFormat.CSV; + } + } + } + + SessionConfig sessionConfig = new SessionConfig(response.getWriter(), format); + + // Now that format is set, output the content-type + switch (format) { + case ADM: + response.setContentType("application/x-adm"); + break; + case JSON: + response.setContentType("application/json"); + break; + case CSV: { + // Check for header parameter or in Accept:. + if ("present".equals(request.getParameter("header")) || + (accept != null && accept.contains("header=present"))) { + response.setContentType("text/csv; header=present"); + sessionConfig.set(SessionConfig.FORMAT_CSV_HEADER, true); + } + else { + response.setContentType("text/csv; header=absent"); + } + } + }; + + return sessionConfig; + } + @Override protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { @@ -72,28 +133,7 @@ abstract class RESTAPIServlet extends HttpServlet { public void handleRequest(HttpServletRequest request, HttpServletResponse response, String query) throws IOException { - response.setCharacterEncoding("utf-8"); - - PrintWriter out = response.getWriter(); - APIFramework.OutputFormat format; - // QQQ For now switch solely based on Accept header. Later will add - // an "output" query parameter. - String accept = request.getHeader("Accept"); - if ((accept == null) || (accept.contains("application/x-adm"))) { - format = OutputFormat.ADM; - response.setContentType("application/x-adm"); - } else if (accept.contains("text/html")) { - format = OutputFormat.HTML; - response.setContentType("text/html"); - } else if (accept.contains("text/csv")) { - format = OutputFormat.CSV; - response.setContentType("text/csv; header=present"); - } else { - // JSON output is the default; most generally useful for a - // programmatic HTTP API - format = APIFramework.OutputFormat.JSON; - response.setContentType("application/json"); - } + SessionConfig sessionConfig = initResponse(request, response); AqlTranslator.ResultDelivery resultDelivery = whichResultDelivery(request); ServletContext context = getServletContext(); @@ -113,21 +153,19 @@ abstract class RESTAPIServlet extends HttpServlet { AQLParser parser = new AQLParser(query); List aqlStatements = parser.parse(); if (!containsForbiddenStatements(aqlStatements)) { - SessionConfig sessionConfig = new SessionConfig(true, false, false, false, false, false, true, true, - false); MetadataManager.INSTANCE.init(); - AqlTranslator aqlTranslator = new AqlTranslator(aqlStatements, out, sessionConfig, format); + AqlTranslator aqlTranslator = new AqlTranslator(aqlStatements, sessionConfig); aqlTranslator.compileAndExecute(hcc, hds, resultDelivery); } } catch (ParseException | TokenMgrError | edu.uci.ics.asterix.aqlplus.parser.TokenMgrError pe) { GlobalConfig.ASTERIX_LOGGER.log(Level.SEVERE, pe.getMessage(), pe); String errorMessage = ResultUtils.buildParseExceptionMessage(pe, query); JSONObject errorResp = ResultUtils.getErrorResponse(2, errorMessage, "", ""); - out.write(errorResp.toString()); + sessionConfig.out().write(errorResp.toString()); response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR); } catch (Exception e) { GlobalConfig.ASTERIX_LOGGER.log(Level.SEVERE, e.getMessage(), e); - ResultUtils.apiErrorHandler(out, e); + ResultUtils.apiErrorHandler(sessionConfig.out(), e); response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR); } } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/api/java/AsterixJavaClient.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/api/java/AsterixJavaClient.java b/asterix-app/src/main/java/edu/uci/ics/asterix/api/java/AsterixJavaClient.java index e0ba829..54804e7 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/api/java/AsterixJavaClient.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/api/java/AsterixJavaClient.java @@ -19,9 +19,9 @@ import java.io.Reader; import java.util.List; import edu.uci.ics.asterix.api.common.APIFramework; -import edu.uci.ics.asterix.api.common.APIFramework.OutputFormat; import edu.uci.ics.asterix.api.common.Job; import edu.uci.ics.asterix.api.common.SessionConfig; +import edu.uci.ics.asterix.api.common.SessionConfig.OutputFormat; import edu.uci.ics.asterix.aql.base.Statement; import edu.uci.ics.asterix.aql.parser.AQLParser; import edu.uci.ics.asterix.aql.parser.ParseException; @@ -76,21 +76,25 @@ public class AsterixJavaClient { } MetadataManager.INSTANCE.init(); - SessionConfig pc = new SessionConfig(optimize, false, printRewrittenExpressions, printLogicalPlan, - printOptimizedPlan, printPhysicalOpsOnly, true, generateBinaryRuntime, printJob); + SessionConfig conf = new SessionConfig(writer, OutputFormat.ADM, optimize, true, generateBinaryRuntime); + conf.setOOBData(false, printRewrittenExpressions, printLogicalPlan, + printOptimizedPlan, printJob); + if (printPhysicalOpsOnly) { + conf.set(SessionConfig.FORMAT_ONLY_PHYSICAL_OPS, true); + } - AqlTranslator aqlTranslator = new AqlTranslator(aqlStatements, writer, pc, OutputFormat.ADM); + AqlTranslator aqlTranslator = new AqlTranslator(aqlStatements, conf); aqlTranslator.compileAndExecute(hcc, null, AqlTranslator.ResultDelivery.SYNC); writer.flush(); } public void execute() throws Exception { if (dmlJobs != null) { - APIFramework.executeJobArray(hcc, dmlJobs, writer, OutputFormat.ADM); + APIFramework.executeJobArray(hcc, dmlJobs, writer); } if (queryJobSpec != null) { - APIFramework.executeJobArray(hcc, new JobSpecification[] { queryJobSpec }, writer, OutputFormat.ADM); + APIFramework.executeJobArray(hcc, new JobSpecification[] { queryJobSpec }, writer); } } -} \ No newline at end of file +} http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java b/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java index 52fc25e..0184a65 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/aql/translator/AqlTranslator.java @@ -37,9 +37,9 @@ import org.json.JSONException; import org.json.JSONObject; import edu.uci.ics.asterix.api.common.APIFramework; -import edu.uci.ics.asterix.api.common.APIFramework.OutputFormat; import edu.uci.ics.asterix.api.common.Job; import edu.uci.ics.asterix.api.common.SessionConfig; +import edu.uci.ics.asterix.api.common.SessionConfig.OutputFormat; import edu.uci.ics.asterix.aql.base.Statement; import edu.uci.ics.asterix.aql.expression.CompactStatement; import edu.uci.ics.asterix.aql.expression.ConnectFeedStatement; @@ -188,18 +188,14 @@ public class AqlTranslator extends AbstractAqlTranslator { public static final boolean IS_DEBUG_MODE = false;//true private final List aqlStatements; - private final PrintWriter out; private final SessionConfig sessionConfig; - private final OutputFormat pdf; private Dataverse activeDefaultDataverse; private final List declaredFunctions; - public AqlTranslator(List aqlStatements, PrintWriter out, SessionConfig pc, APIFramework.OutputFormat pdf) + public AqlTranslator(List aqlStatements, SessionConfig conf) throws MetadataException, AsterixException { this.aqlStatements = aqlStatements; - this.out = out; - this.sessionConfig = pc; - this.pdf = pdf; + this.sessionConfig = conf; declaredFunctions = getDeclaredFunctions(aqlStatements); } @@ -1740,8 +1736,7 @@ public class AqlTranslator extends AbstractAqlTranslator { CompiledLoadFromFileStatement cls = new CompiledLoadFromFileStatement(dataverseName, loadStmt .getDatasetName().getValue(), loadStmt.getAdapter(), loadStmt.getProperties(), loadStmt.dataIsAlreadySorted()); - JobSpecification spec = APIFramework.compileQuery(null, metadataProvider, null, 0, null, sessionConfig, - out, pdf, cls); + JobSpecification spec = APIFramework.compileQuery(null, metadataProvider, null, 0, null, sessionConfig, cls); MetadataManager.INSTANCE.commitTransaction(mdTxnCtx); bActiveTxn = false; if (spec != null) { @@ -1837,12 +1832,12 @@ public class AqlTranslator extends AbstractAqlTranslator { // Query Rewriting (happens under the same ongoing metadata transaction) Pair reWrittenQuery = APIFramework.reWriteQuery(declaredFunctions, metadataProvider, query, - sessionConfig, out, pdf); + sessionConfig); // Query Compilation (happens under the same ongoing metadata // transaction) JobSpecification spec = APIFramework.compileQuery(declaredFunctions, metadataProvider, reWrittenQuery.first, - reWrittenQuery.second, stmt == null ? null : stmt.getDatasetName(), sessionConfig, out, pdf, stmt); + reWrittenQuery.second, stmt == null ? null : stmt.getDatasetName(), sessionConfig, stmt); return spec; @@ -2187,8 +2182,8 @@ public class AqlTranslator extends AbstractAqlTranslator { handle.put(jobId.getId()); handle.put(metadataProvider.getResultSetId().getId()); response.put("handle", handle); - out.print(response); - out.flush(); + sessionConfig.out().print(response); + sessionConfig.out().flush(); hcc.waitForCompletion(jobId); break; case SYNC: @@ -2198,10 +2193,11 @@ public class AqlTranslator extends AbstractAqlTranslator { // In this case (the normal case), we don't use the // "response" JSONObject - just stream the results // to the "out" PrintWriter - if (pdf == OutputFormat.CSV) { - ResultUtils.displayCSVHeader(metadataProvider.findOutputRecordType(), out); + if (sessionConfig.fmt() == OutputFormat.CSV && + sessionConfig.is(SessionConfig.FORMAT_CSV_HEADER)) { + ResultUtils.displayCSVHeader(metadataProvider.findOutputRecordType(), sessionConfig); } - ResultUtils.displayResults(resultReader, out, pdf); + ResultUtils.displayResults(resultReader, sessionConfig); hcc.waitForCompletion(jobId); break; @@ -2211,8 +2207,8 @@ public class AqlTranslator extends AbstractAqlTranslator { handle.put(metadataProvider.getResultSetId().getId()); response.put("handle", handle); hcc.waitForCompletion(jobId); - out.print(response); - out.flush(); + sessionConfig.out().print(response); + sessionConfig.out().flush(); break; default: break; @@ -2715,7 +2711,7 @@ public class AqlTranslator extends AbstractAqlTranslator { private JobId runJob(IHyracksClientConnection hcc, JobSpecification spec, boolean waitForCompletion) throws Exception { - JobId[] jobIds = executeJobArray(hcc, new Job[] { new Job(spec) }, out, waitForCompletion); + JobId[] jobIds = executeJobArray(hcc, new Job[] { new Job(spec) }, sessionConfig.out(), waitForCompletion); return jobIds[0]; } http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/hyracks/bootstrap/FeedLifecycleListener.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/hyracks/bootstrap/FeedLifecycleListener.java b/asterix-app/src/main/java/edu/uci/ics/asterix/hyracks/bootstrap/FeedLifecycleListener.java index 9c7ff81..4cdc91c 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/hyracks/bootstrap/FeedLifecycleListener.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/hyracks/bootstrap/FeedLifecycleListener.java @@ -1104,7 +1104,7 @@ public class FeedLifecycleListener implements IJobLifecycleListener, IClusterEve public void reviveFeed(String dataverse, String feedName, String dataset, String feedPolicy) { PrintWriter writer = new PrintWriter(System.out, true); - SessionConfig pc = new SessionConfig(true, false, false, false, false, false, true, true, false); + SessionConfig conf = new SessionConfig(writer, SessionConfig.OutputFormat.ADM); try { DataverseDecl dataverseDecl = new DataverseDecl(new Identifier(dataverse)); ConnectFeedStatement stmt = new ConnectFeedStatement(new Identifier(dataverse), @@ -1113,7 +1113,7 @@ public class FeedLifecycleListener implements IJobLifecycleListener, IClusterEve List statements = new ArrayList(); statements.add(dataverseDecl); statements.add(stmt); - AqlTranslator translator = new AqlTranslator(statements, writer, pc, APIFramework.OutputFormat.ADM); + AqlTranslator translator = new AqlTranslator(statements, conf); translator.compileAndExecute(AsterixAppContextInfo.getInstance().getHcc(), null, AqlTranslator.ResultDelivery.SYNC); if (LOGGER.isLoggable(Level.INFO)) { @@ -1147,7 +1147,7 @@ public class FeedLifecycleListener implements IJobLifecycleListener, IClusterEve private void endFeed(FeedInfo feedInfo) { MetadataTransactionContext ctx = null; PrintWriter writer = new PrintWriter(System.out, true); - SessionConfig pc = new SessionConfig(true, false, false, false, false, false, true, true, false); + SessionConfig conf = new SessionConfig(writer, SessionConfig.OutputFormat.ADM); try { ctx = MetadataManager.INSTANCE.beginTransaction(); DisconnectFeedStatement stmt = new DisconnectFeedStatement(new Identifier( @@ -1159,7 +1159,7 @@ public class FeedLifecycleListener implements IJobLifecycleListener, IClusterEve new Identifier(feedInfo.feedConnectionId.getDataverse())); statements.add(dataverseDecl); statements.add(stmt); - AqlTranslator translator = new AqlTranslator(statements, writer, pc, APIFramework.OutputFormat.ADM); + AqlTranslator translator = new AqlTranslator(statements, conf); translator.compileAndExecute(AsterixAppContextInfo.getInstance().getHcc(), null, AqlTranslator.ResultDelivery.SYNC); if (LOGGER.isLoggable(Level.INFO)) { http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/java/edu/uci/ics/asterix/result/ResultUtils.java ---------------------------------------------------------------------- diff --git a/asterix-app/src/main/java/edu/uci/ics/asterix/result/ResultUtils.java b/asterix-app/src/main/java/edu/uci/ics/asterix/result/ResultUtils.java index f558eb6..326697f 100644 --- a/asterix-app/src/main/java/edu/uci/ics/asterix/result/ResultUtils.java +++ b/asterix-app/src/main/java/edu/uci/ics/asterix/result/ResultUtils.java @@ -32,7 +32,8 @@ import org.json.JSONArray; import org.json.JSONException; import org.json.JSONObject; -import edu.uci.ics.asterix.api.common.APIFramework; +import edu.uci.ics.asterix.api.common.SessionConfig; +import edu.uci.ics.asterix.api.common.SessionConfig.OutputFormat; import edu.uci.ics.asterix.api.http.servlet.APIServlet; import edu.uci.ics.asterix.om.types.ARecordType; import edu.uci.ics.hyracks.algebricks.common.exceptions.AlgebricksException; @@ -61,22 +62,29 @@ public class ResultUtils { return s; } - public static void displayCSVHeader(ARecordType recordType, PrintWriter out) { + public static void displayCSVHeader(ARecordType recordType, SessionConfig conf) { + // If HTML-ifying, we have to output this here before the header - + // pretty ugly + if (conf.is(SessionConfig.FORMAT_HTML)) { + conf.out().println("

    Results:

    "); + conf.out().println("
    ");
+        }
+
         String[] fieldNames = recordType.getFieldNames();
         boolean notfirst = false;
         for (String name : fieldNames) {
             if (notfirst) {
-                out.print(',');
+                conf.out().print(',');
             }
             notfirst = true;
-            out.print('"');
-            out.print(name.replace("\"", "\"\""));
-            out.print('"');
+            conf.out().print('"');
+            conf.out().print(name.replace("\"", "\"\""));
+            conf.out().print('"');
         }
-        out.print("\r\n");
+        conf.out().print("\r\n");
     }
 
-    public static void displayResults(ResultReader resultReader, PrintWriter out, APIFramework.OutputFormat pdf)
+    public static void displayResults(ResultReader resultReader, SessionConfig conf)
             throws HyracksDataException {
         IFrameTupleAccessor fta = resultReader.getFrameTupleAccessor();
 
@@ -90,11 +98,15 @@ public class ResultUtils {
         // Whether this is the first instance being output
         boolean notfirst = false;
 
-        switch (pdf) {
-            case HTML:
-                out.println("
    Results:
    ");
-                out.println("
    ");
-                // Fall through
+        // If we're outputting CSV with a header, the HTML header was already
+        // output by displayCSVHeader(), so skip it here
+        if (conf.is(SessionConfig.FORMAT_HTML) &&
+            ! (conf.fmt() == OutputFormat.CSV && conf.is(SessionConfig.FORMAT_CSV_HEADER))) {
+            conf.out().println("
    Results:
    ");
+            conf.out().println("
    ");
+        }
+
+        switch (conf.fmt()) {
             case CSV:
                 need_commas = false;
                 break;
@@ -103,7 +115,7 @@ public class ResultUtils {
                 // Conveniently, JSON and ADM have the same syntax for an
                 // "ordered list", and our representation of the result of a
                 // statement is an ordered list of instances.
-                out.print("[ ");
+                conf.out().print("[ ");
                 break;
         }
 
@@ -119,19 +131,19 @@ public class ResultUtils {
                         bbis.setByteBuffer(buffer, start);
                         byte[] recordBytes = new byte[length];
                         int numread = bbis.read(recordBytes, 0, length);
-                        if (pdf == APIFramework.OutputFormat.CSV) {
+                        if (conf.fmt() == OutputFormat.CSV) {
                             if ( (numread > 0) && (recordBytes[numread-1] == '\n') ) {
                                 numread--;
                             }
                         }
                         result = new String(recordBytes, 0, numread, UTF_8);
                         if (need_commas && notfirst) {
-                            out.print(", ");
+                            conf.out().print(", ");
                         }
                         notfirst = true;
-                        out.print(result);
-                        if (pdf == APIFramework.OutputFormat.CSV) {
-                            out.print("\r\n");
+                        conf.out().print(result);
+                        if (conf.fmt() == OutputFormat.CSV) {
+                            conf.out().print("\r\n");
                         }
                     }
                     buffer.clear();
@@ -145,21 +157,21 @@ public class ResultUtils {
             } while (resultReader.read(buffer) > 0);
         }
 
-        out.flush();
+        conf.out().flush();
 
-        switch (pdf) {
-            case HTML:
-                out.println("
    ");
-                break;
+        switch (conf.fmt()) {
             case JSON:
             case ADM:
-                out.println(" ]");
+                conf.out().println(" ]");
                 break;
             case CSV:
                 // Nothing to do
                 break;
         }
 
+        if (conf.is(SessionConfig.FORMAT_HTML)) {
+            conf.out().println("
    ");
+        }
     }
 
     public static JSONObject getErrorResponse(int errorCode, String errorMessage, String errorSummary,

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/main/resources/webui/querytemplate.html
----------------------------------------------------------------------
diff --git a/asterix-app/src/main/resources/webui/querytemplate.html b/asterix-app/src/main/resources/webui/querytemplate.html
index 0adb518..bcd177c 100644
--- a/asterix-app/src/main/resources/webui/querytemplate.html
+++ b/asterix-app/src/main/resources/webui/querytemplate.html
@@ -186,15 +186,23 @@ $(document).ready(function() {
               
             
             
-          
    
-            
-            
-            
-            
-          
    
+            
    
+              
+              
+              
+              
+            
    
 
             
    
+              
               
               
               

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/test/resources/runtimets/results/csv/basic-types-header/basic-types.1.csv
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/runtimets/results/csv/basic-types-header/basic-types.1.csv b/asterix-app/src/test/resources/runtimets/results/csv/basic-types-header/basic-types.1.csv
new file mode 100644
index 0000000..941639e
--- /dev/null
+++ b/asterix-app/src/test/resources/runtimets/results/csv/basic-types-header/basic-types.1.csv
@@ -0,0 +1,2 @@
+"id","name","money"
+12345,Chris,18.25

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/test/resources/runtimets/results/csv/basic-types/basic-types.1.csv
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/runtimets/results/csv/basic-types/basic-types.1.csv b/asterix-app/src/test/resources/runtimets/results/csv/basic-types/basic-types.1.csv
index 941639e..c7fe1a0 100644
--- a/asterix-app/src/test/resources/runtimets/results/csv/basic-types/basic-types.1.csv
+++ b/asterix-app/src/test/resources/runtimets/results/csv/basic-types/basic-types.1.csv
@@ -1,2 +1 @@
-"id","name","money"
 12345,Chris,18.25

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-app/src/test/resources/runtimets/testsuite.xml
----------------------------------------------------------------------
diff --git a/asterix-app/src/test/resources/runtimets/testsuite.xml b/asterix-app/src/test/resources/runtimets/testsuite.xml
index 214056f..a22eb35 100644
--- a/asterix-app/src/test/resources/runtimets/testsuite.xml
+++ b/asterix-app/src/test/resources/runtimets/testsuite.xml
@@ -6569,6 +6569,11 @@
                 basic-types
             
         
+        
+            
+                basic-types-header
+            
+        
     
     
         

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-common/src/test/java/edu/uci/ics/asterix/test/aql/TestsUtils.java
----------------------------------------------------------------------
diff --git a/asterix-common/src/test/java/edu/uci/ics/asterix/test/aql/TestsUtils.java b/asterix-common/src/test/java/edu/uci/ics/asterix/test/aql/TestsUtils.java
index 99afe9a..70e2f6f 100644
--- a/asterix-common/src/test/java/edu/uci/ics/asterix/test/aql/TestsUtils.java
+++ b/asterix-common/src/test/java/edu/uci/ics/asterix/test/aql/TestsUtils.java
@@ -245,7 +245,7 @@ public class TestsUtils {
     }
 
     //Executes AQL in either async or async-defer mode.
-    public static InputStream executeAnyAQLAsync(String str, boolean defer) throws Exception {
+    public static InputStream executeAnyAQLAsync(String str, boolean defer, OutputFormat fmt) throws Exception {
         final String url = "http://localhost:19002/aql";
 
         // Create a method instance.
@@ -256,6 +256,7 @@ public class TestsUtils {
             method.setQueryString(new NameValuePair[] { new NameValuePair("mode", "asynchronous") });
         }
         method.setRequestEntity(new StringRequestEntity(str));
+        method.setRequestHeader("Accept", fmt.mimeType());
 
         // Provide custom retry handler is necessary
         method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false));
@@ -265,16 +266,17 @@ public class TestsUtils {
         String theHandle = IOUtils.toString(resultStream, "UTF-8");
 
         //take the handle and parse it so results can be retrieved 
-        InputStream handleResult = getHandleResult(theHandle);
+        InputStream handleResult = getHandleResult(theHandle, fmt);
         return handleResult;
     }
 
-    private static InputStream getHandleResult(String handle) throws Exception {
+    private static InputStream getHandleResult(String handle, OutputFormat fmt) throws Exception {
         final String url = "http://localhost:19002/query/result";
 
         // Create a method instance.
         GetMethod method = new GetMethod(url);
         method.setQueryString(new NameValuePair[] { new NameValuePair("handle", handle) });
+        method.setRequestHeader("Accept", fmt.mimeType());
 
         // Provide custom retry handler is necessary
         method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false));
@@ -417,9 +419,9 @@ public class TestsUtils {
                             if (ctx.getType().equalsIgnoreCase("query"))
                                 resultStream = executeQuery(statement, fmt);
                             else if (ctx.getType().equalsIgnoreCase("async"))
-                                resultStream = executeAnyAQLAsync(statement, false);
+                                resultStream = executeAnyAQLAsync(statement, false, fmt);
                             else if (ctx.getType().equalsIgnoreCase("asyncdefer"))
-                                resultStream = executeAnyAQLAsync(statement, true);
+                                resultStream = executeAnyAQLAsync(statement, true, fmt);
 
                             if (queryCount >= expectedResultFileCtxs.size()) {
                                 throw new IllegalStateException("no result file for " + testFile.toString());

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-doc/src/site/markdown/csv.md
----------------------------------------------------------------------
diff --git a/asterix-doc/src/site/markdown/csv.md b/asterix-doc/src/site/markdown/csv.md
index 338aa55..6e830f0 100644
--- a/asterix-doc/src/site/markdown/csv.md
+++ b/asterix-doc/src/site/markdown/csv.md
@@ -73,13 +73,9 @@ ignore this header by adding the parameter `"header"="true"`, eg.
      ("format"="delimited-text"),
      ("header"="true"));
 
-This is useful when the CSV file was produced from an earlier
-AsterixDB operation, as AsterixDB's CSV output always has a header
-line.
-
 CSV data may also be loaded from HDFS; see [Accessing External
 Data](aql/externaldata.html) for details.  However please note that
-CSV files on HDFS cannot have headers; attempting to specify
+CSV files on HDFS cannot have headers. Attempting to specify
 "header"="true" when reading from HDFS could result in non-header
 lines of data being skipped as well.
 
@@ -139,22 +135,35 @@ vital for the CSV to make any sense).
 #### Request the CSV Output Format
 
 When sending requests to the Asterix HTTP API, Asterix decides what
-format to use for rendering the results based on the `Accept` HTTP
-header. By default, Asterix will produce JSON output, and this can be
-requested explicitly by specifying the MIME type `application/json`.
-To select CSV output, set the `Accept` header on your request to the
-MIME type `text/csv`. The details of how to accomplish this will of
-course depend on what tools you are using to contact the HTTP API.
-Here is an example from a Unix shell prompt using the command-line
-utility "curl":
-
-    curl -G -H "Accept: text/csv" "http://localhost:19002/query" --data-urlencode '
-        query=use dataverse csv;
+format to use for rendering the results in one of two ways:
+
+* A HTTP query parameter named "output", which must be set to one of
+  the following values: `JSON`, `CSV`, or `ADM`.
+
+* Based on the [`Accept` HTTP header](http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1)
+
+By default, Asterix will produce JSON output.  To select CSV output,
+pass the parameter `output=CSV`, or set the `Accept` header on your
+request to the MIME type `text/csv`. The details of how to accomplish
+this will of course depend on what tools you are using to contact the
+HTTP API.  Here is an example from a Unix shell prompt using the
+command-line utility "curl" and specifying the "output query parameter:
+
+    curl -G "http://localhost:19002/query" \
+        --data-urlencode 'output=CSV' \
+        --data-urlencode 'query=use dataverse csv;
+              set output-record-type "csv_type";
+              for $n in dataset csv_set return $n;'
+
+Alternately, the same query using the `Accept` header:
+
+    curl -G -H "Accept: text/csv" "http://localhost:19002/query" \
+        --data-urlencode 'query=use dataverse csv;
               set output-record-type "csv_type";
               for $n in dataset csv_set return $n;'
 
 Similarly, a trivial Java program to execute the above sample query
-would be:
+and selecting CSV output via the `Accept` header would be:
 
     import java.net.HttpURLConnection;
     import java.net.URL;
@@ -184,13 +193,22 @@ would be:
 
 For either of the above examples, the output would be:
 
-    "id","money","name"
     1,18.5,"Peter Krabnitz"
     2,74.5,"Jesse Stevens"
 
 assuming you had already run the previous examples to create the
 dataverse and populate the dataset.
 
+#### Outputting CSV with a Header
+
+By default, AsterixDB will produce CSV results with no header line.
+If you want a header, you may explicitly request it in one of two ways:
+
+* By passing the HTTP query parameter "header" with the value "present"
+
+* By specifying the MIME type {{text/csv; header=present}} in your
+HTTP Accept: header.  This is consistent with RFC 4180.
+
 #### Issues with open datatypes and optional fields
 
 As mentioned earlier, CSV is a rigid format. It cannot express records

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-test-framework/src/main/java/edu/uci/ics/asterix/testframework/context/TestCaseContext.java
----------------------------------------------------------------------
diff --git a/asterix-test-framework/src/main/java/edu/uci/ics/asterix/testframework/context/TestCaseContext.java b/asterix-test-framework/src/main/java/edu/uci/ics/asterix/testframework/context/TestCaseContext.java
index 7589909..6488c95 100644
--- a/asterix-test-framework/src/main/java/edu/uci/ics/asterix/testframework/context/TestCaseContext.java
+++ b/asterix-test-framework/src/main/java/edu/uci/ics/asterix/testframework/context/TestCaseContext.java
@@ -36,7 +36,8 @@ public class TestCaseContext {
         NONE  ("", ""),
         ADM   ("adm", "application/x-adm"),
         JSON  ("json", "application/json"),
-        CSV   ("csv", "text/csv");
+        CSV   ("csv", "text/csv"),
+        CSV_HEADER ("csv-header", "text/csv; header=present");
 
         private final String extension;
         private final String mimetype;
@@ -62,6 +63,8 @@ public class TestCaseContext {
                 return OutputFormat.JSON;
             case CSV:
                 return OutputFormat.CSV;
+            case CSV_HEADER:
+                return OutputFormat.CSV_HEADER;
             case INSPECT:
             case IGNORE:
                 return OutputFormat.NONE;

http://git-wip-us.apache.org/repos/asf/incubator-asterixdb/blob/833774e0/asterix-test-framework/src/main/resources/Catalog.xsd
----------------------------------------------------------------------
diff --git a/asterix-test-framework/src/main/resources/Catalog.xsd b/asterix-test-framework/src/main/resources/Catalog.xsd
index a33399c..e43acdc 100644
--- a/asterix-test-framework/src/main/resources/Catalog.xsd
+++ b/asterix-test-framework/src/main/resources/Catalog.xsd
@@ -188,6 +188,7 @@
          
          
          
+