Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 50E34200C28 for ; Mon, 13 Mar 2017 22:19:22 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 4F4B6160B6C; Mon, 13 Mar 2017 21:19:22 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7E3C2160B5D for ; Mon, 13 Mar 2017 22:19:19 +0100 (CET) Received: (qmail 49695 invoked by uid 500); 13 Mar 2017 21:19:18 -0000 Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list notifications@asterixdb.apache.org Received: (qmail 49686 invoked by uid 99); 13 Mar 2017 21:19:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Mar 2017 21:19:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 101561AFA03 for ; Mon, 13 Mar 2017 21:19:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.919 X-Spam-Level: X-Spam-Status: No, score=0.919 tagged_above=-999 required=6.31 tests=[SPF_FAIL=0.919] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id BE-kn8P9XJyI for ; Mon, 13 Mar 2017 21:19:01 +0000 (UTC) Received: from unhygienix.ics.uci.edu (unhygienix.ics.uci.edu [128.195.14.130]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id E2CED5F5C4 for ; Mon, 13 Mar 2017 21:18:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by unhygienix.ics.uci.edu (Postfix) with ESMTP id CD6B9240B91; Mon, 13 Mar 2017 14:18:58 -0700 (PDT) Date: Mon, 13 Mar 2017 14:18:58 -0700 From: "Till Westmann (Code Review)" Message-ID: Reply-To: tillw@apache.org X-Gerrit-MessageType: newchange Subject: Change in asterixdb[master]: Fix async result delivery for compilation errors X-Gerrit-Change-Id: Ib594cdceb8ff2801f3e2af37be68c1609bef2a11 X-Gerrit-ChangeURL: X-Gerrit-Commit: 3fe18aa0e3c9b749fa87adba10d7c3d1aaadfa37 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 To: undisclosed-recipients:; archived-at: Mon, 13 Mar 2017 21:19:22 -0000 Till Westmann has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/1575 Change subject: Fix async result delivery for compilation errors ...................................................................... Fix async result delivery for compilation errors - Request submission returns after successful compilation or returns the compilation error. Change-Id: Ib594cdceb8ff2801f3e2af37be68c1609bef2a11 --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/AbstractQueryApiServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryResultApiServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryStatusApiServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RestApiServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultHandle.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultUtil.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/CancellationTestExecutor.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/ResultExtractor.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_1/query_result_1.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_2/query_result_2.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_3/query_result_3.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_4/query_result_4.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_1/query_status_1.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_2/query_status_2.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_3/query_status_3.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_4/query_status_4.1.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.5.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.7.pollget.http M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.8.get.http M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.1.async.sqlpp C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.uri C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.1.async.sqlpp C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.2.pollget.uri R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.3.get.uri C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.4.get.uri C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.5.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.1.async.sqlpp D asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.http R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.uri D asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.http R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.uri C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.uri M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.1.async.sqlpp D asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.http C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.uri R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.3.get.uri M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.1.deferred.sqlpp R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.2.get.uri M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.4.deferred.sqlpp R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.uri M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.6.async.sqlpp D asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.http C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.uri D asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.http C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.uri D asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.json A asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.regex A asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.1.ignore A asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.2.regex A asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.3.json A asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.4.json A asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.5.json M asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-running/async-running.2.json M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/Servlets.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/DatasetJobRecord.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/IDatasetPartitionManager.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java M hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties D hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/DatasetClientContext.java M hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDataset.java M hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDatasetReader.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/dataset/DatasetDirectoryService.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/DatasetPartitionManager.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/ResultState.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/result/ResultWriterOperatorDescriptor.java M hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/AbstractServlet.java 70 files changed, 613 insertions(+), 455 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/75/1575/1 diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/AbstractQueryApiServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/AbstractQueryApiServlet.java index f156de5..2b7429e 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/AbstractQueryApiServlet.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/AbstractQueryApiServlet.java @@ -21,27 +21,20 @@ import static org.apache.asterix.api.http.servlet.ServletConstants.HYRACKS_CONNECTION_ATTR; import static org.apache.asterix.api.http.servlet.ServletConstants.HYRACKS_DATASET_ATTR; -import java.io.IOException; import java.io.PrintWriter; import java.util.UUID; import java.util.concurrent.ConcurrentMap; -import java.util.logging.Level; -import java.util.logging.Logger; import org.apache.asterix.app.result.ResultReader; +import org.apache.asterix.app.result.ResultUtil; import org.apache.asterix.common.exceptions.ErrorCode; import org.apache.asterix.common.exceptions.RuntimeDataException; import org.apache.hyracks.api.client.IHyracksClientConnection; import org.apache.hyracks.api.dataset.IHyracksDataset; import org.apache.hyracks.client.dataset.HyracksDataset; -import org.apache.hyracks.http.api.IServletRequest; import org.apache.hyracks.http.server.AbstractServlet; -import com.fasterxml.jackson.core.JsonProcessingException; -import com.fasterxml.jackson.databind.JsonNode; -import com.fasterxml.jackson.databind.ObjectMapper; - -class AbstractQueryApiServlet extends AbstractServlet { +public class AbstractQueryApiServlet extends AbstractServlet { public enum ResultFields { REQUEST_ID("requestID"), @@ -83,6 +76,22 @@ } } + public enum ErrorField { + CODE("code"), + MSG("msg"), + STACK("stack"); + + private final String str; + + ErrorField(String str) { + this.str = str; + } + + public String str() { + return str; + } + } + AbstractQueryApiServlet(ConcurrentMap ctx, String[] paths) { super(ctx, paths); } @@ -112,55 +121,14 @@ } } - protected static JsonNode parseHandle(ObjectMapper om, String strHandle, Logger logger) throws IOException { - if (strHandle == null) { - logger.log(Level.WARNING, "No handle provided"); - } else { - try { - JsonNode handleObj = om.readTree(strHandle); - return handleObj.get("handle"); - } catch (JsonProcessingException e) { // NOSONAR - logger.log(Level.WARNING, "Invalid handle: \"" + strHandle + "\""); - } - } - return null; - } - protected static UUID printRequestId(PrintWriter pw) { UUID requestId = UUID.randomUUID(); - printField(pw, ResultFields.REQUEST_ID.str(), requestId.toString()); + ResultUtil.printField(pw, ResultFields.REQUEST_ID.str(), requestId.toString()); return requestId; } - protected static void printStatus(PrintWriter pw, ResultStatus rs) { - printField(pw, ResultFields.STATUS.str(), rs.str()); - } - - protected static void printHandle(PrintWriter pw, String handle) { - printField(pw, ResultFields.HANDLE.str(), handle); - } - - protected static void printField(PrintWriter pw, String name, String value) { - printField(pw, name, value, true); - } - - protected static void printField(PrintWriter pw, String name, String value, boolean comma) { - printFieldInternal(pw, name, "\"" + value + "\"", comma); - } - - protected static void printField(PrintWriter pw, String name, long value, boolean comma) { - printFieldInternal(pw, name, String.valueOf(value), comma); - } - - protected static void printFieldInternal(PrintWriter pw, String name, String value, boolean comma) { - pw.print("\t\""); - pw.print(name); - pw.print("\": "); - pw.print(value); - if (comma) { - pw.print(','); - } - pw.print('\n'); + protected static void printHandle(PrintWriter pw, String handle, boolean comma) { + ResultUtil.printField(pw, ResultFields.HANDLE.str(), handle, comma); } } diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryResultApiServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryResultApiServlet.java index 292dd2a..bfc67cf 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryResultApiServlet.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryResultApiServlet.java @@ -23,10 +23,12 @@ import java.util.logging.Level; import java.util.logging.Logger; +import org.apache.asterix.app.result.ResultHandle; import org.apache.asterix.app.result.ResultReader; import org.apache.asterix.app.result.ResultUtil; import org.apache.asterix.translator.IStatementExecutor.Stats; import org.apache.asterix.translator.SessionConfig; +import org.apache.hyracks.api.dataset.DatasetJobRecord; import org.apache.hyracks.api.dataset.IHyracksDataset; import org.apache.hyracks.api.dataset.ResultSetId; import org.apache.hyracks.api.exceptions.ErrorCode; @@ -50,23 +52,46 @@ @Override protected void get(IServletRequest request, IServletResponse response) throws Exception { - response.setStatus(HttpResponseStatus.OK); // TODO this seems wrong ... HttpUtil.setContentType(response, HttpUtil.ContentType.TEXT_HTML, HttpUtil.Encoding.UTF8); - String strHandle = request.getParameter("handle"); PrintWriter out = response.writer(); + final String strHandle = localPath(request); + final ResultHandle handle = ResultHandle.parse(strHandle); + if (handle == null) { + response.setStatus(HttpResponseStatus.BAD_REQUEST); + return; + } + + IHyracksDataset hds = getHyracksDataset(); + ResultReader resultReader = new ResultReader(hds, handle.getJobId(), handle.getResultSetId()); + + try { - JsonNode handle = parseHandle(new ObjectMapper(), strHandle, LOGGER); - if (handle == null) { - response.setStatus(HttpResponseStatus.BAD_REQUEST); + DatasetJobRecord.Status status = resultReader.getStatus(); + + final HttpResponseStatus httpStatus; + if (status == null) { + httpStatus = HttpResponseStatus.NOT_FOUND; + } else { + switch (status.getState()) { + case SUCCESS: + httpStatus = HttpResponseStatus.OK; + break; + case RUNNING: + case IDLE: + case FAILED: + httpStatus = HttpResponseStatus.NOT_FOUND; + break; + default: + httpStatus = HttpResponseStatus.INTERNAL_SERVER_ERROR; + break; + } + } + response.setStatus(httpStatus); + if (httpStatus != HttpResponseStatus.OK) { return; } - JobId jobId = new JobId(handle.get(0).asLong()); - ResultSetId rsId = new ResultSetId(handle.get(1).asLong()); - - IHyracksDataset hds = getHyracksDataset(); - ResultReader resultReader = new ResultReader(hds, jobId, rsId); // QQQ The output format is determined by the initial // query and cannot be modified here, so calling back to @@ -94,4 +119,5 @@ LOGGER.warning("Error flushing output writer for \"" + strHandle + "\""); } } + } diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java index 42bb4f9..0bf58d3 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryServiceServlet.java @@ -55,7 +55,6 @@ import org.apache.hyracks.http.server.utils.HttpUtil; import com.fasterxml.jackson.core.JsonParseException; -import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.JsonMappingException; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; @@ -124,22 +123,6 @@ } } - private enum ErrorField { - CODE("code"), - MSG("msg"), - STACK("stack"); - - private final String str; - - ErrorField(String str) { - this.str = str; - } - - public String str() { - return str; - } - } - private enum Metrics { ELAPSED_TIME("elapsedTime"), EXECUTION_TIME("executionTime"), @@ -186,11 +169,14 @@ } static class RequestParameters { + String host; + String path; String statement; String format; boolean pretty; String clientContextID; String mode; + @Override public String toString() { @@ -199,6 +185,8 @@ public StringBuilder append(final StringBuilder sb) { sb.append("{ "); + sb.append("\"host\": \"").append(host).append("\", "); + sb.append("\"path\": \"").append(path).append("\", "); sb.append("\"statement\": \""); JSONUtil.escape(sb, statement); sb.append("\", "); @@ -249,7 +237,8 @@ return SessionConfig.OutputFormat.CLEAN_JSON; } - private static SessionConfig createSessionConfig(RequestParameters param, PrintWriter resultWriter) { + private static SessionConfig createSessionConfig(RequestParameters param, String handleUrl, PrintWriter + resultWriter) { SessionConfig.ResultDecorator resultPrefix = new SessionConfig.ResultDecorator() { int resultNo = -1; @@ -268,8 +257,8 @@ SessionConfig.ResultDecorator resultPostfix = app -> app.append("\t,\n"); SessionConfig.ResultDecorator handlePrefix = - app -> app.append("\t\"").append(ResultFields.HANDLE.str()).append("\": "); - SessionConfig.ResultDecorator handlePostfix = app -> app.append(",\n"); + app -> app.append("\t\"").append(ResultFields.HANDLE.str()).append("\": \"").append(handleUrl); + SessionConfig.ResultDecorator handlePostfix = app -> app.append("\",\n"); SessionConfig.OutputFormat format = getFormat(param.format); SessionConfig sessionConfig = @@ -285,43 +274,27 @@ private static void printClientContextID(PrintWriter pw, RequestParameters params) { if (params.clientContextID != null && !params.clientContextID.isEmpty()) { - printField(pw, ResultFields.CLIENT_ID.str(), params.clientContextID); + ResultUtil.printField(pw, ResultFields.CLIENT_ID.str(), params.clientContextID); } } private static void printSignature(PrintWriter pw) { - printField(pw, ResultFields.SIGNATURE.str(), "*"); + ResultUtil.printField(pw, ResultFields.SIGNATURE.str(), "*"); } private static void printType(PrintWriter pw, SessionConfig sessionConfig) { switch (sessionConfig.fmt()) { case ADM: - printField(pw, ResultFields.TYPE.str(), HttpUtil.ContentType.APPLICATION_ADM); + ResultUtil.printField(pw, ResultFields.TYPE.str(), HttpUtil.ContentType.APPLICATION_ADM); break; case CSV: String contentType = HttpUtil.ContentType.CSV + "; header=" + (sessionConfig.is(SessionConfig.FORMAT_CSV_HEADER) ? "present" : "absent"); - printField(pw, ResultFields.TYPE.str(), contentType); + ResultUtil.printField(pw, ResultFields.TYPE.str(), contentType); break; default: break; } - } - - private static void printError(PrintWriter pw, Throwable e) throws JsonProcessingException { - Throwable rootCause = ResultUtil.getRootCause(e); - if (rootCause == null) { - rootCause = e; - } - final boolean addStack = false; - pw.print("\t\""); - pw.print(ResultFields.ERRORS.str()); - pw.print("\": [{ \n"); - printField(pw, ErrorField.CODE.str(), "1"); - final String msg = rootCause.getMessage(); - printField(pw, ErrorField.MSG.str(), JSONUtil.escape(msg != null ? msg : rootCause.getClass().getSimpleName()), - addStack); - pw.print("\t}],\n"); } private static void printMetrics(PrintWriter pw, long elapsedTime, long executionTime, long resultCount, @@ -330,13 +303,13 @@ pw.print(ResultFields.METRICS.str()); pw.print("\": {\n"); pw.print("\t"); - printField(pw, Metrics.ELAPSED_TIME.str(), TimeUnit.formatNanos(elapsedTime)); + ResultUtil.printField(pw, Metrics.ELAPSED_TIME.str(), TimeUnit.formatNanos(elapsedTime)); pw.print("\t"); - printField(pw, Metrics.EXECUTION_TIME.str(), TimeUnit.formatNanos(executionTime)); + ResultUtil.printField(pw, Metrics.EXECUTION_TIME.str(), TimeUnit.formatNanos(executionTime)); pw.print("\t"); - printField(pw, Metrics.RESULT_COUNT.str(), resultCount, true); + ResultUtil.printField(pw, Metrics.RESULT_COUNT.str(), resultCount, true); pw.print("\t"); - printField(pw, Metrics.RESULT_SIZE.str(), resultSize, false); + ResultUtil.printField(pw, Metrics.RESULT_SIZE.str(), resultSize, false); pw.print("\t}\n"); } @@ -355,6 +328,8 @@ int sep = contentTypeParam.indexOf(';'); final String contentType = sep < 0 ? contentTypeParam.trim() : contentTypeParam.substring(0, sep).trim(); RequestParameters param = new RequestParameters(); + param.host = host(request); + param.path = servletPath(request); if (HttpUtil.ContentType.APPLICATION_JSON.equals(contentType)) { try { JsonNode jsonRequest = new ObjectMapper().readTree(getRequestBody(request)); @@ -394,6 +369,32 @@ } } + private static String handlePath(ResultDelivery delivery) { + switch (delivery) { + case ASYNC: + return "/status/"; + case DEFERRED: + return "/result/"; + case IMMEDIATE: + default: + return ""; + } + } + + /** + * Determines the URL for a result handle based on the host and the path of the incoming request and the result + * delivery mode. Usually there will be a "status" endpoint for ASYNC requests that exposes the status of the + * execution and a "result" endpoint for DEFERRED requests that will deliver the result for a successful execution. + * + * @param host hostname used for this request + * @param path servlet path for this request + * @param delivery ResultDelivery mode for this request + * @return a handle (URL) that allows a client to access further information for this request + */ + protected String getHandleUrl(String host, String path, ResultDelivery delivery) { + return "http://" + host + path + handlePath(delivery); + } + private void handleRequest(RequestParameters param, IServletResponse response) throws IOException { LOGGER.info(param.toString()); long elapsedStart = System.nanoTime(); @@ -402,7 +403,8 @@ ResultDelivery delivery = parseResultDelivery(param.mode); - SessionConfig sessionConfig = createSessionConfig(param, resultWriter); + String handleUrl = getHandleUrl(param.host, param.path, delivery); + SessionConfig sessionConfig = createSessionConfig(param, handleUrl, resultWriter); HttpUtil.setContentType(response, HttpUtil.ContentType.APPLICATION_JSON, HttpUtil.Encoding.UTF8); HttpResponseStatus status = HttpResponseStatus.OK; @@ -433,16 +435,18 @@ translator.compileAndExecute(getHyracksClientConnection(), getHyracksDataset(), delivery, stats, param.clientContextID, queryCtx); execEnd = System.nanoTime(); - printStatus(resultWriter, ResultDelivery.ASYNC == delivery ? ResultStatus.RUNNING : ResultStatus.SUCCESS); + if (ResultDelivery.IMMEDIATE == delivery || ResultDelivery.DEFERRED == delivery) { + ResultUtil.printStatus(resultWriter, ResultStatus.SUCCESS); + } } catch (AsterixException | TokenMgrError | org.apache.asterix.aqlplus.parser.TokenMgrError pe) { GlobalConfig.ASTERIX_LOGGER.log(Level.SEVERE, pe.getMessage(), pe); - printError(resultWriter, pe); - printStatus(resultWriter, ResultStatus.FATAL); + ResultUtil.printError(resultWriter, pe); + ResultUtil.printStatus(resultWriter, ResultStatus.FATAL); status = HttpResponseStatus.BAD_REQUEST; } catch (Exception e) { GlobalConfig.ASTERIX_LOGGER.log(Level.SEVERE, e.getMessage(), e); - printError(resultWriter, e); - printStatus(resultWriter, ResultStatus.FATAL); + ResultUtil.printError(resultWriter, e); + ResultUtil.printStatus(resultWriter, ResultStatus.FATAL); status = HttpResponseStatus.INTERNAL_SERVER_ERROR; } finally { if (execStart == -1) { diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryStatusApiServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryStatusApiServlet.java index 9aa74c5..4f575c9 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryStatusApiServlet.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryStatusApiServlet.java @@ -18,23 +18,23 @@ */ package org.apache.asterix.api.http.server; +import static org.apache.asterix.api.http.server.AbstractQueryApiServlet.ResultStatus.FAILED; + import java.io.PrintWriter; import java.io.StringWriter; +import java.util.List; import java.util.concurrent.ConcurrentMap; import java.util.logging.Level; import java.util.logging.Logger; +import org.apache.asterix.app.result.ResultHandle; import org.apache.asterix.app.result.ResultReader; +import org.apache.asterix.app.result.ResultUtil; import org.apache.hyracks.api.dataset.DatasetJobRecord; import org.apache.hyracks.api.dataset.IHyracksDataset; -import org.apache.hyracks.api.dataset.ResultSetId; -import org.apache.hyracks.api.job.JobId; import org.apache.hyracks.http.api.IServletRequest; import org.apache.hyracks.http.api.IServletResponse; import org.apache.hyracks.http.server.utils.HttpUtil; - -import com.fasterxml.jackson.databind.JsonNode; -import com.fasterxml.jackson.databind.ObjectMapper; import io.netty.handler.codec.http.HttpResponseStatus; @@ -47,26 +47,25 @@ @Override protected void get(IServletRequest request, IServletResponse response) throws Exception { - String strHandle = request.getParameter("handle"); - ObjectMapper om = new ObjectMapper(); - JsonNode handle = parseHandle(om, strHandle, LOGGER); + final String strHandle = localPath(request); + final ResultHandle handle = ResultHandle.parse(strHandle); if (handle == null) { response.setStatus(HttpResponseStatus.BAD_REQUEST); return; } - JobId jobId = new JobId(handle.get(0).asLong()); - ResultSetId rsId = new ResultSetId(handle.get(1).asLong()); IHyracksDataset hds = getHyracksDataset(); - ResultReader resultReader = new ResultReader(hds, jobId, rsId); + ResultReader resultReader = new ResultReader(hds, handle.getJobId(), handle.getResultSetId()); - ResultStatus resultStatus = resultStatus(resultReader.getStatus()); - - if (resultStatus == null) { + final DatasetJobRecord.Status resultReaderStatus = resultReader.getStatus(); + if (resultReaderStatus == null) { LOGGER.log(Level.INFO, "No results for: \"" + strHandle + "\""); response.setStatus(HttpResponseStatus.NOT_FOUND); return; } + + ResultStatus resultStatus = resultStatus(resultReaderStatus); + Exception ex = extractException(resultReaderStatus); final StringWriter stringWriter = new StringWriter(); final PrintWriter resultWriter = new PrintWriter(stringWriter); @@ -75,12 +74,14 @@ HttpResponseStatus httpStatus = HttpResponseStatus.OK; resultWriter.print("{\n"); - printStatus(resultWriter, resultStatus); + ResultUtil.printStatus(resultWriter, resultStatus, (ex != null) || ResultStatus.SUCCESS == resultStatus); if (ResultStatus.SUCCESS == resultStatus) { String servletPath = servletPath(request).replace("status", "result"); - String resHandle = "http://" + host(request) + servletPath + localPath(request); - printHandle(resultWriter, resHandle); + String resHandle = "http://" + host(request) + servletPath + strHandle; + printHandle(resultWriter, resHandle, false); + } else if (ex != null) { + ResultUtil.printError(resultWriter, ex, false); } resultWriter.print("}\n"); @@ -95,19 +96,29 @@ } ResultStatus resultStatus(DatasetJobRecord.Status status) { - if (status == null) { - return null; - } - switch (status) { + switch (status.getState()) { case IDLE: case RUNNING: return ResultStatus.RUNNING; case SUCCESS: return ResultStatus.SUCCESS; case FAILED: - return ResultStatus.FAILED; + return FAILED; default: return ResultStatus.FATAL; } } + + Exception extractException(DatasetJobRecord.Status status) { + switch (status.getState()) { + case FAILED: + List exceptions = status.getExceptions(); + if (exceptions != null && !exceptions.isEmpty()) { + return exceptions.get(0); + } + return null; + default: + return null; + } + } } diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RestApiServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RestApiServlet.java index d7edb23..7a587ef 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RestApiServlet.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RestApiServlet.java @@ -114,8 +114,8 @@ } SessionConfig.ResultDecorator handlePrefix = - (AlgebricksAppendable app) -> app.append("{ \"").append("handle").append("\": "); - SessionConfig.ResultDecorator handlePostfix = (AlgebricksAppendable app) -> app.append(" }"); + (AlgebricksAppendable app) -> app.append("{ \"").append("handle").append("\": \""); + SessionConfig.ResultDecorator handlePostfix = (AlgebricksAppendable app) -> app.append("\" }"); SessionConfig sessionConfig = new SessionConfig(response.writer(), format, null, null, handlePrefix, handlePostfix); diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultHandle.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultHandle.java index 05eb967..7809a84 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultHandle.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultHandle.java @@ -24,16 +24,51 @@ import org.apache.hyracks.api.job.JobId; public class ResultHandle { - private long jobId; - private long resultSetId; + private final JobId jobId; + private final ResultSetId resultSetId; public ResultHandle(JobId jobId, ResultSetId resultSetId) { - this.jobId = jobId.getId(); - this.resultSetId = resultSetId.getId(); + this.jobId = jobId; + this.resultSetId = resultSetId; + } + + public ResultHandle(long jobId, long resultSetId) { + this(new JobId(jobId), new ResultSetId(resultSetId)); + } + + public static ResultHandle parse(String str) { + int dash = str.indexOf('-'); + if (dash < 1) { + return null; + } + int start = 0; + while (str.charAt(start) == '/') { + ++start; + } + String jobIdStr = str.substring(start, dash); + String resIdStr = str.substring(dash + 1); + try { + return new ResultHandle(Long.parseLong(jobIdStr), Long.parseLong(resIdStr)); + } catch (NumberFormatException e) { + return null; + } + } + + public JobId getJobId() { + return jobId; + } + + public ResultSetId getResultSetId() { + return resultSetId; + } + + @Override + public String toString() { + return Long.toString(jobId.getId()) + "-" + Long.toString(resultSetId.getId()); + } public AlgebricksAppendable append(AlgebricksAppendable app) throws AlgebricksException { - return app.append("[").append(String.valueOf(jobId)).append(", ").append(String.valueOf(resultSetId)) - .append("]"); + return app.append(toString()); } } diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultUtil.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultUtil.java index b730989..7d1e0b0 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultUtil.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/ResultUtil.java @@ -33,6 +33,10 @@ import java.util.stream.Collectors; import java.util.stream.Stream; +import com.fasterxml.jackson.core.JsonProcessingException; +import org.apache.asterix.api.http.server.AbstractQueryApiServlet; +import org.apache.asterix.api.http.server.QueryServiceServlet; +import org.apache.asterix.common.utils.JSONUtil; import org.apache.asterix.om.types.ARecordType; import org.apache.asterix.translator.IStatementExecutor.Stats; import org.apache.asterix.translator.SessionConfig; @@ -92,6 +96,58 @@ } catch (AlgebricksException e) { throw new HyracksDataException(e); } + } + + public static void printStatus(PrintWriter pw, AbstractQueryApiServlet.ResultStatus rs) { + printStatus(pw, rs, true); + } + + public static void printStatus(PrintWriter pw, AbstractQueryApiServlet.ResultStatus rs, boolean comma) { + printField(pw, AbstractQueryApiServlet.ResultFields.STATUS.str(), rs.str(), comma); + } + + public static void printError(PrintWriter pw, Throwable e) { + printError(pw, e, true); + } + + public static void printError(PrintWriter pw, Throwable e, boolean comma) { + Throwable rootCause = getRootCause(e); + if (rootCause == null) { + rootCause = e; + } + final boolean addStack = false; + pw.print("\t\""); + pw.print(AbstractQueryApiServlet.ResultFields.ERRORS.str()); + pw.print("\": [{ \n"); + printField(pw, QueryServiceServlet.ErrorField.CODE.str(), "1"); + final String msg = rootCause.getMessage(); + printField(pw, QueryServiceServlet.ErrorField.MSG.str(), JSONUtil + .escape(msg != null ? msg : rootCause.getClass().getSimpleName()), + addStack); + pw.print(comma ? "\t}],\n" : "\t}]\n"); + } + + public static void printField(PrintWriter pw, String name, String value) { + printField(pw, name, value, true); + } + + public static void printField(PrintWriter pw, String name, String value, boolean comma) { + printFieldInternal(pw, name, "\"" + value + "\"", comma); + } + + public static void printField(PrintWriter pw, String name, long value, boolean comma) { + printFieldInternal(pw, name, String.valueOf(value), comma); + } + + protected static void printFieldInternal(PrintWriter pw, String name, String value, boolean comma) { + pw.print("\t\""); + pw.print(name); + pw.print("\": "); + pw.print(value); + if (comma) { + pw.print(','); + } + pw.print('\n'); } public static ObjectNode getErrorResponse(int errorCode, String errorMessage, String errorSummary, @@ -259,4 +315,5 @@ } return errorTemplate; } + } diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java index 26a6ebd..cf9578d 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java @@ -45,6 +45,7 @@ import org.apache.asterix.active.IActiveEventSubscriber; import org.apache.asterix.algebra.extension.IExtensionStatement; import org.apache.asterix.api.common.APIFramework; +import org.apache.asterix.api.http.server.AbstractQueryApiServlet; import org.apache.asterix.api.http.server.ApiServlet; import org.apache.asterix.app.result.ResultHandle; import org.apache.asterix.app.result.ResultReader; @@ -164,6 +165,7 @@ import org.apache.asterix.utils.FeedOperations; import org.apache.asterix.utils.FlushDatasetUtil; import org.apache.commons.lang3.StringUtils; +import org.apache.commons.lang3.mutable.Mutable; import org.apache.commons.lang3.mutable.MutableBoolean; import org.apache.commons.lang3.mutable.MutableObject; import org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint; @@ -2405,11 +2407,12 @@ switch (resultDelivery) { case ASYNC: MutableBoolean printed = new MutableBoolean(false); + Mutable jobId = new MutableObject<>(JobId.INVALID); executorService.submit(() -> { - JobId jobId = null; try { - jobId = createAndRunJob(hcc, compiler, locker, resultDelivery, id -> { + createAndRunJob(hcc, jobId, compiler, locker, resultDelivery, id -> { final ResultHandle handle = new ResultHandle(id, resultSetId); + ResultUtil.printStatus(sessionConfig.out(), AbstractQueryApiServlet.ResultStatus.RUNNING); ResultUtil.printResultHandle(handle, sessionConfig); synchronized (printed) { printed.setTrue(); @@ -2417,8 +2420,20 @@ } }, clientContextId, ctx); } catch (Exception e) { - GlobalConfig.ASTERIX_LOGGER.log(Level.SEVERE, - resultDelivery.name() + " job " + "with id " + jobId + " failed", e); + synchronized (jobId) { + if (JobId.INVALID.equals(jobId.getValue())) { + // compilation failed + ResultUtil.printStatus(sessionConfig.out(), AbstractQueryApiServlet.ResultStatus.FATAL); + ResultUtil.printError(sessionConfig.out(), e); + synchronized (printed) { + printed.setTrue(); + printed.notify(); + } + } else { + GlobalConfig.ASTERIX_LOGGER.log(Level.SEVERE, resultDelivery.name() + " job " + + "with id " + jobId.getValue() + " " + "failed", e); + } + } } }); synchronized (printed) { @@ -2428,14 +2443,14 @@ } break; case IMMEDIATE: - createAndRunJob(hcc, compiler, locker, resultDelivery, id -> { + createAndRunJob(hcc, null, compiler, locker, resultDelivery, id -> { final ResultReader resultReader = new ResultReader(hdc, id, resultSetId); ResultUtil.printResults(resultReader, sessionConfig, stats, metadataProvider.findOutputRecordType()); }, clientContextId, ctx); break; case DEFERRED: - createAndRunJob(hcc, compiler, locker, resultDelivery, id -> { + createAndRunJob(hcc, null, compiler, locker, resultDelivery, id -> { ResultUtil.printResultHandle(new ResultHandle(id, resultSetId), sessionConfig); }, clientContextId, ctx); break; @@ -2444,19 +2459,23 @@ } } - private static JobId createAndRunJob(IHyracksClientConnection hcc, IStatementCompiler compiler, + private static void createAndRunJob(IHyracksClientConnection hcc, Mutable jId, IStatementCompiler compiler, IMetadataLocker locker, ResultDelivery resultDelivery, IResultPrinter printer, String clientContextId, IStatementExecutorContext ctx) throws Exception { locker.lock(); try { final JobSpecification jobSpec = compiler.compile(); if (jobSpec == null) { - return JobId.INVALID; + return; } final JobId jobId = JobUtils.runJob(hcc, jobSpec, false); - if (ctx != null && clientContextId != null) { ctx.put(clientContextId, jobId); // Adds the running job into the context. + } + if (jId != null) { + synchronized (jId) { + jId.setValue(jobId); + } } if (ResultDelivery.ASYNC == resultDelivery) { printer.print(jobId); @@ -2465,7 +2484,6 @@ hcc.waitForCompletion(jobId); printer.print(jobId); } - return jobId; } finally { // No matter the job succeeds or fails, removes it into the context. if (ctx != null && clientContextId != null) { diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/CancellationTestExecutor.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/CancellationTestExecutor.java index 97101ba..fdda44c 100644 --- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/CancellationTestExecutor.java +++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/CancellationTestExecutor.java @@ -46,9 +46,8 @@ List params, boolean jsonEncoded, boolean cancellable) throws Exception { String clientContextId = UUID.randomUUID().toString(); - if (cancellable) { - setParam(params, "client_context_id", clientContextId); - } + final List newParams = + cancellable ? upsertParam(params, "client_context_id", clientContextId) : params; Callable query = () -> { try { return CancellationTestExecutor.super.executeQueryService(str, fmt, uri, params, jsonEncoded, true); diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/ResultExtractor.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/ResultExtractor.java index 53a0f6c..1d50c26 100644 --- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/ResultExtractor.java +++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/ResultExtractor.java @@ -107,7 +107,7 @@ } break; default: - throw new AsterixException(field + " unanticipated field"); + throw new AsterixException("Unanticipated field \"" + field + "\""); } } @@ -119,9 +119,16 @@ ObjectMapper om = new ObjectMapper(); String result = IOUtils.toString(resultStream, utf8); ObjectNode resultJson = om.readValue(result, ObjectNode.class); - JsonNode handle = resultJson.get("handle"); - ObjectNode res = om.createObjectNode(); - res.set("handle", handle); - return om.writeValueAsString(res); + final JsonNode handle = resultJson.get("handle"); + if (handle != null) { + return handle.asText(); + } else { + JsonNode errors = resultJson.get("errors"); + if (errors != null) { + JsonNode msg = errors.get(0).get("msg"); + throw new AsterixException(msg.asText()); + } + } + return null; } } diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java index 608547c..9a3de14 100644 --- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java +++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java @@ -76,6 +76,7 @@ import org.apache.http.util.EntityUtils; import org.apache.hyracks.util.StorageUtil; import org.junit.Assert; + import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.DeserializationFeature; import com.fasterxml.jackson.databind.JsonMappingException; @@ -461,43 +462,44 @@ protected InputStream executeQueryService(String str, OutputFormat fmt, URI uri, List params, boolean jsonEncoded, boolean cancellable) throws Exception { - setParam(params, "format", fmt.mimeType()); - HttpUriRequest method = jsonEncoded ? constructPostMethodJson(str, uri, "statement", params) - : constructPostMethodUrl(str, uri, "statement", params); + final List newParams = upsertParam(params, "format", fmt.mimeType()); + HttpUriRequest method = jsonEncoded ? constructPostMethodJson(str, uri, "statement", newParams) + : constructPostMethodUrl(str, uri, "statement", newParams); // Set accepted output response type method.setHeader("Accept", OutputFormat.CLEAN_JSON.mimeType()); HttpResponse response = executeHttpRequest(method); return response.getEntity().getContent(); } - protected void setParam(List params, String name, String value) { + protected List upsertParam(List params, String name, + String value) { + boolean replaced = false; + List result = new ArrayList<>(); for (CompilationUnit.Parameter param : params) { + CompilationUnit.Parameter newParam = new CompilationUnit.Parameter(); + newParam.setName(param.getName()); if (name.equals(param.getName())) { - param.setValue(value); - return; + newParam.setValue(value); + replaced = true; + } else { + newParam.setValue(param.getValue()); } + result.add(newParam); } - CompilationUnit.Parameter formatParam = new CompilationUnit.Parameter(); - formatParam.setName(name); - formatParam.setValue(value); - params.add(formatParam); - } - - private List injectStatement(String statement, String stmtParamName, - List otherParams) { - CompilationUnit.Parameter stmtParam = new CompilationUnit.Parameter(); - stmtParam.setName(stmtParamName); - stmtParam.setValue(statement); - List params = new ArrayList<>(otherParams); - params.add(stmtParam); - return params; + if (!replaced) { + CompilationUnit.Parameter newParam = new CompilationUnit.Parameter(); + newParam.setName(name); + newParam.setValue(value); + result.add(newParam); + } + return result; } private HttpUriRequest constructHttpMethod(String statement, URI uri, String stmtParam, boolean postStmtAsParam, List otherParams) throws URISyntaxException { if (statement.length() + uri.toString().length() < MAX_URL_LENGTH) { // Use GET for small-ish queries - return constructGetMethod(uri, injectStatement(statement, stmtParam, otherParams)); + return constructGetMethod(uri, upsertParam(otherParams, stmtParam, statement)); } else { // Use POST for bigger ones to avoid 413 FULL_HEAD String stmtParamName = (postStmtAsParam ? stmtParam : null); @@ -541,7 +543,7 @@ List otherParams) { RequestBuilder builder = RequestBuilder.post(uri); if (stmtParam != null) { - for (CompilationUnit.Parameter param : injectStatement(statement, stmtParam, otherParams)) { + for (CompilationUnit.Parameter param : upsertParam(otherParams, stmtParam, statement)) { builder.addParameter(param.getName(), param.getValue()); } builder.addParameter(stmtParam, statement); @@ -561,7 +563,7 @@ RequestBuilder builder = RequestBuilder.post(uri); ObjectMapper om = new ObjectMapper(); ObjectNode content = om.createObjectNode(); - for (CompilationUnit.Parameter param : injectStatement(statement, stmtParam, otherParams)) { + for (CompilationUnit.Parameter param : upsertParam(otherParams, stmtParam, statement)) { content.put(param.getName(), param.getValue()); } try { @@ -609,10 +611,13 @@ HttpResponse response = executeAndCheckHttpRequest(request); InputStream resultStream = response.getEntity().getContent(); - String handle = IOUtils.toString(resultStream, "UTF-8"); + String resultStr = IOUtils.toString(resultStream, "UTF-8"); + ObjectNode resultJson = new ObjectMapper().readValue(resultStr, ObjectNode.class); + final JsonNode jsonHandle = resultJson.get("handle"); + final String strHandle = jsonHandle.asText(); if (handleVar != null) { - variableCtx.put(handleVar, handle); + variableCtx.put(handleVar, strHandle); return resultStream; } return null; @@ -776,6 +781,7 @@ long startTime = System.currentTimeMillis(); long limitTime = startTime + TimeUnit.SECONDS.toMillis(timeoutSecs); ctx.setType(ctx.getType().substring("poll".length())); + boolean expectedException = false; Exception finalException; LOGGER.fine("polling for up to " + timeoutSecs + " seconds w/ " + retryDelaySecs + " second(s) delay"); while (true) { @@ -785,6 +791,11 @@ finalException = null; break; } catch (Exception e) { + if (isExpected(e, cUnit)) { + expectedException = true; + finalException = e; + break; + } if ((System.currentTimeMillis() > limitTime)) { finalException = e; break; @@ -793,7 +804,9 @@ Thread.sleep(TimeUnit.SECONDS.toMillis(retryDelaySecs)); } } - if (finalException != null) { + if (expectedException) { + throw finalException; + } else if (finalException != null){ throw new Exception("Poll limit (" + timeoutSecs + "s) exceeded without obtaining expected result", finalException); } @@ -838,8 +851,8 @@ resultStream = ResultExtractor.extract(resultStream); } else { String handleVar = getHandleVariable(statement); - setParam(params, "mode", delivery); - resultStream = executeQueryService(statement, fmt, uri, params, true); + resultStream = + executeQueryService(statement, fmt, uri, upsertParam(params, "mode", delivery), true); String handle = ResultExtractor.extractHandle(resultStream); Assert.assertNotNull("no handle for " + reqType + " test " + testFile.toString(), handleVar); variableCtx.put(handleVar, handle); @@ -946,19 +959,32 @@ break; case "get": case "post": - if (!"http".equals(ctx.extension())) { + fmt = OutputFormat.forCompilationUnit(cUnit); + String handleVar = getHandleVariable(statement); + final String trimmedPathAndQuery = stripLineComments(stripJavaComments(statement)).trim(); + final String variablesReplaced = replaceVarRef(trimmedPathAndQuery, variableCtx); + if ("http".equals(ctx.extension())) { + resultStream = executeHttp(ctx.getType(), variablesReplaced, fmt); + } else if ("uri".equals(ctx.extension())) { + resultStream = executeURI(ctx.getType(), URI.create(variablesReplaced), fmt); + } else { throw new IllegalArgumentException( "Unexpected format for method " + ctx.getType() + ": " + ctx.extension()); } - fmt = OutputFormat.forCompilationUnit(cUnit); - final String trimmedPathAndQuery = stripLineComments(stripJavaComments(statement)).trim(); - final String variablesReplaced = replaceVarRef(trimmedPathAndQuery, variableCtx); - resultStream = executeHttp(ctx.getType(), variablesReplaced, fmt); - expectedResultFile = expectedResultFileCtxs.get(queryCount.intValue()).getFile(); - actualResultFile = testCaseCtx.getActualResultFile(cUnit, expectedResultFile, new File(actualPath)); - writeOutputToFile(actualResultFile, resultStream); - runScriptAndCompareWithResult(testFile, new PrintWriter(System.err), expectedResultFile, - actualResultFile); + if (handleVar != null) { + String handle = ResultExtractor.extractHandle(resultStream); + if (handle != null) { + variableCtx.put(handleVar, handle); + } else { + throw new Exception("no handle for test " + testFile.toString()); + } + } else { + expectedResultFile = expectedResultFileCtxs.get(queryCount.intValue()).getFile(); + actualResultFile = testCaseCtx.getActualResultFile(cUnit, expectedResultFile, new File(actualPath)); + writeOutputToFile(actualResultFile, resultStream); + runScriptAndCompareWithResult(testFile, new PrintWriter(System.err), expectedResultFile, + actualResultFile); + } queryCount.increment(); break; case "server": // (start @@ -1052,6 +1078,16 @@ } } + protected static boolean isExpected(Exception e, CompilationUnit cUnit) { + final List expErrors = cUnit.getExpectedError(); + for (String exp : expErrors) { + if (e.toString().contains(exp)) { + return true; + } + } + return false; + } + protected int getTimeoutSecs(String statement) { final Matcher timeoutMatcher = POLL_TIMEOUT_PATTERN.matcher(statement); if (timeoutMatcher.find()) { @@ -1087,6 +1123,10 @@ protected InputStream executeHttp(String ctxType, String endpoint, OutputFormat fmt) throws Exception { String[] split = endpoint.split("\\?"); URI uri = new URI("http", null, host, port, split[0], split.length > 1 ? split[1] : null, null); + return executeURI(ctxType, uri, fmt); + } + + private InputStream executeURI(String ctxType, URI uri, OutputFormat fmt) throws Exception { switch (ctxType) { case "get": return executeJSONGet(fmt, uri); diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_1/query_result_1.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_1/query_result_1.1.get.http index d48dbe5..c676d2d 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_1/query_result_1.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_1/query_result_1.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 25th February 2017 */ -/query/result?handle={"handle":[18,0]} +/query/service/result/18-0 diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_2/query_result_2.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_2/query_result_2.1.get.http index 07b7556..effb6a5 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_2/query_result_2.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_2/query_result_2.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 25th February 2017 */ -/query/result?handle={"handle":[18,0] +/query/service/result/18_0 diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_3/query_result_3.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_3/query_result_3.1.get.http index c39b87e..33cebe9 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_3/query_result_3.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_3/query_result_3.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 25th February 2017 */ -/query/result?handle +/query/service/result/ diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_4/query_result_4.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_4/query_result_4.1.get.http index 88c4814..a949a28 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_4/query_result_4.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_result_4/query_result_4.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 25th February 2017 */ -/query/result?handl={"handle":[18,0]} +/query/service/result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_1/query_status_1.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_1/query_status_1.1.get.http index c18a55b..66428a8 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_1/query_status_1.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_1/query_status_1.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 7th September 2016 */ -/query/status?handle={"handle":[18,0]} +/query/service/status/18-0 diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_2/query_status_2.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_2/query_status_2.1.get.http index d7ece4c..60634d3 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_2/query_status_2.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_2/query_status_2.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 7th September 2016 */ -/query/status?handle={"handle":[18,0] +/query/service/status/18_0 diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_3/query_status_3.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_3/query_status_3.1.get.http index a7ff551..d7d7a0d 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_3/query_status_3.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_3/query_status_3.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 7th September 2016 */ -/query/status?handle +/query/service/status/ diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_4/query_status_4.1.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_4/query_status_4.1.get.http index 5831686..23930dd 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_4/query_status_4.1.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/api/query_status_4/query_status_4.1.get.http @@ -22,4 +22,4 @@ * Expected Result : Negative * Date : 7th September 2016 */ -/query/status?handl={"handle":[18,0]} +/query/service/status diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.5.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.5.get.http index a88991c..8417a7e 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.5.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.5.get.http @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +/query/service/result/$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.7.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.7.pollget.http index 5d59ca3..bcc0edc 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.7.pollget.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.7.pollget.http @@ -19,4 +19,4 @@ //polltimeoutsecs=10 -/query/status?handle=$handle +/query/service/status/$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.8.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.8.get.http index a88991c..8417a7e 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.8.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.8.get.http @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +/query/service/result/$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.1.async.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.1.async.sqlpp index 89ef35e..2f0feff 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.1.async.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.1.async.sqlpp @@ -17,7 +17,7 @@ * under the License. */ -#handlevariable=handle +#handlevariable=status set `import-private-functions` `true`; select value inject_failure(sleep("result", 5000), true); diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.uri similarity index 93% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.uri index a88991c..e20319a 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.uri @@ -17,4 +17,7 @@ * under the License. */ -/query/result?handle=$handle +#polltimeoutsecs=30 +#handlevariable=result + +$status diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.1.async.sqlpp similarity index 91% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.1.async.sqlpp index a88991c..c9a2958 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.1.async.sqlpp @@ -17,4 +17,6 @@ * under the License. */ -/query/result?handle=$handle +#handlevariable=status + +select i, i * i as i2 from range(1, 10) i; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.2.pollget.uri similarity index 93% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.2.pollget.uri index a88991c..bf3f04c 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.2.pollget.uri @@ -17,4 +17,7 @@ * under the License. */ -/query/result?handle=$handle +#polltimeoutsecs=20 +#handlevariable=result + +$status diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.3.get.uri similarity index 95% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.3.get.uri index a88991c..b613531 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.3.get.uri @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +$result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.4.get.uri similarity index 95% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.4.get.uri index a88991c..b613531 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.4.get.uri @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +$result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.5.query.sqlpp similarity index 94% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.5.query.sqlpp index a88991c..e452678 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.3.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-repeated/async-repeated.5.query.sqlpp @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +select i, i * i as i2 from range(1, 10) i; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.1.async.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.1.async.sqlpp index 866b388..5237950 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.1.async.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.1.async.sqlpp @@ -17,6 +17,6 @@ * under the License. */ -#handlevariable=handle +#handlevariable=status -select value sleep("result", 3000); +select value sleep("result", 5000); diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.http deleted file mode 100644 index 916aadf..0000000 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.http +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -#polltimeoutsecs=10 - -/query/status?handle=$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.uri similarity index 96% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.http rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.uri index a88991c..c83909b 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.2.pollget.uri @@ -17,4 +17,6 @@ * under the License. */ -/query/result?handle=$handle +#polltimeoutsecs=10 + +$status diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.http deleted file mode 100644 index 916aadf..0000000 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.http +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -#polltimeoutsecs=10 - -/query/status?handle=$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.uri similarity index 95% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.uri index 916aadf..12fcdfd 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.3.pollget.uri @@ -18,5 +18,6 @@ */ #polltimeoutsecs=10 +#handlevariable=result -/query/status?handle=$handle +$status diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.uri similarity index 96% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.uri index a88991c..b613531 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-running/async-running.4.get.uri @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +$result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.1.async.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.1.async.sqlpp index a44b911..c9a2958 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.1.async.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.1.async.sqlpp @@ -17,6 +17,6 @@ * under the License. */ -#handlevariable=handle +#handlevariable=status select i, i * i as i2 from range(1, 10) i; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.http deleted file mode 100644 index 916aadf..0000000 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.http +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -#polltimeoutsecs=10 - -/query/status?handle=$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.uri similarity index 95% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.uri index 916aadf..12fcdfd 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.2.pollget.uri @@ -18,5 +18,6 @@ */ #polltimeoutsecs=10 +#handlevariable=result -/query/status?handle=$handle +$status diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.3.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.3.get.uri similarity index 96% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.3.get.http rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.3.get.uri index a88991c..b613531 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.3.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async/async.3.get.uri @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +$result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.1.deferred.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.1.deferred.sqlpp index a44b911..815e49e 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.1.deferred.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.1.deferred.sqlpp @@ -17,6 +17,6 @@ * under the License. */ -#handlevariable=handle +#handlevariable=result select i, i * i as i2 from range(1, 10) i; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.2.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.2.get.uri similarity index 96% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.2.get.http rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.2.get.uri index a88991c..b613531 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.2.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/deferred/deferred.2.get.uri @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +$result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.4.deferred.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.4.deferred.sqlpp index df7826f..d834704 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.4.deferred.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.4.deferred.sqlpp @@ -22,7 +22,7 @@ * Date : 09/17/2013 */ -#handlevariable=handle +#handlevariable=result use test; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.uri similarity index 96% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.http rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.uri index a88991c..b613531 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.uri @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +$result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.6.async.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.6.async.sqlpp index df7826f..706a92b 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.6.async.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.6.async.sqlpp @@ -22,7 +22,7 @@ * Date : 09/17/2013 */ -#handlevariable=handle +#handlevariable=status use test; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.http deleted file mode 100644 index 916aadf..0000000 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.http +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -#polltimeoutsecs=10 - -/query/status?handle=$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.uri similarity index 95% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.uri index 916aadf..12fcdfd 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/async-deferred/async-failed/async-failed.2.pollget.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.7.pollget.uri @@ -18,5 +18,6 @@ */ #polltimeoutsecs=10 +#handlevariable=result -/query/status?handle=$handle +$status diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.http deleted file mode 100644 index a88991c..0000000 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.http +++ /dev/null @@ -1,20 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/query/result?handle=$handle diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.uri similarity index 96% copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.http copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.uri index a88991c..b613531 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.get.http +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.8.get.uri @@ -17,4 +17,4 @@ * under the License. */ -/query/result?handle=$handle +$result diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.json b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.json deleted file mode 100644 index 246785b..0000000 --- a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "status": "failed", -} diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.regex b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.regex new file mode 100644 index 0000000..66de2a0 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-failed/async-failed.2.regex @@ -0,0 +1,2 @@ +/"status": "failed"/ +/"errors": ".*"/ diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.1.ignore b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.1.ignore new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.1.ignore diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.2.regex b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.2.regex new file mode 100644 index 0000000..4308ba2 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.2.regex @@ -0,0 +1,2 @@ +/"status": "success"/ +/"handle": ".*"/ diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.3.json b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.3.json new file mode 100644 index 0000000..09e86cc --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.3.json @@ -0,0 +1,10 @@ +{ "i": 1, "i2": 1 } +{ "i": 2, "i2": 4 } +{ "i": 3, "i2": 9 } +{ "i": 4, "i2": 16 } +{ "i": 5, "i2": 25 } +{ "i": 6, "i2": 36 } +{ "i": 7, "i2": 49 } +{ "i": 8, "i2": 64 } +{ "i": 9, "i2": 81 } +{ "i": 10, "i2": 100 } diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.4.json b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.4.json new file mode 100644 index 0000000..09e86cc --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.4.json @@ -0,0 +1,10 @@ +{ "i": 1, "i2": 1 } +{ "i": 2, "i2": 4 } +{ "i": 3, "i2": 9 } +{ "i": 4, "i2": 16 } +{ "i": 5, "i2": 25 } +{ "i": 6, "i2": 36 } +{ "i": 7, "i2": 49 } +{ "i": 8, "i2": 64 } +{ "i": 9, "i2": 81 } +{ "i": 10, "i2": 100 } diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.5.json b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.5.json new file mode 100644 index 0000000..09e86cc --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-repeated/async-repeated.5.json @@ -0,0 +1,10 @@ +{ "i": 1, "i2": 1 } +{ "i": 2, "i2": 4 } +{ "i": 3, "i2": 9 } +{ "i": 4, "i2": 16 } +{ "i": 5, "i2": 25 } +{ "i": 6, "i2": 36 } +{ "i": 7, "i2": 49 } +{ "i": 8, "i2": 64 } +{ "i": 9, "i2": 81 } +{ "i": 10, "i2": 100 } diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-running/async-running.2.json b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-running/async-running.2.json index 2dc2832..272762e 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-running/async-running.2.json +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/async-deferred/async-running/async-running.2.json @@ -1,3 +1,3 @@ { - "status": "running", + "status": "running" } diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml index 431b215..d26e0d5 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml +++ b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml @@ -25,7 +25,7 @@ async-failed - Error in processing tuple 0 + Injected failure in asterix:inject-failure @@ -39,6 +39,11 @@ + + async-repeated + + + async-running diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/Servlets.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/Servlets.java index 6223f36..3047ef5 100644 --- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/Servlets.java +++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/Servlets.java @@ -28,8 +28,8 @@ public static final String SQLPP_QUERY = "/query/sqlpp"; public static final String SQLPP_UPDATE = "/update/sqlpp"; public static final String SQLPP_DDL = "/ddl/sqlpp"; - public static final String QUERY_STATUS = "/query/status"; - public static final String QUERY_RESULT = "/query/result"; + public static final String QUERY_STATUS = "/query/service/status/*"; + public static final String QUERY_RESULT = "/query/service/result/*"; public static final String QUERY_SERVICE = "/query/service"; public static final String CONNECTOR = "/connector"; public static final String SHUTDOWN = "/admin/shutdown"; diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/DatasetJobRecord.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/DatasetJobRecord.java index f29ff4a..55f1d7c 100644 --- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/DatasetJobRecord.java +++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/DatasetJobRecord.java @@ -18,6 +18,9 @@ */ package org.apache.hyracks.api.dataset; +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Arrays; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -26,11 +29,50 @@ import org.apache.hyracks.api.exceptions.HyracksDataException; public class DatasetJobRecord implements IDatasetStateRecord { - public enum Status { + public enum State { IDLE, RUNNING, SUCCESS, FAILED + } + + public static class Status implements Serializable { + + private static final long serialVersionUID = 1L; + + State state = State.IDLE; + + private List exceptions; + + public State getState() { + return state; + } + + void setState(State state) { + this.state = state; + } + + public List getExceptions() { + return exceptions; + } + + void setExceptions(List exceptions) { + this.exceptions = exceptions; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + sb.append("{ \"state\": \"").append(state.name()).append("\""); + if (exceptions != null && !exceptions.isEmpty()) { + sb.append(", \"exceptions\": "); + List msgs = new ArrayList<>(); + exceptions.forEach(e -> msgs.add("\"" + e.getMessage() + "\"")); + sb.append(Arrays.toString(msgs.toArray())); + } + sb.append(" }"); + return sb.toString(); + } } private static final long serialVersionUID = 1L; @@ -39,38 +81,35 @@ private Status status; - private List exceptions; - private Map resultSetMetadataMap = new HashMap<>(); public DatasetJobRecord() { this.timestamp = System.currentTimeMillis(); - this.status = Status.IDLE; + this.status = new Status(); } - private void updateStatus(Status newStatus) { + private void updateState(State newStatus) { // FAILED is a stable status - if (status != Status.FAILED) { - status = newStatus; + if (status.state != State.FAILED) { + status.setState(newStatus); } } public void start() { - updateStatus(Status.RUNNING); + updateState(State.RUNNING); } public void success() { - updateStatus(Status.SUCCESS); + updateState(State.SUCCESS); } public void fail(ResultSetId rsId, int partition) { getOrCreateDirectoryRecord(rsId, partition).fail(); - status = Status.FAILED; } public void fail(List exceptions) { - status = Status.FAILED; - this.exceptions = exceptions; + updateState(State.FAILED); + status.setExceptions(exceptions); } @Override @@ -84,15 +123,15 @@ @Override public String toString() { - return resultSetMetadataMap.toString(); + StringBuilder sb = new StringBuilder(); + sb.append("{ \"status\": ").append(status.toString()).append(", "); + sb.append("\"timestamp\": ").append(timestamp).append(", "); + sb.append("\"resultsets\": ").append(Arrays.toString(resultSetMetadataMap.entrySet().toArray())).append(" }"); + return sb.toString(); } - public List getExceptions() { - return exceptions; - } - - public void setResultSetMetaData(ResultSetId rsId, boolean orderedResult, int nPartitions) throws - HyracksDataException { + public void setResultSetMetaData(ResultSetId rsId, boolean orderedResult, int nPartitions) + throws HyracksDataException { ResultSetMetaData rsMd = resultSetMetadataMap.get(rsId); if (rsMd == null) { resultSetMetadataMap.put(rsId, new ResultSetMetaData(nPartitions, orderedResult)); @@ -114,16 +153,16 @@ return records[partition]; } - public synchronized DatasetDirectoryRecord getDirectoryRecord(ResultSetId rsId, int partition) throws - HyracksDataException { + public synchronized DatasetDirectoryRecord getDirectoryRecord(ResultSetId rsId, int partition) + throws HyracksDataException { DatasetDirectoryRecord[] records = getResultSetMetaData(rsId).getRecords(); if (records[partition] == null) { - throw new HyracksDataException("no record for partition " + partition + " of result set " + rsId); + throw HyracksDataException.create(ErrorCode.RESULT_NO_RECORD, partition, rsId); } return records[partition]; } - public synchronized void updateStatus(ResultSetId rsId) { + public synchronized void updateState(ResultSetId rsId) { int successCount = 0; DatasetDirectoryRecord[] records = getResultSetMetaData(rsId).getRecords(); for (DatasetDirectoryRecord record : records) { diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/IDatasetPartitionManager.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/IDatasetPartitionManager.java index fa22d8e..f79ce53 100644 --- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/IDatasetPartitionManager.java +++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/IDatasetPartitionManager.java @@ -43,7 +43,5 @@ public void abortReader(JobId jobId); - public IWorkspaceFileFactory getFileFactory(); - public void close(); } diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java index 401103b..d2c6d81 100644 --- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java +++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java @@ -60,6 +60,7 @@ public static final int NO_RESULTSET = 24; public static final int JOB_CANCELED = 25; public static final int NODE_FAILED = 26; + public static final int RESULT_NO_RECORD = 27; // Compilation error codes. public static final int RULECOLLECTION_NOT_INSTANCE_OF_LIST = 10001; diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties b/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties index 12601fb..f49a9a9 100644 --- a/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties +++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties @@ -45,5 +45,6 @@ 24 = No result set for job %1$s 25 = Job %1$s has been cancelled by a user 26 = Node %1$s failed +27 = No record for partition %1$s of result set %2$s 10000 = The given rule collection %1$s is not an instance of the List class. diff --git a/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/DatasetClientContext.java b/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/DatasetClientContext.java deleted file mode 100644 index 9eb5914..0000000 --- a/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/DatasetClientContext.java +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.hyracks.client.dataset; - -import org.apache.hyracks.api.context.IHyracksCommonContext; -import org.apache.hyracks.api.io.IIOManager; -import org.apache.hyracks.control.nc.resources.memory.FrameManager; - -public class DatasetClientContext extends FrameManager implements IHyracksCommonContext { - private final int frameSize; - - public DatasetClientContext(int frameSize) { - super(frameSize); - this.frameSize = frameSize; - } - - @Override - public IIOManager getIOManager() { - return null; - } - -} diff --git a/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDataset.java b/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDataset.java index f60b3c3..5f038b2 100644 --- a/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDataset.java +++ b/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDataset.java @@ -20,25 +20,28 @@ import org.apache.hyracks.api.client.IHyracksClientConnection; import org.apache.hyracks.api.comm.NetworkAddress; +import org.apache.hyracks.api.context.IHyracksCommonContext; import org.apache.hyracks.api.dataset.IHyracksDataset; import org.apache.hyracks.api.dataset.IHyracksDatasetDirectoryServiceConnection; import org.apache.hyracks.api.dataset.IHyracksDatasetReader; import org.apache.hyracks.api.dataset.ResultSetId; import org.apache.hyracks.api.exceptions.HyracksDataException; +import org.apache.hyracks.api.io.IIOManager; import org.apache.hyracks.api.job.JobId; import org.apache.hyracks.client.net.ClientNetworkManager; +import org.apache.hyracks.control.nc.resources.memory.FrameManager; public class HyracksDataset implements IHyracksDataset { private final IHyracksDatasetDirectoryServiceConnection datasetDirectoryServiceConnection; private final ClientNetworkManager netManager; - private final DatasetClientContext datasetClientCtx; + private final IHyracksCommonContext datasetClientCtx; public HyracksDataset(IHyracksClientConnection hcc, int frameSize, int nReaders) throws Exception { NetworkAddress ddsAddress = hcc.getDatasetDirectoryServiceInfo(); - datasetDirectoryServiceConnection = new HyracksDatasetDirectoryServiceConnection - (ddsAddress.getAddress(), ddsAddress.getPort()); + datasetDirectoryServiceConnection = + new HyracksDatasetDirectoryServiceConnection(ddsAddress.getAddress(), ddsAddress.getPort()); netManager = new ClientNetworkManager(nReaders); netManager.start(); @@ -57,4 +60,17 @@ } return reader; } + + static class DatasetClientContext extends FrameManager implements IHyracksCommonContext { + + DatasetClientContext(int frameSize) { + super(frameSize); + } + + @Override + public IIOManager getIOManager() { + return null; + } + } + } diff --git a/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDatasetReader.java b/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDatasetReader.java index e377a50..fdac7f1 100644 --- a/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDatasetReader.java +++ b/hyracks-fullstack/hyracks/hyracks-client/src/main/java/org/apache/hyracks/client/dataset/HyracksDatasetReader.java @@ -32,6 +32,7 @@ import org.apache.hyracks.api.comm.FrameHelper; import org.apache.hyracks.api.comm.IFrame; import org.apache.hyracks.api.comm.NetworkAddress; +import org.apache.hyracks.api.context.IHyracksCommonContext; import org.apache.hyracks.api.dataset.DatasetDirectoryRecord; import org.apache.hyracks.api.dataset.DatasetJobRecord.Status; import org.apache.hyracks.api.dataset.IDatasetInputChannelMonitor; @@ -53,7 +54,7 @@ private final ClientNetworkManager netManager; - private final DatasetClientContext datasetClientCtx; + private final IHyracksCommonContext datasetClientCtx; private JobId jobId; @@ -72,7 +73,7 @@ private static int NUM_READ_BUFFERS = 1; public HyracksDatasetReader(IHyracksDatasetDirectoryServiceConnection datasetDirectoryServiceConnection, - ClientNetworkManager netManager, DatasetClientContext datasetClientCtx, JobId jobId, + ClientNetworkManager netManager, IHyracksCommonContext datasetClientCtx, JobId jobId, ResultSetId resultSetId) throws Exception { this.datasetDirectoryServiceConnection = datasetDirectoryServiceConnection; @@ -125,10 +126,8 @@ resultChannel.registerMonitor(lastMonitor); resultChannel.open(datasetClientCtx); return true; - } catch (HyracksDataException e) { - throw e; } catch (Exception e) { - throw new HyracksDataException(e); + throw HyracksDataException.create(e); } } diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/dataset/DatasetDirectoryService.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/dataset/DatasetDirectoryService.java index 98c0697..927d499 100644 --- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/dataset/DatasetDirectoryService.java +++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/dataset/DatasetDirectoryService.java @@ -18,6 +18,7 @@ */ package org.apache.hyracks.control.cc.dataset; +import java.io.PrintWriter; import java.util.Arrays; import java.util.HashMap; import java.util.LinkedHashMap; @@ -31,7 +32,7 @@ import org.apache.hyracks.api.comm.NetworkAddress; import org.apache.hyracks.api.dataset.DatasetDirectoryRecord; import org.apache.hyracks.api.dataset.DatasetJobRecord; -import org.apache.hyracks.api.dataset.DatasetJobRecord.Status; +import org.apache.hyracks.api.dataset.DatasetJobRecord.State; import org.apache.hyracks.api.dataset.IDatasetStateRecord; import org.apache.hyracks.api.dataset.ResultSetId; import org.apache.hyracks.api.dataset.ResultSetMetaData; @@ -83,7 +84,7 @@ } @Override - public void notifyJobStart(JobId jobId) throws HyracksException { + public synchronized void notifyJobStart(JobId jobId) throws HyracksException { jobResultLocations.get(jobId).getRecord().start(); } @@ -138,7 +139,7 @@ throws HyracksDataException { DatasetJobRecord djr = getNonNullDatasetJobRecord(jobId); djr.getDirectoryRecord(rsId, partition).writeEOS(); - djr.updateStatus(rsId); + djr.updateState(rsId); notifyAll(); } @@ -159,29 +160,30 @@ djr.fail(exceptions); } // TODO(tillw) throwing an NPE here hangs the system, why? + // TODO(tillw) still run into NPE here .. jobResultLocations.get(jobId).setException(exceptions.isEmpty() ? null : exceptions.get(0)); notifyAll(); } @Override - public synchronized Status getResultStatus(JobId jobId, ResultSetId rsId) throws HyracksDataException { + public synchronized DatasetJobRecord.Status getResultStatus(JobId jobId, ResultSetId rsId) + throws HyracksDataException { return getNonNullDatasetJobRecord(jobId).getStatus(); } @Override - public Set getJobIds() { + public synchronized Set getJobIds() { return jobResultLocations.keySet(); } @Override - public IDatasetStateRecord getState(JobId jobId) { + public synchronized IDatasetStateRecord getState(JobId jobId) { return getDatasetJobRecord(jobId); } @Override - public void deinitState(JobId jobId) { - // See ASTERIXDB-1614 - DatasetDirectoryService.deinitState() fix intermittently fails - // jobResultLocations.remove(jobId); + public synchronized void deinitState(JobId jobId) { + jobResultLocations.remove(jobId); } @Override @@ -217,8 +219,8 @@ DatasetDirectoryRecord[] knownRecords) throws HyracksDataException { DatasetJobRecord djr = getNonNullDatasetJobRecord(jobId); - if (djr.getStatus() == Status.FAILED) { - List caughtExceptions = djr.getExceptions(); + if (djr.getStatus().getState() == State.FAILED) { + List caughtExceptions = djr.getStatus().getExceptions(); if (caughtExceptions != null && !caughtExceptions.isEmpty()) { final Exception cause = caughtExceptions.get(caughtExceptions.size() - 1); if (cause instanceof HyracksDataException) { @@ -237,6 +239,16 @@ DatasetDirectoryRecord[] records = resultSetMetaData.getRecords(); return Arrays.equals(records, knownRecords) ? null : records; + } + + public PrintWriter print(PrintWriter pw) { + for (JobId jId : getJobIds()) { + pw.print(jId.toString()); + pw.print(" - "); + pw.println(String.valueOf(getDatasetJobRecord(jId))); + } + pw.flush(); + return pw; } } @@ -277,6 +289,11 @@ } } } + + @Override + public String toString() { + return record.toString(); + } } class Waiters extends HashMap { diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/DatasetPartitionManager.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/DatasetPartitionManager.java index 5fac823..930a3e0 100644 --- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/DatasetPartitionManager.java +++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/DatasetPartitionManager.java @@ -124,29 +124,28 @@ @Override public void initializeDatasetPartitionReader(JobId jobId, ResultSetId resultSetId, int partition, IFrameWriter writer) throws HyracksException { - ResultState resultState; - synchronized (this) { - ResultSetMap rsIdMap = (ResultSetMap) partitionResultStateMap.get(jobId); - - if (rsIdMap == null) { - throw new HyracksException("Unknown JobId " + jobId); - } - - ResultState[] resultStates = rsIdMap.getResultStates(resultSetId); - if (resultStates == null) { - throw new HyracksException("Unknown JobId: " + jobId + " ResultSetId: " + resultSetId); - } - - resultState = resultStates[partition]; - if (resultState == null) { - throw new HyracksException("No DatasetPartitionWriter for partition " + partition); - } - } - + ResultState resultState = getResultState(jobId, resultSetId, partition); DatasetPartitionReader dpr = new DatasetPartitionReader(this, datasetMemoryManager, executor, resultState); dpr.writeTo(writer); LOGGER.fine("Initialized partition reader: JobId: " + jobId + ":ResultSetId: " + resultSetId + ":partition: " + partition); + } + + protected synchronized ResultState getResultState(JobId jobId, ResultSetId resultSetId, int partition) + throws HyracksException { + ResultSetMap rsIdMap = (ResultSetMap) partitionResultStateMap.get(jobId); + if (rsIdMap == null) { + throw new HyracksException("Unknown JobId " + jobId); + } + ResultState[] resultStates = rsIdMap.getResultStates(resultSetId); + if (resultStates == null) { + throw new HyracksException("Unknown JobId: " + jobId + " ResultSetId: " + resultSetId); + } + ResultState resultState = resultStates[partition]; + if (resultState == null) { + throw new HyracksException("No DatasetPartitionWriter for partition " + partition); + } + return resultState; } @Override @@ -163,11 +162,6 @@ if (rsIdMap != null) { rsIdMap.abortAll(); } - } - - @Override - public IWorkspaceFileFactory getFileFactory() { - return fileFactory; } @Override diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/ResultState.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/ResultState.java index c501b5b..25b79c8 100644 --- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/ResultState.java +++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/ResultState.java @@ -107,6 +107,7 @@ } catch (IOException e) { // Since file handle could not be closed, just ignore. } + writeFileHandle = null; } } @@ -152,6 +153,7 @@ public synchronized void readClose() throws HyracksDataException { if (readFileHandle != null) { ioManager.close(readFileHandle); + readFileHandle = null; } } @@ -324,4 +326,16 @@ readFileHandle = ioManager.open(fileRef, IIOManager.FileReadWriteMode.READ_ONLY, IIOManager.FileSyncMode.METADATA_ASYNC_DATA_ASYNC); } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + sb.append("{ "); + sb.append('"').append("rspid").append("\":\"").append(resultSetPartitionId).append("\", "); + sb.append('"').append("async").append("\":").append(asyncMode).append(", "); + sb.append('"').append("eos").append("\":").append(eos).append(", "); + sb.append('"').append("failed").append("\":").append(failed).append(", "); + sb.append('"').append("fileRef").append("\":\"").append(String.valueOf(fileRef)).append("\" }"); + return sb.toString(); + } } diff --git a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/result/ResultWriterOperatorDescriptor.java b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/result/ResultWriterOperatorDescriptor.java index b422ef4..f1a777b 100644 --- a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/result/ResultWriterOperatorDescriptor.java +++ b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/result/ResultWriterOperatorDescriptor.java @@ -21,6 +21,7 @@ import java.io.IOException; import java.io.PrintStream; import java.nio.ByteBuffer; +import java.util.logging.Logger; import org.apache.hyracks.api.comm.IFrame; import org.apache.hyracks.api.comm.IFrameWriter; @@ -91,7 +92,7 @@ datasetPartitionWriter.open(); resultSerializer.init(); } catch (HyracksException e) { - throw new HyracksDataException(e); + throw HyracksDataException.create(e); } } @@ -128,6 +129,16 @@ datasetPartitionWriter.close(); } } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + sb.append("{ "); + sb.append("\"rsId\": \"").append(rsId).append("\", "); + sb.append("\"ordered\": ").append(ordered).append(", "); + sb.append("\"asyncMode\": ").append(asyncMode).append(" }"); + return sb.toString(); + } }; } } diff --git a/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/AbstractServlet.java b/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/AbstractServlet.java index e4d9005..252328c 100644 --- a/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/AbstractServlet.java +++ b/hyracks-fullstack/hyracks/hyracks-http/src/main/java/org/apache/hyracks/http/server/AbstractServlet.java @@ -143,7 +143,7 @@ int trim = -1; if (paths.length > 1) { for (int i = 0; i < paths.length; i++) { - String path = paths[i].indexOf('*') >= 0 ? paths[i].substring(0, paths[i].indexOf('*')) : paths[0]; + String path = paths[i].indexOf('*') >= 0 ? paths[i].substring(0, paths[i].indexOf('*')) : paths[i]; if (uri.indexOf(path) == 0) { trim = trims[i]; break; -- To view, visit https://asterix-gerrit.ics.uci.edu/1575 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib594cdceb8ff2801f3e2af37be68c1609bef2a11 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Till Westmann