asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Murtadha Hubail <hubail...@gmail.com>
Subject Re: Test Framework Improvements
Date Thu, 28 Jan 2016 04:08:04 GMT
Hi Taewoo,

No, this wasn’t caused by the improvements. However, before the improvements, in case of
ParseException we were not returning a summary at all in the JSON output and we were getting
JSONException all the time. I have just submitted a patch[1] to add the stack trace in the
JSON output and now you should be able to see the original exception stack trace in the "Failure
Trace" of Unit.

You may checkout it and test it.

Cheers,
Murtadha

[1] https://asterix-gerrit.ics.uci.edu/#/c/599/ <https://asterix-gerrit.ics.uci.edu/#/c/599/>

> On Jan 27, 2016, at 10:14 AM, Taewoo Kim <wangsaeu@gmail.com> wrote:
> 
> This is indeed a good improvement. Regarding this, I have a question. After
> merging the current master into my branch, I executed "executionTest" in
> Eclipse. And I found that "Failure Trace"  stack trace in JUnit shows not
> the origin of failure, rather it always shows "executeTest" like the
> following for each test cases. Originally, I could be able to see the
> origin - in this case - AQL Parser. But, now, with this summary, it is hard
> to identify the origin. I have to execute each of failed test one by one
> and need to check console to see the real origin, not
> "TestExecutor.executeTest". I am wondering this is the result of test frame
> work change?
> 
> 
> ExecutionTest
> org.apache.asterix.test.runtime.ExecutionTest
> [ExecutionTest 0: feeds: feeds_01]
> test[ExecutionTest 0: feeds:
> feeds_01](org.apache.asterix.test.runtime.ExecutionTest)
> java.lang.Exception: Test
> "src/test/resources/runtimets/queries/feeds/feeds_01/feeds_01.1.ddl.aql"
> FAILED!
> at
> org.apache.asterix.test.aql.TestExecutor.executeTest(TestExecutor.java:594)
> at org.apache.asterix.test.runtime.ExecutionTest.test(ExecutionTest.java:96)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> Caused by: java.lang.Exception: HTTP operation failed: 2
> STATUS LINE: HTTP/1.1 500 Server Error
> SUMMARY: SyntaxError: Encountered " "text" "text "" at line 32, column 3.
> Was expecting one of:
>    &ltSTRING_LITERAL&gt ...
>    &ltIDENTIFIER&gt ...
> 
> ==>   text : string,
> STACKTRACE:
> at
> org.apache.asterix.test.aql.TestExecutor.executeHttpMethod(TestExecutor.java:228)
> at
> org.apache.asterix.test.aql.TestExecutor.executeDDL(TestExecutor.java:329)
> at
> org.apache.asterix.test.aql.TestExecutor.executeTest(TestExecutor.java:430)
> ... 34 more
> 
> 
> Best,
> Taewoo
> 
> On Tue, Jan 12, 2016 at 12:46 PM, Murtadha Hubail <hubailmor@gmail.com>
> wrote:
> 
>> Hi All,
>> 
>> Last night Abdullah and I merged the Asterix test framework improvement
>> change. So, don’t be surprised if you have a test case that worked
>> yesterday but not on the current master.
>> 
>> During the development of the change, we discovered and fixed critical
>> issues such as:
>> 1. Failed Jobs' threads that are never terminated and their operators keep
>> waiting forever.
>> 2. Many operators that do not comply with the IFrameWriter contract.
>> 3. Indeterministic result delivery on synchronous queries.
>> 4. Invalid number of active operations on an index after a transaction
>> aborts.
>> 
>> All of these issues and many others were not detected before because of
>> the state the test framework was in. I encountered test cases that were
>> miraculously passing just because they were expected to throw an exception.
>> Just to give you an example, a test case was using the DDL API to run a
>> query, and an exception was thrown saying "SyntaxError: Invalid statement:
>> Non-Update statement” and it was considered as passing, even though the
>> expected exception was “MetadataException: Unknown dataverse”.
>> 
>> Due to those reasons, if you are adding any new test cases, you will have
>> to provide the following:
>> 1. For each query in the test case, an expected result file (even queries
>> that are expected to return no records).
>> 2. For each expected exception in the test case, you have to provide the
>> complete exception type as well as the expected error message. If you have
>> multiple expected exceptions in the same test case, they need to be ordered
>> in the same order they will be raised.
>> 
>> Unfortunately, currently we don’t have standard error codes for exceptions
>> (e.g. ASX-0001: “Duplicate key exception”), so any changes to the
>> exceptions messages will need to be reflected on the test cases, but this
>> is better than having a broken test framework.
>> 
>> Cheers,
>> Murtadha


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message