incubator-bigtop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wing Yew Poon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BIGTOP-476) Improvement of BigTop iTest framework
Date Wed, 21 Mar 2012 18:15:41 GMT

    [ https://issues.apache.org/jira/browse/BIGTOP-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234783#comment-13234783
] 

Wing Yew Poon commented on BIGTOP-476:
--------------------------------------

I am not a fan of writing tests in XML.
Using an external data file (which happens to be XML) in TestCLI makes some sense, in that
it is about running many FsShell commands and parsing the results of the commands. It is specialized
for running FsShell commands.
Of course, the idea may be generalized to running arbitrary shell commands, but if I understood
this proposal correctly, it is about building a framework for writing tests in XML by running
arbitrary shell commands and then comparing the output.
Even if such an idea has merit, I'd prefer that it uses YAML or a DSL, instead of XML.

On the subject of TestCLI, it has a shortcoming, and that is it counts as s single JUnit test.
In actual fact, it consists of more than a hundred testcases. If one test case or 100 test
cases fail, it is reported as a single failure, which is not useful. This is something that
could use improvement.
                
> Improvement of BigTop iTest framework
> -------------------------------------
>
>                 Key: BIGTOP-476
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-476
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 0.3.0
>         Environment: All
>            Reporter: Lei Zou
>            Assignee: Lei Zou
>            Priority: Minor
>             Fix For: 0.3.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Current BigTop test framework has a limitation to handle dynamic generated data. It's
flexibility can be improved. 
> For org.apache.bigtop.itest.hadoopexamples.TestHadoopExamples
> Limitation: if someone wants to make any changes, he/she needs to modify 
> ./bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy.
It requires compilation before running. 
> For org.apache.bigtop.itest.hadooptests.TestTestCLI. The configuration file,
> ./build/hadoop/deb/hadoop-1.0.1/src/test/org/apache/hadoop/cli/testConf.xml, has entries
like the following: 
>    <test> <!-- TESTED -->
>       <description>ls: file using relative path</description>
>       <test-commands>
>         <command>-fs NAMENODE -touchz file1</command>
>         <command>-fs NAMENODE -ls file1</command>
>       </test-commands>
>       <cleanup-commands>
>         <command>-fs NAMENODE -rm file1</command>
>       </cleanup-commands>
>       <comparators>
>         <comparator>
>           <type>TokenComparator</type>
>           <expected-output>Found 1 items</expected-output>
>         </comparator>
>         <comparator>
>           <type>RegexpComparator</type>
>           <expected-output>^-rw-r--r--( )*1( )*[a-z]*( )*supergroup( )*0( )*[0-9]{4,}-[0-9]{2,}-[0-9]{2,}
[0-9]{2,}:[0-9]{2,}( )*/user/[a-z]*/file1</expected-output>
>         </comparator>
>       </comparators>
>     </test>
> Limitation: Put the expected-output, then perform string comparison is good, but still
not flexible enough to handle dynamic generated data. For example, a program randomly generate
key/value pairs, then submit M/R job to calculate sum (average) for each key. There is no
way to calculate the result in advance to put down as expected-output. 
> I am proposing an improvement for BigTop's integration test. We can put all test cases
in a XML file, which contains a list of command-sets; each command-set has command, command-comparator-type,
command-comparator-compare-to. The command is for hadoop/hbase/hive command; command-comparator-type
to specify Java class to perform comparison; command-comparator-compare-to is used to specify
the shell command to generate expected output. 
> I put down 3 cases below:
> <?xml version="1.0" encoding="ISO-8859-1"?>
> <bigtop-itest-suite>
> 	<bigtop-itest-suite-test>
> 		<test-name>Calculate summation in MR</test-name>
> 		<test-desc>Here is simple MR test to calculate sum</test-desc>
> 		<test-pre-integration-test>
> 		</test-pre-integration-test>
> 		<test-integration-test>
>             <command-set>
>             <command>hadoop jar ./target/LeiBigTop-1.1.jar com.lei.bigtop.hadoop.calsum.CalSum
./data ./output</command>
>             <command-comparator-type>com.lei.bigtop.hadoop.integration.test.ExtactComparatorIgnoreWhiteSpace</command-comparator-type>
>             <command-comparator-compare-to><![CDATA[ cat ./output/* ]]></command-comparator-compare-to>
>             </command-set>
> 		</test-integration-test>
> 		<test-post-integration-test>
> 		</test-post-integration-test>
>         </bigtop-itest-suite-test>
>         <bigtop-itest-suite-test>
>             <test-name>calculate pi</test-name>
>             <test-desc>calculate pi using hadoop MR</test-desc>
>             <test-pre-integration-test>
>             </test-pre-integration-test>
>             <test-integration-test>
>                 <command-set>
>                 <command>hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar pi 5
5</command>
>                 <command-comparator-type>org.apache.hadoop.cli.util.SubstringComparator</command-comparator-type>
>                 <command-comparator-compare-to><![CDATA[echo "Pi is 3.68"]]></command-comparator-compare-to>
>                 </command-set>
>             </test-integration-test>
>             <test-post-integration-test>
>             </test-post-integration-test>
>         </bigtop-itest-suite-test>
>         <bigtop-itest-suite-test>
>             <test-name>count word in MR</test-name>
>             <test-desc>count word in Hadoop MR</test-desc>
>             <test-pre-integration-test>
>                 <command-set><command>rm -rf ./wordcount</command></command-set>
>                 <command-set><command>rm -rf ./wordcount_out</command></command-set>
>                 <command-set><command>mkdir ./wordcount</command></command-set>
>                 <command-set><command><![CDATA[curl http://www.meetup.com/HandsOnProgrammingEvents/events/53837022/
| sed -e :a -e 's/<[^>]*>//g;/</N;//ba' | sed 's/&nbsp//g' | sed 's/^[ \t]*//;s/[
\t]*$//'  | sed '/^$/d' | sed '/"http[^"]*"/d' > ./wordcount/content]]></command></command-set>
>                 <command-set><command>hadoop fs -mkdir /wordcount</command></command-set>
>                 <command-set><command>hadoop fs -put ./wordcount/* /wordcount</command></command-set>
>             </test-pre-integration-test>
>                 <test-integration-test>
>                     <command-set><command>hadoop jar $HADOOP_HOME/hadoop-examples-0.*.0.jar
wordcount /wordcount /wordcount_out</command></command-set>
>                     <command-set><command>mkdir ./wordcount_out</command></command-set>
>                     <command-set><command>hadoop fs -get /wordcount_out/*
./wordcount_out</command></command-set>
>                     <command-set><command>hadoop fs -rmr /wordcount</command></command-set>
>                     <command-set><command>hadoop fs -rmr /wordcount_out/</command></command-set>
>                 </test-integration-test>
>                 <test-post-integration-test>
>                     <command-set>
>                     <command>cat ./wordcount_out/* | grep  Roman | sed 's/[^0-9.]*\([0-9.]*\).*/\1/'</command>
>                     <command-comparator-type>com.lei.bigtop.hadoop.integration.test.ExtactComparatorIgnoreWhiteSpace</command-comparator-type>
>                     <command-comparator-compare-to><![CDATA[cat wordcount/*
| grep -c Roman]]></command-comparator-compare-to>
>                     </command-set>
>                 </test-post-integration-test>
>         </bigtop-itest-suite-test>
> </bigtop-itest-suite>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message