pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhang, Liyun" <liyun.zh...@intel.com>
Subject some question about streaming_local.conf
Date Fri, 26 Dec 2014 08:47:37 GMT
Hi all:
  I have 2 questiones about pig/test/e2e/pig/tests/streaming_local.conf:
1.
$cfg = {
        'driver' => 'Pig',
        'nummachines' => 5,

        'groups' => [
                                {
                                # This group is for local mode testing
                                'name' => 'StreamingLocal',
                'sortBenchmark' => 1,
                'sortResults' => 1,
                'floatpostprocess' => 1,
                'delimiter' => '   ',
                'tests' => [
                                                {
                                        #Section 1.1: perl script, no parameters
                        'num' => 1,
                        'execonly' => 'local',  // this line
                        'pig' => q#

all e2e test cases are only executed in local mode now. Can these e2e tests run in other mode,
like "mapreduce","tez","spark"?
when i replace 'execonly'=>'local' with 'execonly'=>'spark', all cases pass when POStream
is implemented in spark mode.
I think we can remove 'execonly'=>'local' and can test these e2e tests in other modes.

2. when using ship with streaming, it will send streaming binary and supporting files, if
any, from the client node to the compute nodes..I found we use "perl ./libexec/GroupBy.pl"
in StreamingLocal_3.pig, this path is a relative path to current executed path. can we use
"perl GroupBy.pl" because i think the file "./libexec/GroupBy.pl" has been shipped to compute
nodes.
StreamingLocal_3.pig
/test/e2e/pigMD `perl ./libexec/GroupBy.pl '\t' 0` ship('./libexec/GroupBy.pl');
A = load './data/singlefile/studenttab10k';
B = group A by $0;
C = foreach B generate flatten(A);
D = stream C through CMD;
store D into './testout/root-1419582821-streaming_local.conf/StreamingLocal_3.out';



Best regards
Zhang,Liyun


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message