hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From neelesh gadhia <ngad...@yahoo.com>
Subject Re: [SQLWindowing] Windowing function output path syntax (#26)
Date Mon, 25 Feb 2013 18:55:51 GMT
my bad. After setting the value to 16MB in bytes (16777216) it worked fine. 

Set hive.ptf.partition.persistence.memsize=16777216;


So does this mean default value of 64MB was causing issue? if yes, can you please elaborate a little how were you able to troubleshoot/debut the issue to this parameter setting?

Thanks,
Neelesh



________________________________
 From: "Butani, Harish" <harish.butani@sap.com>
To: neelesh gadhia <ngadhia@yahoo.com> 
Cc: "user@hive.apache.org" <user@hive.apache.org>; Ashutosh Chauhan <hashutosh@apache.org> 
Sent: Monday, February 25, 2013 9:53 AM
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)
 

How are you setting the value? It needs to be set in bytes.
From: neelesh gadhia <ngadhia@yahoo.com>
Reply-To: neelesh gadhia <ngadhia@yahoo.com>
Date: Sunday, February 24, 2013 10:01 PM
To: SAP SAP <harish.butani@sap.com>
Cc: "user@hive.apache.org" <user@hive.apache.org>, "hashutosh@apache.org" <hashutosh@apache.org>
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)


treid 16 and 32 mb and get a different error.

hive> Set hive.ptf.partition.persistence.memsize=16;
hive> select mid, tdate, tamt,sum(tamt) as com_sum over (rows between unbounded preceding and current row)
    > from t_enc
    > distribute by mid
    > sort by mid, tdate;

1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.SEL :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

4.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
    ]

5.PTF :
RowResolver::
    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding unbounded) current))) -> com_sum
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current)))
    ]


1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.SEL :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

4.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
    ]

5.PTF :
RowResolver::
    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding unbounded) current))) -> com_sum
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current)))
    ]

6.SEL :
RowResolver::
    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
    Aliases:[
        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum -> _col3
    ]
    columns mapped to expressions:[
    ]

7.FS :
RowResolver::
    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
    Aliases:[
        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum -> _col3
    ]
    columns mapped to expressions:[
    ]

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201302242150_0002, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201302242150_0002
Kill Command = /usr/local/Cellar/hadoop/1.1.1/libexec/bin/../bin/hadoop job  -kill job_201302242150_0002
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-02-24 21:58:36,550 Stage-1 map = 0%,  reduce = 0%
2013-02-24 21:58:38,560 Stage-1 map = 100%,  reduce = 0%
2013-02-24 21:58:45,595 Stage-1 map = 100%,  reduce = 33%
2013-02-24 21:58:48,616 Stage-1 map = 100%,  reduce = 100%
2013-02-24 21:59:01,693 Stage-1 map = 100%,  reduce = 0%
2013-02-24 21:59:08,723 Stage-1 map = 100%,  reduce = 33%
2013-02-24 21:59:11,738 Stage-1 map = 100%,  reduce = 100%
2013-02-24 21:59:25,805 Stage-1 map = 100%,  reduce = 0%
2013-02-24 21:59:32,833 Stage-1 map = 100%,  reduce = 33%
2013-02-24 21:59:35,852 Stage-1 map = 100%,  reduce = 100%
2013-02-24 21:59:49,912 Stage-1 map = 100%,  reduce = 0%
2013-02-24 21:59:56,941 Stage-1 map = 100%,  reduce = 33%
2013-02-24 21:59:59,954 Stage-1 map = 100%,  reduce = 100%
2013-02-24 22:00:14,026 Stage-1 map = 100%,  reduce = 0%
2013-02-24 22:00:15,030 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201302242150_0002 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201302242150_0002
Examining task ID: task_201302242150_0002_m_000002 (and more) from job job_201302242150_0002

Task with the most failures(4): 
-----
Task ID:
  task_201302242150_0002_r_000000

URL:
  http://localhost:50030/taskdetails.jsp?jobid=job_201302242150_0002&tipid=task_201302242150_0002_r_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: java.lang.StackOverflowError
    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:274)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.StackOverflowError
    at sun.nio.cs.UTF_8.updatePositions(UTF_8.java:58)
    at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:392)
    at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:447)
    at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544)
    at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:240)
    at java.lang.StringCoding.encode(StringCoding.java:272)
    at java.lang.String.getBytes(String.java:946)
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.checkAndCreate(File.java:1705)
    at java.io.File.createTempFile0(File.java:1726)
    at java.io.File.createTempFile(File.java:1803)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.addPartition(PTFPersistence.java:420)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:495)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.append(PTFPersistence.java:496)
    at org.apache.hadoop.hive.ql.exec.PTFPer

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1  Reduce: 1   HDFS Read: 1011 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive> 




________________________________
 From: "Butani, Harish" <harish.butani@sap.com>
To: neelesh gadhia <ngadhia@yahoo.com> 
Cc: "user@hive.apache.org" <user@hive.apache.org>; Ashutosh Chauhan <hashutosh@apache.org>; neelesh gadhia <ngadhia@yahoo.com> 
Sent: Sunday, February 24, 2013 9:22 PM
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)


Can you try
Set hive.ptf.partition.persistence.memsize=something lower

The default is 64mb; will fix it so that the default is lower

On Feb 24, 2013, at 8:26 PM, "neelesh gadhia" <ngadhia@yahoo.com> wrote:


Hi Harish,
>
>I downloaded the latest code from https://github.com/apache/hive/tree/ptf-windowing ( ptf-windowing branch). 
>
>I ran the same sql and get a different error now. Please advise.
>
>hive> select mid, tdate, tamt,sum(tamt) as com_sum over (rows between unbounded preceding and current row)
>    > from t_enc
>    > distribute by mid
>    > sort by mid, tdate;
>
>1.TS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>2.SEL :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>3.RS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>4.EX :
>RowResolver::
>    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
>    Aliases:[
>        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
>    ]
>    columns mapped to expressions:[
>    ]
>
>5.PTF :
>RowResolver::
>    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
>    Aliases:[
>        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding unbounded) current))) -> com_sum
>        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
>    ]
>    columns mapped to expressions:[
>        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current)))
>    ]
>
>
>1.TS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>2.SEL :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>3.RS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>4.EX :
>RowResolver::
>    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
>    Aliases:[
>        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
>    ]
>    columns mapped to expressions:[
>    ]
>
>5.PTF :
>RowResolver::
>    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
>    Aliases:[
>        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding unbounded) current))) -> com_sum
>        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file -> _col3, input__file__name -> _col4
>    ]
>    columns mapped to expressions:[
>        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current)))
>    ]
>
>6.SEL :
>RowResolver::
>    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
>    Aliases:[
>        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum -> _col3
>    ]
>    columns mapped to expressions:[
>    ]
>
>7.FS :
>RowResolver::
>    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
>    Aliases:[
>        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum -> _col3
>    ]
>    columns mapped to expressions:[
>    ]
>
>Total MapReduce jobs = 1
>Launching Job 1 out of 1
>Number of reduce tasks not specified. Estimated from input data size: 1
>In order to change the average load for a reducer (in bytes):
>  set hive.exec.reducers.bytes.per.reducer=<number>
>In order to limit the maximum number of reducers:
>  set hive.exec.reducers.max=<number>
>In order to set a constant number of reducers:
>  set mapred.reduce.tasks=<number>
>Starting Job = job_201302242020_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201302242020_0001
>Kill Command = /usr/local/Cellar/hadoop/1.1.1/libexec/bin/../bin/hadoop job  -kill job_201302242020_0001
>Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
>2013-02-24 20:22:38,431 Stage-1 map = 0%,  reduce = 0%
>2013-02-24 20:22:40,451 Stage-1 map = 100%,  reduce = 0%
>2013-02-24 20:22:48,505 Stage-1 map = 100%,  reduce = 33%
>2013-02-24 20:22:52,530 Stage-1 map = 100%,  reduce = 0%
>2013-02-24 20:23:00,566 Stage-1 map = 100%,  reduce = 33%
>2013-02-24 20:23:04,586 Stage-1 map = 100%,  reduce = 0%
>2013-02-24 20:23:12,625 Stage-1 map = 100%,  reduce = 33%
>2013-02-24 20:23:16,645 Stage-1 map = 100%,  reduce = 0%
>2013-02-24 20:23:23,675 Stage-1 map = 100%,  reduce = 33%
>2013-02-24 20:23:27,694 Stage-1 map = 100%,  reduce = 0%
>2013-02-24 20:23:28,699 Stage-1 map = 100%,  reduce = 100%
>Ended Job = job_201302242020_0001 with errors
>Error during job, obtaining debugging information...
>Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201302242020_0001
>Examining task ID: task_201302242020_0001_m_000002 (and more) from job job_201302242020_0001
>
>Task with the most failures(4): 
>-----
>Task ID:
>  task_201302242020_0001_r_000000
>
>URL:
>  http://localhost:50030/taskdetails.jsp?jobid=job_201302242020_0001&tipid=task_201302242020_0001_r_000000
>-----
>Diagnostic Messages for this Task:
>java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":1002,"reducesinkkey1":"2012/03/01 00:01:00"},"value":{"_col0":1002,"_col1":"2012/03/01 00:01:00","_col2":110.0,"_col3":32,"_col4":"hdfs://localhost:8020/user/hive/warehouse/t_enc/ngg.txt"},"alias":0}
>    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:274)
>    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
>    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
>    at org.apache.hadoop.mapred.Child.main(Child.java:249)
>Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":1002,"reducesinkkey1":"2012/03/01 00:01:00"},"value":{"_col0":1002,"_col1":"2012/03/01 00:01:00","_col2":110.0,"_col3":32,"_col4":"hdfs://localhost:8020/user/hive/warehouse/t_enc/ngg.txt"},"alias":0}
>    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:262)
>    ... 7 more
>Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.reflect.InvocationTargetException
>    at org.apache.hadoop.hive.ql.exec.PTFPersistence.createList(PTFPersistence.java:65)
>    at org.apache.hadoop.hive.ql.exec.PTFPartition.init(PTFPartition.java:60)
>    at org.apache.hadoop.hive.ql.exec.PTFPartition.<init>(PTFPartition.java:53)
>    at org.apache.hadoop.hive.ql.exec.PTFOperator.createFirstPartitionForChain(PTFOperator.java:430)
>    at org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:140)
>    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:521)
>    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:849)
>    at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
>    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:521)
>    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:253)
>    ... 7 more
>Caused by: java.lang.reflect.InvocationTargetException
>    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>    at org.apache.hadoop.hive.ql.exec.PTFPersistence.createList(PTFPersistence.java:61)
>    ... 16 more
>Caused by: java.lang.OutOfMemoryError: Java heap space
>    at org.apache.hadoop.hive.ql.exec.PTFPersistence$ByteBasedList.<init>(PTFPersistence.java:95)
>    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.addPartition(PTFPersistence.java:425)
>    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.<init>(PTFPersistence.java:404)
>    ... 21 more
>
>
>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>MapReduce Jobs Launched: 
>Job 0: Map: 1  Reduce: 1   HDFS Read: 1011 HDFS Write: 0 FAIL
>Total MapReduce CPU Time Spent: 0 msec
>hive> 
>
>
>
>
>
>
>________________________________
> From: neelesh gadhia <ngadhia@yahoo.com>
>To: "Butani, Harish" <harish.butani@sap.com>; "user@hive.apache.org" <user@hive.apache.org>; Ashutosh Chauhan <hashutosh@apache.org> 
>Sent: Friday, February 22, 2013 11:00 PM
>Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)
>
>
>Hi Harish,
>
>I did download the latest code from ptf-windowing branch on Feb 19 1:14 am PST. See attached screen shot of the location I used to download.
>
>I will try to download again and build it and update you with failed task log if I am able to reproduce the issue.
>
>thanks,
>Neelesh
>
>
>
>________________________________
> From: "Butani, Harish" <harish.butani@sap.com>
>To: "user@hive.apache.org" <user@hive.apache.org>; neelesh gadhia <ngadhia@yahoo.com>; Ashutosh Chauhan <hashutosh@apache.org> 
>Sent: Friday, February 22, 2013 10:40 PM
>Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)
>
>
> The plan dump indicates that you are not using the latest code from the ptf-windowing branch. So first of all, if you can please try with the latest code.
>Otherwise can you post the log file of the failed task; and also tell us which version of the code you are using. 
>
>
>Regards,
>Harish.
>
>From: neelesh gadhia <ngadhia@yahoo.com>
>Reply-To: "user@hive.apache.org" <user@hive.apache.org>, neelesh gadhia <ngadhia@yahoo.com>
>Date: Friday, February 22, 2013 2:55 PM
>To: "user@hive.apache.org" <user@hive.apache.org>, "hashutosh@apache.org" <hashutosh@apache.org>
>Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)
>
>
>
>Thanks Harish for your quick response.
>
>I tried it with the new syntax using one of the example in ptf_general_queries.q and get the following error. Am I still doing something wrong here?
>
>hive> select mid, tdate, tamt,sum(tamt) as com_sum over (rows between unbounded preceding and current row)
>    > from t_enc
>    > distribute by mid 
>    > sort by mid, tdate;
>
>1.TS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>2.RS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>3.EX :
>RowResolver::
>    columns:[t_enc._col0, t_enc._col1, t_enc._col2]
>    Aliases:[
>        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2
>    ]
>    columns mapped to expressions:[
>    ]
>
>4.PTF :
>RowResolver::
>    columns:[<null>._col0, t_enc._col1, t_enc._col2, t_enc._col3]
>    Aliases:[
>        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding unbounded) current))) -> _col0
>        t_enc:[mid -> _col1, tdate -> _col2, tamt -> _col3
>    ]
>    columns mapped to expressions:[
>        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current)))
>    ]
>
>insclause-0:
>Def ObjectInspector:[_col0, _col1, _col2]
>SerDe:org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
>windowingtablefunction:
>Def ObjectInspector:[com_sum, _col0, _col1, _col2]
>SerDe:org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
>Evaluator Output ObjectInspector:[com_sum, _col0, _col1, _col2]
>SelectList:_col0, _col1, _col2, _col3
>
>1.TS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>2.RS :
>RowResolver::
>    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
>    Aliases:[
>        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file -> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
>    ]
>    columns mapped to expressions:[
>    ]
>
>3.EX :
>RowResolver::
>    columns:[t_enc._col0, t_enc._col1, t_enc._col2]
>    Aliases:[
>        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2
>    ]
>    columns mapped to expressions:[
>    ]
>
>4.PTF :
>RowResolver::
>    columns:[<null>._col0, t_enc._col1, t_enc._col2, t_enc._col3]
>    Aliases:[
>        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding unbounded) current))) -> _col0
>        t_enc:[mid -> _col1, tdate -> _col2, tamt -> _col3
>    ]
>    columns mapped to expressions:[
>        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding unbounded) current)))
>    ]
>
>5.SEL :
>RowResolver::
>    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
>    Aliases:[
>        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum -> _col3
>    ]
>    columns mapped to expressions:[
>    ]
>
>6.FS :
>RowResolver::
>    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
>    Aliases:[
>        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum -> _col3
>    ]
>    columns mapped to expressions:[
>    ]
>
>Total MapReduce jobs = 1
>Launching Job 1 out of 1
>Number of reduce tasks not specified. Estimated from input data size: 1
>In order to change the average load for a reducer (in bytes):
>  set hive.exec.reducers.bytes.per.reducer=<number>
>In order to limit the maximum number of reducers:
>  set hive.exec.reducers.max=<number>
>In order to set a constant number of reducers:
>  set mapred.reduce.tasks=<number>
>Starting Job = job_201302221435_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201302221435_0001
>Kill Command = /usr/local/Cellar/hadoop/1.1.1/libexec/bin/../bin/hadoop job  -kill job_201302221435_0001
>Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
>2013-02-22 14:52:43,467 Stage-1 map = 0%,  reduce = 0%
>2013-02-22 14:53:05,568 Stage-1 map = 100%,  reduce = 100%
>Ended Job = job_201302221435_0001 with errors
>Error during job, obtaining debugging information...
>Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201302221435_0001
>Examining task ID: task_201302221435_0001_m_000002 (and more) from job job_201302221435_0001
>
>Task with the most failures(4): 
>-----
>Task ID:
>  task_201302221435_0001_m_000000
>
>URL:
>  http://localhost:50030/taskdetails.jsp?jobid=job_201302221435_0001&tipid=task_201302221435_0001_m_000000
>-----
>Diagnostic Messages for this Task:
>java.lang.RuntimeException: java.util.NoSuchElementException
>    at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:228)
>    at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>    at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
>    at org.apache.hadoop.mapred.Child.main(Child.java:249)
>Caused by: java.util.NoSuchElementException
>    at java.util.Vector.lastElement(Vector.java:456)
>    at com.sun.beans.ObjectHandler.lastExp(ObjectHandler.java:134)
>    at com.sun.beans.ObjectHandler.addArg(ObjectHandler.java:119)
>    at com.sun.beans.ObjectHandler.endElement(ObjectHandler.java:374)
>    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:593)
>    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
>    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2939)
>    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:647)
>    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
>    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
>    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
>    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
>    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
>    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
>    at javax.xml.parsers.SAXParser.parse(SAXParser.java:364)
>    at javax.xml.parsers.SAXParser.parse(SAXParser.java:142)
>    at java.beans.XMLDecoder$1.run(XMLDecoder.java:248)
>    at java.beans.XMLDecoder$1.run(XMLDecoder.java:242)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at java.beans.XMLDecoder.getHandler(XMLDecoder.java:242)
>    at java.beans.XMLDecoder.close(XMLDecoder.java:155)
>    at org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:525)
>    at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:220)
>    ... 12 more
>
>
>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>MapReduce Jobs Launched: 
>Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
>Total MapReduce CPU Time Spent: 0 msec
>hive> 
>
>
>
>
>
>
>
>
>________________________________
> From: "Butani, Harish" <harish.butani@sap.com>
>To: "user@hive.apache.org" <user@hive.apache.org>; neelesh gadhia <ngadhia@yahoo.com>; Ashutosh Chauhan <hashutosh@apache.org> 
>Sent: Friday, February 22, 2013 2:44 PM
>Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)
>
>
>Hi Neelesh,
>
>
>You are using the syntax from the SQLWindowing project; which was done on top of HQL. 
>Now the syntax is standard SQL; see ptf_general_queries.q for examples. Your e.g can be expressed as:
>
>
>select sum(tamt) over (partition by mid order by mid rows between unbounded preceding and current row) as cum_amt,mid,tdate,tamt,cum_amt
> 
>From t_enc
> 
>
>
>Regards,
>Harish.From: neelesh gadhia <ngadhia@yahoo.com>
>Reply-To: "user@hive.apache.org" <user@hive.apache.org>, neelesh gadhia <ngadhia@yahoo.com>
>Date: Friday, February 22, 2013 2:05 PM
>To: "hashutosh@apache.org" <hashutosh@apache.org>, "user@hive.apache.org" <user@hive.apache.org>
>Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)
>
>
>
>Hello,
>
>
>I downloaded the source code from ptf-windowing branch and build the dist based on that. 
>
>
>
>Now when I try to make use of windowing function using the following ql, I get an error as shown below. Am I missing anything here? Please advise.
>
>
>
>
>
>from <select mid, tdate, tamt from t_enc >
>partition by mid
>order by mid
>with
>sum(tamt) over rows between
>unbounded preceding and current row as cum_amt
>select mid,tdate,tamt,cum_amt;
>
>
>
>
>hive> from <select mid, tdate, tamt from t_enc >    
>    > partition by mid
>    > order by mid
>    > with
>    > sum(tamt) over rows between
>    > unbounded preceding and current row as cum_amt
>    > select mid,tdate,tamt,cum_amt;
>NoViableAltException(258@[])
>    at org.apache.hadoop.hive.ql.parse.HiveParser.joinSource(HiveParser.java:32612)
>    at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:32498)
>    at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatement(HiveParser.java:26832)
>    at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:26716)
>    at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:981)
>    at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:687)
>    at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:444)
>    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416)
>    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
>    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:898)
>    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
>    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
>    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>    at java.lang.reflect.Method.invoke(Method.java:597)
>    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>FAILED: ParseException line 1:5 cannot recognize input near '<' 'select' 'mid' in join source
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>________________________________
> From: hbutani <notifications@github.com>
>To: hbutani/SQLWindowing <SQLWindowing@noreply.github.com> 
>Cc: ngadhia <ngadhia@yahoo.com> 
>Sent: Sunday, February 17, 2013 4:50 PM
>Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)
>
>
>Hi,
>We don't actively support this library anymore. This functionality is in the process of being folded into hive. You can see the latest code at https://github.com/apache/hive, the ptf-windowing branch. Also checkout the Jiras in Hive Jira: look for Jiras whose Component is PTF-Windowing.
>regards,
>Harish.
>—
>Reply to this email directly or view it on GitHub. 
>
>
>
>
>
>
>
>
Mime
View raw message