hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Butani, Harish" <harish.but...@sap.com>
Subject Re: [SQLWindowing] Windowing function output path syntax (#26)
Date Mon, 25 Feb 2013 05:22:25 GMT
Can you try
Set hive.ptf.partition.persistence.memsize=something lower

The default is 64mb; will fix it so that the default is lower

On Feb 24, 2013, at 8:26 PM, "neelesh gadhia" <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>
wrote:

Hi Harish,

I downloaded the latest code from https://github.com/apache/hive/tree/ptf-windowing ( ptf-windowing
branch).

I ran the same sql and get a different error now. Please advise.

hive> select mid, tdate, tamt,sum(tamt) as com_sum over (rows between unbounded preceding
and current row)
    > from t_enc
    > distribute by mid
    > sort by mid, tdate;

1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.SEL :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

4.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file
-> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
    ]

5.PTF :
RowResolver::
    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding
unbounded) current))) -> com_sum
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file
-> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding
unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE
(preceding unbounded) current)))
    ]


1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.SEL :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

4.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file
-> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
    ]

5.PTF :
RowResolver::
    columns:[<null>.com_sum, t_enc._col0, t_enc._col1, t_enc._col2, t_enc._col3, t_enc._col4]
    Aliases:[
        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding
unbounded) current))) -> com_sum
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2, block__offset__inside__file
-> _col3, input__file__name -> _col4
    ]
    columns mapped to expressions:[
        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding
unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE
(preceding unbounded) current)))
    ]

6.SEL :
RowResolver::
    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
    Aliases:[
        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum ->
_col3
    ]
    columns mapped to expressions:[
    ]

7.FS :
RowResolver::
    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
    Aliases:[
        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum ->
_col3
    ]
    columns mapped to expressions:[
    ]

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201302242020_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201302242020_0001
Kill Command = /usr/local/Cellar/hadoop/1.1.1/libexec/bin/../bin/hadoop job  -kill job_201302242020_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-02-24 20:22:38,431 Stage-1 map = 0%,  reduce = 0%
2013-02-24 20:22:40,451 Stage-1 map = 100%,  reduce = 0%
2013-02-24 20:22:48,505 Stage-1 map = 100%,  reduce = 33%
2013-02-24 20:22:52,530 Stage-1 map = 100%,  reduce = 0%
2013-02-24 20:23:00,566 Stage-1 map = 100%,  reduce = 33%
2013-02-24 20:23:04,586 Stage-1 map = 100%,  reduce = 0%
2013-02-24 20:23:12,625 Stage-1 map = 100%,  reduce = 33%
2013-02-24 20:23:16,645 Stage-1 map = 100%,  reduce = 0%
2013-02-24 20:23:23,675 Stage-1 map = 100%,  reduce = 33%
2013-02-24 20:23:27,694 Stage-1 map = 100%,  reduce = 0%
2013-02-24 20:23:28,699 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201302242020_0001 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201302242020_0001
Examining task ID: task_201302242020_0001_m_000002 (and more) from job job_201302242020_0001

Task with the most failures(4):
-----
Task ID:
  task_201302242020_0001_r_000000

URL:
  http://localhost:50030/taskdetails.jsp?jobid=job_201302242020_0001&tipid=task_201302242020_0001_r_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0) {"key":{"reducesinkkey0":1002,"reducesinkkey1":"2012/03/01
00:01:00"},"value":{"_col0":1002,"_col1":"2012/03/01 00:01:00","_col2":110.0,"_col3":32,"_col4":"hdfs://localhost:8020/user/hive/warehouse/t_enc/ngg.txt"},"alias":0}
    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:274)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing
row (tag=0) {"key":{"reducesinkkey0":1002,"reducesinkkey1":"2012/03/01 00:01:00"},"value":{"_col0":1002,"_col1":"2012/03/01
00:01:00","_col2":110.0,"_col3":32,"_col4":"hdfs://localhost:8020/user/hive/warehouse/t_enc/ngg.txt"},"alias":0}
    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:262)
    ... 7 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.hive.ql.exec.PTFPersistence.createList(PTFPersistence.java:65)
    at org.apache.hadoop.hive.ql.exec.PTFPartition.init(PTFPartition.java:60)
    at org.apache.hadoop.hive.ql.exec.PTFPartition.<init>(PTFPartition.java:53)
    at org.apache.hadoop.hive.ql.exec.PTFOperator.createFirstPartitionForChain(PTFOperator.java:430)
    at org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:140)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:521)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:849)
    at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:521)
    at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:253)
    ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence.createList(PTFPersistence.java:61)
    ... 16 more
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$ByteBasedList.<init>(PTFPersistence.java:95)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.addPartition(PTFPersistence.java:425)
    at org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList.<init>(PTFPersistence.java:404)
    ... 21 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   HDFS Read: 1011 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive>


________________________________
From: neelesh gadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>
To: "Butani, Harish" <harish.butani@sap.com<mailto:harish.butani@sap.com>>; "user@hive.apache.org<mailto:user@hive.apache.org>"
<user@hive.apache.org<mailto:user@hive.apache.org>>; Ashutosh Chauhan <hashutosh@apache.org<mailto:hashutosh@apache.org>>
Sent: Friday, February 22, 2013 11:00 PM
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

Hi Harish,

I did download the latest code from ptf-windowing branch on Feb 19 1:14 am PST. See attached
screen shot of the location I used to download.

I will try to download again and build it and update you with failed task log if I am able
to reproduce the issue.

thanks,
Neelesh

________________________________
From: "Butani, Harish" <harish.butani@sap.com<mailto:harish.butani@sap.com>>
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>;
neelesh gadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>; Ashutosh Chauhan
<hashutosh@apache.org<mailto:hashutosh@apache.org>>
Sent: Friday, February 22, 2013 10:40 PM
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

 The plan dump indicates that you are not using the latest code from the ptf-windowing branch.
So first of all, if you can please try with the latest code.
Otherwise can you post the log file of the failed task; and also tell us which version of
the code you are using.

Regards,
Harish.

From: neelesh gadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>,
neelesh gadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>
Date: Friday, February 22, 2013 2:55 PM
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>,
"hashutosh@apache.org<mailto:hashutosh@apache.org>" <hashutosh@apache.org<mailto:hashutosh@apache.org>>
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

Thanks Harish for your quick response.

I tried it with the new syntax using one of the example in ptf_general_queries.q and get the
following error. Am I still doing something wrong here?

hive> select mid, tdate, tamt,sum(tamt) as com_sum over (rows between unbounded preceding
and current row)
    > from t_enc
    > distribute by mid
    > sort by mid, tdate;

1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2
    ]
    columns mapped to expressions:[
    ]

4.PTF :
RowResolver::
    columns:[<null>._col0, t_enc._col1, t_enc._col2, t_enc._col3]
    Aliases:[
        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding
unbounded) current))) -> _col0
        t_enc:[mid -> _col1, tdate -> _col2, tamt -> _col3
    ]
    columns mapped to expressions:[
        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding
unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE
(preceding unbounded) current)))
    ]

insclause-0:
Def ObjectInspector:[_col0, _col1, _col2]
SerDe:org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
windowingtablefunction:
Def ObjectInspector:[com_sum, _col0, _col1, _col2]
SerDe:org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Evaluator Output ObjectInspector:[com_sum, _col0, _col1, _col2]
SelectList:_col0, _col1, _col2, _col3

1.TS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

2.RS :
RowResolver::
    columns:[t_enc.mid, t_enc.tdate, t_enc.tamt, t_enc.BLOCK__OFFSET__INSIDE__FILE, t_enc.INPUT__FILE__NAME]
    Aliases:[
        t_enc:[mid -> mid, tdate -> tdate, tamt -> tamt, block__offset__inside__file
-> BLOCK__OFFSET__INSIDE__FILE, input__file__name -> INPUT__FILE__NAME
    ]
    columns mapped to expressions:[
    ]

3.EX :
RowResolver::
    columns:[t_enc._col0, t_enc._col1, t_enc._col2]
    Aliases:[
        t_enc:[mid -> _col0, tdate -> _col1, tamt -> _col2
    ]
    columns mapped to expressions:[
    ]

4.PTF :
RowResolver::
    columns:[<null>._col0, t_enc._col1, t_enc._col2, t_enc._col3]
    Aliases:[
        :[(tok_function sum (tok_table_or_col tamt) (tok_windowspec (tok_windowrange (preceding
unbounded) current))) -> _col0
        t_enc:[mid -> _col1, tdate -> _col2, tamt -> _col3
    ]
    columns mapped to expressions:[
        (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE (preceding
unbounded) current))) -> (TOK_FUNCTION sum (TOK_TABLE_OR_COL tamt) (TOK_WINDOWSPEC (TOK_WINDOWRANGE
(preceding unbounded) current)))
    ]

5.SEL :
RowResolver::
    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
    Aliases:[
        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum ->
_col3
    ]
    columns mapped to expressions:[
    ]

6.FS :
RowResolver::
    columns:[<null>._col0, <null>._col1, <null>._col2, <null>._col3]
    Aliases:[
        <null>:[mid -> _col0, tdate -> _col1, tamt -> _col2, com_sum ->
_col3
    ]
    columns mapped to expressions:[
    ]

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201302221435_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201302221435_0001
Kill Command = /usr/local/Cellar/hadoop/1.1.1/libexec/bin/../bin/hadoop job  -kill job_201302221435_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-02-22 14:52:43,467 Stage-1 map = 0%,  reduce = 0%
2013-02-22 14:53:05,568 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201302221435_0001 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201302221435_0001
Examining task ID: task_201302221435_0001_m_000002 (and more) from job job_201302221435_0001

Task with the most failures(4):
-----
Task ID:
  task_201302221435_0001_m_000000

URL:
  http://localhost:50030/taskdetails.jsp?jobid=job_201302221435_0001&tipid=task_201302221435_0001_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: java.util.NoSuchElementException
    at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:228)
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
    at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.util.NoSuchElementException
    at java.util.Vector.lastElement(Vector.java:456)
    at com.sun.beans.ObjectHandler.lastExp(ObjectHandler.java:134)
    at com.sun.beans.ObjectHandler.addArg(ObjectHandler.java:119)
    at com.sun.beans.ObjectHandler.endElement(ObjectHandler.java:374)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:593)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2939)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:647)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
    at javax.xml.parsers.SAXParser.parse(SAXParser.java:364)
    at javax.xml.parsers.SAXParser.parse(SAXParser.java:142)
    at java.beans.XMLDecoder$1.run(XMLDecoder.java:248)
    at java.beans.XMLDecoder$1.run(XMLDecoder.java:242)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.beans.XMLDecoder.getHandler(XMLDecoder.java:242)
    at java.beans.XMLDecoder.close(XMLDecoder.java:155)
    at org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:525)
    at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:220)
    ... 12 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive>





________________________________
From: "Butani, Harish" <harish.butani@sap.com<mailto:harish.butani@sap.com>>
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>;
neelesh gadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>; Ashutosh Chauhan
<hashutosh@apache.org<mailto:hashutosh@apache.org>>
Sent: Friday, February 22, 2013 2:44 PM
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

Hi Neelesh,

You are using the syntax from the SQLWindowing project; which was done on top of HQL.
Now the syntax is standard SQL; see ptf_general_queries.q for examples. Your e.g can be expressed
as:

select sum(tamt) over (partition by mid order by mid rows between unbounded preceding and
current row) as cum_amt,
mid,tdate,tamt,cum_amt
>From t_enc

Regards,
Harish.
From: neelesh gadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>,
neelesh gadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>
Date: Friday, February 22, 2013 2:05 PM
To: "hashutosh@apache.org<mailto:hashutosh@apache.org>" <hashutosh@apache.org<mailto:hashutosh@apache.org>>,
"user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

Hello,

I downloaded the source code from ptf-windowing branch and build the dist based on that.

Now when I try to make use of windowing function using the following ql, I get an error as
shown below. Am I missing anything here? Please advise.


from <select mid, tdate, tamt from t_enc >
partition by mid
order by mid
with
sum(tamt) over rows between
unbounded preceding and current row as cum_amt
select mid,tdate,tamt,cum_amt;


hive> from <select mid, tdate, tamt from t_enc >
    > partition by mid
    > order by mid
    > with
    > sum(tamt) over rows between
    > unbounded preceding and current row as cum_amt
    > select mid,tdate,tamt,cum_amt;
NoViableAltException(258@[])
    at org.apache.hadoop.hive.ql.parse.HiveParser.joinSource(HiveParser.java:32612)
    at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:32498)
    at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatement(HiveParser.java:26832)
    at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:26716)
    at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:981)
    at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:687)
    at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:444)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:898)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
FAILED: ParseException line 1:5 cannot recognize input near '<' 'select' 'mid' in join
source







________________________________
From: hbutani <notifications@github.com<mailto:notifications@github.com>>
To: hbutani/SQLWindowing <SQLWindowing@noreply.github.com<mailto:SQLWindowing@noreply.github.com>>
Cc: ngadhia <ngadhia@yahoo.com<mailto:ngadhia@yahoo.com>>
Sent: Sunday, February 17, 2013 4:50 PM
Subject: Re: [SQLWindowing] Windowing function output path syntax (#26)

Hi,
We don't actively support this library anymore. This functionality is in the process of being
folded into hive. You can see the latest code at https://github.com/apache/hive, the ptf-windowing
branch. Also checkout the Jiras in Hive Jira: look for Jiras whose Component is PTF-Windowing.
regards,
Harish.
-
Reply to this email directly or view it on GitHub<https://github.com/hbutani/SQLWindowing/issues/26#issuecomment-13701539>.










Mime
View raw message