tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JaeHwa Jung (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-144) Implement INSERT OVERWRITE clause
Date Tue, 03 Sep 2013 15:08:51 GMT

    [ https://issues.apache.org/jira/browse/TAJO-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756684#comment-13756684
] 

JaeHwa Jung commented on TAJO-144:
----------------------------------

I found two kinds of errors.

First, If I use "SELECT * FROM" clause, it made NullPointerException as follows:
{code:xml} 
tajo> insert overwrite into table1 select * from table2;
java.lang.NullPointerException
	at org.apache.tajo.engine.planner.rewrite.ProjectionPushDownRule.rewrite(ProjectionPushDownRule.java:71)
	at org.apache.tajo.engine.planner.rewrite.BasicQueryRewriteEngine.rewrite(BasicQueryRewriteEngine.java:63)
	at org.apache.tajo.engine.planner.LogicalOptimizer.optimize(LogicalOptimizer.java:40)
	at org.apache.tajo.master.GlobalEngine.createLogicalPlan(GlobalEngine.java:189)
	at org.apache.tajo.master.GlobalEngine.executeQuery(GlobalEngine.java:110)
	at org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.submitQuery(TajoMasterClientService.java:135)
	at org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:241)
	at org.apache.tajo.rpc.ProtoBlockingRpcServer$ServerHandler.messageReceived(ProtoBlockingRpcServer.java:91)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:680)</test> 
{code} 

Second, If I write each columns, it can't progress more than zero percent as follows:
{code:xml} 
tajo> insert overwrite into table4 select id, name, score, type from table2;
2013-09-03 23:51:26,073 INFO  client.TajoClient (TajoClient.java:connectionToQueryMaster(190))
- Connected to Query Master (qid=q_1378219667565_0004, addr=192.168.1.103:26121)
Progress: 0%, response time: 1.313 sec
Progress: 0%, response time: 2.315 sec
Progress: 0%, response time: 3.317 sec
Progress: 0%, response time: 4.318 sec
Progress: 0%, response time: 5.319 sec
Progress: 0%, response time: 6.321 sec
Progress: 0%, response time: 7.323 sec
Progress: 0%, response time: 8.325 sec
Progress: 0%, response time: 9.327 sec
Progress: 0%, response time: 10.328 sec
Progress: 0%, response time: 11.33 sec
{code}

In second case, Tajo master write log file as follows:
{code:xml}
2013-09-03 23:51:22,023 INFO  master.GlobalEngine (GlobalEngine.java:executeQuery(106)) -
SQL: insert overwrite into table4 select id, name, score, type from table2
2013-09-03 23:51:22,041 INFO  rewrite.BasicQueryRewriteEngine (BasicQueryRewriteEngine.java:rewrite(64))
- The rule "ProjectionPushDown " rewrites the query.
2013-09-03 23:51:22,047 INFO  master.GlobalEngine (GlobalEngine.java:createLogicalPlan(190))
- LogicalPlan:
Logical Plan Root

INSERT OVERWRITE INTO{
  "tableName": "table4",
  "uri": {},
  "meta": {
    "schema": {
      "fields": [
        {
          "name": "table4.id",
          "dataType": {}
        },
        {
          "name": "table4.name",
          "dataType": {}
        },
        {
          "name": "table4.score",
          "dataType": {}
        },
        {
          "name": "table4.type",
          "dataType": {}
        }
      ],
      "fieldsByName": {
        "table4.score": 2,
        "table4.name": 1,
        "table4.id": 0,
        "table4.type": 3
      }
    },
    "storeType": "CSV",
    "options": {
      "keyVals": {
        "csvfile.delimiter": "|"
      }
    },
    "stat": {
      "numRows": 0,
      "numBytes": 0,
      "numBlocks": 0,
      "numPartitions": 0,
      "avgRows": 0,
      "columnStats": []
    }
  }
} [SUBQUERY]
2013-09-03 23:51:22,051 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoAsyncDispatcher
is inited.
2013-09-03 23:51:22,051 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.querymaster.QueryInProgress
is inited.
2013-09-03 23:51:22,052 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoAsyncDispatcher
is started.
2013-09-03 23:51:22,052 INFO  master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:start(101))
- AsyncDispatcher started:QueryInProgress:q_1378219667565_0004
2013-09-03 23:51:22,052 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.querymaster.QueryInProgress
is started.
2013-09-03 23:51:22,052 INFO  querymaster.QueryInProgress (QueryInProgress.java:startQueryMaster(145))
- Initializing QueryInProgress for QueryID=q_1378219667565_0004
2013-09-03 23:51:22,052 INFO  rm.YarnTajoResourceManager (YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(204))
- Allocate and launch ApplicationMaster for QueryMaster: queryId=q_1378219667565_0004, appId=application_1378219667565_0004
2013-09-03 23:51:22,053 INFO  master.ContainerProxy (YarnContainerProxy.java:createCommonContainerLaunchContext(233))
- Set the environment for the application master
2013-09-03 23:51:22,053 INFO  master.ContainerProxy (YarnContainerProxy.java:createCommonContainerLaunchContext(301))
- defaultFS: hdfs://localhost:9010
2013-09-03 23:51:22,053 INFO  master.ContainerProxy (YarnContainerProxy.java:createCommonContainerLaunchContext(302))
- defaultFS: hdfs://localhost:9010
2013-09-03 23:51:22,055 INFO  master.ContainerProxy (YarnContainerProxy.java:createCommonContainerLaunchContext(316))
- Writing a QueryConf to HDFS and add to local environment
2013-09-03 23:51:22,134 INFO  rm.YarnTajoResourceManager (YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(265))
- Completed setting up QueryMasterRunner command ${JAVA_HOME}/bin/java -Xmx2000m org.apache.tajo.worker.TajoWorker
qm q_1378219667565_0004 blrunnerui-MacBook-Pro.local:9005 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr

2013-09-03 23:51:22,135 INFO  rm.YarnTajoResourceManager (YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(283))
- Submitting QueryMaster to ResourceManager
2013-09-03 23:51:22,137 INFO  client.YarnClientImpl (YarnClientImpl.java:submitApplication(134))
- Submitted application application_1378219667565_0004 to ResourceManager at /0.0.0.0:8032
2013-09-03 23:51:22,139 INFO  rm.YarnTajoResourceManager (YarnTajoResourceManager.java:monitorApplication(303))
- Got application report from ASM for, appId=4, appAttemptId=appattempt_1378219667565_0004_000001,
clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=0,
appStartTime=1378219882136, yarnAppState=SUBMITTED, distributedFinalState=UNDEFINED, appTrackingUrl=blrunnerui-MacBook-Pro.local:8088/proxy/application_1378219667565_0004/,
appUser=blrunner
2013-09-03 23:51:22,242 INFO  rm.YarnTajoResourceManager (YarnTajoResourceManager.java:monitorApplication(303))
- Got application report from ASM for, appId=4, appAttemptId=appattempt_1378219667565_0004_000001,
clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=0,
appStartTime=1378219882136, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=blrunnerui-MacBook-Pro.local:8088/proxy/application_1378219667565_0004/,
appUser=blrunner
2013-09-03 23:51:22,242 INFO  rm.YarnTajoResourceManager (YarnTajoResourceManager.java:allocateAndLaunchQueryMaster(289))
- Launching QueryMaster with appAttemptId: appattempt_1378219667565_0004_000001
2013-09-03 23:51:22,242 INFO  querymaster.QueryInProgress (QueryInProgress.java:submmitQueryToMaster(199))
- No QueryMaster conneciton info.
2013-09-03 23:51:25,769 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:25,769 INFO  querymaster.QueryInProgress (QueryInProgress.java:connectQueryMaster(182))
- Connect to QueryMaster:blrunnerui-MacBook-Pro.local/192.168.1.103:26122
2013-09-03 23:51:25,773 INFO  querymaster.QueryInProgress (QueryInProgress.java:submmitQueryToMaster(203))
- Call executeQuery to :blrunnerui-MacBook-Pro.local:26122,q_1378219667565_0004
2013-09-03 23:51:27,496 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:29,497 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:31,498 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:33,499 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:35,500 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:37,502 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:39,503 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
2013-09-03 23:51:41,504 INFO  querymaster.QueryInProgress (QueryInProgress.java:heartbeat(233))
- Received QueryMaster heartbeat:q_1378219667565_0004, queryMaster=host:blrunnerui-MacBook-Pro.local,
port=26122,26121,0, slots=0:0:0, used=0:0:0
{code}
                
> Implement INSERT OVERWRITE clause
> ---------------------------------
>
>                 Key: TAJO-144
>                 URL: https://issues.apache.org/jira/browse/TAJO-144
>             Project: Tajo
>          Issue Type: New Feature
>          Components: planner/optimizer
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.2-incubating
>
>         Attachments: TAJO-144.patch
>
>
> 'INSERT INTO' appends data into existing table. In contrast, 'INSERT OVERWRITE' will
overwrite existing table data. It is used as an idiom in analytical fields based HDFS. We
need it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message