tajo-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-1397) Resource allocation should be fine grained.
Date Mon, 20 Jul 2015 08:55:04 GMT

    [ https://issues.apache.org/jira/browse/TAJO-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633191#comment-14633191
] 

Hudson commented on TAJO-1397:
------------------------------

ABORTED: Integrated in Tajo-master-CODEGEN-build #393 (See [https://builds.apache.org/job/Tajo-master-CODEGEN-build/393/])
TAJO-1397: Resource allocation should be fine grained. (jinho) (jhkim: rev 5a02873d5f0bd8fedd8527808bd3d4ecbe7d8af3)
* tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
* tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java
* tajo-core/src/main/proto/QueryCoordinatorProtocol.proto
* tajo-core/src/test/java/org/apache/tajo/worker/TestHistory.java
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterMergeJoinExec.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java
* tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
* tajo-core/src/main/resources/webapps/admin/cluster.jsp
* tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptToSchedulerEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java
* tajo-core/src/test/java/org/apache/tajo/querymaster/TestKillQuery.java
* tajo-core/src/main/java/org/apache/tajo/master/event/GrouppedContainerAllocatorEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/QMResourceAllocateEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatus.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleScheduler.java
* tajo-core/src/main/proto/ContainerProtocol.proto
* tajo-core/src/main/resources/webapps/admin/querytasks.jsp
* tajo-common/src/main/java/org/apache/tajo/util/StringUtils.java
* tajo-core/src/main/java/org/apache/tajo/engine/planner/enforce/Enforcer.java
* tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/AsyncRpcServer.java
* tajo-core/src/main/java/org/apache/tajo/master/event/ContainerEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
* tajo-common/src/main/java/org/apache/tajo/util/TUtil.java
* tajo-core/src/test/java/org/apache/tajo/worker/TestTaskExecutor.java
* tajo-core/src/main/proto/QueryMasterProtocol.proto
* tajo-core/src/main/java/org/apache/tajo/session/Session.java
* tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEvent.java
* tajo-core/src/main/resources/webapps/worker/queryplan.jsp
* tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStopEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/TaskManagerEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/TaskExecutorEvent.java
* tajo-core/src/main/resources/webapps/admin/index.jsp
* tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java
* tajo-core/src/test/java/org/apache/tajo/worker/MockTaskManager.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStopEvent.java
* tajo-core/src/test/java/org/apache/tajo/worker/MockTaskExecutor.java
* tajo-core/src/main/java/org/apache/tajo/master/event/TaskFatalErrorEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java
* tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerLivelinessMonitor.java
* tajo-core/src/main/proto/TajoWorkerProtocol.proto
* tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerReconnectEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java
* tajo-core/src/main/resources/webapps/admin/query.jsp
* tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/AbstractQueryScheduler.java
* tajo-core/src/test/java/org/apache/tajo/util/metrics/TestSystemMetrics.java
* tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/TaskStartEvent.java
* tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequestImpl.java
* tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java
* tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEventType.java
* tajo-core/src/test/java/org/apache/tajo/worker/MockExecutionBlock.java
* tajo-core/src/test/java/org/apache/tajo/engine/query/TestOuterJoinQuery.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerEvent.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterManagerService.java
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterHashJoinExec.java
* .travis.yml
* tajo-core/src/main/java/org/apache/tajo/ws/rs/resources/ClusterResource.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/QuerySchedulingInfo.java
* CHANGES
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueInfo.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
* tajo-core/src/main/resources/webapps/admin/query_executor.jsp
* tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
* tajo-core/src/main/java/org/apache/tajo/worker/TaskManager.java
* tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java
* tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptStatusUpdateEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/TajoResourceScheduler.java
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockErrorEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/SchedulingAlgorithms.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEvent.java
* tajo-core/src/test/java/org/apache/tajo/querymaster/TestQueryState.java
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterMergeJoinExec.java
* tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java
* tajo-core/src/test/java/org/apache/tajo/worker/MockNodeResourceManager.java
* tajo-core/src/main/java/org/apache/tajo/worker/LegacyTaskImpl.java
* tajo-core/src/main/java/org/apache/tajo/master/event/QueryStartEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerIdPBImpl.java
* tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestSimpleScheduler.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatusEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java
* tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
* tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoAdmin.java
* tajo-core/src/main/resources/webapps/worker/taskhistory.jsp
* tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
* tajo-core/src/main/resources/webapps/worker/index.jsp
* tajo-core/src/main/java/org/apache/tajo/master/event/StageContainerAllocationEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEventType.java
* tajo-core/src/main/resources/webapps/admin/task.jsp
* tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
* tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java
* tajo-common/src/main/java/org/apache/tajo/SessionVars.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/NodeReconnectEvent.java
* tajo-core/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java
* tajo-core/src/main/resources/webapps/worker/task.jsp
* tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
* tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/NodeState.java
* tajo-core/src/main/java/org/apache/tajo/master/event/StageEventType.java
* tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStartEvent.java
* tajo-core/src/main/proto/ResourceProtos.proto
* tajo-core/src/main/resources/webapps/worker/querydetail.jsp
* tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
* tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java
* tajo-core/src/main/java/org/apache/tajo/master/event/StageShuffleReportEvent.java
* tajo-core/src/main/proto/ResourceTrackerProtocol.proto
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java
* tajo-core/src/main/java/org/apache/tajo/metrics/WorkerResourceMetricsGaugeSet.java
* tajo-core/src/main/java/org/apache/tajo/util/JSPUtil.java
* tajo-core/src/main/resources/webapps/worker/tasks.jsp
* tajo-core/src/main/resources/webapps/worker/taskdetail.jsp
* tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ScanExec.java
* tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
* tajo-core/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java
* tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptEventType.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStartEvent.java
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/QueryStopEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/event/TaskCompletionEvent.java
* tajo-core/pom.xml
* tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java
* tajo-core/src/main/java/org/apache/tajo/util/history/HistoryReader.java
* tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocatorEventType.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEventType.java
* tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
* tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
* tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java
* tajo-project/pom.xml
* tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java
* tajo-core/src/main/resources/webapps/admin/querydetail.jsp
* tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/TaskContainer.java
* tajo-core/src/main/java/org/apache/tajo/ws/rs/responses/WorkerResponse.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceManager.java
* tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java
* tajo-core/src/test/java/org/apache/tajo/worker/MockWorkerContext.java
* tajo-docs/src/main/sphinx/configuration/worker_configuration.rst
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterTask.java
* tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
* tajo-core/src/main/java/org/apache/tajo/worker/FetchImpl.java
* tajo-core/src/main/java/org/apache/tajo/master/QueryCoordinatorService.java
* tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java
* tajo-core/src/main/resources/webapps/worker/taskcontainers.jsp
* tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java
* tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerLauncher.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/ResourceReserveSchedulerEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/NodeLivelinessMonitor.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerStatusEvent.java
* tajo-core/src/test/java/org/apache/tajo/worker/TestTaskManager.java
* tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java
* tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
* tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
* tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
* tajo-client/src/main/proto/ClientProtos.proto
* tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocationEvent.java
* tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerState.java
* tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/AbstractTaskScheduler.java
* tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java
* tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
* tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
* tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterHashJoinExec.java
* tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java
* tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java
* tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java
* tajo-core/src/main/java/org/apache/tajo/master/cluster/WorkerConnectionInfo.java
* tajo-plan/src/main/proto/Plan.proto
* tajo-core/src/main/java/org/apache/tajo/worker/Task.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEvent.java
* tajo-core/src/main/java/org/apache/tajo/worker/TaskExecutor.java
* tajo-core/src/main/resources/webapps/worker/querytasks.jsp
* tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java
* tajo-core/src/main/java/org/apache/tajo/engine/planner/global/builder/DistinctGroupbyBuilder.java
* tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestFifoScheduler.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueState.java
* tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java
* tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleFifoScheduler.java


> Resource allocation should be fine grained.
> -------------------------------------------
>
>                 Key: TAJO-1397
>                 URL: https://issues.apache.org/jira/browse/TAJO-1397
>             Project: Tajo
>          Issue Type: Improvement
>          Components: QueryMaster, resource manager, Worker
>            Reporter: Hyunsik Choi
>            Assignee: Jinho Kim
>             Fix For: 0.11.0
>
>         Attachments: ResoruceSequence.jpg, TAJO-1397.patch, TAJO-1397_2.patch, TAJO-1397_3.patch,
TAJO-1397_4.patch, old_resource_circuit.png, resource_circuit.png
>
>
> See the comment:
> https://issues.apache.org/jira/browse/TAJO-540?focusedCommentId=14359478&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14359478
> From the discussion in TAJO-540
> {quote}
> In general, query (or job) scheduler aims at the maximum resource utilization. For multi-tenancy,
we also need to consider the fairness for multiple users (or queries). BTW, the maximum resource
utilization and fairness are usually conflict to each other in many cases. To mitigate this
problem, many scheduler seems to use preemption approach.
> In this point, our resource and scheduler system has the following problems:
>  * A query exclusively uses allocated resources at the first time until the query is
completed or failed.
>  * There is no mechanism to deallocate resources during query processing.
>  * Preempt is also not allowed.
> To achieve the multi tenancy, we should change our resource circulation. Especially,
resource allocation must be fine grained instead of per query.
> So, I'll create a jira issue to change the resource circulation. We have to do this issue
firstly in my opinion. If we achieve this, implementing multi-tenant scheduler would be much
easier than now. It would be a good starting point of this issue.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message