aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aurora ReviewBot <wfar...@apache.org>
Subject Re: Review Request 46459: Schema changes for resource management refactoring
Date Thu, 21 Apr 2016 22:17:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46459/#review129971
-----------------------------------------------------------



Master (95dcca5) is red with this patch.
  ./build-support/jenkins/build.sh

                     src/test/python/apache/thermos/common/test_pathspec.py::test_exception_on_none_keys
PASSED
                     src/test/python/apache/thermos/common/test_planner.py::test_planner_empty
PASSED
                     src/test/python/apache/thermos/common/test_planner.py::test_planner_unordered
PASSED
                     src/test/python/apache/thermos/common/test_planner.py::test_planner_ordered
PASSED
                     src/test/python/apache/thermos/common/test_planner.py::test_planner_mixed
PASSED
                     src/test/python/apache/thermos/common/test_planner.py::test_planner_unsatisfiables
PASSED
                     
                     ==================== FAILURES ====================
                     ____ TestInspectCommand.test_inspect_job_raw _____
                     
                     self = <aurora.client.cli.test_inspect.TestInspectCommand testMethod=test_inspect_job_raw>
                     
                         def test_inspect_job_raw(self):
                           mock_stdout = []
                           def mock_print_out(msg, indent=0):
                             indent_str = " " * indent
                             mock_stdout.append("%s%s" % (indent_str, msg))
                           job_config = self.get_job_config()
                           with contextlib.nested(
                               patch('apache.aurora.client.cli.context.AuroraCommandContext.print_out',
                                   side_effect=mock_print_out),
                               patch('apache.aurora.client.cli.context.AuroraCommandContext.get_job_config',
                                   return_value=job_config)):
                             cmd = AuroraCommandLine()
                             assert cmd.execute(['job', 'inspect', '--raw', 'west/bozo/test/hello',
'config.aurora']) == 0
                             output = '\n'.join(mock_stdout)
                     >       assert output == str(job_config.job())
                     E       AssertionError: assert "JobConfigura...r='jenkins'))"
== "JobConfigurat...r='jenkins'))"
                     E         Detailed information truncated, use "-vv" to show
                     
                     src/test/python/apache/aurora/client/cli/test_inspect.py:92: AssertionError
                      generated xml file: /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml

                      1 failed, 655 passed, 6 skipped, 1 warnings in 197.57 seconds

                     
FAILURE


               Waiting for background workers to finish.
22:17:21 04:03   [complete]
               FAILURE


I will refresh this build result if you post a review containing "@ReviewBot retry"

- Aurora ReviewBot


On April 21, 2016, 9:57 p.m., Maxim Khutornenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46459/
> -----------------------------------------------------------
> 
> (Updated April 21, 2016, 9:57 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> This is the first patch towards the generic resource management approach described [here](https://docs.google.com/document/d/1J9SIswRMpVKQpnlvJAMAJtKfPP7ZARFknuyXl-2aZ-M/edit#).
> 
> Apologies for the large diff but this is the minimal logically correct set of changes
to ensure proper schema migration/backfilling. 
> 
> A few design/implementation notes:
> - All resources are stored in a single table with values converted to String. This way
we can handle additional resource types by just adding a `ResourceTypeConverter` for the new
type. Among other benefits the increased human readability of SQL data simplifying ad-hoc
analysis
> - The requested ports have been flatmapped together with other resource types to simplify
handling
> - Resource details are described in `ResourceType` that provides translation bridge between
generic representation and specific resource types
> - `IResource` immutable wrapper got couple of new methods:
>   - `public static Resource newBuilder(int id, Object value)`: for creating mutable resource
values in a generic way when reloading resources from DB;
>   - `public Object getRawValue()`: for accessing resource value in a generic way when
storing it in DB;
> 
> 
> Notes on data migration/backfilling:
> - The `resource_types` table needs to be populated before any migration happens as it
provides foreign keys for the `task_resource`
> - Next, `task_resource` data migration covering task configs already present in the DB
(e.g. those inserted by job updates or in case of `DBTaskStore` enabled)
> - `SnapshotStoreImpl` backfilling covers data not present in the dbsnapshot (e.g. `DBTaskStore`
disabled)
> - `LogStorage` backfilling covers transaction replays
> - `ConfigurationManager` backfilling covers external RPC handling between v-1/v+1 client/server
combinations
> 
> Also, made some test changes to ensure our tests pass with or without `DBTaskStore` enabled.
> 
> 
> Diffs
> -----
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 0cf4d6e428806c2b17bd8a824a0bd4fc83c80766

>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java d91c742c7783905214de563321771ce33dba74a3

>   src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 041cec38f1f96985bbf85468ece0ebbcef2f2486

>   src/main/java/org/apache/aurora/scheduler/http/api/GsonMessageBodyHandler.java 44295f80ba8b464d502e72c11c517fac28716c59

>   src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java b6fc949d31371304c2e71363dd61fbf40cfd0ac8

>   src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 360914ede6f1978a071a9f7a94d1f328bc252e60

>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 25160df1be9539c1ecd4613db5deb99a9606a512

>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java e778a39ef61e456ad9d2d2aa6f3e5d69e44bf02c

>   src/main/java/org/apache/aurora/scheduler/storage/db/migration/V003_CreateResourceTypesTable.java
PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/migration/V004_CreateTaskResourceTable.java
PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ResourceTypeHandler.java
PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java
ed561c65947d41861d80229ad459392a4bceadf1 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java PRE-CREATION

>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java cdd1060401a38e26cd726ec95e2bb617ccf4708c

>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f586186b61a6025ffccef8cafe480aa6dbf9681c

>   src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java b6922e1d5bdd188304359b5646be437ed09ec8d1

>   src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java PRE-CREATION

>   src/main/python/apache/aurora/config/thrift.py be0cd68674a71bd4baadf276f40a4bc0223ce4be

>   src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py 3465fe9ab4a4403270854d20dbc1d20dfc40a80d

>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 36df7ed5e23e360f23e6130dca7ae1ce234ee2ce

>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 92a0798d4d97920b786c4713f8779a7a32767652

>   src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java c31069047a9d59d6d635a86f673dd23c2b6868f1

>   src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
1ccfe01111e6c7c18046a8193245e537ae207a13 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java
PRE-CREATION 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java PRE-CREATION

>   src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java a33f6f7cbfc1e8262585fdf6c59e14c2afc661e0

>   src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java bf9479d60531324578354c4a15fcfdac040e7ffc

>   src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java ff9c1d08a9a99a69e94d634c895d20fa6c8c2f88

>   src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java PRE-CREATION

>   src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java fd6a40c8a6f0a16f0914b8025fd6207770bd740e

>   src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 50a1fdbb3dad501bf12fc245e32123a8630dbba5

>   src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
e6e79a0547cafeee80d656dff09d7adb52e4c420 
>   src/test/python/apache/aurora/config/test_thrift.py 88292d3c4423c0555088a0adaee3c0e62ed0567e

> 
> Diff: https://reviews.apache.org/r/46459/diff/
> 
> 
> Testing
> -------
> 
> Manual scheduler upgrades/rollbacks in Vagrant.
> 
> 
> Thanks,
> 
> Maxim Khutornenko
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message