aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Sirois <jsir...@apache.org>
Subject Re: Review Request 45193: Treat empty and null collections equivalently in task queries.
Date Wed, 23 Mar 2016 16:49:38 GMT


> On March 23, 2016, 8:35 a.m., John Sirois wrote:
> > I think this change stands on its own aside from the current state of the generated
Go thrift bindings, but there has been a good deal of discussion about those bindings offline.
 Some homework below.
> > 
> > For the case of the `TaskQuery` thrift struct, thrift 0.9.3 generates the following
Go struct:
> > ```go
> > type TaskQuery struct {
> > 	// unused field # 1
> > 	JobName string `thrift:"jobName,2" json:"jobName"`
> > 	// unused field # 3
> > 	TaskIds  map[string]bool         `thrift:"taskIds,4" json:"taskIds"`
> > 	Statuses map[ScheduleStatus]bool `thrift:"statuses,5" json:"statuses"`
> > 	// unused field # 6
> > 	InstanceIds map[int32]bool `thrift:"instanceIds,7" json:"instanceIds"`
> > 	// unused field # 8
> > 	Environment string           `thrift:"environment,9" json:"environment"`
> > 	SlaveHosts  map[string]bool  `thrift:"slaveHosts,10" json:"slaveHosts"`
> > 	JobKeys     map[*JobKey]bool `thrift:"jobKeys,11" json:"jobKeys"`
> > 	Offset      int32            `thrift:"offset,12" json:"offset"`
> > 	Limit       int32            `thrift:"limit,13" json:"limit"`
> > 	Role        string           `thrift:"role,14" json:"role"`
> > }
> > ```
> > 
> > This is reasonable since `api.TaskQuery{}.TaskIds == nil` is true; ie the collections
(maps represent sets here) zero to nil.
> > The issue comes in the serialization for these fields, `taskIds` is shown below
as an example:
> > ```go
> > func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) {
> > 	if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err != nil {
> > 		return thrift.PrependError(fmt.Sprintf("%T write field begin error 4:taskIds:
", p), err)
> > 	}
> > 	if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); err != nil {
> > 		return thrift.PrependError("error writing set begin: ", err)
> > 	}
> > 	for v, _ := range p.TaskIds {
> > 		if err := oprot.WriteString(string(v)); err != nil {
> > 			return thrift.PrependError(fmt.Sprintf("%T. (0) field write error: ", p), err)
> > 		}
> > 	}
> > 	if err := oprot.WriteSetEnd(); err != nil {
> > 		return thrift.PrependError("error writing set end: ", err)
> > 	}
> > 	if err := oprot.WriteFieldEnd(); err != nil {
> > 		return thrift.PrependError(fmt.Sprintf("%T write field end error 4:taskIds: ",
p), err)
> > 	}
> > 	return err
> > }
> > ```
> > 
> > So, since its safe to do so in Go (`len(p.TaskIds) == 0` and `for v, _ := range
p.TaskIds {` loops 0 times for `p.TaskIds == nil`), the code always emits the `taskIds` field,
whether nil or not, which presents on the other end of the wire as an empty set (as opposed
to a null or un-set set).  This does seem like a clear bug in the thrift compiler.  https://issues.apache.org/jira/browse/THRIFT-3700
is similar, but on the deserialization side of things so I've filed https://issues.apache.org/jira/browse/THRIFT-3752.

Hrm, so this may not be a thrift compiler bug per-se.  Marking all TaskQuery thrift fields
as optional yields:
```go
type TaskQuery struct {
	// unused field # 1
	JobName *string `thrift:"jobName,2" json:"jobName,omitempty"`
	// unused field # 3
	TaskIds  map[string]bool         `thrift:"taskIds,4" json:"taskIds,omitempty"`
	Statuses map[ScheduleStatus]bool `thrift:"statuses,5" json:"statuses,omitempty"`
	// unused field # 6
	InstanceIds map[int32]bool `thrift:"instanceIds,7" json:"instanceIds,omitempty"`
	// unused field # 8
	Environment *string          `thrift:"environment,9" json:"environment,omitempty"`
	SlaveHosts  map[string]bool  `thrift:"slaveHosts,10" json:"slaveHosts,omitempty"`
	JobKeys     map[*JobKey]bool `thrift:"jobKeys,11" json:"jobKeys,omitempty"`
	Offset      *int32           `thrift:"offset,12" json:"offset,omitempty"`
	Limit       *int32           `thrift:"limit,13" json:"limit,omitempty"`
	Role        *string          `thrift:"role,14" json:"role,omitempty"`
}
```

So map (set) fields are unchanged (still `nil`able), but primitives - not `nil`able before,
are now represented
as `nil`able pointers.  This also has the effect of emitting `IsSet*` methods and respecting
these methods as a
gate for field serialization:
```go
func (p *TaskQuery) IsSetTaskIds() bool {
	return p.TaskIds != nil
}

func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) {
	if p.IsSetTaskIds() {
		if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err != nil {
			return thrift.PrependError(fmt.Sprintf("%T write field begin error 4:taskIds: ", p), err)
		}
		if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); err != nil {
			return thrift.PrependError("error writing set begin: ", err)
		}
		for v, _ := range p.TaskIds {
			if err := oprot.WriteString(string(v)); err != nil {
				return thrift.PrependError(fmt.Sprintf("%T. (0) field write error: ", p), err)
			}
		}
		if err := oprot.WriteSetEnd(); err != nil {
			return thrift.PrependError("error writing set end: ", err)
		}
		if err := oprot.WriteFieldEnd(); err != nil {
			return thrift.PrependError(fmt.Sprintf("%T write field end error 4:taskIds: ", p), err)
		}
	}
	return err
}
```

In other words, the thirft compiler for Go has a different notion of unset requiredness than
the java compiler - and the perils of the global lack of specificity as to how to handle optional
vs required vs <none> is well documented :/


- John


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125024
-----------------------------------------------------------


On March 23, 2016, 10:35 a.m., John Sirois wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> -----------------------------------------------------------
> 
> (Updated March 23, 2016, 10:35 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java                  |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java                    |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java             |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java      |  6 ++++--
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml     |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java | 20 +++++++++++++++++---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -----
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242

>   src/main/java/org/apache/aurora/scheduler/base/Query.java ee01eaa4d0230d6bf0909b6460f27a74f03240db

>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java ac0bb374842741d7ccb7a83c574a90ac156af0f9

>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 231a55615abfbb483667f5f8ef71d2709fc16a88

>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 684614ffc42dd6778c7675a6c2f81cb72c106c0e

>   src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815

> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> -------
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message