kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xiongqi wu <xiongq...@gmail.com>
Subject Re: [DISCUSS] KIP-354 Time-based log compaction policy
Date Fri, 17 Aug 2018 04:17:27 GMT
on 2)
The offsetmap is built starting from dirty segment.
The compaction starts from the beginning of the log partition.  That's how
it ensure the deletion of tomb keys.
I will double check tomorrow.

Xiongqi (Wesley) Wu


On Thu, Aug 16, 2018 at 6:46 PM Brett Rann <brann@zendesk.com.invalid>
wrote:

> To just clarify a bit on 1.  whether there's an external storage/DB isn't
> relevant here.
> Compacted topics allow a tombstone record to be sent (a null value for a
> key) which
> currently will result in old values for that key being deleted if some
> conditions are met.
> There are existing controls to make sure the old values will stay around
> for a minimum
> time at least, but no dedicated control to ensure the tombstone will delete
> within a
> maximum time.
>
> One popular reason that maximum time for deletion is desirable right now is
> GDPR with
> PII. But we're not proposing any GDPR awareness in kafka, just being able
> to guarantee
> a max time where a tombstoned key will be removed from the compacted topic.
>
> on 2)
> huh, i thought it kept track of the first dirty segment and didn't
> recompact older "clean" ones.
> But I didn't look at code or test for that.
>
> On Fri, Aug 17, 2018 at 10:57 AM xiongqi wu <xiongqiwu@gmail.com> wrote:
>
> > 1, Owner of data (in this sense, kafka is the not the owner of data)
> > should keep track of lifecycle of the data in some external storage/DB.
> > The owner determines when to delete the data and send the delete request
> to
> > kafka. Kafka doesn't know about the content of data but to provide a mean
> > for deletion.
> >
> > 2 , each time compaction runs, it will start from first segments (no
> > matter if it is compacted or not). The time estimation here is only used
> > to determine whether we should run compaction on this log partition. So
> we
> > only need to estimate uncompacted segments.
> >
> > On Thu, Aug 16, 2018 at 5:35 PM, Dong Lin <lindong28@gmail.com> wrote:
> >
> > > Hey Xiongqi,
> > >
> > > Thanks for the update. I have two questions for the latest KIP.
> > >
> > > 1) The motivation section says that one use case is to delete PII
> > (Personal
> > > Identifiable information) data within 7 days while keeping non-PII
> > > indefinitely in compacted format. I suppose the use-case depends on the
> > > application to determine when to delete those PII data. Could you
> explain
> > > how can application reliably determine the set of keys that should be
> > > deleted? Is application required to always messages from the topic
> after
> > > every restart and determine the keys to be deleted by looking at
> message
> > > timestamp, or is application supposed to persist the key-> timstamp
> > > information in a separate persistent storage system?
> > >
> > > 2) It is mentioned in the KIP that "we only need to estimate earliest
> > > message timestamp for un-compacted log segments because the deletion
> > > requests that belong to compacted segments have already been
> processed".
> > > Not sure if it is correct. If a segment is compacted before user sends
> > > message to delete a key in this segment, it seems that we still need to
> > > ensure that the segment will be compacted again within the given time
> > after
> > > the deletion is requested, right?
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Thu, Aug 16, 2018 at 10:27 AM, xiongqi wu <xiongqiwu@gmail.com>
> > wrote:
> > >
> > > > Hi Xiaohe,
> > > >
> > > > Quick note:
> > > > 1) Use minimum of segment.ms and max.compaction.lag.ms
> > > > <http://max.compaction.ms
> > <http://max.compaction.ms>>
> > > >
> > > > 2) I am not sure if I get your second question. first, we have jitter
> > > when
> > > > we roll the active segment. second, on each compaction, we compact
> upto
> > > > the offsetmap could allow. Those will not lead to perfect compaction
> > > storm
> > > > overtime. In addition, I expect we are setting max.compaction.lag.ms
> > on
> > > > the order of days.
> > > >
> > > > 3) I don't have access to the confluent community slack for now. I am
> > > > reachable via the google handle out.
> > > > To avoid the double effort, here is my plan:
> > > > a) Collect more feedback and feature requriement on the KIP.
> > > > b) Wait unitl this KIP is approved.
> > > > c) I will address any additional requirements in the implementation.
> > (My
> > > > current implementation only complies to whatever described in the KIP
> > > now)
> > > > d) I can share the code with the you and community see you want to
> add
> > > > anything.
> > > > e) submission through committee
> > > >
> > > >
> > > > On Wed, Aug 15, 2018 at 11:42 PM, XIAOHE DONG <dannyrivclo@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi Xiongqi
> > > > >
> > > > > Thanks for thinking about implementing this as well. :)
> > > > >
> > > > > I was thinking about using `segment.ms` to trigger the segment
> roll.
> > > > > Also, its value can be the largest time bias for the record
> deletion.
> > > For
> > > > > example, if the `segment.ms` is 1 day and `max.compaction.ms` is
> 30
> > > > days,
> > > > > the compaction may happen around 31 days.
> > > > >
> > > > > For my curiosity, is there a way we can do some performance test
> for
> > > this
> > > > > and any tools you can recommend. As you know, previously, it is
> > cleaned
> > > > up
> > > > > by respecting dirty ratio, but now it may happen anytime if max lag
> > has
> > > > > passed for each message. I wonder what would happen if clients send
> > > huge
> > > > > amount of tombstone records at the same time.
> > > > >
> > > > > I am looking forward to have a quick chat with you to avoid double
> > > effort
> > > > > on this. I am in confluent community slack during the work time.
My
> > > name
> > > > is
> > > > > Xiaohe Dong. :)
> > > > >
> > > > > Rgds
> > > > > Xiaohe Dong
> > > > >
> > > > >
> > > > >
> > > > > On 2018/08/16 01:22:22, xiongqi wu <xiongqiwu@gmail.com> wrote:
> > > > > > Brett,
> > > > > >
> > > > > > Thank you for your comments.
> > > > > > I was thinking since we already has immediate compaction setting
> by
> > > > > setting
> > > > > > min dirty ratio to 0, so I decide to use "0" as disabled state.
> > > > > > I am ok to go with -1(disable), 0 (immediate) options.
> > > > > >
> > > > > > For the implementation, there are a few differences between
mine
> > and
> > > > > > "Xiaohe Dong"'s :
> > > > > > 1) I used the estimated creation time of a log segment instead
of
> > > > largest
> > > > > > timestamp of a log to determine the compaction eligibility,
> > because a
> > > > log
> > > > > > segment might stay as an active segment up to "max compaction
> lag".
> > > > (see
> > > > > > the KIP for detail).
> > > > > > 2) I measure how much bytes that we must clean to follow the
"max
> > > > > > compaction lag" rule, and use that to determine the order of
> > > > compaction.
> > > > > > 3) force active segment to roll to follow the "max compaction
> lag"
> > > > > >
> > > > > > I can share my code so we can coordinate.
> > > > > >
> > > > > > I haven't think about a new API to force a compaction. what
is
> the
> > > use
> > > > > case
> > > > > > for this one?
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 15, 2018 at 5:33 PM, Brett Rann
> > > <brann@zendesk.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > We've been looking into this too.
> > > > > > >
> > > > > > > Mailing list:
> > > > > > > https://lists.apache.org/thread.html/
> > <https://lists.apache.org/thread.html/>
> > > ed7f6a6589f94e8c2a705553f364ef
> > > > > > > 599cb6915e4c3ba9b561e610e4@%3Cdev.kafka.apache.org%3E
> > > > > > > jira wish: https://issues.apache.org/jira/browse/KAFKA-7137
> > <https://issues.apache.org/jira/browse/KAFKA-7137>
> > > > > > > confluent slack discussion:
> > > > > > > https://confluentcommunity.slack.com/archives/C49R61XMM/
> > <https://confluentcommunity.slack.com/archives/C49R61XMM/>
> > > > > p1530760121000039
> > > > > > >
> > > > > > > A person on my team has started on code so you might want
to
> > > > > coordinate:
> > > > > > > https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log-
> > <https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log->
> > > > > > > cleaner-compaction-max-lifetime-2.0
> > > > > > >
> > > > > > > He's been working with Jason Gustafson and James Chen around
> the
> > > > > changes.
> > > > > > > You can ping him on confluent slack as Xiaohe Dong.
> > > > > > >
> > > > > > > It's great to know others are thinking on it as well.
> > > > > > >
> > > > > > > You've added the requirement to force a segment roll which
we
> > > hadn't
> > > > > gotten
> > > > > > > to yet, which is great. I was content with it not including
the
> > > > active
> > > > > > > segment.
> > > > > > >
> > > > > > > > Adding topic level configuration "max.compaction.lag.ms",
> and
> > > > > > > corresponding broker configuration "
> > log.cleaner.max.compaction.la
> > > > g.ms
> > > > > ",
> > > > > > > which is set to 0 (disabled) by default.
> > > > > > >
> > > > > > > Glancing at some other settings convention seems to me
to be -1
> > for
> > > > > > > disabled (or infinite, which is more meaningful here).
0 to me
> > > > implies
> > > > > > > instant, a little quicker than 1.
> > > > > > >
> > > > > > > We've been trying to think about a way to trigger compaction
as
> > > well
> > > > > > > through an API call, which would need to be flagged somewhere
> (ZK
> > > > > admin/
> > > > > > > space?) but we're struggling to think how that would be
> > coordinated
> > > > > across
> > > > > > > brokers and partitions. Have you given any thought to that?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 16, 2018 at 8:44 AM xiongqi wu <
> xiongqiwu@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Eno, Dong,
> > > > > > > >
> > > > > > > > I have updated the KIP. We decide not to address the
issue
> that
> > > we
> > > > > might
> > > > > > > > have for both compaction and time retention enabled
topics
> (see
> > > the
> > > > > > > > rejected alternative item 2). This KIP will only ensure
log
> can
> > > be
> > > > > > > > compacted after a specified time-interval.
> > > > > > > >
> > > > > > > > As suggested by Dong, we will also enforce "
> > > max.compaction.lag.ms"
> > > > > is
> > > > > > > not
> > > > > > > > less than "min.compaction.lag.ms".
> > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > > Time-based
> > > > > > > log
> > > > > > > > compaction policy
> > > > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > > Time-based
> > > > > > > log compaction policy>
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu <
> > xiongqiwu@gmail.com
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Per discussion with Dong, he made a very good
point that if
> > > > > compaction
> > > > > > > > > and time based retention are both enabled on
a topic, the
> > > > > compaction
> > > > > > > > might
> > > > > > > > > prevent records from being deleted on time. The
reason is
> > when
> > > > > > > compacting
> > > > > > > > > multiple segments into one single segment, the
newly
> created
> > > > > segment
> > > > > > > will
> > > > > > > > > have same lastmodified timestamp as latest original
> segment.
> > We
> > > > > lose
> > > > > > > the
> > > > > > > > > timestamp of all original segments except the
last one. As
> a
> > > > > result,
> > > > > > > > > records might not be deleted as it should be
through time
> > based
> > > > > > > > retention.
> > > > > > > > >
> > > > > > > > > With the current KIP proposal, if we want to
ensure timely
> > > > > deletion, we
> > > > > > > > > have the following configurations:
> > > > > > > > > 1) enable time based log compaction only : deletion
is done
> > > > though
> > > > > > > > > overriding the same key
> > > > > > > > > 2) enable time based log retention only: deletion
is done
> > > though
> > > > > > > > > time-based retention
> > > > > > > > > 3) enable both log compaction and time based
retention:
> > > Deletion
> > > > > is not
> > > > > > > > > guaranteed.
> > > > > > > > >
> > > > > > > > > Not sure if we have use case 3 and also want
deletion to
> > happen
> > > > on
> > > > > > > time.
> > > > > > > > > There are several options to address deletion
issue when
> > enable
> > > > > both
> > > > > > > > > compaction and retention:
> > > > > > > > > A) During log compaction, looking into record
timestamp to
> > > delete
> > > > > > > expired
> > > > > > > > > records. This can be done in compaction logic
itself or use
> > > > > > > > > AdminClient.deleteRecords() . But this assumes
we have
> record
> > > > > > > timestamp.
> > > > > > > > > B) retain the lastModifed time of original segments
during
> > log
> > > > > > > > compaction.
> > > > > > > > > This requires extra meta data to record the information
or
> > not
> > > > > grouping
> > > > > > > > > multiple segments into one during compaction.
> > > > > > > > >
> > > > > > > > > If we have use case 3 in general, I would prefer
solution A
> > and
> > > > > rely on
> > > > > > > > > record timestamp.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Two questions:
> > > > > > > > > Do we have use case 3? Is it nice to have or
must have?
> > > > > > > > > If we have use case 3 and want to go with solution
A,
> should
> > we
> > > > > > > introduce
> > > > > > > > > a new configuration to enforce deletion by timestamp?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu <
> > > xiongqiwu@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Dong,
> > > > > > > > >>
> > > > > > > > >> Thanks for the comment.
> > > > > > > > >>
> > > > > > > > >> There are two retention policy: log compaction
and time
> > based
> > > > > > > retention.
> > > > > > > > >>
> > > > > > > > >> Log compaction:
> > > > > > > > >>
> > > > > > > > >> we have use cases to keep infinite retention
of a topic
> > (only
> > > > > > > > >> compaction). GDPR cares about deletion of
PII (personal
> > > > > identifiable
> > > > > > > > >> information) data.
> > > > > > > > >> Since Kafka doesn't know what records contain
PII, it
> relies
> > > on
> > > > > upper
> > > > > > > > >> layer to delete those records.
> > > > > > > > >> For those infinite retention uses uses, kafka
needs to
> > > provide a
> > > > > way
> > > > > > > to
> > > > > > > > >> enforce compaction on time. This is what
we try to address
> > in
> > > > this
> > > > > > > KIP.
> > > > > > > > >>
> > > > > > > > >> Time based retention,
> > > > > > > > >>
> > > > > > > > >> There are also use cases that users of Kafka
might want to
> > > > expire
> > > > > all
> > > > > > > > >> their data.
> > > > > > > > >> In those cases, they can use time based retention
of their
> > > > topics.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Regarding your first question, if a user
wants to delete a
> > key
> > > > in
> > > > > the
> > > > > > > > >> log compaction topic, the user has to send
a deletion
> using
> > > the
> > > > > same
> > > > > > > > key.
> > > > > > > > >> Kafka only makes sure the deletion will happen
under a
> > certain
> > > > > time
> > > > > > > > >> periods (like 2 days/7 days).
> > > > > > > > >>
> > > > > > > > >> Regarding your second question. In most cases,
we might
> want
> > > to
> > > > > delete
> > > > > > > > >> all duplicated keys at the same time.
> > > > > > > > >> Compaction might be more efficient since
we need to scan
> the
> > > log
> > > > > and
> > > > > > > > find
> > > > > > > > >> all duplicates. However, the expected use
case is to set
> the
> > > > time
> > > > > > > based
> > > > > > > > >> compaction interval on the order of days,
and be larger
> than
> > > > 'min
> > > > > > > > >> compaction lag". We don't want log compaction
to happen
> > > > frequently
> > > > > > > since
> > > > > > > > >> it is expensive. The purpose is to help low
production
> rate
> > > > topic
> > > > > to
> > > > > > > get
> > > > > > > > >> compacted on time. For the topic with "normal"
incoming
> > > message
> > > > > > > message
> > > > > > > > >> rate, the "min dirty ratio" might have triggered
the
> > > compaction
> > > > > before
> > > > > > > > this
> > > > > > > > >> time based compaction policy takes effect.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Eno,
> > > > > > > > >>
> > > > > > > > >> For your question, like I mentioned we have
long time
> > > retention
> > > > > use
> > > > > > > case
> > > > > > > > >> for log compacted topic, but we want to provide
ability to
> > > > delete
> > > > > > > > certain
> > > > > > > > >> PII records on time.
> > > > > > > > >> Kafka itself doesn't know whether a record
contains
> > sensitive
> > > > > > > > information
> > > > > > > > >> and relies on the user for deletion.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Mon, Aug 13, 2018 at 6:58 PM, Dong Lin
<
> > > lindong28@gmail.com>
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hey Xiongqi,
> > > > > > > > >>>
> > > > > > > > >>> Thanks for the KIP. I have two questions
regarding the
> > > use-case
> > > > > for
> > > > > > > > >>> meeting
> > > > > > > > >>> GDPR requirement.
> > > > > > > > >>>
> > > > > > > > >>> 1) If I recall correctly, one of the
GDPR requirement is
> > that
> > > > we
> > > > > can
> > > > > > > > not
> > > > > > > > >>> keep messages longer than e.g. 30 days
in storage (e.g.
> > > Kafka).
> > > > > Say
> > > > > > > > there
> > > > > > > > >>> exists a partition p0 which contains
message1 with key1
> and
> > > > > message2
> > > > > > > > with
> > > > > > > > >>> key2. And then user keeps producing messages
with
> key=key2
> > to
> > > > > this
> > > > > > > > >>> partition. Since message1 with key1 is
never overridden,
> > > sooner
> > > > > or
> > > > > > > > later
> > > > > > > > >>> we
> > > > > > > > >>> will want to delete message1 and keep
the latest message
> > with
> > > > > > > key=key2.
> > > > > > > > >>> But
> > > > > > > > >>> currently it looks like log compact logic
in Kafka will
> > > always
> > > > > put
> > > > > > > > these
> > > > > > > > >>> messages in the same segment. Will this
be an issue?
> > > > > > > > >>>
> > > > > > > > >>> 2) The current KIP intends to provide
the capability to
> > > delete
> > > > a
> > > > > > > given
> > > > > > > > >>> message in log compacted topic. Does
such use-case also
> > > require
> > > > > Kafka
> > > > > > > > to
> > > > > > > > >>> keep the messages produced before the
given message? If
> > yes,
> > > > > then we
> > > > > > > > can
> > > > > > > > >>> probably just use AdminClient.deleteRecords()
or
> time-based
> > > log
> > > > > > > > retention
> > > > > > > > >>> to meet the use-case requirement. If
no, do you know what
> > is
> > > > the
> > > > > > > GDPR's
> > > > > > > > >>> requirement on time-to-deletion after
user explicitly
> > > requests
> > > > > the
> > > > > > > > >>> deletion
> > > > > > > > >>> (e.g. 1 hour, 1 day, 7 day)?
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>> Dong
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Aug 13, 2018 at 3:44 PM, xiongqi
wu <
> > > > xiongqiwu@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> > Hi Eno,
> > > > > > > > >>> >
> > > > > > > > >>> > The GDPR request we are getting
here at linkedin is if
> we
> > > > get a
> > > > > > > > >>> request to
> > > > > > > > >>> > delete a record through a null key
on a log compacted
> > > topic,
> > > > > > > > >>> > we want to delete the record via
compaction in a given
> > time
> > > > > period
> > > > > > > > >>> like 2
> > > > > > > > >>> > days (whatever is required by the
policy).
> > > > > > > > >>> >
> > > > > > > > >>> > There might be other issues (such
as orphan log
> segments
> > > > under
> > > > > > > > certain
> > > > > > > > >>> > conditions) that lead to GDPR problem
but they are more
> > > like
> > > > > > > > >>> something we
> > > > > > > > >>> > need to fix anyway regardless of
GDPR.
> > > > > > > > >>> >
> > > > > > > > >>> >
> > > > > > > > >>> > -- Xiongqi (Wesley) Wu
> > > > > > > > >>> >
> > > > > > > > >>> > On Mon, Aug 13, 2018 at 2:56 PM,
Eno Thereska <
> > > > > > > > eno.thereska@gmail.com>
> > > > > > > > >>> > wrote:
> > > > > > > > >>> >
> > > > > > > > >>> > > Hello,
> > > > > > > > >>> > >
> > > > > > > > >>> > > Thanks for the KIP. I'd like
to see a more precise
> > > > > definition of
> > > > > > > > what
> > > > > > > > >>> > part
> > > > > > > > >>> > > of GDPR you are targeting as
well as some sort of
> > > > > verification
> > > > > > > that
> > > > > > > > >>> this
> > > > > > > > >>> > > KIP actually addresses the
problem. Right now I find
> > > this a
> > > > > bit
> > > > > > > > >>> vague:
> > > > > > > > >>> > >
> > > > > > > > >>> > > "Ability to delete a log message
through compaction
> in
> > a
> > > > > timely
> > > > > > > > >>> manner
> > > > > > > > >>> > has
> > > > > > > > >>> > > become an important requirement
in some use cases
> > (e.g.,
> > > > > GDPR)"
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > > Is there any guarantee that
after this KIP the GDPR
> > > problem
> > > > > is
> > > > > > > > >>> solved or
> > > > > > > > >>> > do
> > > > > > > > >>> > > we need to do something else
as well, e.g., more
> KIPs?
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > > Thanks
> > > > > > > > >>> > >
> > > > > > > > >>> > > Eno
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > > On Thu, Aug 9, 2018 at 4:18
PM, xiongqi wu <
> > > > > xiongqiwu@gmail.com>
> > > > > > > > >>> wrote:
> > > > > > > > >>> > >
> > > > > > > > >>> > > > Hi Kafka,
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > This KIP tries to address
GDPR concern to fulfill
> > > > deletion
> > > > > > > > request
> > > > > > > > >>> on
> > > > > > > > >>> > > time
> > > > > > > > >>> > > > through time-based log
compaction on a compaction
> > > enabled
> > > > > > > topic:
> > > > > > > > >>> > > >
> > > > > > > > >>> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP->
> > > > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP->>
> > > > > > > > >>> > > > 354%3A+Time-based+log+compaction+policy
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > Any feedback will be appreciated.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > Xiongqi (Wesley) Wu
> > > > > > > > >>> > > >
> > > > > > > > >>> > >
> > > > > > > > >>> >
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Xiongqi (Wesley) Wu
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Xiongqi (Wesley) Wu
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Xiongqi (Wesley) Wu
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Brett Rann
> > > > > > >
> > > > > > > Senior DevOps Engineer
> > > > > > >
> > > > > > >
> > > > > > > Zendesk International Ltd
> > > > > > >
> > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> > > > > > >
> > > > > > > Mobile: +61 (0) 418 826 017
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Xiongqi (Wesley) Wu
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Xiongqi (Wesley) Wu
> > > >
> > >
> >
> >
> >
> > --
> > Xiongqi (Wesley) Wu
> >
>
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message