Mailing-List: contact user-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@beam.apache.org
MIME-Version: 1.0
In-Reply-To: <AM3PR05MB3215990D23AE1A49CA1E45DD47F0@AM3PR05MB321.eurprd05.prod.outlook.com>
References: <CACoXVvrkB8BE1WOciLybHAxCSLb1326-u9WyONMaC4Y0dZc=zA@mail.gmail.com>
 <AM3PR05MB3215990D23AE1A49CA1E45DD47F0@AM3PR05MB321.eurprd05.prod.outlook.com>
From: Derek Hao Hu <phoenixinter@gmail.com>
Date: Mon, 2 Oct 2017 09:01:10 -0700
Message-ID: <CACoXVvppnfsxrA0RD6RqooiHZQyKg=5HmOf4dDTeiGoc2Q4EmA@mail.gmail.com>
Subject: Re: How to get the PublishTime of PubsubMessages?
To: user@beam.apache.org
Content-Type: multipart/alternative; boundary="001a113f7a80ee2e99055a927c55"
archived-at: Mon, 02 Oct 2017 16:01:38 -0000

--001a113f7a80ee2e99055a927c55
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks Reinier! Is it easy to get the [MessageId](
http://googleapis.github.io/googleapis/java/proto-google-cloud-pubsub-v1/0.=
1.19/apidocs/com/google/pubsub/v1/PubsubMessage.html#getMessageId--)
of the PubsubMessages?

PS: If you are familiar with this part of the Beam implementation, can you
also point out where it gets the timestamp attribute? I'm trying to
understand the implementation as well. :)

Cheers,

Derek

On Sat, Sep 30, 2017 at 5:58 AM, Reinier Kip <rkip@bol.com> wrote:

> Hi Derek,
>
>
> The publish time is the element's timestamp, retrievable via
> DoFn.ProcessContext#timestamp, unless you've configured a timestamp
> attribute with PubsubIO.Read#withTimestampAttribute. As far as I know,
> payload, attributes and publish time is all you get. What other informati=
on
> from Google's PubsubMessage are you looking for?
>
>
> Reinier
> ------------------------------
> *From:* Derek Hao Hu <phoenixinter@gmail.com>
> *Sent:* 30 September 2017 03:18:13
> *To:* user@beam.apache.org
> *Subject:* How to get the PublishTime of PubsubMessages?
>
> Hi,
>
> =E2=80=8BI'm trying to use [PubsubIO](=E2=80=8Bhttps://beam.apache.org/do=
cumentation/sdks/
> javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html) to read
> [PubsubMessage](https://beam.apache.org/documentation/sdks/
> javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.html).
>
> I've noticed that the PubsubMessage class in Beam only exposes Attribute
> and Payload but the [PubSubMessage in Google Cloud](http://googleapis.
> github.io/googleapis/java/proto-google-cloud-pubsub-v1/
> 0.1.19/apidocs/com/google/pubsub/v1/PubsubMessage.html?is-external=3Dtrue=
)
> exposes a lot more information like `getPublishTime()`.
>
> Is there a way for us to get the same amount information in Beam easily?
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>


--=20
Derek Hao Hu

Software Engineer | Snapchat
Snap Inc.

--001a113f7a80ee2e99055a927c55
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:georgia,=
serif">Thanks Reinier! Is it easy to get the [MessageId](<a href=3D"http://=
googleapis.github.io/googleapis/java/proto-google-cloud-pubsub-v1/0.1.19/ap=
idocs/com/google/pubsub/v1/PubsubMessage.html#getMessageId--">http://google=
apis.github.io/googleapis/java/proto-google-cloud-pubsub-v1/0.1.19/apidocs/=
com/google/pubsub/v1/PubsubMessage.html#getMessageId--</a>) of the PubsubMe=
ssages?</div><div class=3D"gmail_default" style=3D"font-family:georgia,seri=
f"><br></div><div class=3D"gmail_default" style=3D"font-family:georgia,seri=
f">PS: If you are familiar with this part of the Beam implementation, can y=
ou also point out where it gets the timestamp attribute? I&#39;m trying to =
understand the implementation as well. :)</div><div class=3D"gmail_default"=
 style=3D"font-family:georgia,serif"><br></div><div class=3D"gmail_default"=
 style=3D"font-family:georgia,serif">Cheers,</div><div class=3D"gmail_defau=
lt" style=3D"font-family:georgia,serif"><br></div><div class=3D"gmail_defau=
lt" style=3D"font-family:georgia,serif">Derek</div></div><div class=3D"gmai=
l_extra"><br><div class=3D"gmail_quote">On Sat, Sep 30, 2017 at 5:58 AM, Re=
inier Kip <span dir=3D"ltr">&lt;<a href=3D"mailto:rkip@bol.com" target=3D"_=
blank">rkip@bol.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quo=
te" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"=
>


<div>

<div id=3D"m_-5530863974341173987divtagdefaultwrapper" style=3D"font-size:1=
2pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir=3D"ltr">
<p>Hi Derek,</p>
<p><br>
</p>
<p>The publish time is the element&#39;s timestamp, retrievable via=C2=A0<s=
pan>DoFn.ProcessContext#<wbr>timestamp</span>, unless you&#39;ve configured=
 a timestamp attribute with=C2=A0<span>PubsubIO.Read#<wbr>withTimestampAttr=
ibute. As far as I know, payload, attributes and publish time
 is all you get.=C2=A0What other information from Google&#39;s PubsubMessag=
e=C2=A0are you looking for?</span></p>
<p><span><br>
</span></p>
<p><span>Reinier</span></p>
</div>
<hr style=3D"display:inline-block;width:98%">
<div id=3D"m_-5530863974341173987divRplyFwdMsg" dir=3D"ltr"><font face=3D"C=
alibri, sans-serif" style=3D"font-size:11pt" color=3D"#000000"><b>From:</b>=
 Derek Hao Hu &lt;<a href=3D"mailto:phoenixinter@gmail.com" target=3D"_blan=
k">phoenixinter@gmail.com</a>&gt;<br>
<b>Sent:</b> 30 September 2017 03:18:13<br>
<b>To:</b> <a href=3D"mailto:user@beam.apache.org" target=3D"_blank">user@b=
eam.apache.org</a><br>
<b>Subject:</b> How to get the PublishTime of PubsubMessages?</font>
<div>=C2=A0</div>
</div><div><div class=3D"h5">
<div>
<div dir=3D"ltr">
<div class=3D"gmail_default" style=3D"font-family:georgia,serif">Hi,</div>
<div><br>
</div>
<div>
<div class=3D"gmail_default" style=3D"font-family:georgia,serif">=E2=80=8BI=
&#39;m trying to use [PubsubIO](=E2=80=8B<a href=3D"https://beam.apache.org=
/documentation/sdks/javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubI=
O.html" target=3D"_blank">https://beam.<wbr>apache.org/documentation/sdks/<=
wbr>javadoc/2.1.0/org/apache/beam/<wbr>sdk/io/gcp/pubsub/PubsubIO.<wbr>html=
</a>)
 to read [PubsubMessage](<a href=3D"https://beam.apache.org/documentation/s=
dks/javadoc/2.1.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.html" tar=
get=3D"_blank">https://beam.<wbr>apache.org/documentation/sdks/<wbr>javadoc=
/2.1.0/org/apache/beam/<wbr>sdk/io/gcp/pubsub/<wbr>PubsubMessage.html</a>).=
</div>
<div class=3D"gmail_default" style=3D"font-family:georgia,serif"><br>
</div>
<div class=3D"gmail_default" style=3D"font-family:georgia,serif">I&#39;ve n=
oticed that the PubsubMessage class in Beam only exposes Attribute and Payl=
oad but the [PubSubMessage in Google Cloud](<a href=3D"http://googleapis.gi=
thub.io/googleapis/java/proto-google-cloud-pubsub-v1/0.1.19/apidocs/com/goo=
gle/pubsub/v1/PubsubMessage.html?is-external=3Dtrue" target=3D"_blank">http=
://googleapis.<wbr>github.io/googleapis/java/<wbr>proto-google-cloud-pubsub=
-v1/<wbr>0.1.19/apidocs/com/google/<wbr>pubsub/v1/PubsubMessage.html?<wbr>i=
s-external=3Dtrue</a>)
 exposes a lot more information like `getPublishTime()`.</div>
<div class=3D"gmail_default" style=3D"font-family:georgia,serif"><br>
</div>
<div class=3D"gmail_default" style=3D"font-family:georgia,serif">Is there a=
 way for us to get the same amount information in Beam easily?</div>
</div>
-- <br>
<div class=3D"m_-5530863974341173987gmail_signature">
<div dir=3D"ltr">
<div><font face=3D"georgia, serif">Derek Hao Hu<br>
<br>
Software Engineer | Snapchat<br>
Snap Inc.</font></div>
</div>
</div>
</div>
</div>
</div></div></div>

</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><d=
iv><font face=3D"georgia, serif">Derek Hao Hu<br><br>Software Engineer | Sn=
apchat<br>Snap Inc.</font></div></div></div>
</div>

--001a113f7a80ee2e99055a927c55--