beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-4707) Change fields and table names in nexmark perfkit tables
Date Thu, 05 Jul 2018 14:15:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-4707?focusedWorklogId=119376&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119376
]

ASF GitHub Bot logged work on BEAM-4707:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 05/Jul/18 14:14
            Start Date: 05/Jul/18 14:14
    Worklog Time Spent: 10m 
      Work Description: echauchot commented on a change in pull request #5889: [BEAM-4707]
insert timestamp to bigQuery as seconds as expected by BigQuery
URL: https://github.com/apache/beam/pull/5889#discussion_r200367555
 
 

 ##########
 File path: sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
 ##########
 @@ -164,7 +164,7 @@ public NexmarkPerf decode(InputStream inStream)
           NexmarkPerf nexmarkPerf = input.getValue();
           TableRow row =
               new TableRow()
-                  .set("timestamp", start.getMillis())
+                  .set("timestamp", start.getEpochSecond())
 
 Review comment:
   BQ docs say that timestamp need to be epoch in seconds. I would also have preferred testing
on a real instance in unit test rather that the mock but there is no embedded BQ available
for tests, and obviously using a ProcessBuilder to fork a BQ process is not possible (BQ is
not OSS) either. I agree, there should be more type checking (see https://issues.apache.org/jira/browse/BEAM-4734).
Yes I tested it, write then read test passes with timestamp serialized json value of timestamp=1454284800

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 119376)
    Time Spent: 3.5h  (was: 3h 20m)

> Change fields and table names in nexmark perfkit tables
> -------------------------------------------------------
>
>                 Key: BEAM-4707
>                 URL: https://issues.apache.org/jira/browse/BEAM-4707
>             Project: Beam
>          Issue Type: Improvement
>          Components: testing
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: Major
>             Fix For: 2.6.0
>
>          Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Nexmark BQ tables for perfkit lack timestamp field. Also the table name contains a boolean
than shows the mode of execution. It would be better to have batch or streaming label in place
of the boolean.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message