hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] vinothchandar commented on issue #1384: [SUPPORT] Hudi datastore missing updates for many records
Date Sun, 08 Mar 2020 16:55:33 GMT
vinothchandar commented on issue #1384: [SUPPORT] Hudi datastore missing updates for many records
URL: https://github.com/apache/incubator-hudi/issues/1384#issuecomment-596226463
 
 
   Hmmm the datasource does fail the commit if there are such errors..
   
   ```
    } else {
         log.error(s"$operation failed with $errorCount errors :")
         if (log.isTraceEnabled) {
           log.trace("Printing out the top 100 errors")
           writeStatuses.rdd.filter(ws => ws.hasErrors)
             .take(100)
             .foreach(ws => {
               log.trace("Global error :", ws.getGlobalError)
               if (ws.getErrors.size() > 0) {
                 ws.getErrors.foreach(kt =>
                   log.trace(s"Error for key: ${kt._1}", kt._2))
               }
             })
         }
         false
       }
   ``` 
   
   In any case, having some information on workload,  MOR vs COW and % of missing records
would help debug more.. Did you also have the issue on 0.4.7? or only after you upgraded to
0.5.1? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message