hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] bwu2 commented on issue #1328: Hudi upsert hangs
Date Fri, 21 Feb 2020 00:58:42 GMT
bwu2 commented on issue #1328: Hudi upsert hangs
URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-589446887
 
 
   Thanks for your replies!
   
    @lamber-ken I will try again with that setting. Does increasing the memory available by
setting `option("hoodie.memory.merge.max.size", "2004857600000")` work better than increasing
executor memory or`hoodie.memory.merge.fraction`? Will it not result in an OOME? 
   
   @vinothchandar Yes, we are re-running against real datasets in the next couple of days.
I will report back. But do note that the original problem actually arose on a real dataset
not a degenerate or artificial workload: it was not a particularly wide table (with 6 or 7
fields), with about 4m rows bulk inserted and then another 4m upserted (of which most were
the same rows).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message