apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raja.Aravapalli <Raja.Aravapa...@target.com>
Subject hdfs file write operator is increasing the latency - resulting entire DAG to fail
Date Thu, 13 Jul 2017 15:13:28 GMT

We have an apex application that is reading from Kafka and wring to HDFS.

The  data flow for kafka topic is very huge… say 2500 messages per sec!!

The issue we are facing is:

The operator (which extends AbstractFileOutputOperator) is writing to hdfs is building latency
over time and failing eventually. Can someone pls share your thoughts on how I can handle
this ?

Thanks a lot.

View raw message