drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hsuan Yi Chu <hyi...@maprtech.com>
Subject Can we pass the #skipped records with RecordBatch?
Date Tue, 01 Dec 2015 19:35:00 GMT
I am trying to deal with the following scenario:

A bunch of minor fragments are doing things in parallel. Each of them could
skip some records. Since the downstream minor fragment needs to know the
sum of skipped-record-counts (in order to just display or see if the number
exceeds the threshold) in the upstreams, each upstream minor fragment needs
to pass this scalar with RecordBatch.

Since this seems impacting the protocol of RecordBatch, I am looking for
some advice here.

Thanks.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message