apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: FileSplitter/BlockReader
Date Wed, 24 Feb 2016 04:45:11 GMT
That's one option, though stats listener may provide better context for
decision making.

This is needed for better batch support.

--
sent from mobile
On Feb 23, 2016 8:36 PM, "Sandesh Hegde" <sandesh@datatorrent.com> wrote:

> Experimental, Can we use iteration to send the feedback and based on that
> adjust the bandwidth?
>
> On Tue, Feb 23, 2016 at 8:32 PM Thomas Weise <thomas@datatorrent.com>
> wrote:
>
>> Priyanka, that's right. In this case we want to process as fast as
>> possible without overwhelming the downstream operators.
>>
>> --
>> sent from mobile
>> On Feb 23, 2016 8:25 PM, "Priyanka Gugale" <priyanka@datatorrent.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have seen that code in Ingestion, and as I remember it had some issues
>>> and wasn't working.
>>> I don't have exactly what Thomas asked, but we can use Bandwidth limit
>>> feature, where splitter itself emits based on configured limit, but yes it
>>> isn't based on load on readers.
>>>
>>> -Priyanka
>>>
>>> On Wed, Feb 24, 2016 at 5:33 AM, Chandni Singh <chandni@datatorrent.com>
>>> wrote:
>>>
>>>> Hi Thomas,
>>>>
>>>> I had written a StatsListener which when set on FileSplitter and
>>>> BlockReader would automatically throttle block emission rate. The stats
>>>> listener will send a setProperty command to FileSplitter however if I
>>>> recall correctly, this changed the property only on the logical operator
>>>> and wasn't propagated to the physical operator.
>>>>
>>>> It was a while back so I am not sure if that was still the issue or
>>>> there was something else wrong my implementation.
>>>>
>>>> Thanks,
>>>> Chandni
>>>>
>>>> On Tue, Feb 23, 2016 at 3:48 PM, Thomas Weise <thomas@datatorrent.com>
>>>> wrote:
>>>>
>>>>> Is there any support for the splitter to automatically throttle block
>>>>> emission rate based on the readers throughput?
>>>>>
>>>>> --
>>>>> sent from mobile
>>>>>
>>>>
>>>>
>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message