manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Scheduling Info
Date Thu, 29 Jan 2015 08:12:40 GMT
Hi Jitu,

In ManifoldCF, threads are not allocated to specific jobs, so that is
definitely the wrong way to look at it.  There is a queue and work is
pulled from the queue whenever a thread is ready.  The work is scheduled
using the document priority mechanism, which deliberately distributes
priority values so that all document bins get an equal shot.

Karl



On Thu, Jan 29, 2015 at 1:06 AM, Jitu <abjitu@gmail.com> wrote:

> Hi Karl,
>
> Thanks for your quick reply. Please correct me if i am wrong. For 1000
> jobs assuming every repository connector is different from other and max
> thread per repository connector is 10 then will there be 10,000 threads
> spanned for repository connections? how many threads will be spanned for
> 1000 jobs?
>
> Thanks,
> Jitu
>
> On Wed, Jan 28, 2015 at 6:31 PM, Karl Wright <daddywri@gmail.com> wrote:
>
>> Yes, if I understand the user's usage correctly, all 1000 jobs are all
>> active at the same time.
>>
>> Note that you can scale MCF in several different ways if you need to (and
>> I am not clear at all at the moment that your case would require that).
>> You can increase thread count somewhat (I wouldn't go beyond 100).  You can
>> add more agents processes.  But frankly 90 jobs sounds pretty small to me.
>>
>> Karl
>>
>>
>> On Wed, Jan 28, 2015 at 6:38 AM, Jitu <abjitu@gmail.com> wrote:
>>
>>> Hi Karl,
>>>           Assuming all these 1000 jobs are running concurrently.
>>>
>>> Thanks,
>>> Jitu
>>>
>>>
>>> On Wed, Jan 28, 2015 at 5:07 PM, Jitu <abjitu@gmail.com> wrote:
>>>
>>>> Hi Karl,
>>>>
>>>> Thanks for the reply. these thousands of jobs run on single machine?
>>>> wont there be any issue with number of threads spanned. will there be any
>>>> issue with throughput of the jobs.
>>>>
>>>> Thanks,
>>>> Jitu
>>>>
>>>> On Wed, Jan 28, 2015 at 4:29 PM, Karl Wright <daddywri@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Jitu,
>>>>>
>>>>> We have people with hundreds of connections and thousands of jobs.
>>>>> They work fine.  Why do you think that 90 jobs would be a problem?
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jan 28, 2015 at 5:09 AM, Jitu <abjitu@gmail.com> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>> We have a requirement where We have around 30 clients and each client
>>>>>> has Google drive, wiki and box connectors. if i run all the clients
job in
>>>>>> one machine say 30 * 3 = 90 jobs running concurrently may not scale
>>>>>> assuming all clients scheduled at same time.
>>>>>>
>>>>>> please advice on how to handle many jobs concurrently.
>>>>>>
>>>>>> Thanks,
>>>>>> Jitu
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message