manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject RE: Scheduling Info
Date Thu, 29 Jan 2015 10:31:15 GMT
No.
Karl

Sent from my Windows Phone
------------------------------
From: Jitu
Sent: 1/29/2015 3:30 AM
To: user@manifoldcf.apache.org
Subject: Re: Scheduling Info

Hi Karl,

Thanks for your quick reply. Your answer helped me in understanding how
threads are managed for jobs.
Please correct me if i am wrong. For 1000 jobs assuming every repository
connector is different from other and max thread per repository connector
is 10 then will there be 10,000 threads spanned for repository connections?

Thanks,
Jitu

On Thu, Jan 29, 2015 at 1:42 PM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Jitu,
>
> In ManifoldCF, threads are not allocated to specific jobs, so that is
> definitely the wrong way to look at it.  There is a queue and work is
> pulled from the queue whenever a thread is ready.  The work is scheduled
> using the document priority mechanism, which deliberately distributes
> priority values so that all document bins get an equal shot.
>
> Karl
>
>
>
> On Thu, Jan 29, 2015 at 1:06 AM, Jitu <abjitu@gmail.com> wrote:
>
>> Hi Karl,
>>
>> Thanks for your quick reply. Please correct me if i am wrong. For 1000
>> jobs assuming every repository connector is different from other and max
>> thread per repository connector is 10 then will there be 10,000 threads
>> spanned for repository connections? how many threads will be spanned for
>> 1000 jobs?
>>
>> Thanks,
>> Jitu
>>
>> On Wed, Jan 28, 2015 at 6:31 PM, Karl Wright <daddywri@gmail.com> wrote:
>>
>>> Yes, if I understand the user's usage correctly, all 1000 jobs are all
>>> active at the same time.
>>>
>>> Note that you can scale MCF in several different ways if you need to
>>> (and I am not clear at all at the moment that your case would require
>>> that).  You can increase thread count somewhat (I wouldn't go beyond 100).
>>> You can add more agents processes.  But frankly 90 jobs sounds pretty small
>>> to me.
>>>
>>> Karl
>>>
>>>
>>> On Wed, Jan 28, 2015 at 6:38 AM, Jitu <abjitu@gmail.com> wrote:
>>>
>>>> Hi Karl,
>>>>           Assuming all these 1000 jobs are running concurrently.
>>>>
>>>> Thanks,
>>>> Jitu
>>>>
>>>>
>>>> On Wed, Jan 28, 2015 at 5:07 PM, Jitu <abjitu@gmail.com> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>> Thanks for the reply. these thousands of jobs run on single machine?
>>>>> wont there be any issue with number of threads spanned. will there be
any
>>>>> issue with throughput of the jobs.
>>>>>
>>>>> Thanks,
>>>>> Jitu
>>>>>
>>>>> On Wed, Jan 28, 2015 at 4:29 PM, Karl Wright <daddywri@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Jitu,
>>>>>>
>>>>>> We have people with hundreds of connections and thousands of jobs.
>>>>>> They work fine.  Why do you think that 90 jobs would be a problem?
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 28, 2015 at 5:09 AM, Jitu <abjitu@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Karl,
>>>>>>>
>>>>>>> We have a requirement where We have around 30 clients and each
>>>>>>> client has Google drive, wiki and box connectors. if i run all
the clients
>>>>>>> job in one machine say 30 * 3 = 90 jobs running concurrently
may not scale
>>>>>>> assuming all clients scheduled at same time.
>>>>>>>
>>>>>>> please advice on how to handle many jobs concurrently.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jitu
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message