james-server-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <e...@apache.org>
Subject Re: inmemory mailbox wrong configuration
Date Thu, 07 Jun 2012 14:35:08 GMT
Hi Tim,

1. Yes,
2. Yes

3. StoreMessageManager in mailbox-store has the boiler-plate for you. If 
your implementation blocks (lets say in a YourImplMessageMapper method), 
the thread will block. There are 20 threads defined by default, so if 
the first thread blocks, the other one are still there to consume the 
mails from the AMQ queue.  Of course, if all the 20 threads block, 
spooling is stopped. Btw, there is no time-out on a thread blocking (But 
you should not implement your mailbox storage with a blocking mechanism 
- I suppose you are asking to anticipate a database, network connection 
not being released and being blocked?)

The <threads>20</threads> is for the AMQ consuming. For the remote 
delivery, it works completely differently, based on the 
RemoteDeliveryMailet where you can define there the number of concurrent 
threads running for remote delivery.

Thx for your interest :)

Eric

On 06/07/2012 03:16 PM, Timothy Prepscius wrote:
> Thanks for this!
>
> I am translating this in my head this way:
>
> 1. Mail Receiver receives a mail and queues it in ActiveMQ.
> 2. Mail Processor has multiple consumer threads watching the ActiveM queue.
> 3. If one mailbox  write operation blocks for 5 minutes, the other consumer threads of
the Mail Processor will continue.
>
> Is this roughly correct?
>
> Is this relevant or is the spool outgoing: (below xml of mailetcontainer-template.conf
)
>
> <!-- This block is responsible for processing messages on the spool. -->
>
> <mailetcontainer enableJmx="true">
>
>      <!-- MailAddress used for PostMaster -->
> 	<context>
> 	<postmaster>postmaster@localhost</postmaster>
> 	</context>
>
>      <!-- Number of spool threads -->
>      <spooler>
> 	<threads>20</threads>
> 	</spooler>
>
>
>
> Thanks again for your time!
>
> -tim
>
> On Jun 6, 2012, at 8:43 AM, Eric Charles wrote:
>
>> Hi Tim,
>>
>> 1. James doesn't block on connection (but don't forget the intermediate spooling
via ActiveMQ).
>>
>> 2. To answer this question, you need to realize James reads the mails via a Camel
context, getting them from ActiveMQ.
>>
>> Sorry if these response are too short, I was just wanting to take more time to provide
you with a complete picture, but I understand you need to go forward.
>>
>> What you can do it put a break point in the append method, and send mails to James
(running in your IDE). You will see the various threads and also that it is non-blocking....
exception made of the various lock with do via the JVMMailboxPathLocker, see for example this
snippet from StoreMessageManager
>>
>>             final Message<Id>  message = createMessage(internalDate, size,
bodyStartOctet, contentIn, flags, propertyBuilder);
>>             return locker.executeWithLock(mailboxSession, new StoreMailboxPath<Id>(getMailboxEntity()),
new MailboxPathLocker.LockAwareExecution<Long>() {
>>
>>                 @Override
>>                 public Long execute() throws MailboxException {
>>                     MessageMetaData data = appendMessageToStore(message, mailboxSession);
>>
>>                     SortedMap<Long, MessageMetaData>  uids = new TreeMap<Long,
MessageMetaData>();
>>                     uids.put(data.getUid(), data);
>>                     dispatcher.added(mailboxSession, uids, getMailboxEntity());
>>                     return data.getUid();
>>                 }
>>             }, true);
>>
>>
>> This being said, you will also need to  take care of the various locking/concurrent
access depending on the technology you use to store your mails (database, file, jcr,...).
>>
>> I don't have a Yes/No answer, it must be looked and analysed in context.
>>
>> Happy to further discuss,
>>
>> Eric
>>
>>
>>
>> On 06/06/2012 02:24 PM, Timothy Prepscius wrote:
>>> Ok, I'm sorry to spam about this question, but I would really like a response.
>>> I am making assumptions, which may be incorrect.
>>> Because of maven it is difficult for me to get a full source tree to read.
>>>
>>>
>>> 1.  Does james block on a single connection?
>>> (I assume no, because this would be just be crazy)
>>>
>>> 2.  Does james block on a transaction within a connection, meaning server X talks
to you on a single connection, transmits M1, M2, M3.
>>> Does james receive M1, queue a transaction in a different thread, and continue
on to M2, -OR- does it receive M1, block on transaction processor, and then continue onto
M2?
>>>
>>> (I hope doesn't block, if so I should implement a thread pool for the mailbox-write.)
>>>
>>>
>>> Thanks,
>>>
>>> -tim
>>>
>>> On Jun 4, 2012, at 9:28 AM, Timothy Prepscius wrote:
>>>
>>>>
>>>> I got rid of the NPE on java7 instead of switching to java6.  So no problems
there anymore.
>>>> I will investigate how to do the jira in a minute.
>>>>
>>>>
>>>> I have (perhaps) only one more question:
>>>>
>>>> Should I implement the Mailbox as a multi-threaded non blocking pool?
>>>> (Does James block the entire server on a Mailbox write?)
>>>>
>>>> Or should I implement the Mailbox as a single-threaded blocking write mechanism?
>>>> (Is there somewhere in James a thread pool handling requests from the smtp
socket?)
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> -tim
>>>>
>>>> On Jun 4, 2012, at 4:49 AM, Eric Charles wrote:
>>>>
>>>>> Hi Timothy,
>>>>>
>>>>> 1.- I have committed your patch [1], thx for this.
>>>>> 2.- Do you still have NPE with JDK6. If yes, can you post here the full
stacktrace?
>>>>> 3.- Please open a JIRA [2] for your proposal for getFullContent() and
explain there in a few words why it would be more optimal.
>>>>>
>>>>> Thx, Eric
>>>>>
>>>>> [1] http://svn.apache.org/viewvc?rev=1345862&view=rev
>>>>> [2] https://issues.apache.org/jira/browse/MAILBOX
>>>>>
>>>>>
>>>>> On 06/04/2012 05:00 AM, Timothy Prepscius wrote:
>>>>>>
>>>>>> On Jun 3, 2012, at 10:59 PM, Timothy Prepscius wrote:
>>>>>>
>>>>>>> Sorry, hit the wrong button I guess.
>>>>>>>
>>>>>>> On Jun 3, 2012, at 10:40 PM, Eric Charles wrote:
>>>>>>>
>>>>>>>> Hi Timothy,
>>>>>>>> Can you post to the mailing list so everyone can comment
on this?
>>>>>>>> Thx,  Eric
>>>>>>>>
>>>>>>>> On 06/03/2012 07:23 PM, Timothy Prepscius wrote:
>>>>>>>>> hey, could I also offer one code change, which will not
conflict with anything anywhere (I think) ?
>>>>>>>>>
>>>>>>>>> It optimizes the getFullContent for the SimpleMessage
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> tprepscius$ svn diff
>>>>>>>>> Index: src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMessage.java
>>>>>>>>> ===================================================================
>>>>>>>>> --- src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMessage.java
(revision 1345622)
>>>>>>>>> +++ src/main/java/org/apache/james/mailbox/store/mail/model/impl/SimpleMessage.java
(working copy)
>>>>>>>>> @@ -96,6 +96,10 @@
>>>>>>>>>         }
>>>>>>>>>     }
>>>>>>>>>
>>>>>>>>> +    @Override
>>>>>>>>> +    public InputStream getFullContent() throws IOException
{
>>>>>>>>> +    	return content.newStream(0, -1);
>>>>>>>>> +    }
>>>>>>>>>
>>>>>>>>>     public Date getInternalDate() {
>>>>>>>>>         return internalDate;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jun 3, 2012, at 1:14 PM, Timothy Prepscius wrote:
>>>>>>>>>
>>>>>>>>>> hmm, I just added httpclient-4.0.2.jar to the class
path and got the same error.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> a second ago I unzipped the httpclient jar
>>>>>>>>>> removed the entire META-INF/* dir tree
>>>>>>>>>> and zipped it up again manually and now it works.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> hmmm
>>>>>>>>>>
>>>>>>>>>> not sure what that means :-)
>>>>>>>>>>
>>>>>>>>>> -tim
>>>>>>>>>>
>>>>>>>>>> On Jun 3, 2012, at 1:03 PM, Timothy Prepscius wrote:
>>>>>>>>>>
>>>>>>>>>>> vanilla is fine, no prob
>>>>>>>>>>>
>>>>>>>>>>> it is indeed 1.7
>>>>>>>>>>>
>>>>>>>>>>> tprepscius@blue:~/Projects/James/apache-james/bin$
java -version
>>>>>>>>>>> java version "1.7.0_03"
>>>>>>>>>>> OpenJDK Runtime Environment (IcedTea7 2.1.1pre)
(7~u3-2.1.1~pre1-1ubuntu3)
>>>>>>>>>>> OpenJDK Client VM (build 22.0-b10, mixed mode,
sharing)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Jun 3, 2012, at 12:47 PM, Eric Charles wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thx Timothy, I will commit your patch in
trunk tomorrow.
>>>>>>>>>>>>
>>>>>>>>>>>> For NPE, do you run fine vanilla james? We
had such NPE when running with jdk7, although it was running fine with JDK6.
>>>>>>>>>>>> Thx, Eric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 06/03/2012 03:28 PM, Timothy Prepscius
wrote:
>>>>>>>>>>>>> yah:
>>>>>>>>>>>>>
>>>>>>>>>>>>>      <constructor-arg index="0" ref="memory-sessionMapperFactory"/>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hey- I have a question:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I used another mail package to test
my ideas, called, JES-2, and now I'm moving to James for production.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I want to create a new james-mailbox.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I struggled for many many hours doing
this and hit the same road block over and over.
>>>>>>>>>>>>> Unfortunately it is a configuration problem,
not a coding problem, so it becomes an indeterminate time problem- which isn't fun ;-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here is what I did:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1.  tar/untar clone the james-mailbox/memory-mailbox
to james-mailbox/custom-mailbox
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2.  rename the java packages and class
names to xxx.customer.CustomerBlahBlahBlah.
>>>>>>>>>>>>> (things compile and have different names/packages
than the memory)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3.  cloned the spring-mailbox-memory.xml
to spring-mailbox-custom.xml, changed names inside to reflect names of #2
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4.  modified the spring-mailbox.xml to
include this new spring-mailbox-custom.xml
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------
>>>>>>>>>>>>>
>>>>>>>>>>>>> 5.  mvn'd the code, shut down the mail
server, transferred the jars for mailbox-custom and the modified mailbox-spring.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (at other points I was transferring everything
I could find, but it made no difference)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 6.  modified the run.sh to include the
jar in the class path.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 7. ran sudo ./run.sh
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Error is something like:
>>>>>>>>>>>>>
>>>>>>>>>>>>> a NullPointerException when parsing XML
for spring-server.xml.
>>>>>>>>>>>>> with a huge stack trace which doesn't
yield too much interesting (to my eyes) except that I know it is init-ing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can't remember where that XML file
is.  I found it yesterday, but now I've forgotten where it is.
>>>>>>>>>>>>> Anyhow, when I found it I noticed that
it had an import to the spring-mailbox.xml…
>>>>>>>>>>>>>
>>>>>>>>>>>>> But I've unzipped those jars I transferred,
and the spring-mailbox.xml seems to be there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any thoughts?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is the guy/girl who wrote the hbase on
this list?
>>>>>>>>>>>>> Do you have any notes on the process
of integrating that mailbox into james?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> (just noticed my writing is incredibly
bad this morning, couldn't sleep, apologize if I'm writing like a 2 year old.)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jun 3, 2012, at 3:17 AM, Eric Charles
wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Tim,
>>>>>>>>>>>>>> Thx for the feedback.
>>>>>>>>>>>>>> Any patch to correct this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thx, Eric
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 06/03/2012 08:06 AM, Timothy Prepscius
wrote:
>>>>>>>>>>>>>>> http://svn.apache.org/repos/asf/james/mailbox/trunk/spring/src/main/resources/META-INF/org/apache/james/spring-mailbox-memory.xml
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>      <constructor-arg index="0"
ref="maildir-sessionMapperFactory"/>
>>>>>>>>>>>>>>> might be wrong.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -tim
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> eric | http://about.echarles.net
| @echarles
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
server-user-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> eric | http://about.echarles.net | @echarles
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> eric | http://about.echarles.net | @echarles
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>> --
>>>>> eric | http://about.echarles.net | @echarles
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>> --
>> eric | http://about.echarles.net | @echarles
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>

-- 
eric | http://about.echarles.net | @echarles

---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org


Mime
View raw message