james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manuel Carrasco Moñino <man...@apache.org>
Subject Re: NoSQL backend for James
Date Mon, 17 Oct 2011 11:51:42 GMT
I have added the Stress test and I see the exception, not sure how to fix yet.
https://github.com/manolo/james-couchdb/commit/cf9a41116fca458c440a28ec937b83aeccd7967a

- Manolo

Exception in thread "pool-5-thread-61"
org.ektorp.UpdateConflictException: document update conflict: id:
5e04b2a2c6fd920bc09af5a675000e7d rev:
3-ce227ff7c45a6905e010684f08011d05
	at org.ektorp.impl.StdCouchDbConnector$6.error(StdCouchDbConnector.java:319)
	at org.ektorp.impl.StdCouchDbConnector$6.error(StdCouchDbConnector.java:308)
	at org.ektorp.http.RestTemplate.handleResponse(RestTemplate.java:98)
	at org.ektorp.http.RestTemplate.put(RestTemplate.java:38)
	at org.ektorp.impl.StdCouchDbConnector.update(StdCouchDbConnector.java:307)
	at org.ektorp.support.CouchDbRepositorySupport.update(CouchDbRepositorySupport.java:155)
	at org.apache.james.mailbox.couchdb.mail.CouchDbMailboxMapper.save(CouchDbMailboxMapper.java:90)
	at org.apache.james.mailbox.couchdb.mail.CouchDbUidProvider.nextUid(CouchDbUidProvider.java:39)
	at org.apache.james.mailbox.store.mail.AbstractMessageMapper.add(AbstractMessageMapper.java:125)
	at org.apache.james.mailbox.store.StoreMessageManager$4.run(StoreMessageManager.java:532)
	at org.apache.james.mailbox.store.StoreMessageManager$4.run(StoreMessageManager.java:1)
	at org.apache.james.mailbox.store.transaction.TransactionalMapper.execute(TransactionalMapper.java:38)
	at org.apache.james.mailbox.store.StoreMessageManager.appendMessageToStore(StoreMessageManager.java:529)
	at org.apache.james.mailbox.store.StoreMessageManager$1.execute(StoreMessageManager.java:326)
	at org.apache.james.mailbox.store.StoreMessageManager$1.execute(StoreMessageManager.java:1)
	at org.apache.james.mailbox.store.AbstractMailboxPathLocker.executeWithLock(AbstractMailboxPathLocker.java:41)
	at org.apache.james.mailbox.store.StoreMessageManager.appendMessage(StoreMessageManager.java:322)
	at org.apache.james.mailbox.AbstractStressTest$2.run(AbstractStressTest.java:92)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

On Sun, Oct 16, 2011 at 6:24 PM, Pepijn de Vos <pepijndevos@yahoo.com> wrote:
> Thanks man, really appreciated.
>
> I too think my ModSeq impl looks good, but incrementing the modSeq 4 times in parallel,
I noticed that
>
> the value is incremented only 2-3 times.
> _rev is incremented even less.
> additional documents get created.
>
> What happens is that the same mailbox object gets passed to all invocations of nextModSeq.
This means that incrementing them in one increments it everywhere. Normally this would only
create conflicts in all but one, but because this mailbox does not have an id yet, it creates
a new mailbox for every conflict.
>
> So I assume this is not a problem in practice? Because after your fix, all created mailboxes
actually do have an id. I'll test that later, and then try to figure out JUnit.
>
> Pepijn
>
> On Oct 16, 2011, at 4:23 PM, Manuel Carrasco Moñino wrote:
>
>> On Sat, Oct 15, 2011 at 5:40 PM, Pepijn de Vos <pepijndevos@yahoo.com> wrote:
>>> So, the result and questions for today:
>>>
>>> https://github.com/pepijndevos/james-couchdb
>>>
>>> What is serialVersionUID?
>>
>> It is a good practice to define it for serializable classes
>> http://www.javapractices.com/topic/TopicAction.do?Id=45
>>
>>>
>>> Is UidValidity something else than what is provided by CouchDbUidProvider?
>>
>> It's a number that changes when all uids become invalid. The server
>> cannot update this number during a session.
>> http://www.afterlogic.com/mailbee-net/docs/MailBee.ImapMail.Imap.UidValidity.html
>>
>>>
>>> I had a look at the InMemory test, and it seems to fail on remainders of the
InMemory implementation(mentions of ConcurrentHashmap). I'm not sure what it tests, but I
think it's more than just CouchDbMailboxManager.
>>>
>>> Today I tried to implement ModSeq and Uid. They work fine until you introduce
concurrency. I'd be glad if someone would take a look at nextModSeq.
>>
>> Your implementation looks good.
>>
>>>
>>> I'll try to write a test for it tomorrow. The thing is that the test would need
the database, right? How do you handle that?
>>
>> Difficult since it seems there is no way to install couchdb and start
>> it from maven.
>> Right now tests should be ignored when ddbb is not running (see
>> maildir tests which are omitted in windows systems)
>> I've added in the setup a call to empty the 'james' database when the
>> test run, also I have added  initStandardDesignDocument() to
>> initialize database.
>> Be careful before run the test if you do not want the ddbb be re-created.
>>
>> Happy coding.
>> - Manolo
>>
>>>
>>> Pepijn
>>>
>>> On Oct 15, 2011, at 11:41 AM, Pepijn de Vos wrote:
>>>
>>>> Thanks! :)
>>>>
>>>> I have no idea what you did with the generics though. I read an article about
them, and thought I understood them.
>>>>
>>>> Basically they are just statically checked casts, right? So what does it
mean to write X implements Y<Z>?
>>>>
>>>> The tests are just leftovers from InMemory. I'm not doing TDD, but I rather
see tests as a frozen REPL. I write code, test it in the REPL, if I find myself repeating
pieces in the REPL, I make them into a test. I didn't get to the stage where I could even
compile anything.
>>>>
>>>> Pepijn
>>>>
>>>> On Oct 14, 2011, at 8:14 PM, Manuel Carrasco Moñino wrote:
>>>>
>>>>> You are right, you already sent the github link.
>>>>>
>>>>> I've checked out the code and I've made some changes to make it
>>>>> compile, also I have changed id signature in messages and mailboxes to
>>>>> String since couchdb uses string. I have hardcoded a user and password
>>>>> in the Utils class, it should go in the configuration files though.
>>>>>
>>>>> I just have sent you a pull request
>>>>> https://github.com/manolo/james-couchdb/commit/2b6b0ebce5c78198b14d1e7ec05c178d56482919
>>>>>
>>>>> The unique Test in the tree do no pass, I think you can start fixing
>>>>> the test so as it should be easier to follow the code taking a look to
>>>>> tests.
>>>>>
>>>>> - Manolo
>>>>>
>>>>>
>>>>> On Thu, Oct 13, 2011 at 9:54 PM, Pepijn de Vos <pepijndevos@yahoo.com>
wrote:
>>>>>> I added the github link to the beautifully formatted original email,
but I think Google mistreated my message.
>>>>>>
>>>>>> Her it is again: https://github.com/pepijndevos/james-couchdb
>>>>>>
>>>>>> Pepijn
>>>>>>
>>>>>> On Oct 13, 2011, at 9:28 PM, Manuel Carrasco Moñino wrote:
>>>>>>
>>>>>>> Hi Pepjin, could be possible to share your code anywhere, so
as I
>>>>>>> could checkout it and take a look?
>>>>>>>
>>>>>>> Don't worry about if the code is ok or not, I think github could
be
>>>>>>> ok, but you could send a compressed file via email or whatever
you
>>>>>>> prefer.
>>>>>>>
>>>>>>> - Manolo
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 13, 2011 at 8:12 PM, Pepijn de Vos <pepijndevos@yahoo.com>
wrote:
>>>>>>>> Okay, I put up the result for the day.
>>>>>>>> I made a CouchDbMailbox with Ektorp @annotations.
>>>>>>>> I made an attempt to make the MailboxMapper, but I got stuck
at the <type> casting sugar which I don't grok. My IDE keeps complaining it can't resolve
the incompatible types, while both are just Mailboxes.
>>>>>>>> I stuffed the CouchDB connection in a class, not happy with
it.
>>>>>>>> I'm not sure how to implement the findMailboxWithPathLike
and hasChildren methods.
>>>>>>>> Any help appreciated, especially with the... <> things.
list() is the only one that's red wiggly lines, the others are just unchecked casts. I've
been adding random casts and <> left and right.
>>>>>>>>
>>>>>>>> https://github.com/pepijndevos/james-couchdb
>>>>>>>>
>>>>>>>> Pepijn
>>>>>>>>
>>>>>>>> On Oct 13, 2011, at 6:23 PM, Pepijn de Vos wrote:
>>>>>>>>
>>>>>>>>> More questions. I started hacking!
>>>>>>>>>
>>>>>>>>> I'm going with Ektorp. I figured out most of it, I think.
Except that I don't understand the configuration. It does have a Spring module. Any pointers
on how to organize the config and connections?
>>>>>>>>>
>>>>>>>>> What does findMailboxWithPathLike do? The implementations
seem to do weird things with regexes. Preferably I make that into a nice CouchDB view. CouchDB
can't do fulltext search. As far as I can tell, the IMAP RFC doesn't say anything about it.
>>>>>>>>>
>>>>>>>>> To have custom message and mailbox classes, do I need
to do anything else besides subclassing the corresponding *Manager class to return one?
>>>>>>>>>
>>>>>>>>> I'm getting there!
>>>>>>>>>
>>>>>>>>> Pepijn
>>>>>>>>>
>>>>>>>>> On Oct 12, 2011, at 9:35 PM, Manuel Carrasco Moñino
wrote:
>>>>>>>>>
>>>>>>>>>> Actually Ektorp is not a full implementation of JPA,
but it provides a
>>>>>>>>>> JPA like API with support to many of its annotations
etc.
>>>>>>>>>>
>>>>>>>>>> Anyway, based on my experience, Ektorp simplifies
the access from java
>>>>>>>>>> to couchdb and the bootstrap of couchdb, so as theoretically
when
>>>>>>>>>> James starts the first time, the database, views,
design, mapreduce,
>>>>>>>>>> etc should be created.
>>>>>>>>>>
>>>>>>>>>> - Manolo
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 12, 2011 at 1:05 PM, Pepijn de Vos <pepijndevos@yahoo.com>
wrote:
>>>>>>>>>>> Ektorp seems nice, but I'm more comfortable just
using something that resembles the HTTP API, since I'm not familiar with JPA. Haven't decided
yet.
>>>>>>>>>>>
>>>>>>>>>>> Pepijn
>>>>>>>>>>>
>>>>>>>>>>> On Oct 11, 2011, at 5:08 PM, Manuel Carrasco
Moñino wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Pepijn
>>>>>>>>>>>>
>>>>>>>>>>>> Which java library are you considering to
use to connect with couchdb?
>>>>>>>>>>>> I'm using [1] ektorp and makes really easy
to map domain models.
>>>>>>>>>>>>
>>>>>>>>>>>> - Manolo
>>>>>>>>>>>>
>>>>>>>>>>>> [1] http://www.ektorp.org/reference_documentation.html#d0e532
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 10, 2011 at 10:44 PM, Pepijn
de Vos <pepijndevos@yahoo.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Oct 10, 2011, at 8:24 PM, Ioan Eugen
Stan wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have patience. You will need it if
you wish to complete something.
>>>>>>>>>>>>>> Patience and perseverance or else
you'll be just another one who
>>>>>>>>>>>>>> tried.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't expect to have it finished by
the end of the week, but if I'm still completely clueless by then, it's just not worth the
effort.
>>>>>>>>>>>>> I don't have the ambition to become a
James commiter or even a Java dev, I just thought it would be nice to use CouchDB for my application.
>>>>>>>>>>>>> Somewhere is a point where pragmatism
beats learning. There isn't any technical reason why I can't use JPA.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can I just copy an existing one
and rename stuff? In other words, how are the modules glued into the whole? How does the server
know which class to load? It's not in the pom.xml, afaict.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Not sure what you mean by that. It
uses dependency injection provided
>>>>>>>>>>>>>> by Spring framework (and soon Guice)
to inject object references into
>>>>>>>>>>>>>> other objects at runtime.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ah, dependency injection. *googles* So
just the fact that I implement the interface is enough to @autowire it into James?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The sample config is gone btw:
http://james.apache.org/server/3/config-mailbox.html
>>>>>>>>>>>>>>> Do I inherit tests as well? I
would imagine that a lot of tests are common to all mailbox implementations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think this is because the configuration
changed and now it's spring
>>>>>>>>>>>>>> based, and more modular. I see you
are very ambitious but I sense you
>>>>>>>>>>>>>> have a lot of catching up. James
is complex so give it time, if you
>>>>>>>>>>>>>> expect too much from yourself and
fail you will probably be too
>>>>>>>>>>>>>> disappointed.
>>>>>>>>>>>>> Yea, I read a Java book long ago, never
did any big projects with it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Make a public repo, commit something
and ask if you get stuck. I will
>>>>>>>>>>>>>> try to help when/if I can. I suggest
you start with simple
>>>>>>>>>>>>>> implementation that passes some unit
tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So If I take any mailbox impl, put it
in a separate repo, will it work? All sorts of things refer to the parent pom. I'll put something
on github once I figure it out. I think it'll work out once I get to the point where I can
write some code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> See for example the unit tests I
did for Mailbox interface in HBase
>>>>>>>>>>>>>> implementation:
>>>>>>>>>>>>>> http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/mail/model/HBaseMailboxTest.java
>>>>>>>>>>>>> What I mean with inheriting tests is
that these all look very generic. They look like they could test any mailbox implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pepijn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ioan Eugen Stan
>>>>>>>>>>>>>> http://ieugen.blogspot.com/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
server-dev-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message