ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From akaptsan <akapt...@mail.ru>
Subject RE: Is there limitation on Semaphores and Queues?
Date Sat, 09 Jan 2016 10:19:41 GMT
Denis

 

1. One queue per table would prevent concurrent multithreaded modification of objects in this
table. I can’t afford have table-level synchronization, it must be row level

 

2. It not suitable. Replication must be asynchronous: 

- update and commit on 1-st database 

- send change message to 2-nd 

- apply change to 2-nd database  

 

Would it be better to modify my algorithm of applying changes:

-          Create or lock object’s semaphore

-          Process all messages from object’s queue

-          Close the queue

-          Close the semaphore

So I will have much less opened semaphores and queues. But semaphore and queue will be created
and closed almost on every change 

 

Regards, 

Anatoly

 

 

From: Denis Magda [via Apache Ignite Users] [mailto:ml-node+s70518n2455h20@n6.nabble.com]

Sent: Saturday, January 09, 2016 12:55 PM
To: akaptsan
Subject: Re: Is there limitation on Semaphores and Queues?

 

Anatoly,

 

I see. So the rest of the options I have in my mind are the following.

 

1) Queues & semaphores per object. This solution doesn't look scalable cause the more
objects you have the more resources (queues & semaphores) you're allocating. If you still
want to go for queues I would have one per table/cache and not per object. Instead of semaphores
I would use plain Ignite distributed locks [1] per table/cache. In this case you will have
per table/cache synchronization and this can affect performance but probably the difference
won't be so much or even better in your use case, you just need to measure.

 

2) As another one approach you can do all the modifications through Ignite cluster transactionally
[2] thus guaranteeing that there won't be concurrent updates of an object. Ignites caches,
that will be updated, will work with your own implementation of a custom persistent storage
[3] that will send updates to both DBs and a transaction that is started at Ignite's cache
layer will be committed only if DBs are updated as well.

 

Hope this helps to come to a satisfactory solution on your side.

 

[1] https://ignite.apache.org/releases/1.5.0.final/javadoc/org/apache/ignite/IgniteCache.html#lock(K)

[2] https://apacheignite.readme.io/docs/transactions

[3] https://apacheignite.readme.io/docs/persistent-store

 

Regards,

Denis

 

On Sat, Jan 9, 2016 at 11:30 AM, akaptsan <[hidden email]> wrote:

Denis

 

Thank you so much for helping me!

But none of your suggestions really work for me L.

 

I can’t now switch my application to Ignate as primary storage. But I do have this idea
in remote plans.

 

I can’t use native RDBMS replication, because databases are not 100% identical (in fact
probably less then 50% of DB objects are identical). 

I do need application-level capturing and applying of changes. 

 

And main reason why I can’t use Kafka or similar messaging system is: I need to eliminate
simultaneous change of one object in the two DBs. If it happened, then the object state in
DBs would be inconsistent.

That’s why I want to use Ignite semaphore

   

 

 

From: Denis Magda [via Apache Ignite Users] [mailto:[hidden email] <http://user/SendEmail.jtp?type=node&node=2454&i=0>
] 
Sent: Saturday, January 09, 2016 9:26 AM
To: akaptsan
Subject: Re: Is there limitation on Semaphores and Queues?

 

Anatoly, 

Please properly subscribe to the user list (this way we will not have to 
manually approve your emails and you will get answers on your questions quicker). All you
need to do is send an email to “ 
[hidden email] <http://user/SendEmail.jtp?type=node&node=2454&i=1> ” and follow
simple instructions in the 
reply 

Since the content in the databases is almost identical and you have a load balancer that directs
queries to one of the systems I would review current architecture design and probably switch
to the following one. You can still have a single Ignite in-memory cluster that will persist
data [1] in one or several databases depending on your requirements and configuration. In
such a design all business model related queries will go directly to the cluster and you don't
need to care about replication cause everything will be already there. Ignite supports ANSI-99
SQL so your SQL queries should work fine as well[2]. 

If your systems are web applications then you may want to use web session clustering in addition
[3]. 

On the other hand, if you can't switch to such an architecture then basing on your description
you are trying to implement a replication between the databases and probably solutions like
Kafka Connect [4] should work perfectly fine for you. Also different RDBMS vendors provide
native tools for replication between their databases. As an example replication between Oracle
databases can be done using Oracle Golden Gate product. I don't get why you need to lock an
Ignite object when you're applying changes, stored in queues, from one database to another
that's why consider that Kafka or RDBMS native replication tool will be enough. 

Does any of suggestions work for you? 

[1] https://apacheignite.readme.io/docs/persistent-store
[2] https://apacheignite.readme.io/docs/sql-queries
[3] https://apacheignite.readme.io/docs/web-session-clustering
[4] http://kafka.apache.org/documentation.html 

  _____  

If you reply to this email, your message will be added to the discussion below:

http://apache-ignite-users.70518.x6.nabble.com/Is-there-limitation-on-Semaphores-and-Queues-tp2415p2453.html


To unsubscribe from Is there limitation on Semaphores and Queues?, click here.
 <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
NAML 

 

  _____  

View this message in context: RE: Is there limitation on Semaphores and Queues? <http://apache-ignite-users.70518.x6.nabble.com/Is-there-limitation-on-Semaphores-and-Queues-tp2415p2454.html>



Sent from the Apache Ignite Users mailing list archive <http://apache-ignite-users.70518.x6.nabble.com/>
 at Nabble.com.

 

 

  _____  

If you reply to this email, your message will be added to the discussion below:

http://apache-ignite-users.70518.x6.nabble.com/Is-there-limitation-on-Semaphores-and-Queues-tp2415p2455.html


To unsubscribe from Is there limitation on Semaphores and Queues?, click here <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2415&code=YWthcHRzYW5AbWFpbC5ydXwyNDE1fC0xNzU0MjEzMTg3>
.
 <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
NAML 





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Is-there-limitation-on-Semaphores-and-Queues-tp2415p2456.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Mime
View raw message