Internally we have a need to hook into certain changes within some data that is housed in Oracle. Our solution will be to use Oracle CDC(Change Data Capture) to grab the info we need right from the redo logs and push that data into a table. We're going use the table to provide those who would like to do ETL a simple way to grab data that is not on the live tables and only contains the changes. From this table we'll push the rows into Oracle AQ(Advanced Queuing). From here we have the opportunity to drive that data right onto NSB much like the Service Broker is being used in this article.
Thanks to the contributors over at NServiceBus/NServiceBus-Contrib I have a great head start. I started out going down the NSB 3.0 route since the barrier to entry has been lowered greatly when attempting to create a custom Transport. Inadvertently I realized that I'm going to need to support NSB 2.5 seeing as how most of my shop is still on 2.0 and moving to 2.5, not only that 3.0 is still considered alpha or pre-alpha even. I also ran into an interesting serialization error in 3.0 anyway, once which I'll have to revisit later. The message headers on TransportMessage in 3.0 is now a Dictionary(used to be a List) which the serializer does not like one bit.
Luckily I was able to re-use the code from the 3.0 implementation. Going back to the 2.5 methodology has been a bit painful, and I'm glad that the custom Transport story is changing drastically. I have everything all set up and I'm ready to test. I'm even running Oracle 10g Express locally so this thing should be easy to try out for everyone. I'm interested to see if I can clean up some of the serialization work from the Service Broker implementation. Oracle AQ stores everything in a special table that backs the queue. You can tell it to make the column that stores the data to be an XMLType column or RAW, which I'm hoping plays more nicely with the serializers.
Once I get something more solid my plan is to put the code out on my personal repository in github and then work on getting it integrated into NServiceBus-Contrib. Let me know if anyone has already built this or is trying to, I'm always on the lookout for any problems or suggestions that have come up.
This is great news! We're forced to use OracleAQ at work, and it prevented us from using nServiceBus, which turned out to be a huge mistake. I can't wait to try it out.
ReplyDeleteHi Adam,
ReplyDeleteThanks very much for this. Seems to be working (mostly) for me. Two areas where I am having problems are when using distributed transactions (I changed the code to use TransactionScope to get around this), and when it then attempts to move a message to the error queue (an exception with msg.Payload is invalid as the detail is thrown).
Have you experienced these issues?
I have not had an issue with transactions. Underneath it all it is just using a basic ADO.NET transaction. I also did not have an issue with the error queue. What part of the code did you have to change?
ReplyDeleteI reverted my transaction changes, it turned out I just needed to set Enlist=dynamic in my connectionstring.
ReplyDeleteStill having problems with the error queue though, you mentioned you had problems with that, how did you get around it?
What problems are you having? I don't recall having issues with it....
ReplyDeleteWhen the MoveToErrorQueue method tries to Enqueue the message it throws an exception. The only info I can get out of the exception is:
ReplyDeletemsg.Payload is Invalid.