March 29, 2014 1 Comment
The adapter provides a very easy to use approach to receiving and sending messages to and from a service bus queue/topic/subscription but, as it turns out, it might be that it is a little bit simplistic or – at the very least – some caution is required as I’ve learnt with a customer yesterday –
The customer uses BizTalk to read a message from a Service Bus queue using the adapter. In the pipeline they ‘debatch’ the incoming message into multiple messages that get published to BizTalk’s message box.
This all worked just fine during testing, but when the customer moved to performance testing they hit a point after which they got a lot of errors such as the follows –
The adapter "SB-Messaging" raised an error message. Details "System.ObjectDisposedException: Cannot access a disposed object.
Object name: ‘Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement+RedirectContainerChannelFactory`1[System.ServiceModel.Channels.IRequestSessionChannel]’.
The adapter "SB-Messaging" raised an error message. Details "Microsoft.ServiceBus.Messaging.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.
When reading from the Service Bus the client can employ one of two strategies – a desructive read whereby the message is remove from there queue as soon as it is being read or a ‘peek lock’ strategy where the read message is locked and effectively hidden on the queue for a duration, allowing the client to process the message and then come back and either confirm that it had successfully processed the message or abandon the message, effectively putting it back on the queue.
The peek-lock strategy is very useful to ensure no message loss and BizTalk’s SB-Messaging adapter makes a good use of it – until the received message is persisted in the message box it is not removed from the queue and hence zero-message loss can be guaranteed.
On the flip side, it means that a message may be delivered more than once and this has to be catered for.
In this customer’s case, the debatching they performed meant that, under load, it took BIzTalk too much time to complete the debatching and persisting all the messages to the message box. That meant that it did not go back to the queue in time to release the lock resulting with the error above. unfortunately this also caused a snow-ball effect as it meant that another BizTalk instance picked up the same message and the issue repeated itself, with the large number of messages being persisted to the message box increasing the load on the server and with it the duration of the receive pipeline processing….
Once identified they have found a quick workaround, at least in the first instance – remove the debatching from the service bus receive port and move it to a later stage in their BizTalk processing. This, I think, is a good appraoch, as it lets them ‘take ownership’ of the message in BizTalk, removing it from the Service Bus, before starting the heavy processing.
Another approach is to change the lock duration property on the queue/subscription in question. the default is 60 seconds. This could by time and with careful load-testing a reasonable value may be found, but – of course – it does not eliminate the risk, just pushes it out further.
Asking in the virtual corridors of Solidsoft it appears we have seen this issue with at least two other customers. I think that the SB-Messaging adapter should support both strategies and allow the customer the choice, and perhaps flagging the need to set the lock duration value on the queue/subscription as required.
Cross posted on the Solidsoft blog