Skip to main content

Watch your minions!

We have a Sitecore Commerce 9.0.3 implementation where minions play a somewhat more significant role than they were probably initially designed for. They synchronize catalog with PIM, pricing with external price system, promotions with external marketing system, etc. One of the most important roles is pushing the queue of created orders to the fulfillment provider.
Once the order volume passed the initial rate, the fulfillment provider started getting duplicate orders. This was somewhat random, and hard to catch. The duplicate orders would always be sent within less than a second from each other. The minion logs on our side would show no duplicate attempts to send anything, and the code itself made that impossible. To add to the mystery - rebooting the server would fix the issue but only until the next deployment. After much investigation and some lucky coincidence, we've found out the following:
  • Authoring role would suddenly start running minions;
  • The minions in Authoring and Minions roles would run almost simultaneously, trying to grab the same items from the same lists at the same time. If item was not removed from the list immediately after reading, it would be red and processed twice;
  • There were no minions defined for Authoring (besides the standard NodeHeartBeat);
  • Killing and restarting the Authoring node showed no minions configurations loaded.
So how was this happening? By running the Bootstrap. You see, our deployment scripts run Bootstrap against the Authoring role (as the first available). Guess what - running Bootstrap loads all configurations in this node, including minions. And it runs them! Killing the process and letting it re-start creates a "normal" Authoring instance that does not fight your minions.
Such a nice undocumented feature...

By the way, this also immediately cleared our logs from those nasty unexplainable "Entity version supplied is not current..." exceptions.

TL;DR: Running Bootstrap() against a Commerce Engine instance loads all configurations into that instance and might result in duplicate minions running. Once done with Bootstrap - kill that node process with fire. Then restart normally. Or use Ops instance for bootstrap and keep it shut down during the normal operations.

Labels

Labels:

Comments

Post a Comment

Popular posts from this blog

SharePoint 2013 and "search scopes"

While SharePoint 2013 introduced some undeniably handy features in the content discovery department, some of the old functionality has changed so much that you'll have to forget many of the design tricks that worked in 2007 and 2010.  One of such pain points is deprecating search scopes in SharePoint 2013. The requirement Let's start with a trivial situation. You have a requirement that states: "A user should be able to perform different types of search from any page/sub site on the portal by selecting a type of search and specifying keyword(s)". This would be considered an out-of-box functionality in 2007/2010: you just enable search scopes dropdown (considering the search box control is already in the master page), and define scopes. Piece of cake! Well, not so much in 2013.   The problem The thing is - there's no search scopes in SharePoint 2013. OK, to be exact - there's no user-definable search scopes, as we still have a couple of built-in ...
14 comments
Read more

Navigation Settings in SharePoint 2013 Site Definitions

Who likes creating SharePoint site definitions from scratch? Well, it's not my favorite part of the SharePoint development either. An initially good idea of setting up your whole site through an XML file, received implementation and support far from being perfect. That quickly lead developers to look for alternative ways of defining and provisioning sites, primarily through the SharePoint API. A simple ONET.XML with beefy code-packed features became de-facto standard. What was supposed to be a template became compiled code. I was hoping SharePoint 2013 would at least attempt to do something about it… Tasked with creating a few publishing portal site definitions, I was unpleasantly surprised to find out that any attempt to use my new site definition to provision a site collection root results in both Global and Current Navigation switched to "Managed Navigation" mode, while I needed it to stay in the good old "Structural Navigation" one. This mode is new to Sh...
Post a Comment
Read more

Catalog versioning in Sitecore Commerce Engine

Since the release of Sitecore Experience Commerce 9.0 Update-2 entity versioning is enabled by default for all catalog-related commerce entities: Catalogs , Categories and SellableItems . While certainly being a desirable feature, versioning can have a negative effect in certain use cases. Here are a couple of examples: One of the clients decided to use the BizFx business tools as a Product Information Management system. While one can argue on effectiveness of such decision, the real pain came up after the 9.0.2 upgrade when a quick 2-step process of updating the product description suddenly became an unintuitive and clumsy struggle through a dozen of steps. The client absolutely hated that experience and did not really care about versioning. Another client was using an external PIM that we were integrating with by synchronizing the deltas with SXC Catalog system. In 9.0.2 that resulted in creating a new version for every entity update. Not only these versions were useless (sin...
Post a Comment
Read more