All for one, one for all? When Microsoft CSP subscriptions get confusing
Technology Management MD James Crowter recalls when a customer's Dynamics 365 system went down following an unpaid bill
A couple of weeks ago we had a nightmare of an evening; a real heart-stopping moment I should share because there are lessons to be learned for our whole community, partners and end users. I was sitting in the bar of a hotel with a couple of my colleagues after a client visit when one of them had a WhatsApp message saying that a customer's Dynamics 365 customer service system was down. What made me really start to sweat was the reason. The account was suspended "because of non-payment".
Cue lots of tapping the screen of my phone to check the status of all the vendor accounts we have with Microsoft. Thank goodness for Business Central's phone app - it got a pounding right there. We are a Direct CSP (Cloud Solution Provider) partner, so we pay Microsoft directly for clients' CSP use including Dynamics 365 services. None of those accounts had anything close to due, let alone overdue. Invoices were on for every period there should have been; our admin team hadn't dropped the ball.
For the next 45 minutes, my team exchanged lots of messages. It might have been after seven in the evening but I'm proud of the response. Anyone who might have been able to help, was on it.
The cause? It turned out the customer uses Office 365, which we thought was provided through another partner. In fact it was a 'web-direct' purchase direct from Microsoft. The subscription was overdue for renewal and the original credit card it was bought with had expired.
What surprised me was that this stopped the company's Dynamics 365 instance as well. Surely it's in the same AzureAD organisation to get single sign-on and simplify admin. They also have some PowerBI charts displaying in CS and NAV 2017.
Once the cause was identified and the card updated, the fun did not end there. The O365 account came back online a few hours later but Microsoft had deprovisioned the tenant. They were using V8 of Dynamics 365 Customer Service and we had deliberately deferred the upgrade because we knew we needed to move their NAV 2017 to a later cumulative update to be compatible with the authentication changes forced by D365 v9.
Yes, you guessed it, when a tenant is deprovisioned there is no option to restore to anything but the 'latest build'. The result was disconnection from the NAV system sitting in the Azure environment we manage, until later the next day when we could finish deploying the roll-up. Fingers were firmly crossed as we did a lot less testing than we like to.
Imagine your client not being able to tell who their debtors were, because one of the required subscriptions hadn't been paid by someone in their supply chain.
My first takeaway is that this is a very good reason why the client should have all its CSP subscriptions in one place. Dare I say a better reason than saving a few pennies. Maybe I'm biased but isn't it best to have it all with the partner who understands the most complex and business critical parts of their tenant, ie Dynamics?
Customers - you're not buying paper clips here. Remind your purchasing bulldog, determined to drive the cost down, that if they get this wrong they won't know what price was agreed as the purchase order system will be offline. CSP is very competitive anyway. Microsoft don't give much margin to anyone, so price differences will be tiny. That being the case, is it worth the risk?
Secondly, shouldn't each organisation understand who they have a dependency on and ideally try to minimise that? Do you buy from a direct CSP partner like us or from an indirect CSP partner who in turn purchases from a CSP distributor? If that's the case, should you know who your critical systems are dependent on and be able to monitor their financial health? Do you want to have to start asking questions when you have no access?
And shouldn't client users ask what protection there is in that dependency channel? With CSP you only need to pay monthly. Some partners are asking for a year in advance so they take that cash and then pay monthly to their supplier. Your money would be lost if they didn't make that payment.
Should there be an escrow account in the mix where Microsoft gets paid even if something happens in a partner chain link? If you're paying monthly then the maximum risk is a month's subscription.
Microsoft's terms and conditions details that it will warn 28 days before 'deprovisioning'. Who gets those notifications? Even if your tenant is fully managed by a partner, every client should have someone on the distribution list.
Senior management need to know who to call, not just the IT manager. With Dynamics down you won't be able to transact - no sales, no business records. In fact, in the UK you're in breach of your director's responsibilities and could end up with a ban because you haven't ensured your business records are safe. At the least expect the tax authorities to take a very dim view.
Moving to the cloud makes you massively more secure. In fact, I'd say that you only need to make sure the bill is paid. To do that though, you need to know how it's paid. Ask the questions now, while it's not life and death for your business.
This article first appeared on LinkedIn
James Crowter is managing director of Technology Management