Yesterday I had a meeting with a stakeholder and a senior representative from a vendor that supplies a SaaS solution. This particular stakeholder has had many challenges with the vendor in question. In fact, I’d go so far as to say they’ve had continual issues, since implementation. It was an honest meeting about the problems that have occurred and it was all pretty much laid out on the table. I’m not looking to pick on the vendor here, so I’m just going to call them vendor.
Vendor had promised to deliver a Root Cause Analysis for each of three significant incidents that have occurred in the last few months. Incidents 2 and 3 have reasonable causes, but Incident 1 is problematic. To sum it up, there was a significant change in security which exposed sensitive data to part of the population which should never see the data in question. Vendor’s senior executive explained that they had reviewed all of the available logs and could not see when the change occurred and that they couldn’t find evidence that there was a change. We know that there was because when we’d done full UAT less than 12 months before, it wasn’t that way.
Coming from an IT background, I had some questions. Did you restore the system from a point in time, say 3 or 6 months ago, and verify that the system was working or wasn’t? The answer, “That can’t be done; it’s not feasible to restore historically as it’s out of our disaster recovery plan.” Their system logs? Only a few weeks worth are kept. Since they aren’t willing to do the restore and research the issue through the historical data, they’ve decided to call this an anomaly and close the case.
The principle of causality states that all things (i.e. effects), with the exception of the first cause, have a cause. In other words, my existence is caused by my father loving my mother. Dad’s cause is my Grandma being wooed by my Old Spice wearing Grandpa, and so it goes back to the first cause. This relevant in doing a Root Cause Analysis because to be successful, you have to keep digging through those causes and effects until you get back to the last time the world was right. The vendor has made a conscious decision not to do that, not that they can’t, but that they won’t because they don’t see the value.
Our SaaS providers are entrusted with our data and our confidence. They have given us assurances that they will guard both of these and provide our end users with the best experience possible. To do that, you have to be unyielding in your willingness to track down the root cause of issues and do everything you can to make sure it doesn’t happen again. Calling it an anomaly just won’t do. We are relying on the word of our vendors, but that’s just not doing justice to the promise of customer satisfaction. It begs the question, are we short changing our end users by handing over control to our vendors?