A strategy for SaaS validation
Rajesh Ranjan examines the best path to follow in testing times
There is so much happening in the SaaS space these days. Differentiating between SaaS, PaaS and IaaS is getting more difficult, with everyone trying to do more and take more control. So when the services and offerings are undergoing this change, how does one keep pace in engineering, especially in testing?
Primarily, it is important to understand the difference between quality control and quality assurance. Add quality engineering in here, and you begin to appreciate that it is a blurred picture. If one has to go with the earlier explanation of cloud being the delivery medium for SaaS, the need for validation becomes even broader.
Every solution needs to be validated, both in functionality and architecture. Functional validation certifies the functional requirements and validates if the solution actually does what it is supposed to do.
But that is not sufficient. As a SaaS solution is hosted – or rather controlled – by the ISV, it is not just what it does but also how it does it that needs validation.
This is where architecture validation comes into play, as it defines the boundary under which the solution has to perform. Some might argue that these are operations or IT issues and do not come under the provenance of engineering.
But as the lines between SaaS, PaaS and IaaS are being blurred, so are the roles and responsibilities of different teams – especially engineering and operations.
Creating test cases for the functional requirements is easy, but what becomes the differentiator is the operational part, where non-functional requirements have a lot of impact. And these need validation too.
Including operation in the validation scope for a SaaS solution adds many dimensions to the overall strategy. Not just how, but where, becomes important.
For example: every solution uses logging and most of them have multiple logs: system logs, audit logs, operations logs, application logs, error logs and so on.
But in a multi-tenant environment, do logs also need to be tenant aware? Does it matter where the logs are maintained? Indeed, does it matter how they are maintained or who has access, and does that need validation? In my opinion, yes.
The old-school thought process for validating security for web application has become more complex. Not only the functional security needs to be validated, but also OWASP, which has proved to be popular and covers many important aspects of security.
Add cloud as the delivery medium for SaaS and the complexity increases manifold. The ecosystem, as well as the solution itself, now comes under the purview of validation, along with data ownership, user identity propagation, multi-tenancy, data security and data contamination.
To have a successful and long-lasting solution, a provider must understand these aspects and include them in the validation strategy.
There are still many dimensions that need validation. These may include licensing, provisioning, metering, cross-boundary transactions and more. One can argue that they may also have functional requirements and will come under the functional validation, but they must be included in the overall validation strategy one way or another.
Creating test cases for non-functional and operational requirements is not easy, and the validation is even more challenging. Combine this with agile, in terms of SDLC, and the complexities and costs multiply.
If an ISV goes to a customer and explains the different areas in which the solution has been validated, the customer is likely to not only appreciate the due diligence, but become an effective marketing tool – even though he or she has not yet started using the solution.
Indeed, if a process is set one time and has been driven into the various teams, adhering and implementing these validations is not so difficult. An inclusive mindset is needed, along with a proactive validation strategy and a collaborative team to make it happen.
Rajesh Ranjan is programme director and head of ESP Labs at Mindtree