What are we testing in a SaaS implementation?

By Raj Bajaj, Liqueo Senior Test Consultant

This article looks at changes in Quality Assurance practices with a focus on SaaS implementations.

There has been a drive for organisations to move away from traditionally internally hosted and supported systems to a more dynamic vendor-hosted solution. The theory is that Software as a Service (SaaS) allows organisations to focus on their core business and deliver a working system much faster. The organisation also benefits from the vendor’s schedule of upgrades, increased scalability, and reduced maintenance and hardware costs.

The transition to SaaS implementations has created a mindset shift around Quality Assurance. As SaaS products are hosted and supported by the vendor, it has raised questions around the scope of quality that should be ensured by an organisation’s QA engineers.

There are two fundamental differences to the Testing function in a change project when implementing a Service over a System:

1) What is possible to Test

2) What is necessary to Test

What is it possible to test?

The vendor will always lock the system down to protect themselves. Vendors will generally lock a system down so that only the intended User Interfaces can be accessed to ensure that Client Users do not break the system by unintended use. This Locking down makes some testing impossible – for example, checking the Database for results.

In return the Vendor will agree to an SLA (Service Level Agreement) stating at least ‘up time’ and percentages, and ‘resolution time’ for issues

What is it necessary to test?

QA engineers are responsible and accountable for ensuring the whole implementation satisfies the exit criteria of any phase of deployment. This would usually mean a full execution of pre-defined sets of tests.

Providing the resolution is delivered within the SLA, should QA worry if a system fails?

When the internal delivery team is no longer responsible for either the build or support of the SaaS product, clear QA roles and responsibilities must be defined between the client and the SaaS vendor up-front. This should ensure no gaps remain. This should also form part of any contractual arrangements and commitments with the SaaS provider, including the SLA.

It is more important to test that the vendor can respond to an issue with their system within the SLA than it is to test that those issues do not exist.

The vendor should be made responsible for the validation of performance and other non-functional activities of the SaaS application. This will ensure overall resilience can be achieved and maintained, including the integrity of all their integration touch points. This part of the solution is expected to ‘simply work’ – and in any SaaS solution it is the responsibility of the vendor to guarantee this. The core QA function would then focus on the integration points between any in house/on-prem applications and those that are now SaaS hosted, ensuring that these work effectively. Typically they will cover areas such as:

- workflow
- business logic
- user experience
- data processing
- configuration
- access management and security

Does this work with an Agile team?

Where testing is integrated into an Agile team, all functional and non-functional testing should be adopted as part of their iterative and incremental delivery. Tests are written as part of each sprint and, in order to be easily repeatable and increase the ability to deliver quickly, are automated as much as possible. Adopting Agile methodologies embeds the quality assurance element upfront. This means testing takes place continually throughout development to de-risk delivery (rather than having the QA ‘phase’ late in the development cycle as you would in a traditional Waterfall approach). The Liqueo Ways of Working team have written an article on Agile and DevOps, which explores this in more depth.

For an Agile team integrating with a SaaS solution, their tests would cover the same scenarios for allintegration points between in house, on prem and SaaS  applications as a Waterfall team. The difference is that they would build up the automated test suite in each iteration, based on priority.

Becoming more Agile in your test approach is critical when integrating with SaaS solutions. As the SaaS provider releases new versions, an automated regression test suite helps pinpoint any issues or inconsistencies early, so they can be raised with the vendor and fixed quickly. The SaaS vendor would be expected to provide a high level of confidence that the supported application meets any
pre-agreed Service Level Agreement (SLA).

In a nutshell, we should avoid assumptions about testing when implementing SaaS solutions. Ensure that your organisation has an agreement on division of responsibilities between the vendor and the QA engineers. This will ensure that there are no gaps in your quality assurance process and full integration can be achieved. Meanwhile, the vendor ensures that they meet their obligations and
service level agreements.

If you would like to know more about QA and SaaS testing – please Contact Us | Liqueo | London, UK