|
The good news: SaaS solutions are for the most part built to deliver excellent security and performance levels: making applications run well from a central, secure point is the core business of a SaaS provider. Large investments to ensure service levels can be shared across many customers, and often create better operational results than systems deployed in house. Nonetheless, not all SaaS providers are created equal.
Delivering on a Service Proposition
Look at the language the company uses about service, and consider how well it listens to your needs and understands the value you seek to create in your environment. Do they offer live sales and support people to assist with questions? How earnestly and directly are issues handled? Does the company offer service level commitments and publish its performance statistics for customers to see? With that in mind, here are some guidelines that a thoughtful consumer can follow when verifying a potential solution's operational effectiveness. We call it seeking "P.R.A.I.S.E."
Performance: Measurement, metrics and monitoring that raise the bar
Reliability: Problem submission, determination, resolution and tracking Availability: Service commitments for when you need it most Information Stewardship: Obligation to protect your information now and in for the future
Scalability: Capacity programs encompassing a 3608 view
Enterprise Dependability: Customer growth, financial strength and value of SaaS
Performance:
When assessing performance, what you are really asking for is a guarantee that the SaaS provider will deliver its service consistently and seamlessly, responding quickly to the "clicks" of your users. Speed matters: studies show that, on average, a user expects a response in no more than 2 seconds, is willing to wait up to 5 seconds, and has almost always given up by the time 8 to 10 seconds has elapsed.
Many companies measure server performance alone, or the speed at which the back end server responds to requests. This does not matter much to end users, however, unless that speed translates into a rapid response to their requests. Vendors also boast of how many transactions per second (TPS) the system will handle - but again, TPS in itself does not necessarily translate into a rapid response to the user. Other factors like the size of the page being returned and the complexity of the logic in that page will also have a significant effect on performance, as do bandwidth and the size of the data set being processed.
It's a fine point, but it pays to ask what the measurement points are. For example, is an end-to-end use case being measured, or just individual page response? In evaluating whether the numbers are meaningful to you, you may want to check to see that the data points are gathered in a way that matches the way you will use the system - for example, are the pages and processes involved in your most frequent transactions the ones that are being measured?
PERFORMANCE: WHAT TO LISTEN FOR
Do you have a service level commitment or agreement? The SaaS provider should clearly outline what performance levels the customer can expect. Ideally, an overall response time for the application should be defined within percentiles:
Currently, a good target for SaaS SLCs is to have 95% of transactions with a server response of less than 5 seconds and 99% of transactions less than 10 seconds. Exceptions depending on load are acceptable, as long as they are clearly identified (e.g., transactions whose response time is dependent upon load-file-upload transaction). There may be other acceptable dependencies based on outside providers, for which the SaaS vendor is not responsible; for example, the speed of the Internet backbone. This kind of exception in a service level commitment, while not ideal, is acceptable if identified up-front.
Note: There is a difference between a service level commitment and an agreement. The former is a non-binding version of the latter. SLAs generally come at a premium price, because of the reporting and management requirements, such as regular reporting that summarizes the performance of the system for that reporting period by customer.
|
