Every MSP knowledge tool looks good in a demo. The tool is pre-loaded with well-organized content, the demo environment is clean, and the person showing it knows exactly what to search for. That is not the environment where your tool actually needs to work.
The real quality bar is: can a solo on-call tech resolve an unfamiliar incident at 2am, without context, without colleagues available, using only your knowledge system?
What 2am looks like operationally
At 2am, the on-call tech is typically not the person who normally handles this client. They might be junior, or they might be senior but unfamiliar with this specific environment. The alert is not descriptive. It says something like "backup job failed" or "VPN gateway unreachable." There is no ticket history for context. They are not going to call a colleague unless they are truly stuck.
They are going to open whatever knowledge system you have, search for something, and make decisions based on what they find. The quality of that decision depends entirely on whether your knowledge system surfaces the right information fast enough and accurately enough for them to act on it.
Three questions to run the test yourself
Question 1: Can they identify the client environment in under 30 seconds? Pull up your knowledge system as if you had never used it before. Navigate to a specific client without using bookmarks or shortcuts. If it takes more than 30 seconds to arrive at that client's relevant section, you have a navigation problem that will compound under the pressure of an active incident.
Question 2: Can they find the relevant runbook from a symptom description? Pick a ticket from last month. Read only the initial description, not the resolution. Go to the knowledge system and try to find a runbook that addresses that symptom. If you cannot find it within 60 seconds, a tech at 2am will not find it either.
Question 3: Can they access the needed credential without asking anyone? Once the runbook is found, does it surface the relevant credentials? Or does it say "you will need the admin credentials" and leave the tech to figure out where those live? The runbook and the credential should be reachable in the same workflow.
Common failure modes this test surfaces
The most common failure: the runbook exists, but it is filed under a name that does not match how the tech would describe the problem. A runbook titled "Exchange Online MX Record Synchronization" will not be found by a tech searching "email not delivering." Runbooks need to be findable by symptom description, not by the technical root cause that only an expert would know in advance.
Second most common: credentials are in a separate system from runbooks, requiring the tech to switch contexts. A tech at 2am navigating between a runbook in one tool and credentials in another is a tech who might grab the wrong credentials for the wrong client.
Third: the runbook assumes context the tech does not have. Steps like "verify the current configuration first" without explaining how are incomplete at 2am. Every non-obvious step needs to be explicit.
What a passing system looks like
A knowledge system that passes the 2am test has three properties. Client context is the navigation anchor: everything the tech needs is accessible from "I am working on Client X." The runbook and credential travel together. And the runbook's steps are written for a tech who knows how to use the tools but does not know this specific environment. That last one is the hard part, and it is the part that gets cut when documentation is written in a hurry.