What Is an External Risk Review? A Guide for Outdoor and Off-Campus Programs
What to expect from an independent risk management review and how it differs from accreditation.
What is an external review, and does our program need one?
Sooner or later, these questions creep up on most program and executive directors' radar. Some are genuinely curious about how their program's risk management compares, and they know that an external perspective can be instrumental in gathering and conveying information in a way that aligns the organization and helps determine overall strategy and priorities. Others haven't yet commissioned one, and now a board or an accreditor is waiting and it has to get done. Many are eyeing several issues and potential solutions, and aren't yet sure how they relate or where to start.
I wrote this guide for all three, so that anyone considering a review knows what they're getting into.
What is a program risk review?
An external risk management review, sometimes also called an audit, is an independent assessment of how a program manages risk. It should always be conducted by a qualified reviewer who is otherwise unaffiliated with the organization. Every review should produce a written report of findings and recommendations, which are submitted directly to the organization. The organization then studies the report and responds to it in writing and by acting on its recommendations in a way that it chooses.
It's important to remember that reviews capture just a snapshot of the program in space and time. It's the reviewers' job to translate and convey that snapshot to a variety of stakeholders in a way that represents the overall operation and its culture. This often takes a back-and-forth with the internal sponsor of the review, who is typically a program director or risk manager. Reviewers manage this limitation by triangulating and generalizing carefully, and by making sure the evidence they've gathered can support each finding before it is finalized and presented to stakeholders.
Three things a review is not
An external program review is sometimes confused with an external incident review; the two serve different purposes. An incident review, often called an independent investigation, is conducted in response to a specific event. High-severity incidents, allegations of wrongdoing, and questions about how an event was handled are all circumstances that call for one, and in some cases and jurisdictions the investigator needs a license to do that work. Conversely, a program risk review examines the whole risk management system rather than a single event.
But the distinction doesn't mean that incidents are out-of-bounds during a program review. Reviewing incident data and further investigating select incidents have an important place in program reviews. For example, a couple of years ago, I was working with a university program, and a few incidents from prior seasons sparked my curiosity. To provide historical context and further, more tangible insight, I wanted to learn why they occurred and whether any correlation existed between them. I also wanted to see how the organization learned from these events and what it put in place since. The entire incident management process, from escalation to management to reporting and review, serves as a helpful set of artifacts about how an organization works and how mature its risk management program is.
Secondly, while it is standard practice to engage a consultant to lead a review, a review is fundamentally different from many other types of consulting engagements. The difference comes down to how well the organization articulates its questions and already understands its problems. Directors who can name and define a problem in their org might benefit by skipping a review altogether and going straight into a consulting engagement to help scope and resolve that one, specific issue. A review better serves a leader who sees several issues and needs help understanding them and prioritizing their solutions. This situation is quite common, no matter how experienced a program or executive director is. In fact, see my post about 8 reasons to initiate an external review to see if a consulting engagement or a review would be more helpful.
Lastly, a review should never substitute for normal business and executive management functions. Occasionally, an administrator or senior leader wants an external consultant, under the guise of a program review, to justify a decision leadership has already made, like budget cuts or staffing changes. These requests come across my desk from time to time and are signs of management problems that should be handled internally. Instead of a review, these issues are better served through consulting or coaching engagements.
The difference between a review and accreditation
This is important to note because it is frequently confused by programs and new reviewers alike. Although they can look similar from the outside, they fundamentally serve different purposes and convey different messages to external audiences.
Accreditation answers whether an organization meets an accrediting body's standards (read: industry standards). Once the organization passes, they receive a credential that they can publicly display for parents, partner schools, and insurers. The message is that the organization meets a set of standards agreed upon by an industry association and is dedicated to planning and operating with diligence and at a high, professional level.
External reviews are informed by established industry standards but are more flexible and malleable. For example, many of the organizations I've reviewed don’t fit neatly into a single association's standards, and they need help to align, mix and match, and identify reasonable norms and practices from comparable organizations. I've also worked with plenty of accredited programs who want a deeper or more targeted external perspective and get a set of recommendations that are more descriptive and fitting for their exact context or set of circumstances. Many accrediting bodies, such as the Association for Experiential Education, the American Camp Association, and the Forum on Education Abroad, require accredited programs to regularly conduct internal and external reviews as part of the continuous improvement cycle.
A review also differs from accreditation in where it starts. Accreditation starts from the standard and asks whether the organization meets it. A review, on the other hand, starts from the organization. It begins by considering its context, constraints, strategy, and the barriers specific to the organization that a generic standard can't account for. In turn, a review can identify vulnerabilities and a way forward fitting to the organization, while still helping it to meet or maintain standards.
The two also work well in sequence, and many organizations use both. Accredited organizations sometimes reach a point where the re-accreditation cycle starts to feel routine and improvement stalls. In this case, a review can go deeper than the pass-or-fail approach accreditations typically take to find deeper layers of improvement, such as determining how developed, mature, and efficient various aspects of the risk management program actually are. Check out a free self-assessment about risk management maturity.
Scope, format, and timing
The next questions are practical ones: how big, in what format, and when.
A common question is "we can't afford a big, comprehensive review" or "we're not ready for one yet, we don't want to pay for someone to come and tell us what we already know."
That makes sense; a review doesn't have to be comprehensive or a big ordeal to be helpful. Plenty of organizations request a focused review of one aspect of their program or operation, like crisis management, child protection, water activities, or incident reporting and learning. A focused review costs less, finishes faster, and can provide meaningful and actionable direction for what to do next. The following year, with a new budget, a new review can be commissioned for another area the organization wants to learn about and improve.
An important question is the format and review methods. It’s easier than ever to do it remotely, where the reviewer reads documents and interviews personnel online. However, there is no replacement for a good site visit, which significantly enhances the review and provides deeper evaluation and more relevant recommendations. Limiting it to online-only access can provide an incomplete view of work and risk management. Traveling to observe staff in action, to see packing, operating vehicles, and briefings, allows asking why tasks are done a certain way and how to improve them. Triangulating among written work as prescribed in manuals and curriculum, observations of work as actually done on site, and face-to-face interviews in the normal work environment offers the most helpful insights for relevant recommendations. Other methods can be helpful and add even further dimension, such as incident analysis, as we previously discussed. I've also designed and implemented staff and participant surveys to capture experiences and various perspectives on safety and work, and I find it helpful to map work using safety science and systems-based methods I studied in graduate school (see my post about systems thinking methods).
Timing is a question to figure out, and it matters more than many think. A common instinct is to schedule site visits during the slow season to allow staff the time and space to host a reviewer and 'show them around.” At the other end of the spectrum, a director might want me to see their program at its worst, like in summer or the height of the busy season when staff are stretched, and all the program’s vulnerabilities are easily exposed. I prefer visiting during normal operations, as it provides a baseline of 'normal' for the program. I aim to observe everything, ask questions, and speak with staff, parents, participants, partners, and administrators to get a sense of what the day-to-day and “optimal” looks and feels like. Sitting all day in a conference room with back-to-back interviews is no more effective than doing the same remotely in pajamas. People should speak openly, without sounding scripted.
Duration and cost depend on the scope. A focused review typically takes about a week, including document analysis, planning beforehand, a few days on-site, and report writing. This is the standard that most reviewers and organizations prefer. For larger, complex organizations, I have overseen comprehensive reviews that span months, with different team members assigned to various areas. Conversely, I have completed reviews of specific program components, such as transportation or equipment protocols, or the program handbook and training curriculum, in as little as a day. The scope of the review is less important than its usefulness to the organization, ensuring findings and recommendations are documented and shared with sponsors or senior leaders, and that the project aligns with available budget and timing constraints.
The report, and who sees it
Every review ends in a report, and the report deserves as much attention as the review itself. In addition to findings and recommendations for improvement, a good report also highlights strengths. Every program should get credit for the good work and diligence already in place, and I find that those strengths are helpful tools for meeting the recommendations. The recommendations are organized in some helpful fashion so all stakeholders can make sense of risk management and the next necessary steps, and are accompanied by supporting resources where they help. My reports also include an analysis of the organization's overall risk management strategy, which is an assessment of everything the review found, synthesized into a tangible articulation of how risk management is working, and the enablers, barriers, and constraints involved, rather than a list of individual fixes. Many reviewers don't provide that level of analysis, and clients consistently tell me it's the most useful part of the report in terms of zooming out to see the big picture, and in getting everyone aligned on the state of their risk management.
It's normal to have some level of anxiety about the final report, like what it will say and who will get to see it. These concerns should be worked out in advance between the reviewer and the internal sponsor, and the chief executive of the organization. As a standard practice, everyone involved in the review should have a chance to read the report before it is finalized and distributed. Their lens, including potential corrections or updates via additional evidence, should be considered and added as necessary. A presentation of the report typically follows, and the organization decides who attends.
After the review
The report belongs to the organization, and so do the implementation steps that come after it. Some recommendations the organization can complete on its own, and it's normal for a few of those to be timely. Others are dependent on resources or other tasks and take a year or multiyear efforts to implement. Organizations typically need help with some recommendations because they need specialist know-how, resources, or just time and capacity. Focused consulting engagements help with these, like writing a policy, compiling and updating a manual, or delivering training. Reaching out to expand their network is also helpful, such as making introductions to specialists in medical, mental health, legal, insurance, or security. Ongoing coaching for senior leaders is another helpful approach after a review, especially around risk management strategy and change management. A good report can offer some guidance on these next steps so the organization can get started on an efficient path.
Whether an organization comes to a review curious about how it compares, answering a board or an accreditor, or facing several issues without a clear starting point, the path begins with the same conversation.
Slay Risk conducts program risk reviews for schools and provider organizations across Asia and North America, with worldwide availability. For more details, visit www.slayrisk.com/program-risk-review. Contact us to quickly determine if a review is beneficial at this moment, and to discuss its scope and requirements.
FAQ
What is an external risk management review?
An external risk management review is an independent assessment of a program's risk management, conducted by a qualified reviewer who is otherwise unaffiliated with the organization. The review compares written systems with actual practice, produces a written report of findings and prioritized recommendations, and the organization responds to the report in writing and by acting on it. Accrediting bodies such as AEE treat these reviews as distinct from accreditation site visits, licensing inspections, and audits required for funding.
What is the difference between a program review and an incident investigation?
An incident investigation responds to a specific event or allegation and establishes what happened, under its own standards of proof and, in some jurisdictions, licensing requirements. A program review assesses the entire risk management system or specific aspects of it. A program review may also examine past incidents to learn from trends and as evidence of how the organization learns.
What does a reviewer need access to?
Documents, people, and normal operations. Organizations share policies, plans, training materials, incident records, and organizational charts, and they interview personnel at all levels, from field staff and participants to executives, and even external personnel from vendors and partners. For on-site reviews, the reviewer also observes regular program activities, such as planning and preparation, field activities, staff, student and parent briefings, debriefs, and transitions.
How should an organization prepare for an external review?
Assign an internal sponsor to liaise with the reviewer(s), set up meetings, and prepare a schedule. Agree on the scope of the review and determine with whom the findings and recommendations will be shared before the review starts. Gather documents and make copies, or otherwise make them available to the reviewers (always retain custody of the originals), and schedule a site visit during normal, representative operations. Before the site visit, inform the staff of the review and its purpose, and ask them to be candid. Scripted answers are not helpful, and a review isn’t supposed to catch anyone out. The best way to prep staff is to be transparent about what the organization wants to learn.
Who sees the review report?
The organization and the reviewer agree on distribution early, before findings are determined. The personnel who participated in the review typically read the draft and can respond before it's final. The organization decides who attends the closing presentation, which often includes board members, executives, and program leadership.
