The OCQAS engages peer-reviewers (auditors) to conduct external quality assurance reviews under the College Quality Assurance Audit Process (CQAAP). The use of external auditors to conduct institutional audits is established in the Quality Assurance Audit Policy 18, which sets the expectation that the OCQAS appoints and trains peer reviewers and ensures that audit panels are composed of qualified individuals of high calibre.

Operational procedures are further defined in Quality Assurance Audit Procedure 18-1 and the CQAAP Guidelines and Framework, which specify that the OCQAS is responsible for the selection, training, and appointment of audit panel members. All auditors must be external to the audited college and free from real or perceived conflicts of interest (see 11.3). Prior to the submission of the audit package, the OCQAS provides the college with the proposed list of potential auditors. Colleges are required to review this list and identify any real, perceived, or potential conflicts of interest, as well as any other concerns regarding the suitability of proposed auditors. Following this review, the OCQAS confirms the final audit panel. This process promotes transparency and confidence in the appointment of peer reviewers.

The OCQAS maintains a roster of qualified auditors (Auditor Biographies), recruited through a publicly accessible process outlined on the OCQAS website. The Auditor Job Description defines the role, responsibilities, qualifications, and conditions of engagement. At the beginning of each academic year, the OCQAS contacts individuals on the roster to confirm (Example – Auditor Confirmation of Commitment) their continued interest and availability. Appointments to specific audit panels are made by the OCQAS based on expertise, availability, and the needs of the audit, ensuring appropriate panel composition and balance of experience.

Training of peer reviewers is described in the CQAAP Guidelines and Framework (pp. 15-17) document. All audit panel members participate in an orientation prior to conducting reviews. This training covers evaluation procedures, the triangulation of evidence, and expectations regarding professional conduct and impartiality.

Building on the commendation received in the 2021 INQAAHE review for its rigorous process for selecting peer reviewers, the OCQAS has embedded and strengthened a structured, competency-based selection and assessment model. Prospective auditors are required to complete a formal training and assessment process prior to appointment to an audit panel.

This process includes:

• The review of a college audit package by prospective auditors
• The submission of an initial independent assessment
• The evaluation of inter-rater reliability by the OCQAS using Fleiss’s kappa to assess consistency of judgments
• Targeted training sessions to address inconsistencies and calibrate understanding of the Standards and Requirements
• The submission of a second assessment of the same audit package
• A re-evaluation using the same statistical measure to confirm improved alignment

Based on the results of this process, the OCQAS selects audit panel members from the pool of trained candidates. This approach ensures that reviewers demonstrate consistent and reliable application of the criteria, supporting fairness, consistency, and evidence-based decision-making.

Audit panel chairs are appointed from the existing pool of experienced auditors. Individuals normally first serve as panel members and subsequently as co-chairs before being appointed as chairs, ensuring they have developed sufficient experience with the evaluation process, standards, and reporting expectations. The co-chair role also serves as a mentorship mechanism, with co-chairs supporting and mentoring new chairs. This staged approach strengthens leadership capacity, promotes consistency across panels, and supports the effective conduct of audits.

The OCQAS provides compensation to audit panel members, reflecting its commitment to attracting and retaining highly qualified reviewers and supporting accountability for the professional and timely performance of their responsibilities.

The quality assurance agency ensures that appointed auditors have appropriate academic background and professional experience through explicit qualification requirements set out in the Auditor Job Description. Required qualifications include recent senior-level experience in the post-secondary education sector or in roles collaborating across institutional units; knowledge of post-secondary education policies, procedures, and practices; the ability to collect and verify audit evidence; strong analytical and report-writing skills; and demonstrated objectivity and impartiality.

Additional requirements apply to Audit Panel Chairs, including demonstrated audit experience and being external to the Ontario Colleges of Applied Arts and Technology system.

The OCQAS also maintains the currency of auditor information through an annual roster update process. At the beginning of each academic year, auditors are contacted and asked to confirm their continued interest and availability (Example – Auditor Confirmation of Commitment), and to provide any updates to their professional information and experience. This process supports the appointment of auditors based on current qualifications and relevant expertise.

For French-language audits, the OCQAS appoints French-speaking auditors. These auditors participate in the same orientation and training processes as all other auditors, and the audits are led and supported by a French-speaking OCQAS staff member to ensure effective communication, consistency in the application of standards, and appropriate support throughout the review process.

The quality assurance agency applies a formal Conflict of Interest Policy 07 that explicitly applies to those conducting audits. The policy defines actual, potential, and perceived conflicts of interest and establishes mandatory disclosure and management procedures.

Auditors are prohibited from auditing their own college or a college where they were employed within the previous five years and must disclose in writing any past or current connections prior to accepting an audit engagement.

Prior to the submission of the audit package, the OCQAS provides the college with the proposed list of potential auditors (Email Example – List of Potential Panel Auditors). Colleges are required to review this list and identify any real, perceived, or potential conflicts of interest, as well as any other concerns regarding the suitability of proposed auditors.

The quality assurance agency provides structured training to ensure auditors understand the agency’s quality assurance procedures and standards before participating in audit activities. Training requirements are documented through the Auditor Information Session and related training materials. The information session introduces the CQAAP process, Standards and Requirements, evidence-based evaluation methods, and expectations related to professional conduct, confidentiality, independence, and communication.

As part of the mandatory training process (CQAAP Guidelines and Framework, pp.15-17), auditors complete an assessment of a college self-study to support the calibration of reviewer judgments and reinforce a shared understanding of the Standards and Requirements. This exercise promotes the consistent application of evaluation criteria across review teams. Prior to participating in these activities, auditors are required to sign a confidentiality agreement.

When changes to the CQAAP process, Standards and Requirements, or related guidance are under review, auditors are included in the consultation process and are therefore aware of the feedback and issues being considered. Any resulting changes, clarifications, or questions are then addressed through the mandatory training session.

The quality assurance agency monitors and evaluates auditor participation and performance through established operational practices. At the end of each review/audit process, the OCQAS distributes a survey to the colleges (College CQAAP Feedback Audits 2024-2025) and auditors (Audit Panel CQAAP Feedback Survey Audits 2024-2025) who participated in audits. The survey collects feedback on audit processes and information related to audit team functioning.

The results of these surveys are used to inform process improvements and to identify strengths and areas for enhancement in the conduct of audits.

In addition to survey-based feedback, the OCQAS employs a continuous, practice-based approach to monitoring reviewer performance. The OCQAS staff participate in all audit activities and are therefore able to observe reviewer engagement, application of standards, and team dynamics directly. This enables real-time guidance, as well as ongoing mentoring and coaching of auditors throughout the audit process. Where appropriate, feedback is provided directly to auditors, including through confidential discussions led by the Executive Director.