Undergraduate Program Review Audit Committee 
Quality Assurance, Undergraduate Program Review and Audits

1.  Quality Assurance

Purposes of Quality Assurance

Quality assurance measures have arisen to ensure and enhance the quality of academic programs. The internal academic approval process establishes standards for new programs, and universities have long been determined to ensure that existing programs continue to meet those standards. Nevertheless, it is the quality improvement dimension, rather than simply confirmation of meeting minimum standards, that offers the greatest benefit and interest to academics. After all, even the best of programs can be made even better, and continuous quality improvement is a distinguishing feature of the most renowned programs and institutions in the higher education community. 

Key Components of Quality Assurance Processes

Experience worldwide demonstrates that effective quality assurance processes (such as program reviews) contain the following three key components:

i. Self-study.  Program review is not something done only by outsiders to the program. It rests in the first instance on the self-appraisal by the members of the program. Legitimate self-appraisal processes are guided by the mission and strategic plan of the university and department and the goals of the program under review. A necessary part of any self-appraisal (or external assessment) is the collection, presentation and analysis of relevant data about the program. Experience has shown that nothing is more important to the successful review of any program than the self-appraisal by its members. However, nothing is more variable in its quality and effectiveness than that self-appraisal. Ineffective self-appraisals are descriptive rather than reflective, analytical, self-critical and evaluative; loaded with data that is presented rather than analyzed; defensive or self-justifying rather than aimed at quality improvement; prepared in a formulaic or mechanical way, as if completing a checklist rather than demonstrating that the members of the program are sensitive to and thinking about the context, mission and objectives of the program; and written by the Chair without evidence of buy-in (or sometimes even knowledge) of faculty and students rather than resulting from a participatory self-critical process. Effective self-appraisals invariably lead to quality improvement, and make the reports of external reviewers more useful to that purpose.

ii. Peer review.  External quality assessment necessarily involves external review. Effective quality assurance processes involve one or more external reviewers who report on a site visit.  Some program review processes also use an internal reviewer, from the institution but from a different program.  Such internal reviewers normally serve as members of the site visit team, and thus contribute to the joint review team’s report, but may in some processes visit and report separately.

iii. Judgment/decision. Effective quality assurance reaches closure. A decision is made by the responsible body as to the quality of the program and any needed remedial actions. It is important that quality improvements indeed occur, rather than linger as unfulfilled recommendations in some limbo of wishful thinking. There needs to be a mechanism for action to determine which of the recommendations arising from the self-study and the reviewers’ reports will be implemented, to what extent, by whom, and on what schedule.

Academic Audits

Under a system of academic audits, universities undertake to review the academic quality of their academic programs according to standards established by the external body responsible for the audit. The audit does not assess the quality of those programs selected for the audit – that review was previously conducted by the university. Nor does the audit second-guess the university review, or serve as a court of appeal for program members who hope for a better deal. Rather the audit tests for the compliance of the university’s program review policies with the standards, schedules, procedures and other aspects enunciated by the external body, and for whether the sample reviews examined by the auditors were actually implemented fully in accordance with the letter and spirit of the university’s policy. Thus, the audit is concerned with process rather than direct assessment of academic quality.  Its contribution to public accountability rests upon its demonstration that the quality of every program in the university is regularly reviewed according to transparent and sound procedures and standards that are verified by external audit.  Its contribution to quality improvement rests upon the cyclical reviews of each individual program, and upon the logic that an improvement in quality assurance processes and attention of program members to quality issues will lead to improved academic outcomes.

2.  Ontario Undergraduate Program Reviews
 
Background

In October 1996, the Council of Ontario Universities (COU) established guidelines for the conduct of periodic quality reviews of undergraduate programs, and committed to a system of regular audits of the Ontario universities’ policies and procedures for these reviews. The body responsible for the conduct of the audits is the Ontario Council of Academic Vice-Presidents (OCAV) operating through its Undergraduate Program Review Audit Committee (UPRAC). The UPRAC audit guidelines are designed to satisfy the need for public accountability identified in the report of the Task Force on University Accountability (the Broadhurst Report) and by the former Ontario Council on University Affairs in its Advisory Memorandum, OCUA 93-VI “Academic Audit Review.” At the same time, the audit process preserves the principles of university self-regulation and autonomy.

Expectations

Reviews of programs and departments are not new to Ontario universities. There is, for example, at many universities a long tradition of conducting a review at the end of the term of a chair.  What is new with the Undergraduate Program Reviews is the expectation that cyclical reviews of the quality of all undergraduate programs become part of the already on-going process by which universities plan for the best use of their resources, engage in self-renewal in order to remain in the forefront of the knowledge expansion and ensure that they are offering undergraduates programs of the highest quality.

 
3. UPRAC Audits

UPRAC audits provide institutions with valuable and objective advice on how they might take steps to improve their processes, and they assure government and the citizens of Ontario that undergraduate program quality is being maintained and enhanced.

Auditors

The audits are conducted independently by three auditors who are appointed by UPRAC. They are all senior university professors, mostly retired, who have had major administrative responsibility in their institutions and have demonstrated strong interest in the development, operation and quality of undergraduate programs.

Cycle of Audits

The first audits were conducted in 1997. With two or, more usually, three audits each year, over the course of seven years the review processes of all Ontario universities are audited after which the cycle is repeated.

UPRAC Audit Guidelines

The UPRAC Review and Audit Guidelines specify the two tests that the auditors must apply. First, they are mandated to examine the conformity of the institution’s review process to the model framework for program reviews laid down in Guidelines. Then they must consider how well institutional procedures and practices conform to the institution’s own review policy. Guidelines state that a review must include a thorough and analytical self-evaluation by the program’s faculty on the strengths and weaknesses of their program; students are to participate in this self-study. The review must also require an assessment of the program by at least one external consultant. A further step in the review process must be an appraisal by faculty who are not from the program under review. The review must conclude with an action plan to deal with the recommendations arising from the review and a summary report must be presented to the senate and board of governors. The audit of the process for the implementation of new programs follows an analogous route.       

Conducting an Audit

An audit entails a careful and detailed review and assessment of the written documentation provided by the university followed by a site visit, and finally, a written report. During the visit the auditors meet the academic vice-president, deans and directors, representatives (including students) from the programs whose reviews were selected for audit and members of the senate committee responsible for curricular and academic standards. In their reports, besides providing commentary, the auditors make recommendations and suggestions designed to improve the processes that institutions have put in place for the reviews of their undergraduate programs. Copies of the auditors’ reports are sent to the Minister of Training, Colleges and Universities and to all Ontario universities. One year after the publication of the report, the institution is required to indicate through the auditors to OCAV and the Minister how it has responded to the auditors’ recommendations.

Click on the link below to obtain the complete text of the UPRAC Review and Audit Guildelines in PDF Format.

UPRAC Review and Audit Guidelines  You will need Adobe Acrobat Reader to view this file.