Happy New Year!!
2013 came to a close with bitter cold and concerns about future of many academic programs at the U of S.
This issue of VOX is another submission from Professor Eric Howe. It follows-up what he previously wrote and presents his thoughts on TransformUS now that the task forces have released their reports.
As we embark on 2014, we on the VOX Editorial Board welcome your comments on TransformUS and other issues. What other weaknesses—or strengths—in the ranking methodology were there in addition to those analyzed by Professor Howe? Are there criteria that should have been considered but were not? Grandiose Administrative initiatives like TransformUS are not unique to U of S. If you know of comparable experiences at other institutions, like York University for example, and would like to share your views, we would be delighted to hear from you.
Whether it is in response to what Professor Howe has written or something else, we encourage you to put pen to paper and provide us with your thoughts on issues impacting you.
If you have something you would like to contribute to VOX, information about making submissions can be found on the last page of this publication.
The Editors
TransformUS: Even Worse Than Predicted
Professor Eric Howe
Department of Economics
University of Saskatchewan
My paper “The Emperor Isn’t Wearing Any Clothes: Intellectually Bankrupt Academic Prioritization” (The Naked Emperor subsequently) presented a number of extremely negative predictions about the outcome of TransformUS. Those predictions were based on the methodology being employed—one developed by Robert Dickeson and presented in his book Prioritizing Academic Programs and Services. TransformUS has now released its prioritization report (The Report subsequently). The results are worse than I predicted. TransformUS will cause irreparable damage to our university and province.
In these litigious times it should be stressed that the following is my interpretation. Just as was the case with The Naked Emperor, I do not deny that others may have different interpretations. Again, too, this paper will be limited to examining the prioritization of academic programs; services will not be included.
It should be noted at the outset that our faculty colleagues who were tied up doing TransformUS for eight months worked hard. They agreed to add TransformUS to their already over-full schedules, so they consequently had to make sacrifices in other areas such as research, scholarly work, teaching, staying current in their discipline, public service, practice of professional skills, grant applications, and living their lives. Their efforts on TransformUS fell short, but it is important to note that the fault lies not with the faculty involved but with the members of Central Administration who selected Dickeson’s methodology. From the moment that Dickeson’s methodology was selected, the completion of TransformUS was a fool’s errand.
There are many programs at the UofS, so there was little time despite the long hours of work from the Task Force. The Report says that 90 hours were devoted to the review period for 485 programs (The Report, pp. 1 and 10), so the review period allowed an average of eleven minutes and 8 seconds per program. Suppose conservatively that a quarter of the review period was given over to other matters—writing The Report, tallying votes, organizing, breaks, guest presentations, eating meals, and recording comments—and that allows 8 minutes and 21 seconds per program. Each program was assessed on the basis of ten criteria, allowing 50 seconds per criterion per program. With twenty members, each Task Force member got to contribute, on average, 25 seconds of discussion per program, or 2.5 seconds per criterion per program. Even when The Report assigns an important program to an indefensibly counter-intuitive quintile, it is difficult to blame the members of the Task Force because 8 minutes and 21 seconds (50 seconds per criterion) is not much time for an informed decision about something as complicated as a program at a research university, nor is 25 seconds (2.5 seconds per criterion) much time for an individual to affect the decision. Again, the fault for TransformUS lies not with the faculty involved but with Dickeson’s methodology.
I resigned from the Academic Task Force of TransformUS at the end of October. It had become clear that the biases in Dickeson’s methodology were not going to be corrected. It would have been professionally unethical for an economist to continue in those circumstances: economists study how to rationally make difficult choices and TransformUS isn’t how it is done. TransformUS is to rational choice what 1+1=3 is to mathematics.
One of the alarming aspects of TransformUS is the ease with which its terms can be redefined and abused. Consistent with its mandate, The Report states that quintile 5 contains programs that are “candidates for phasing out, subject to further review.” However, quintile 5 was redefined in a UofS announcement by the Academic and Finance Vice Presidents on November 5—even before the Task Force had completed its report—to consist of programs that were “recommended for elimination.”
1. An open, transparent process?
Much of the most interesting information about TransformUS has not been released. We were instructed in the initial full-day meetings of the Task Force that we should not even keep notes since they might be accessible subsequently.
Indeed, the specific event which precipitated my resignation from the Task Force was a document which was distributed for signatures by the members of the Task Force at the end of October which included an explicit agreement to destroy all notes and spreadsheets upon completion of The Report. That for a process which was to be open and transparent?
What has been left out of The Report is much of the information which would allow an interested reader to assess the validity of the quintile assignments.
For example, consider that many of the programs placed in quintile 5 in The Report weren’t placed there when the programs were initially scored. The TransformUS process involved initially scoring each of the programs based on the information available in the templates. Those initial scores involved voting and required a 75% majority. The votes were usually preceded by a discussion of the program. Even for programs that were not discussed, the members of the Task Force were asked if they wanted to discuss the program prior to the vote. Subsequently, however, the Task Force moved some programs in a second pass, still using the same templates. Why?
The Report explains the second pass in this way:
A series of subsequent meetings, including a day-long Saturday meeting, was devoted to revisiting the placement of the programs. In the second pass, the programs were considered in relation to other programs within the unit. This enabled task force members to consider the programs more holistically against the background of the unit, to identify any discrepancies between related templates, and to assure ourselves that a consistent approach to evaluation had been used. This second review also gave the task force a chance to consider whether there were additional general themes that should be identified in the report. (The Report, p. 10)
Note that there is nothing in this explanation that would suggest a substantial inequality between number of programs the second pass moved up to a higher quintile and those moved down to a lower.
From the scoring prior to my resignation from TransformUS on October 25, I have the initial quintile assignments for the 375 programs which had been assigned to that point. These can be compared to the final quintile assignments in The Report to examine the effect of the second pass.
Of the 375 programs, 5 were initially placed in quintile 3 but were moved downward to quintile 5 on the second pass; 17 of the programs were initially placed in quintile 4 but were moved downward to quintile 5 on the second pass. In fact, over one in four of the programs assigned to quintile 5 in The Report were initially assigned to higher quintiles. Over one in six of the programs assigned to quintile 4 were initially assigned to higher quintiles. Over one in four of the programs initially assigned to quintile 4 were moved to quintile 5. Of the 375 programs, 2 were moved up and 36 were moved down.
What motivated this movement from the initial assignment to the final?
I believe that the downward revisions in the second pass were driven by Dickeson’s arbitrary dictum that 20% of the programs must go into the “candidates for elimination” category, quintile 5. The first pass placed 15.5% of the 375 programs in quintile 5. To get closer to 20%, some programs were moved down to quintile 5 in the second pass. That left less than 20% in quintile 4, so other programs were also moved down to quintile 4.
Anyone finding their program in quintile 5 may reasonably be left with the question of what that means. Over a quarter of such programs were initially scored to be better than that. It would be very interesting to know how a good labour lawyer would view this situation when it results in faculty being fired. Programs were scored higher but had their scores lowered to put the dictated number of programs in quintile 5. One wonders what Central Administration’s justification would be: “we wanted to fire more faculty than seemed appropriate from a review of the programs.”
2. The size bias
One of the biases in the methodology used by TransformUS is a bias against small programs. Incredibly, The Report dismisses small programs as “boutique” programs (p. 20) and questions whether they should be allowed to continue to exist at the UofS.
It is worth emphasizing what the logically correct consideration is—the one that TransformUS should have used. Obviously, the UofS should seek to maximize the benefit (broadly defined) from spending its budget, so the logically correct evaluation of a program is its benefit per dollar of cost.
In that context, a small program with a small cost can easily have a larger benefit per dollar of cost than its larger cousins. Rather than arbitrarily dismissing small programs as “boutique” and assigning them to quintile 5, the larger programs might just as well have been dismissed as “resource hogs” and assigned to quintile 5.
But they weren’t. As explained in The Naked Emperor, the methodology of TransformUS made it bad to be small and good to be large. Regardless of quality. Empirically, that can be documented in any number of ways. For example, there are 363 programs at the UofS with a positive number of students during the three years of data collected by TransformUS. (For this, take the student headcounts from the templates, summing over the three years, drop the student headcounts which were irregularly included for research programs, and count the number of programs with a positive number of students.) The median headcount per program is 53. There are 180 programs with more than the median of 53 students, and 180 with fewer. The set of larger programs have a median quintile assignment of 2 and make up 90% of the teaching programs which were assigned to quintile 1. The set of smaller programs have a median quintile assignment of 4 and make up 88% of the teaching programs which were assigned to the quintile 5.
Defenders of TransformUS might claim that this just reflects the low quality of small programs. However, The Report seems to contradict that view by noting that the group of small programs contains those for which the “rigour and specific focus … embody important academic values” (The Report, p. 20). The Report continues, saying that the evaluation of the future of the small elite programs at the UofS is beyond their mandate. However, The Report then goes on to evaluate them anyway.
3. The bias against Aboriginal programs
Another bias discussed in The Naked Emperor is one against Aboriginal programs. Aboriginal people will make up the majority of Saskatchewan’s population in the foreseeable future. The average educational level of Aboriginal people—although it is increasing—continues to fall further behind that of Nonaboriginal people—which is increasing faster. Consequently, it is vitally important provincially that Aboriginal educational initiatives are expanded. For this reason, it would be expected that Aboriginal programs would be prominent in quintile 1—the quintile for “candidates for enhanced resources.”
Indeed, The Report claims that Aboriginal programs were placed in “The full range of quintiles” (The Report, p. 19). So how many Aboriginal programs are in quintile 1? Zero. That number again, ZERO!
There are three programs in Quintile 1 which include the word “Aboriginal” in their detailed program list in The Report: the 4 year BSc in Toxicology (4.6% of the student headcount is Aboriginal), the 4 year Honours BA in English (4.1%), and the research program in Physics (0 students whether Aboriginal or Nonaboriginal). Of the 482 programs at the UofS scored in The Report, the 4 year BSc in Toxicology is ranked 81st in its number of Aboriginal students, the 4 year Honours BA in English is 104th, and the research program in Physics is tied for last (The Report, pp. 43-45 together with data from the program templates). Perhaps it is best left to the reader to decide whether these three constitute Aboriginal programs. The detailed program list for Toxicology states there are “plans for incorporation of Aboriginal students.” I have absolutely no intention of denigrating the efforts by the faculty in any of these three programs, but note that many programs on campus tried to spin the information in their templates by stating that they have hopes for future enrollment of Aboriginal students. Whether the hoped for enrollment happens is another matter.
So, how did actual Aboriginal programs do in TransformUS? Here the story is mixed, though many of the Aboriginal programs fared badly. The Native Law Centre was assigned to quintile 4 as were the 4-year Honours BA in Aboriginal Public Administration, the 4-year Honours BA in Native Studies, the 4-year BA in Aboriginal Public Administration, the 3-year BA in Northern Studies, the International Centre for Northern Governance and Development, and the 4-year BA in Northern Studies. The Honours BA in Northern Studies was assigned to quintile 5.
Defenders of TransformUS would perhaps argue that the poor showing of these Aboriginal programs is a reflection of their quality—and take no account that, as shown in The Naked Emperor, the scoring methodology is biased. That argument would be reminiscent of the assertion of segregationists in the American South during Black voter-registration drives that Blacks were illiterate—as supposedly demonstrated by the use of tests which were biased against Blacks.
4. The bias against low-cost programs
One of the laments expressed by members of the Academic Task Force during its meetings was the low quality of the cost data in the templates, and I suspect that the low quality of the data in general partly accounts for the tentative phrasing of quintile 5 (“candidates for phasing out, subject to further review”). The significance of the phrase “subject to further review” deserves some comment. There were several meetings of the Task Force where it was asserted that Central Administration would use “better” cost data before deciding which programs in quintile 5 should actually be eliminated.
Obviously, however, if better cost data exists then the better data should have been used by TransformUS. As shown in The Naked Emperor, the data on program cost used by TransformUS were inherently inaccurate. It would be fascinating to study how the results would have been different if accurate cost data had been utilized in the prioritization. That is not possible since in fact no one knows what the academic programs at the UofS cost.
5. How did the colleges do?
The Report includes scores for 482 programs. How did these work out for the academic units? Table 1 shows the median score for programs by unit, ordered from the worst to the best. (Ties in the ordering of the median were broken using the average; for example programs in both the Colleges of Nursing and of Engineering got assigned to quintiles with a median of 4, but the average quintile for Engineering was somewhat better than that for Nursing.)
Table 1. Median Quintile Assignments by Academic Unit
| Unit | Median quintile |
| College of Graduate Studies and Research | 5 |
| Library | 4 |
| College of Nursing | 4 |
| School of Public Health | 4 |
| College of Medicine | 4 |
| College of Engineering | 4 |
| College of Kinesiology | 3 |
| College of Arts and Science | 3 |
| Western College of Veterinary Medicine | 3 |
| Edwards School of Business | 3 |
| College of Dentistry | 3 |
| College of Law | 3 |
| School of Physical Therapy | 3 |
| College of Agriculture and Bioresources | 2 |
| University-wide Programs | 2 |
| Johnson-Shoyama Graduate School of Public Policy | 2 |
| College of Education | 2 |
| School of Environment and Sustainability | 2 |
| College of Pharmacy and Nutrition | 2 |
Of the principal teaching colleges on campus, the worst one according to The Report is Nursing with a median quintile of 4 and the worst average. The median program on campus is Edwards with a median quintile of 3. The best is Pharmacy and Nutrition with a median quintile of 2 and the best average. If the reader believes that the ranking in Table 1 is based on an appropriate evaluation of appropriate information, then I have some really good land in Florida you will want to buy.
6. Meewasin Valley Community College
There is a straightforward resolution available for TransformUS. This year’s provincial Throne Speech included a pledge to develop a third university in Saskatchewan. The new university will be built from the structure that makes up SIAST. Moving in that direction, SIAST has been renamed Saskatchewan Polytechnic. Suppose that Central Administration accepts the conclusions of The Report and decides that there is no place for nursing education at the UofS. In that event, it would seem reasonable to transfer The College of Nursing to Saskatchewan Polytechnic. Given the significance of Nurses and Nursing provincially, the College’s budget would have to be transferred also. Similar transfers could be made for other colleges which are upset with their ranking or fear that they may be gutted by the arbitrary elimination of important programs.
At the conclusion of TransformUS, with the UofS no longer responding to the urgent social needs of the province, having eliminated its elite “boutique” programs, having transferred away or gutted many of its colleges, it would be appropriate to rename the UofS too. Meewasin Valley Community College would seem to be appropriate.
About VOX
VOX is a forum for the expression of opinions of members of the USFA on topics of general interest to the membership. Submissions to be considered for publication should be sent to the USFA office to the attention of the VOX Editorial Board or they may be sent by email to usfa@usaskfaculty.ca, or to any member of the Editorial Board.
VOX is sponsored by the University of Saskatchewan Faculty Association and is published by an independent Editorial Board, whose members are:
George Khachatourians
Brian Pratt
Howard Woodhouse
VOX may appear up to eight times a year, depending on the volume of submissions. All articles remain the property of the authors, and permission to reprint them should be obtained directly from them. All opinions ex-pressed in VOX are those of the authors, and do not necessarily represent the position of the USFA or the Editorial Board.