The HEA has published its impact analysis of the Programme for Research in Third Level Institutions (h/t Colm Harmon). It is good that government agencies are increasingly open to such evaluation.
From the executive summary, we learn that PRTLI centres and initiatives had a budget of 1.7 bln euro, with 1.2 bln from the Exchequer. 1,700 people were employed, at an exchequer cost of 700,000 euro per job. In 1998, 2,400 full-time academics were employed at the universities and ITs. In 2008, there were 6,200 FTEs.
The commercial impact (a mix of turnover, investment, and cost savings) was 750 million euro, with 1,300 jobs created (or 600,000 euro per job). For the next five years, a further impact of 1.1 bln euro is projected.
In the foreword, John Hennessy (the HEA chairperson) puts on a brave face and lists all the benefits that were not quantified.
Intrigued by the numbers (and their precision; above numbers are rounded) in the executive summary, I read on expecting to find tables and tables with detailed data that would tell me who publishes and who gets cited, which disciplines create economic value, and what universities are motors of development. Unfortunately, such data is not available. The data, by the way, were gathered by questionnaire — that is, companies were asked how many people they additionally employed because of the PRTLI.
Some evaluation is better than no evaluation, but I think that a 1.2 bln euro investment warrants more analysis than what is offered by the HEA.
9 replies on “The impact of the PRTLI”
It would be very interesting to get a breakdown of grant recipients by nationality also. Both via salaries, and grants, we seem to spend a lot on research, yet some areas still have rather a lot of Irish people….which suggests grantfare, rather than global excellence.
Thanks Richard. Overall I share your view that this evaluation was a wasted opportunity (and I say that as someone who had to figure out what to put into the forms!!).
I can only discuss what we (UCD Geary Inst) claimed and indeed didn’t claim.
From the outset it remains very very disappointing that we do not wish to examine the academic output in these exercises, as you note. I am very pleased to note the reference to Jim Heckman whose role at UCD indeed was funded by PRTLI (and of course is now funded by his own ERC grant!), but disappointed at how that was basically it as far as the Institute goes – good as this is, I think we have more to shout about (and dare I say it more than many other PRTLI investments!). To use accounting parlance, the HEA seem to not wish to value the ‘goodwill’ or the intangible asset.
As an economic evaluation it seems amazing not to see reference to the elephant in the room which is the missing counterfactual of what would have happened without PRTLI. That again is a hard thing to do but it does need some effort by evaluators. Simply listing how many papers are produced before and after PRTLI is, as you know, meaningless.
Also as an economic evaluation some calculation of the net costs would have been sensible but this perhaps would work against the Sciences where capital costs outweigh the recurrent (mainly human capital) costs. For Geary the vast bulk of what we received in two PRTLI rounds (1 & 3) was spent on hiring folks who of course pay taxes, spend in the economy etc etc. So the cost per job is high in the evaluation document but perhaps higher than it should have appeared given it includes capital spend and it not net of payroll costs recaptured (incidentally we did an exercise of this nature – mainly because I am married to an accountant! – so it says a lot for our influence).
The evaluation makes light also of the ability to use PRTLI funding to leverage additional (research) funding. Again this may be because the vast bulk of science funding outside of PRTLI is also exchequer funded (SFI/HRB etc) – for Geary the ratio of non exchequer to exchequer was 2.5 to 1 over the PRTLI3 period, including the PRTLI grant (ok ok – cue lots of comments that EU funding is indirectly exchequer but I am taking the goal when I can get it). And again the bulk of that money was spent on (a) uni overheads, and (b) salaries which are spent in the economy, payroll taxes etc etc.
I think PRTLI was a good thing but mainly for forcing some upping of the game and, in our case, bringing on some wonderful smart young folks who have gone on to bigger, better things in academic life and outside. I think these exercises miss that point completely which is a pity.
But upping the game may not be a good enough reason to continue the Programme – that needs debate. Also at some point PRTLI perhaps became a beast that had to be fed by subsequent rounds of….PRTLI (a point I hope not to be true about Geary in that we went as fast as possible for big EU/US/Philanthropic funding). Moreover PRTLI has – as the conclusions hints – become a major tool in the strategic planning toolbox of the HEA in that phrases like ‘national priorities’ and a requirements for collaboration have popped up in funding calls which, for better or worse, is somewhat unusual.
Unfortunately we are still not getting the answer to the question of the ‘knowledge economy’ debate. Your colleagues at the ESRI are far more attuned to that literature than I am, and this evaluation has nothing approximating a model of that nature. If data is missing we need to start gathering it properly. Personally I think all that an exercise like PRTLI can do is create an environment for this sort of research led economy stuff. It can’t be responsible for it. That is the folly of these evaluation exercises.
This is another egregious example of the waste of public money by a state agency.
Guesswork would be a polite term for the estimates and the biggest assumed commercial impact is ahead.
The content is dense and badly presented because it was the type of assignment where you have to work back from the desired result.
No project apparently failed or was abandoned and how reliable is a post hoc estimate from an MNC of the impact of a project that may be just a contribution to a bigger international one?
On Colm Harmon’s point that the cost of labour maybe higher than it should have been, I would expect that is more than offset by ignoring IDA Ireland support provided to MNCs in collaborations with the public sector. So the taxpayer may well be funding 75% of the cost.
The authors make the point several times on how important the R&D infrastructure support is for MNCs but the reader is left no wiser as to what type of research pharmaceutical companies are doing in Ireland at a time when big pharma is cutting R&D spend because of poor outcomes.
Pfizer for example is shedding about 2,000 researchers from its half-century old facility in Sandwich, Kent where it has had up to 2,400 employed.
I wrote a commentary on the report last week:
To understand the ‘knowledge economy’ fantasyworld, read Science Foundation Ireland’s recently published annual report with its effusions on citations, international linkups, collaborations, case studies etc.
Then read its separate census of outcomes.
[…] “The HEA has published its impact analysis of the Programme for Research in Third Level Institutions h/t Colm Harmon. It is good that government agencies are increasingly open to such evaluation …” (more) […]
The ESRI bid for this project but was not successful because of our apparent lack of experience in evaluating public investment programmes
One would think, though, that the one of the 750-odd economists and business scholars in Irish universities could have done a better job.
“It is good that government agencies are increasingly open to such evaluation.”
If by such evalaution Richard means in house evaluations then I doubt if it is any good at all, as indeed his subsequent strictures and those of others suggest. Indeed, getting agencies to hire independent consultants would not be much better. Consultants quickly sense which side of the bread is buttered. In fact such a large evaluation can only usefully done by a totally independent and sceptically inclined entity. The CAG might fit that bill. Better yet would be an institution located abroad.
Reading my copy of Trinity Today (October 2011) I spotted the claim that “in neuroscience research Trinity College Dublin is in the top 0.01 per cent internationally”.
I wonder if the maths department vetted that claim!
Ah, Diogenes, welcome. We have waited some time for you to join us on this board. Your strictures based on ‘he who pays the piper calls the tune’ are timely, but, being the original ‘cosmopolitan’ I’m surprised you reckon foreign consultants would tell the truth.
It is very simple economics which most economists here seem to ignore. There is no supply of genuinely independent, objective consulting services in these areas because there is no demand offering a range of prices for different levels of quantity and quality. But there is no shortage of demand for, or supply of, consulting services that present the answer those commissioning the work require.
In theory, the Oireachtas should be the body that commissions these independent assessments because it, ultimately, authorises all public expenditure. However, it has allowed itself to be deprived of the powers and resources to commission these assessments. And even if it had the necessary powers and resources, the ‘tyranny of faction’ exercised by the governing faction(s) would ensure that the scope and terms of reference of any assessment would prevent the consultants from doing their job properly. And, in any event, they would be left in no doubt about the specific bounds around their expected results.
All these assessments, policy impact assessments and regulatory impact assessments conducted or commissioned by the ‘government machine’ are just a mixture of some facts, estimates and bullshit in varying combinations depending on the answer that is required. It is time-consuming and totally futile to attempt to disentangle the components.
And, because some externally unquantifiable amount of bullshit is mixed in, the entire output is bullshit.
@ Paul Hunt
Give me something I can make and/or and sell.
What a waste of resource (if I have read correctly).