Autumn 2018 consultation response
REF 2021 consultation on the draft guidance and criteria
| If your response is in relation to specific main panels, please indicate which one(s): |
| Relevant to all |
| We are seeking views during the consultation on both the draft guidance on submissions and the draft panel criteria and working methods. Please select the documents for which you would like to provide a response: |
| Both documents |
Guidance on submissions: Part 1: Overview of the assessment framework
| The guidance is clear in ‘Part 1: Overview of the assessment framework’: |
Please provide any comments on Part 1. (Indicative 300 word limit)
As the learned society representing criminologists we are responding mainly on issues that seek to ensure that criminological submissions are assessed as suitably and equitably as possible across the subpanels in the absence of a criminological UoA. We have consulted our membership through a regular item in our Society ebulletin and concerns and comments outside this key issue have also been incorporated. We welcome the REF commitment to publish data from the consultation and the REF itself and be amenable to FOI requests. We hope that the REF team will relate all future decisions relating to criminology to this data. |
Guidance on submissions: Part 3, Section 2: Research outputs (REF2)
|
Please provide any comments on Part 3, Section 2. (Indicative 300 word limit)
We believe that the discrete summary reports promised for Criminology (para 175-77 of Panel Criteria consultation document) would be best facilitated by having submitting HEIs required to ‘flag-up’ their Criminological submissions, just as they are currently asked to ‘flag’ inter-disciplinary work (para. 169 of the panel criteria document) and forensic science submissions (paras.265-66 of the submissions consultation document) on the justification that…. “Identifying these outputs will help to ensure appropriate assessment (for example, via cross-referral, the use of joint assessors, or existing expertise on the panel in which the outputs are submitted). Additionally, it will enable panels to review the health of UK research in this field, and will generate an outputs quality sub-profile for forensic science across the exercise”. All of this applies equally, if not more so, to Criminology. It should be noted here that Criminology was submitted in 2014 to at least one sub panel in each of the four main Panels from Social Work and Social Policy via Psychology, Psychiatry and Neuroscience and Mathematical Science to History and Communication, Cultural and Media Studies (research conducted by BSC from online REF 2014 databases). Such flagging would enable all Criminology to be included in the promised reports. However, we do not believe that such flagging alone will be sufficient to acknowledge the pervasive influence of the discipline, particularly in the face of potential institutional departmental politics. |
| A glossary of output types and collection formats is set out at Annex K, to provide increased clarity to institutions on categorising types of output for submission. Do you have any comments on the clarity and usefulness of this annex? (Indicative 300 word limit) |
| In Annex K of the submission guidance document (p.121-22), there is guidance that Edited collections can be submitted: ‘To submit a work in this category the editor must have had sole responsibility, or be identified as having made a substantial contribution to the editing, choices for inclusion and underpinning process of investigation.’ We think this an appropriate resolution of the question, but we’re not sure it is well known, appreciated or understood and might need greater promulgation. In REF2014 we don’t think it was satisfactorily handled, but we support this proposal. |
|
Panel criteria and working methods: Part 2: Unit of assessment descriptors
| Do the UOA descriptors provide a clear and appropriate description of the disciplines covered by the UOAs? Please include any suggestions for refining the descriptors and state which UOA(s) you are commenting on. |
| UOA 3: Allied Health Professions, Dentistry, Nursing and Pharmacy
UOA 4: Psychology, Psychiatry and Neuroscience
UOA 10: Mathematical Sciences
UOA 14: Geography and Environmental Studies
UOA 16: Economics and Econometrics
UOA 17: Business and Management Studies
UOA 18: Law
UOA 19: Politics and International Studies
UOA 20: Social Work and Social Policy
UOA 21: Sociology
UOA 26: Modern Languages and Linguistics
UOA 27: English Language and Literature
UOA 28: History
UOA 30: Philosophy
UOA 32: Art and Design: History, Practice and Theory
UOA 34: Communication, Cultural and Media Studies, Library and Information Management
Where relevant, please state which UOA(s) you are commenting on.
Criminology as a discipline taught in every university in the UK and with its own QAA benchmarks deserves to be treated as a discipline in its own right named in one of the UoAs and not be an add on treated as merely an inter-disciplinary entity. We have members who feel disaffected by the non-naming of Criminology in a title within the REF and are not persuaded by the REF argument that the research community was overwhelmingly in favour of the status quo being maintained through the REFs and therefore Criminology could not be named even though it is a flourishing discipline, with a huge expansion in the post 1992s. There were three times more Criminology outputs to the Law Panel alone than Film Studies outputs to the Music, Drama and Performing Arts UoA on which it is now named (against the maintaining the status quo rule) (215/697) forming a far larger percentage of the respective totals. As many units and teams within universities do not submit to the same sub Panel in subsequent exercises, the need for continuity for continuity’s sake is not made out. Naming in one Panel does not preclude submissions still going to other UoAs and that happens throughout the REF exercise even for less obviously meta disciplines than Criminology, such as Chemistry for example. . For that reason, too, we think that the Panels to which Criminology currently submits as a joint submission should be entitled to be allowed multiple submissions from the same institution. We have ticked all UoAs to which case studies tagged as criminology were submitted in REF 2014 |
Panel criteria and working methods: Part 3, Section 1: Submissions
| Please comment on the criteria in ‘Part 3, Section 1: Submissions’, in particular on:- where further clarification is required- where refinements could be made- whether there are areas where more consistency across panels could be achieved- whether there are differences between the disciplines that justify further differentiation between the main panel criteria.Where referring to particular main panels, please state which one(s). (Indicative 300 word limit) |
• Following our earlier representations to the REF team, it does appear that a stronger commitment has been given to standardise the assessment of Criminology submissions across the 3 Panel C subpanels Law, Sociology and Social Work & Social Policy (paras 175-77: Consultation on panel criteria and working methods) by improving working practices and specifically meeting co-ordination and (para.177) the commitment for each ‘relevant sub panel’ to prepare a discrete summary report on the Criminological outputs it receives. We welcome these commitments.
• However, in order to both assess Criminology submissions appropriately and to assist in recognising the Criminological input to the research endeavour, we would question the fact that only one Criminology advisor (the same as last time) is being proposed to work across Main Panel C sub panels 18, 20 and 21 and to assist calibration. Clearly, from the evidence of the 2014 submission, Criminology was widespread in its influence and relevance being submitted to at least one sub panel in all the four main panels and it could be surmised that more than one advisor would be necessary and that a case could be made for at least one from each of what the REF team have identified as the relevant sub panels ie 18, 20 and 21 When HEIs are canvassed about their intended submissions during 2019, it will be possible to assess the need for cross-panel Criminology advisors, and if the expected assessment load merits it, more than one ought to be appointed. Furthermore, the cross-panel Criminology advisors should contribute to any Criminology summary reports.
• In addition, we believe that the guidelines should be more specific about what the ‘discrete summary reports’ on Criminology should contain – i.e. quantitative data about the actual numbers of Criminology submissions (detailing not just outputs but impact case studies separately) and their quality profile. This would make any subsequent decision about the place of Criminology in any future REF or REF equivalent more transparent and data driven. Previously, an informal commitment to provide Criminology summary reports produced variation in detail and thus comparability which did not inspire confidence. In the absence of a Criminology sub Panel which could produce its own report, we would argue that it would be beneficial for the REF team to produce one overarching report on the discipline with data from all the sub panels containing Criminology.
We also support the notion of’ flagging’ submissions as we have outlined in the Submissions consultation qu 8. |
|
Panel criteria and working methods: Part 3, Section 2: Outputs
| Please comment on the criteria in ‘Part 3, Section 2: Outputs’, in particular on:- the proposed criteria for double-weighting outputs in Main Panels C and D, and on whether requests to double-weight books should automatically be accepted- whether Annex C ‘Main Panel D – outputs types and submission guidance’ is helpful and clear – where further clarification is required- where refinements could be made- whether there are areas where more consistency across panels could be achieved- whether there are differences between the disciplines that justify further differentiation between the main panel criteria. Where referring to particular main panels, please state which one(s). (Indicative 300 word limit) |
• In relation to the Double-weighting of books/monographs: Para 239 notes that ‘it is expected that most books, monographs… or longer-form outputs warrant double-weighting although claims will not automatically be accepted.’ There were ambiguities and changes during the assessment process on this issue in 2014, and these need sorting out before the process begins this time. We think the proposal is about right, and the consultation document indicates what might be said in claiming double-weighting in the 100 words of text supporting the output, but the expectation of double-weighting is the default position.
• On a more general point, we would also press for the Panels to all agree the same Criteria – some of the requirements for the different panels are repetitive and could all be included in the most rigorous – i.e. a criteria that authors should specify their contribution if there are more than 10 from Panel A could cover all Panels even those for whom it rarely applies. A split discipline like Criminology needs greater consistency between Panels – Criminology outputs went to at least one UoA on each of the four Panels in 2014. For example, Psychology, Psychiatry and Neuroscience, a UoA in Main Panel A, had more Case Studies tagged as Criminology than did the Sociology UoA. |
|
Panel criteria and working methods: Part 3, Section 3: Impact
| Please comment on the criteria in ‘Part 3, Section 3: Impact’, in particular on:- where further clarification is required- where refinements could be made- whether there are areas where more consistency across panels could be achieved- whether there are differences between the disciplines that justify further differentiation between the main panel criteria. Where referring to particular main panels, please state which one(s). (Indicative 300 word limit) |
| • We would press for the flagging up, and having Criminologists doing, the Criminology impact case studies, in the hope that the promised sub panel coordination might make this organisationally less difficult than in 2014. Impact is anecdotally one of the key strengths of the discipline: in 2014 there was evidence that Criminology was significantly useful in attracting research income -‘The total amount of external research income received by HEIs submitting to SP 20 during the REF period was £74.8 million … Criminology and Criminal justice often provided the main source of research to generate external income’ (Main Panel C Summary Report p75) and in the absence of having a Criminology sub Panel it would be useful for REF to generate more data on this. |
|
Panel criteria and working methods: Part 3, Section 4: Environment
| Please comment on the criteria in ‘Part 3, Section 4: Environment’, in particular on:- whether the difference in section weightings across main panels is sufficiently justified by disciplinary difference (paragraphs 322 and 323)- whether the list of quantitative indicators provided at www.ref.ac.uk is clear and helpful- where further clarification is required- where refinements could be made- whether there are areas where more consistency across panels could be achieved- whether there are differences between the disciplines that justify further differentiation between the main panel criteria. Where referring to particular main panels, please state which one(s). (Indicative 300 word limit) |
| We believe it would be beneficial for institutions submitting Criminology to one of the three relevant sub Panels identified by the REF team to have to specify where their Criminologists sit in their department under the People section of the Environment report and if they are represented at all levels of the unit if that unit replicates the REF structure of having Criminology subsumed by Law, Social Policy or Sociology. |
|
Overall panel criteria and working methods
| Overall, the ‘Panel criteria and working methods’ achieves an appropriate balance between consistency and allowing for discipline-based differences between the panels. |
Please comment on the balance between consistency and allowing for discipline-based differences between the main panels. (Indicative 300 word limit)
In the interests pf consistency and parity, we believe all necessary meetings of sub panel members should be fully financed by the REF team and not by subject associations who have differing abilities to reimburse their members. This also applies to dissemination events by REF panel members. |
BSC Submission to the Consultation on the second Research Excellence Framework
Summer 2018
| Q1. Do you have any comments on the proposal to maintain an overall continuity of approach with REF 2014, as outlined in paragraphs 10 and 23? |
This response to the HEFCE Consultation on the REF process has been drawn up on behalf of the BSC national Executive Committee. Executive Committee members had an opportunity to comment on an initial draft which was also forwarded to BSC members who sat on REF Panels or acted as assessors in 2014 (Panels 20, 22 and 23). A revised version of the response was then placed on the members’ area of the BSC website (for 2 weeks) for comments by BSC members in general.
The BSC are broadly in support of continuity in terms of the REF methodology in order to allow appropriate assessment of development. Continuing changes to the format – the REF has only run once – will make comparisons over time and across REFs difficult making it more complex for institutions to judge progress and thereby invalidating the kinds of conclusions people would hope to draw from the process over time. There is much to be said for letting the tried and tested methodology settle.
The argument for continuity made in paragraph 27 of the consultation document is just as valid here regarding REF Methodology. |
|
| Q2. What comments do you have about the unit of assessment structure in REF 2021? |
Despite us urging continuity in REF methodology, the consultation specifically asks about the representation of Criminology in a future REF structure. The BSC Executive view is that the current REF structure disadvantages and obscures criminological work as no REF Sub-Panel identifies criminology in its title. Criminology submissions chiefly went to 3 separate Sub-Panels in 2014 (C22; C20, and C23). Relatively more criminology outputs went to C22.
Our response on this issue reflects some differences of opinion within Criminology itself. We have surveyed opinion, including consulting our membership as a whole, over a provisional response. Following this, the view of the Executive Committee of the British Society of Criminology is unequivocal in reflecting a clear preference that Criminology be acknowledged in one of the Sub-Panel titles, with C22 being perhaps the most appropriate.
Criminology has the hallmarks of a discipline, including its own QAA benchmarks, indicating both coherence and professional identity. Furthermore, the growth and popularity of the discipline continues unabated with the UK also leading the world in many areas of criminological research. Criminological expertise informs national government on a wide range of issues, featuring strongly in a number of Impact Case Studies. Taken together, this amounts to a strong case for formally recognising Criminology within the REF panel structure.
There have been continuing debates about the best way for the interests of the discipline of criminology to be represented since research assessment exercises began. Following RAE 2008, the BSC fought for better representation of criminologists on the existing Sub-Panels to which Criminology was submitted and, as a result of that year’s nomination procedure, more academics with criminological backgrounds were recruited to the existing Sub-Panels. Some of these senior academics have recommended that future criminological submissions continue to be divided between Panels 20, 22 and 23 on the grounds that this best reflects criminology’s institutional foundations, current strengths and contemporary inter-disciplinary character. They have argued problems of assessing criminological submissions in a non-criminological sub-panel could be addressed by ensuring better coordination and consistency across the designated sub-panels reviewing criminological work including co-location and timing of meetings. Furthermore, cross sub-panel calibration should also be enhanced by the ‘flagging’ of criminology submissions.The BSC Executive has considered these views and still overwhelmingly supports criminology named in a specified Sub-Panel title. We do not accept the argument that this will prevent criminology-related submissions going to other sub-panels; on the contrary, developing inter-disciplinarity implies that academic boundaries will become increasingly porous; primarily legal or sociological criminology might still go to other sub-panels, much as some psychological criminology did in 2014, such boundary judgements will always be necessary. Our primary concern is the coherence and academic recognition of our subject area, not hard and fast subject classifications or divisiveness. These respective positions are less opposed than they might appear: whether or not our objective of a named sub-panel is agreed, enhanced collaboration and cross-panel calibration will still be necessary for those criminology submissions going into other Sub-Panels. |
|
| Q3a. Do you agree that the submissions guidance and panel criteria should be developed simultaneously? |
| Yes
Comments:
Drawing upon guidance from former panel members, we are absolutely convinced of the need for sub-panels to develop their own effective ownership of panel criteria, (it is an important aspect of panel calibration too) so we do think the submission guidance and panel criteria need to be tied closely in tandem, however this takes time. Perhaps it need not take quite so long as in 2014 – there was a long ‘phoney war’ period during which these deliberations took place over several meetings. At the same time, as indicated in paragraph 32, there is surely ‘scope for further improvements to consistency’ across sub-panels and orchestrated via the main panel. We are not convinced that any of this has much to do with delaying the appointment of main or sub-panel chairs, or international and research-user main panel members, rather it is about the clear communication between main and sub-panels. |
| Q3b. Do you support the later appointment of sub-panel members, near to the start of the assessment year? |
| No
Comments:
See above |
| Q4. Do you agree with the proposed measures outlined at paragraph 35 for improving representativeness on the panels? |
| Yes
Comments:
We are generally happy with the proposed measures outlined in paragraph 35 for improving representativeness on the panels. |
| Q5a. Based on the options described at paragraphs 36 to 38 what approach do you think should be taken to nominating panel members? |
| We would broadly favour the retention of the nominating process adopted in 2014. We would not favour the inclusion of representatives of mission groups, individual UK HEIs and groups within or subsidiaries of individual UK HEIs within the REF selection process. However, we would support a selection process that reflected a more proportionate distribution of Main and Sub-Panel members across the various HEI sectors from Russell Group universities through to Post-1992 universities. As the REF is part of a research resource distribution exercise, that is surely one of the chief considerations of REF participation demographics. It is, however, one that is scarcely commented upon. We think it should be an issue, and should, for instance form part of the E&D monitoring process referred to in para 37. |
|
| Q5b. Do you agree with the proposal to require nominating bodies to provide equality and diversity information? |
| No
Comments:
The BSC does not centrally collate such data, so would not be able to supply it. |
| Q6. Please comment on any additions or amendments to the list of nominating bodies, provided alongside the consultation document. |
| We have no specific suggestions |
|
| Q7. Do you have any comments on the proposal to use HESA cost centres to map research-active staff to UOAs and are there any alternative approaches that should be considered? |
| See below answer to question 7. |
|
| Q8. What comments do you have on the proposed definition of ‘research-active’ staff described in paragraph 43? |
| We are generally supportive of the Stern principle that ‘it is important that all academic staff who have any significant responsibility to undertake research are returned to the REF’ but would suggest that each submitting university determine for itself the proportion of its staff with a ‘significant responsibility’ for research. Universities with ‘teaching only contracts’ may have advantaged themselves in the ‘research power’ calculation by restricting the ‘research responsibility’ to a subsection of their staff, but this ought not be the only selection criterion, indeed with a sliding scale of outputs (from one to six) the REF could become more open and inclusive, could capture a wider range of the research undertaken reflecting (Q8) a wider definition of research active staff. Such arrangements would help to prevent universities ‘gaming’ the system, whilst leaving them free to select the UoAs to which they submit according to their perception of best advantage. Accordingly we do not support the proposal on HESA cost centres (Q7) which seems only to add another layer of opacity to the process. Furthermore, whilst some disciplines have a clear mapping between UoA and cost centre, others (including criminology) are less straightforward and there are also issues for inter-disciplinary staff who may not have previously been submitted with their home departments. |
|
| Q9a. The proposal to require an average of two outputs per full-time equivalent staff returned? |
| In conjunction with our answers to questions 7 and 8, we would support the broad proposals suggested here, but including an average of 4 outputs per fte research active staff member and, on balance, a maximum of 6 and a minimum of 2. As suggested earlier, the decisions about who and what to submit to which UoAs and Panels should be a university level decision based upon their own aims, goals, strengths and strategic advantage. The REF is an audit of universities not of individual staff, there is/should be no right to be submitted. |
|
| Q9b. The maximum number of outputs for each staff member? |
|
|
| Q9c. Setting a minimum requirement of one for each staff member? |
| We think the minimum should be 2 – see above. |
|
| Q10a. Is acceptance for publication a suitable marker to identify outputs that an institution can submit and how would this apply across different output types? |
| Consistent with our argument in the above paragraph, ‘the REF is an audit of universities not of individual staff’, then it should follow that, as Stern argued, outputs should be submitted only by the institution where the output was demonstrably generated. However we recognise that this is an area in which there are substantial perceptions of unfairness and disadvantage relating especially to early career staff. While little of this seems to be a result of the REF exercise per se, but rather reflects decisions made by universities themselves and how they are understood, a clear preference for allowing portability of outputs has been articulated by the BSC Executive. In due course, with the partial decoupling of outputs and staff this may become less of an issue in the future. |
|
| Q10b. What challenges would your institution face in verifying the eligibility of outputs? |
| No Response |
| Q10c. Would non-portability have a negative impact on certain groups and how might this be mitigated? |
| See above but we would repeat that the REF is an audit of universities, not ‘certain groups’. |
|
| Q10d. What comments do you have on sharing outputs proportionally across institutions? |
| We are not in favour of output sharing, it seems a complex non-solution to a non-problem. |
|
| Q11. Do you support the introduction of a mandatory requirement for the Open Researcher and Contributor ID to be used as the staff identifier, in the event that information about individual staff members continues to be collected in REF 2021? |
Comments:
No particular view. |
| Q12. What comments do you have on the proposal to remove Category C as a category of eligible staff? |
| We support the removal of category C; |
|
| Q13. What comments do you have on the definition of research assistants? |
| Research assistants should not be included in the REF unless there is evidence of them having experience or outputs as independent researchers; |
|
| Q14. What comments do you have on the proposal for staff on fractional contracts and is a minimum of 0.2 FTE appropriate? |
| As in REF2014 we would consider an 0.2 fte sufficient, maybe ensuring the appointment was of some duration, for instance a minimum of 24 months prior to the census date. |
|
| Q15. What are your comments in relation to better supporting collaboration between academia and organisations beyond higher education in REF 2021? |
| While agreeing with the principle of better collaboration outside HE being recognised in the Environment aspect of the REF we have no particular view about how it might be best achieved. |
|
| Q16. Do you agree with the proposal to allow the submission of a reserve output in cases where the publication of the preferred output will post-date the submission deadline? |
Comments:
We agree with the suggestion (para. 67) that Panel C be more encouraging of ‘double-weighted’ outputs (with the proviso of a substitute output if the panel declines double-weighting). Sub-Panel C22 was very discouraging of double-weighted outputs from the start of REF2014, but had a late change of heart, reversing a number of earlier decisions. Above all the arrangements should be clear. In accordance with this, (Q16) we do not think that institutions should be encouraged to submit outputs which knowingly will postdate the submission deadline. Therefore we oppose the proposal to allow the submission of a reserve output in cases where the publication of a preferred output will postdate the submission deadline, an exception might be made in cases where the publication date is not clear. The submission deadline should be the submission deadline. |
| Q17. What are your comments in relation to the assessment of interdisciplinary research in REF 2021? |
| We very much agree in principle with the ambition to encourage inter-disciplinary and cross disciplinary work. We consider that all three (a. b. c.) proposals are all worthy of consideration, although ‘interdisciplinary champions’ should not be seen as over-riding sub-panel decisions, but rather as part of an enhanced process of collegial peer review. Perhaps interdisciplinary research should be specifically flagged up and explained by those submitting it, the explanation to include the choice of UoA panel to which it is submitted. |
|
| Q18. Do you agree with the proposal for using quantitative data to inform the assessment of outputs, where considered appropriate for the discipline? If you agree, have you any suggestions for data that could be provided to the panels at output and aggregate level? |
Comments:
The process of academic peer review should remain the chief mode of REF assessment, if Sub-Panels decide that they could be helped in their determinations by the availability of citation impact data they should be enabled to request it, but should be discouraged from relying, exclusively, upon it. |
| Q19. Do you agree with the proposal to maintain consistency where possible with the REF 2014 impact assessment process? |
| Yes
Comments:
We are roadly in agreement with maintaining consistency with the impact assessment methodology employed in 2014. This is an important area of the REF, politically and academically. Changing the ground rules would create uncertainty, with universities having undergone a sharp learning curve last time, as well as diminishing the comparability of impact across REFs 2014 and 2021. |
| Q20. What comments do you have on the recommendation to broaden and deepen the definition of impact? |
| In our view the definition of impact was already pretty widely drawn in the Social Science (Panel C) area so we are not convinced of the need to broaden and deepen these |
|
| Q21. Do you agree with the proposal for the funding bodies and Research Councils UK to align their definition of academic and wider impact? |
| No Response |
| Q22. What comments do you have on the criteria of reach and significance? |
| The criteria of reach and significance, we believe, were effectively operationalized by the sub-panels, . |
|
| Q23. What do you think about having further guidance for public engagement impacts and what do you think would be helpful? |
| We are not convinced that further guidance would help. In our experience, panels were pretty exhaustive in the guidance they prepared. |
|
| Q24. Do you agree with the proposal that impacts should remain eligible for submission by the institution or institutions in which the underpinning research has been conducted? |
| Yes
Comments:
We would argue against changing the impact assessment arrangements, retain the REF2014 rules. |
| Q25. Do you agree that the approach to supporting and enabling impact should be captured as an explicit section of the environment element of the assessment? |
Comments:
We are happy to see the impact template absorbed into the environment statement and that 100 per cent of the impact sub-profile would be attributed to the impact case studies. However, we are not wholly convinced that some introductory statement regarding the selection of case studies would be useful. If this were incorporated into the Environment statement, that element might need reading first, before the Impact Case Studies. In C22 in REF2014 the Environment statements were read last, that would have to change if the templates were to go. As a seemingly increasingly important variable of the REF process, should universities also be held to account for what they said they would do last time as regards impact – why ask for approaches and stratifies for maximising impact if they don’t intend to judge universities against it. |
| Q26. What comments do you have on the suggested approaches to determining the required number of case studies? Are there alternative approaches that merit consideration? |
| Having already suggested a more inclusive output selection (sliding scale, 2-6 outputs, but an average of 4) we do not feel this would significantly increase the required overall number of case studies. We would suggest minimum changes here, with a view to continuity and comparability with the last REF, but maybe some minor recalibration of staff submitted with case studies. We do not agree with ‘institutional level’ case studies, we believe they should continue to be tied to UoA submissions. |
|
| Q27. Do you agree with the proposal to include a number of mandatory fields in the impact case study template to support the assessment and audit process better (paragraph 96)? |
Comments:
No specific comments to add. |
| Q28. What comments do you have on the inclusion of further optional fields in the impact case study template? |
| No specific comments to add. |
|
| Q29. What comments do you have in relation to the inclusion of examples of impact arising from research activity and bodies of work, as well as from specific research outputs? |
| We would generally urge no change to the rules governing the work underpinning impact case studies. Anything other than the current 2* output rule will likely introduce further vagaries, and poorly evidenced claims, into the assessment process. |
|
| Q30. Do you agree with the proposed timeframe for the underpinning research activity (1 January 2000 – 31 December 2020)? |
| Yes
Comments:
We are happy with the timeframe to be applied to the underpinning work. |
| Q31. What are your views on the suggestion that the threshold criterion for underpinning research, research activity or a body of work should be based on standards of rigour? Do you have suggestions for how rigour could be assessed? |
| See above answer to qu 29 |
|
| Q32a. The suggestion to provide audit evidence to the panels? |
| No Response |
| Q32b. The development of guidelines for the use and standard of quantitative data as evidence for impact? |
| No Response |
| Q32c. Do you have any other comments on evidencing impacts in REF 2021? |
| REF2014 was the first attempt at impact assessment and some HEIs undoubtedly found it challenging (but was it meant to be easy?) and some made mistakes. Hopefully all will be better placed to do themselves justice a second time around. There seems no reason to object to improving the guidance given to submitting HEIs (audit evidence and assessment evidence) but we would urge against widespread reliance on standardised quantitative measures – in the spirit of our response to Q18. |
|
| Q33. What are your views on the issues and rules around submitting examples of impact in REF 2021 that were returned in REF 2014? |
| We consider that some element of additionality should be necessary to the repeated submission of an impact case study. The essence of impact is change, so new impact should represent the ‘gold standard’ whereas ‘continuing’ impact, while valuable, should have to demonstrate some further resultant outcomes, further developmental changes, and/or new audiences – or larger audiences – reached. Impact is often not apparent until long after research has taken place and may require a sustained period of time to come to its full flowering. 2021 Panels should be informed if impact case studies were submitted in 2014, and tasked with ensuring differentiating new and ‘continuing’ impact. |
|
| Q34a. Do you agree with the proposal to change the structure of the environment template by introducing more quantitative data into this aspect of the assessment? |
Comments:
In our recollection the environment section assessment was already significantly structured around research income and, less so, PGR completions. The step-changes in environment scores largely followed the research income. Accordingly, much as we support the incorporation of well-evidenced material into the environment template, it is difficult to see what would plausibly supplant (or complement) the existing data. Process measures (eg. number of ECRs awarded sabbaticals) are interesting, but less valuable than outcome measures. |
| Q34b. Do you have suggestions of data already held by institutions that would provide panels with a valuable insight into the research environment? |
|
|
| Q35. Do you have any comment on the ways in which the environment element can give more recognition to universities’ collaboration beyond higher education? |
| No Response |
| Q36. Do you agree with the proposals for providing additional credit to units for open access? |
| No
Comments:
We disagree, the current issue of Open Access concerns primarily transitional arrangements, this is not likely to be an issue in any future REFs and we are not convinced it is an appropriate use of the REF assessment. |
| Q37. What comments do you have on ways to incentivise units to share and manage their research data more effectively ? |
| No Response |
| Q38. What are your views on the introduction of institutional level assessment of impact and environment? |
| We are not averse to the suggestion about the submission of institutional level impact case studies but don’t see why this should be mandatory and not be for institutions to decide for themselves – offset against the numbers of ICS submissions each HEI would requires. There is, maybe, a recognition that institutional ICSs will be stronger, which should provide enough incentive for HEIs to supply them, up to a maximum of (perhaps) 10% of their overall ICS quota. |
|
| Q39. Do you have any comments on the factors that should be considered when piloting an institutional level assessment? |
| The strength of the REF assessment process is the transparent and evidential foundation upon which it stands. Movement in the direction of institutional assessment has the potential to move worryingly closer to untestable, relatively opaque and perhaps more ‘reputational’ claims. Accordingly we are sceptical about such developments. |
|
| Q40. What comments do you have on the proposed approach to creating the overall quality profile for each submission? |
| On balance, we would support the most robust, transparent and clearly evidenced arrangement of quality weightings. Output weightings to remain at 65%; Impact weightings could stay as they are or rise to 25%, leaving Environment with either 10 or 15%. This balance across the three elements of the REF seems entirely appropriate given the fact that the Environment narrative describes processes and is the most susceptible to reputational interpretation, whereas Outputs and Impact are both outcomes, and therefore the real issues for a research assessment methodology. |
|
| Q41. Given the proposal that the weighting for outputs remain at 65 per cent, do you agree that the overall weighting for impact should remain at 20 per cent? |
| Yes
Comments:
Or be raised to 25% – see answer to qu 40 |
| Q42. Do you agree with the proposed split of the weightings between the institutional and submission level elements of impact and environment? |
| No
Comments:
As per our argument in question 40, we do not agree with the proposed splitting identified in the Figure 1, (page 29) making institutional impact submission mandatory (which we have already rejected – response to Q38, above) but would prefer the REF2014 model 65/20/15, or the variation 65/25/10. |
| Q43. What comments do you have on the proposed timetable for REF 2021? |
| We are content with the proposed timetable |
|
| Q44. Are there proposals not referred to above, or captured in your response so far, that you feel should be considered? If so, what are they and what is the rationale for their inclusion? |
Except in specific cases already identified we would broadly support relatively little change as possible from the arrangements established for REF2014 which worked well and secured the broad confidence of the academic community. In order for there to be meaningful comparison between REF2014 and REF2021, there needs to be as little as possible change.
However, that said, with our discipline specific focus in this submission, we nevertheless strongly reiterate the need for the strength of our discipline, Criminology, to be appropriately reflected in the REF Panel structure in a sub-panel title. |
|
