In this guest blog post, Darren Flynn, Academic Liaison Manager, Library and Learning Services at the University of Northampton, talks about assigned systematic reviews in a higher education and allied health setting. This post is an expansion of Darren’s thoughts that he first expressed in a Twitter thread.
Over recent years it’s become increasing common for student final projects and dissertations to undertake an entirely literature-based study. Particularly in health courses, these often take the form of scaled-down systematic reviews where students independently formulate a topic and research question, develop, test and execute a search strategy then sift, appraise and synthesise a selection of secondary research. From a practical point of view, this form of project has distinct advantages over empirical research: it allows students to research topics not otherwise feasible, presents fewer (though not no) ethical barriers and is less circumscribed by logistical, temporal and geographic barriers. From an information literacy perspective it can represent an effective capstone; bringing together multiple strands and skills developed over an academic programme into a single project. Because of this, as an information literacy professional, I was and am supportive both in conceptual and practical senses of the shift towards this type of project.
However, this support comes with a range of specific caveats. Literature-based studies present particular challenges to the different stakeholders (students, supervisors and librarians) involved that require addressing if such projects are to be successful. In this post (an expansion of a Twitter thread) I hope to share my thoughts and experiences gleaned from my past role as an academic liaison librarian supporting allied health subjects both in terms of direct support to students and in liaison work with academics setting and supervising these projects. The following does not constitute a formal policy or approach, nor a set of “dos and don’ts” but rather a set of considerations that might be made in when approaching these kinds of projects and developing services and practices to support them.
Rationale
As outlined above, there are good pedagogic and practical reasons for using literature-based studies as dissertation projects. These do not however preclude a critical assessment of the reasons underlying the shift towards them as an assessment method. Literature-based studies should not be implicitly or explicitly thought or described as easier or less labour-intensive options than traditional empirical studies. Even when falling short of a full systematic review, these types of project typically require many hours of work and great effort to complete, they can be frustrating, non-linear and require the development of new skills not usually attempted over the course of a typical undergraduate programme. What’s more, the dependence on published literature alone can introduce significant limits on what research questions can be asked and how they may be answered.
Literature-based studies should neither be thought of as an expedient means of reducing staff time (such as supervisor contact time) or physical resources (such as labs). The resource needs of these studies are significant though different, generally requiring input from libraries and librarians to enable them to work efficiently. Academics considering their use should assess the resource requirements in consultation with their librarian, and if necessary, actively advocate for the levels of investment necessary.
In discussing the rationale for literature-based studies we should also consider what the cognitive and conceptual reasons are for completing one. Generally, most academics I’ve spoken to accept that work produced will not meet the standard of a systematic review and it is likely that any outcomes will represent only a partial synthesis of available evidence on a topic. We may accept these limitations and equate them with those of the a typical undergraduate or taught postgraduate empirical study wherein the sample is likely too small and/or unrepresentative to provide for reliable results etc. The learning objective of completing a study though is demonstrating an understanding of the process and gaining first-hand experience of managing a small-scale project. The same logic may be applied to literature-based studies: the point is what students do with the papers they find, are they able to demonstrate the skills of evidence-appraisal and synthesising results to answer a question. Whether or not a search is particularly well-structured or comprehensive might therefore be seen as inconsequential and thus whether or not librarian support is available of relatively minor importance. In theory this approach may appear both logical and pragmatic, however in practice is extremely problematic. Even when the search strategy is not a direct concern in assessment it is in effect foundational to the success of the rest of the project. Without an effective search strategy, students face either not locating all or enough relevant and usable studies (undermining their synthesis and conclusions) or spend many frustrating hours scouring results lists that reach into the 1,000s. Expert and experienced support therefore is a necessary prerequisite for these assessments, even where the component concerned [search strategy] may not be directly assessed.
Terminology
Up to this point I have avoided the thorny issue of the terminology we might use to describe these types of projects, so far using the term literature-based studies as a catch-all term that is benign by virtue of its ambiguity. It is important to acknowledge explicitly that a systematic review is a specific, definable and recognisable research methodology that is highly unlikely to be undertaken by students below PhD-level study. Typically an undergraduate or taught postgraduate student might adopt some of the methods of a systematic review in their project (devising inclusion/exclusion criteria, developing a search strategy etc.) and may also be required to include some of the documentation of a systematic review (reporting of search terms, screening diagrams etc.) but the inclusion of aspects or processes does not make the term systematic review applicable: the whole remains greater than the sum of its parts. The distinction may appear overly-pedantic but misapplication of the term systematic review is wrong both conceptually (by undervaluing the methodology) and practically; a student who thinks they are doing an SR searching for support materials will find information that will run from daunting to terrifying. A librarian supporting the students completing these projects cannot give appropriate advice if they do not have an awareness of the actual type of study the student has been set.
Therefore if we believe accurate terminology matters, we need a suitable term that both describes the project’s specific requirements and is distinct from an systematic review. In my practice the term “systematised literature review” was used to express that: 1) The project was literature-based, 2) The purpose was to review and synthesise literature and 3) That it used some systematic methods of an systematic review without adopting the full methodology. Alternative terms I’ve come across include pragmatic systematised literature review, systematic-style literature review or a documented literature review.
Delimiting a systematic review-style project
If we accept that most projects set for students that are described as systematic reviews are not, we necessarily need to explicitly delimit them for students and set out clear expectations. In a typical systematic review, any papers located that are relevant and conform to inclusion/exclusion criteria must be included. In a student project this is often impractical and inequitable; projects with the same wordcount where one student synthesises three papers and another ten papers cannot be fairly compared against an identical marking scheme. Guidelines are therefore needed as to how many papers are necessary to complete the assessment but in addition advice on what students should do if they exceed the recommended number; do students include the most recent, best quality, the most homogenous selection? Academics should also consider what types of study may be included, is grey literature acceptable? Are pilot studies? What are acceptable time bounds (if any), studies from previous five, ten, fifteen years? When considering originality (i.e. any existing reviews on the same topic) what is an acceptable time period since publication? Must students only use papers published since the last review? Is there an expectation for students to use non-English language publications and if so, how might students translate them? If a student is fluent in additional languages can they use studies published in that language? Will students be expected to gain access to materials the library doesn’t have access to, and if so, does the library offer a free document supply service?
Making such a list of questions can appear deliberately obstructive and captious but these are questions students will inevitably ask and they have the right to expect a definitive and consistent answer. Not having a such boundaries in place and transparently shared risks inequitable treatment based on the subjective judgement of a supervisor or a librarian answering in the moment.
Liaison
It should, but doesn’t always, go without saying that academics planning on using literature-based dissertation projects must engage with the library/their librarian. We would rightly recognise the flaw in setting lab-based assessments without informing laboratory technicians and equally not consulting the librarian responsible for the subject area given their role in supplying and supporting the necessary inputs for these types of projects. In terms of liaison several key discussions are necessary. Librarians need a clear understanding of the scope of the projects and their requirements: what specifically are students being asked to do, what processes and documentation are expected and what are the timescales between project commencement and completion. This information is rarely efficiently conveyed by students themselves and it is a dispiriting experience for all parties when a student needs help but the librarian cannot offer any for fear of giving inaccurate information. From an academic’s point of view they need to know what services and levels of support and input is the librarian able to provide: will this be limited to a single taught session, are research consultations available, how are these booked and what is the expected wait time for these, is there an enquiry service, how might they refer students to additional training. More generally, academics need an awareness of librarian capacity to support these projects given they may be responsible for multiple subject areas setting similar projects. While a literature searching session will often suffice for many students and the more straightforward topics, the nature of systematic review-style projects is that many students will require additional advice and support to check the viability of a project, design a search strategy and to adapt general principles covered in a taught session to the specifics of their project. Moreover, the experience of attending a literature searching session can generate rather than lessen the need for students to seek individual guidance as they realise the potential complexity of the task at hand. In addition to direct student support [by liaising with librarians] supervisors can derive further benefits such as exemplar search strategies outlining effective search strings, guidance on effective assessment of searches and providing support materials that can be shared with students.
On academic liaison I would finally add the need to develop high levels of trust between professionals with different spheres and levels of expertise. A cognitive shift is required by some project supervisors that in regard to some aspects of a systematic review-style project (e.g. scoping a project, developing search strategies) their expertise is generally significantly lower than a librarian’s and they may need to defer and refer as necessary to support students appropriately. A common experience I had as a health librarian was meeting with students to discuss a potential research question which had been provisionally approved by a supervisor (on the basis of its relevance to the programme and an assumption that literature on the topic was extant) but was patently unviable as a project once an initial scoping search was completed (too few published papers, existing recent review etc.) These scenarios contained the potential for conflict; ethically I felt bound to advise the student to adapt or change the topic but doing so could undermine a colleague. For the most part this was managed through strong trust relationships between myself and academics but particularly in the context of high reported levels of imposter syndrome amongst librarians, academics should be explicit in their support of librarians’ judgement in advising students approaching a literature-based study.
Darren Flynn – @patchedelbows
Do you have a question or comment? Feel free to leave a comment on this post to start a conversation!
Do you have a case study that you would like to share? Why not showcase this on the ILG blog? See our guidelines for further information on submissions.
Thanks Darren, this is a really thought provoking blog. As an NHS librarian, I’m often approached by our clinical staff who are undertaking studies and asked to help with systematic reviews. There’s no existing relationship for me with their supervisors, so it can be really difficult to carefully try to reset expectations, or to educate the student so they can confidently go back to their supervisor with what they’ve found. There can be a disconnect for us also because the students aren’t necessarily undertaking their course with a local university.
Librarian capacity for support is also a really good point here; primarily my post is to support patient care queries and it can be difficult to make time for our staff who are also students. We always try! I just don’t think that the people setting the assignments take this into account.