header advert
Bone & Joint Open Logo

Receive monthly Table of Contents alerts from Bone & Joint Open

Comprehensive article alerts can be set up and managed through your account settings

View my account settings

Visit Bone & Joint Open at:



Open Access

General Orthopaedics

Virtual exams: has COVID-19 provided the impetus to change assessment methods in medicine?

Download PDF



The ongoing COVID-19 pandemic has disrupted and delayed medical and surgical examinations where attendance is required in person. Our article aims to outline the validity of online assessment, the range of benefits to both candidate and assessor, and the challenges to its implementation. In addition, we propose pragmatic suggestions for its introduction into medical assessment.


We reviewed the literature concerning the present status of online medical and surgical assessment to establish the perceived benefits, limitations, and potential problems with this method of assessment.


Global experience with online, remote virtual examination has been largely successful with many benefits conferred to the trainee, and both an economic and logistical advantage conferred to the assessor or organization. Advances in online examination software and remote proctoring are overcoming practical caveats including candidate authentication, cheating prevention, cybersecurity, and IT failure.


Virtual assessment provides benefits to both trainee and assessor in medical and surgical examinations and may also result in cost savings. Virtual assessment is likely to be increasingly used in the post-COVID world and we present recommendations for the continued adoption of virtual examination. It is, however, currently unable to completely replace clinical assessment of trainees.

Cite this article: Bone Jt Open 2021;2(2):111–118.

Take home message

The ongoing COVID-19 pandemic stalled medical and surgical examinations where attendance is required in person, and may have delayed career development for some young surgeons. The introduction of online assessment would circumvent the delay in assessment associated with the pandemic and similar events, and additionally provide many benefits both locally and globally if adopted as the ongoing standard.


Medicine is a career of lifelong learning and assessment. Whether in core training, specialist training, or at a consultant level, continual assessment remains central to quality improvement in medicine. A typical Trauma and Orthopaedic (T&O) surgical speciality training programme in the UK exemplifies this. Core surgical training must be completed for two years, and an intercollegiate membership exam, i.e. MRCS, must be passed succesfully prior to entry into any surgical speciality training. Success in the exit exam following speciality training, i.e. FRCS (Tr & Orth) for Trauma and Orthopaedics, is mandatory for completion of specialist training.1 These exams consist of written, clinical, and viva voce assessment. During speciality training the trainee’s progress is continuously monitored in the annual reviews of competency progression (ARCPs), which include reviews of operative performance consisting of procedure-based assessments, and direct observation of procedure skills, along with nonoperative performance, including case-based discussions and clinical evaluation exercises.1 The Fellowship exam is a summative assessment of the clinical skills developed during training and assesses the trainee against the skills necessary to be at the level of a ‘Day 1 Consultant’ in the relevant surgical speciality.

While assessment has evolved over the past century, exams have continued to be administered in a traditional ‘exam hall’ manner.2 The ongoing COVID-19 pandemic has exposed flaws in this system, which poses rigid parameters of time, date, and location for candidates and assessors. All parts of both the MRCS and FRCS exams were postponed due to the COVID-19 pandemic until an alternative COVID-safe format could be devised. The Fellowship section 1 exams, which follow a single best answer (SBA) and extended matching item questions (EMQ) format, were postponed until examination centres were able to put in place COVID-safe measures. The section 2 exams, the clinical component which consists of clinical examination, and scenario- and patient-based viva voce examinations (vivas), were also postponed for the development of COVID-safe examinations and remote assessment options. Going forward, it is time for the surgical profession to adopt online remote assessment as an intrinsic part of both undergraduate and postgraduate medical training, both during and after the COVID-19 pandemic. In this article, we aim to assess the validity of summative online assessments for examinations outside of the hospital clinical environment, the advantages to both trainee and assessor, and the challenges to implementation through a review of the available literature. This will provide a background of evidence and validation for the implementation of online examinations, and may act as a checklist for future and continual implementation.


Literature searches were performed by three independent reviewers (MP, SS, JZ) to identify relevant articles in both medical education and computational technology. Medline and IEEE Xplore were searched using the following terms: ‘computer-based’, ‘online’, ‘virtual’, ‘remote’, ‘computer-based’, ‘medic*’, ‘surg*’, ‘Exam*’, ‘assessment’, ‘valid*’, ‘outcomes’, ‘advantages’, and ‘challenges’. Articles were included if they were related to the validity of summative assessment, the benefits of virtual assessments or the challenges to the implementation of virtual assessment.

This literature search identified 10,836 articles after removal of duplicates, while a further 74 articles were identified from other sources. After title and abstract screening 96 articles remained, and 49 full-text articles were included in this narrative review.


Following the literature search, articles were analyzed and broadly grouped into different categories including 1) assessment methods, 2) trainee and trainer perspective, and 3) challenges to implementation of virtual examination, which are discussed below.

Computer-based assessment and validity

Prior to the COVID-19 pandemic, the majority of computer-based assessments (CBAs) had been trialled in medical schools rather than in postgraduate training, and had been formative in nature.3 The Ottawa consensus statement for good medical assessment identifies construct validity, reproducibility, equivalence, acceptability, feasibility, educational benefit, and timely feedback as key elements for successful medical assessment.4 Kane’s framework provides a construct validity argument in a step-by-step framework and represents the most recent evolution in validity theory (Figure 1).5

Fig. 1 
            Kane’s framework for interpreting the validity argument of an assessment, which it achieves by emphasizing scoring, generalization, extrapolation, and implication arguments.

Fig. 1

Kane’s framework for interpreting the validity argument of an assessment, which it achieves by emphasizing scoring, generalization, extrapolation, and implication arguments.

MCQ and EMQ assessment

The UK and Ireland Royal Colleges of Surgeons (RCS) use MCQ, SBA, and EMQ in the written components of their examinations. Computer-based SBA and EMQ are valid by proxy, as they are currently conducted at CBA centres in the written section 1 of FRCS examinations.6 Studies in higher education confirm that similar exam scores are awarded with both paper- and computer-based MCQ assessment, displaying valid scoring and generalization.7 MCQs also display valid extrapolation and implication, as they correlate well with other measures of clinical performance.8

Viva voce examinations

Viva voce examinations are considered a valid form of assessment, however scoring and generalization may follow a subjective, examiner-dependent process.9 To ensure that the marking in these examinations remains objective, the Intercollegiate Fellowship examiners (FRCS) are trained and provided with an objective scoring system. For computer-based vivas to be valid, interpretation of subjective features should not change with the use of telecommunication. Factors influencing subjective scoring include personal qualities of fluency and creativity, nervousness, incoherence, cultural differences, values, and language factors, including discourse style, which may be perceived differently over video.9 The system should therefore provide sufficient audiovisual quality for the accurate interpretation of language and non-verbal communication, and the examination should be carried out in a controlled environment such that personal qualities are not perceived differently to a traditional interview setting. Sufficient technical quality is available in medical telecommunication. Current technology including smartphones and tablets have been shown to provide accurate image-based teleconsultations between physicians.10,11 Virtual patient consultations additionally report high efficacy across multiple specialties, equalling or improving outcomes compared to face-to-face consultation.12

Clinical ability

The validity of virtual assessments for clinical ability, in both technical and non-technical skills, must be viewed with more caution. Technical skills can be broadly grouped into practical skills and patient assessment. Virtual patient assessments intuitively cannot be valid if physical examination is required, as they cannot be extrapolated to a clinical environment. While a recent literature review highlighted the paucity of evidence for the validity of online case-based simulations in assessment of clinical ability, evidence is emerging for the validity and efficacy of online assessment in clinical reasoning and case presentation given standardized patient encounters, and the use of virtual patients for history taking.13-15 The use of such virtual patient encounters may precede the online adaptation of patient-based objective structured clinical examinations (OSCEs) assessing clinical ability, especially in a context where examination-based findings can be supplemented. This is supported by preliminary evidence validating the use of ‘teleOSCEs’ assessing clinical skills.16

Non-technical skills are wide-ranging and include a core set of communication skills. In this area current computer-based assessments utilize recorded consultations which are later analyzed (Objective Structured Video Examination format), and virtual assessment of active communication skills is now reported.17 It can be hypothesized that standardized online simulated patient encounters may be validated for postgraduate exams. Evidence demonstrates that online teaching of communication skills is as effective as traditional learning in an undergraduate setting, suggesting valid scoring and generalization.18 Virtual patient encounters also appear to validly extrapolate, provoking similar emotional responses to real standardized encounters in communication skills assessment.19 Such simulation has high efficacy for undergraduate communication skills assessment, and evidence exists to suggest that screen-to-screen communication has no difference in patient perceived information exchange, interpersonal relationship building, and shared decision-making versus face-to-face communication.20-22

Trainee and trainer perspective

Virtual assessment will only be adopted if accessible and advantageous to both candidates and examiners.


Online learning and assessment are associated with the ‘any time anywhere’ mantra, and can be taken from anywhere around the world. On a global scale, this could promote professional accreditation and quality improvement, reducing ‘brain drain’ in countries without formal postgraduate training or standardized entry and exit exams.23 In Malawi, online surgical sciences MSc scholarships were awarded to 24 surgical trainees through the RCS Edinburgh.24 Not only did the scholarship candidates score similarly to the cohort average, but they also reported improved patient safety and decision-making skills following completion of the course.

Cost saving

Virtual exams may enable savings for both the assessment body and trainee. One may estimate a significant reduction to cost without the need for exam hall hire, invigilator and examiner employment, travel expenses, exam paper printing, and digitization and marking of exam papers.25 Through a top-down market sizing approach applied to the RCS of England annual report and accounts for the year ending 30 June 2019, we provide an estimated saving for assessment of T&O surgeons by the RCS of England of between GBP £52,000 and £157,000 (Figure 2).26 Trainees will also incur savings related to travel and accommodation.

Fig. 2 
            Cost estimate of savings provided by changing to an online examination system. The most conservative estimate splits to the left, while the best-case estimate splits to the right. Savings were estimated through market sizing strategies and the Royal College of Surgeons of England Annual report and accounts for the year ending 30 June 2019. T&O, trauma and orthopaedics.

Fig. 2

Cost estimate of savings provided by changing to an online examination system. The most conservative estimate splits to the left, while the best-case estimate splits to the right. Savings were estimated through market sizing strategies and the Royal College of Surgeons of England Annual report and accounts for the year ending 30 June 2019. T&O, trauma and orthopaedics.

Efficient administration

A reduction in administration time, including the use of digital marking sheets for vivas, along with automated marking for MCQ, SBA, EMQ, and some SAQ formats, may enable trainees to receive results sooner, if not the same day.2 Using a virtual assessment, the time taken to mark over 14,000 very short answer questions (VSAQs) was reduced from 70 hours to five hours in an undergraduate pathology examination.27 Virtual assessment would also remove the handwriting variability and the halo effect associated with these questions, where examiners associate poor handwriting with lower performance.28 Virtual assessment may therefore increase the feasibility of traditionally labour intensive examination methods such as SAQs.

Challenges to implementation

Practical caveats

Although remote assessments are attractive, variation between candidates’ home environments remains a concern. Noisy, busy households are not conducive to examination. Mitigating such factors may be possible by continuing to provide students with the opportunity to undertake exams at local CBA centres rather than remotely at home, and by requiring a standardized virtual background for viva examination.

Variations in household internet access should also be considered, as unreliable bandwidth may disadvantage candidates. It is possible to deliver online exams ‘offline’ or with adjusted time constraints to counter unreliable or insufficient bandwidth. Services offer downloadable exams and the capability for seamless offline examination with connectivity loss. For stratification of candidates to pre-downloaded remote examinations, bandwidth and connectivity should be registered during the enrolment process. Alternatively, an unexpected drop in bandwidth during a fully online examination can be compensated with a Time Adaptive Mobile E-Exam (TAMEx) format.29 This monitors the examinee’s device, and if the device loses connectivity, the total time of this occurrence is added to the remaining examination time.

Assessment of students across multiple time zones is disadvantageous for some candidates if a uniform examination start time is used, as international candidates take the examination at irregular hours. The optimal examination start time for cognitive functioning, which disadvantages the least number of chronotypes, is around noon.30 A standardized start time in each time zone would therefore be optimal.

In order for time zone varied start times to be enabled without an increased risk of cheating and plagiarism, a different set of questions is required for each exam. This is possible given a large question bank from which an examination software randomly selects a generalisable question set.31 Question bank development may also enable annual integrated continual formative assessment and feedback, improving training quality.32,33 Question bank design is labour-intensive. Importing or generating a question bank requires checking, editing, and tagging for all questions and exam blueprinting.31,34 Additionally, it has been reported that 90.8% of questions in online medical assessments have an associated image.35 If this is representative, administrative costs may increase in relation to data protection, copyright, and server storage.25 It is therefore essential for costings to be performed during procurement.

Ongoing IT support must also be available during exams, along with fail-safe measures to continuously back up progress in case of a systems failure.25 This agrees with university students’ attitudes, whose primary concerns regarding online examinations were data security and data loss.36

The overall wellbeing and performance of candidates is important. Transition of written exams to CBA has received favourable student responses, with no difference in results versus written exams.7 A proportion of candidates do, however, believe that IT skills enable faster and easier exam completion. Additionally, a survey of CBA using individuals’ own devices found IT compatibility to be an issue.37 This would warrant a priori instruction and troubleshooting to reduce anxiety and test day errors. It is also known that viva voce examination elicits higher anxiety than the written equivalent, and may not give an accurate indication of future performance.38 To reduce anxiety, trainees may have an initial group brief in an online conference setting, and structured viva voce (SVV) may be used rather than traditional viva voce (TVV). SVV is a valid assessment method, and is potentially a better discriminator than TTV.39 Additionally, the majority of medical students trialled with SVV reported a preference for this over TVV.

Exam security

Cheating is a primary concern, as it may invalidate assessments, and may occur in many forms. Broadly, an individual may impersonate the candidate, or the candidate may access prohibited materials during the assessment. Such behaviours are overcome or minimized in an examination centre where proctors monitor behaviour. New mitigating strategies are essential in order to reduce cheating during virtual assessments.


Static authentication involves confirmation of the candidate’s identity at the beginning of the examination, while in continuous authentication the candidate’s identity is repeatedly confirmed throughout the examination. Authentication can be through either a knowledge factor (for example username and password combinations), an ownership factor such as an access card, or a unique biometric factor.40 Knowledge and ownership factors are not highly secure and biometric factors tend to be favoured.

Virtual exams may employ a live proctoring system with ID comparison made by an invigilator who monitors candidates remotely. Automated proctoring software is a viable alternative, where biometrics are utilized for live authentication, including: keystroke recognition, facial recognition, fingerprint recognition, and voiceprint or iris scanning.40 These measures require candidate registration prior to examination.

Fingerprint recognition is typically used for static authentication. It requires a universal serial bus fingerprint scanner and is commonly used in conjunction with other biometric measures.41 Keystroke analysis can provide static authentication through the typing of a standardised text, and is able to continuously authenticate if typing is required during the exam.42,43 Keystroke dynamics act as an individual signature, and current analysis using Artificial Neural Networks can extract user patterns with 80% accuracy.43 Static authentication using facial recognition has been possible for almost forty years but requires sufficient camera quality.44 Continuous facial recognition poses the additional problems of movement, angling, and lighting changes. Recently, a continuous authentication algorithm has been developed with 100% accuracy based on a three-minute training video.45 Current models for integrated biometric authentication utilize these three methods.

Other biometric technologies may be integrated in the future. Theoretical models for voice recognition have been proposed and validated, but not widely studied, and carry practical implications for continuous authentication in a silent examination atmosphere.46 Similarly, iris scanning has been proposed, but requires an infrared camera which is not readily accessible.47

Cheating detection

Cheating detection is a continuous process. In a live proctored online exam, the individual will be directly observed at their workspace by a proctor. This carries limitations, as the proctor receives a narrow field of view with typical camera hardware. Additionally, it is costly to acquire staffing. Therefore, an automated proctor system may be used where images and video are flagged for proctor review if suspicious activity is detected by the system.

Eye tracking is an automated proctoring technique which flags viewing of offscreen content, or inappropriate areas of the screen. Initial calibration is required through recording candidate pupil position while they focus on particular onscreen coordinates. In a sample of 30 adults an eye tracking system was able to detect cheating with accuracy over 97%.48 Further theoretical advances to continuous cheating detection include thermal imaging, whereby the intention to cheat may manifest by an increase in the candidate's body temperature.49 Such technology requires validation and benefit to be shown over current biometrics, and provides an insight into innovative future cheating detection solutions.

Furthermore, test and software design can reduce cheating behaviour. Multiple steps have been suggsted by Cluskey et al50 to prevent cheating in un-proctored online examination, with the aim of increasing practical difficulties associated with cheating behaviour (Table I). Finally, a data science approach may be considered, requiring analysis of answer concordance in order to flag excessively similar examination scripts for proctor review.51

Table I.

Question design strategies to mitigate cheating behaviour. Adapted from Cluskey et al.50

All examinees should sit an exam at one particular time slot
The questions should be accessible for a short period of time
The sequence of questions should be randomised
Questions should be presented one at a time, with no going back
Question difficulty should be such that the questions can just be completed without much excess time
The exam should only be accessible once, with no resetting unless proof of technical failure exists
The exam questions should be frequently changed
Students should be required to use a restrictive ‘locked-in’ browser


Cyberattacks may be orchestrated by candidates, or a third party hoping to disrupt the exam for malicious reasons. In the context of virtual medical examination, cyber security measures are increasingly important if patient cases are discussed or presented, to ensure the maintenance of patient anonymity. Software attacks can be mapped using attack-tree frameworks. Rosmansyah et al52 describe such a framework for cyberattacks on virtual exams. Each attack can be carried out in multiple ways for which there are specific cybersecurity countermeasures. For instance, a Denial of Service attack is a web application attack which aims to overload a server by sending large amounts of data. This can be mitigated by incorporation of dynamic path identifiers, fuzzy reasoning-based windows firewall, or channel hopping.53-55


Despite a paucity of literature describing summative online assessment, we have outlined the validity of online written examinations, and of online vivas. There is also a developing argument for the validity of virtual patient encounters to assess clinical skills, which are now being piloted at several institutions. Furthermore, we outlined the benefits of the continued adoption of a virtual examination system, along with the practical issues its adoption faces. Given the potential issues, we have outlined our recommendations for online assessment implementation in Table II.

Table II.

Challenges to the implementation of online assessment, and recommendations for the introduction of online assessment.

Challenges Recommendations
Maintaining exam integrity across multiple time zones Use a standardized examination start time on the same date to disadvantage as few candidates as possible. Utilise question banks, automated exam generation from blueprinting, order randomization, and restrictive browsers to prevent cheating behaviour.
Mitigating the impact of home environments and subjective influences Require candidates to have a standard virtual background or plain wall for viva voce examinations, and maintain a dress code. Use SVV formats to further ensure objectivity. Enable candidates to book education facilities for examinations where a home environment is not suitable.
Poor candidate internet access Provide a system which provides seamless offline-online capability. This may be through pre-downloading and synchronisation or other means. Prior to viva voce examinations, ensure candidates provide evidence of compatible bandwidth.
Question bank development Seek a system which allows automated importation of existing questions from a word format or similar, and allocate administration time to blueprinting and tagging of questions. Perform a costing, and ensure a long-term plan or contract is in place if an external system is used, such that data is not lost at the end of a contract.
Systems failure and compatibility Utilize a system which has continuous back-up of progress, or one which utilizes hard-disk space to enable back-up in the event of internet loss. Ensure candidates have had prior instruction and sample questions to prevent compatibility issues, or user-related failures.
Cybersecurity threats Ensure systems employ a reputable internet security system. We recommend consulting an expert or external company due to the complexities involved.
Questions concerning the validity of online clinical assessment using virtual patients and virtual patient encounters Continue to record and present data to explicitly establish the validity of summative online examinations in the surgical field. Furthermore, in the near future, record data regarding outlier cases who are stratified to remote examination and use this to investigate the validity of virtual patient encounters in surgical examination.
Impersonation Ensure static authentication at the beginning of the exam. This should be at least bimodal, requiring a knowledge factor and/or ownership factor, as well as a biometric factor.
Cheating detection Employ artificial intelligence-driven systems which utilize multiple cheating detection methods to assess the risk of cheating and use statistical analysis to flag excessively similar scripts. Prevent cheating behaviour through exam design, and application lock-in. Produce and keep a webcam recording of the candidate for manual review.
  1. SVV, structured viva voce

Prior to the COVID-19 outbreak, only a small number of medical institutions utilized online platforms. Universities had begun to implement online assessment, including Imperial College London and the University of Edinburgh, which had implemented summative MCQ, EMQ and VSAQ into their undergraduate courses with good outcomes.27,31 The implementation of such systems required question bank importation, and blueprinting of exams. This system enabled a fourteen-fold increase in the speed of marking, along with the integration of previously unused VSAQs. Similarly, a number of Royal Colleges, including the governing bodies for paediatrics, general practice and radiology in England, had implemented online examination systems. These systems offered the ability to blueprint the curriculum, import existing questions from microsoft word, and provide fully online or offline capability provided that Wi-Fi synchronization had occurred. These systems concurrently enabled automated set up of the exam from question banks, and instant marking and feedback for those taking the exam. They were, however, still conducting exams at a centralized examination centre.

Since the introduction of COVID-19 restrictions, we have seen a movement from the Royal Colleges to provide remote examination. This includes the Royal College of Surgeons, who have provided a fully remote online MRCS part A examination. This utilized static authentication in a two-fold process, using a username and password combination, and subsequently biometric facial recognition.56 Remote proctoring was additionally employed, with an artificial intelligence (AI) system tracking on-screen movements, gaze, and audio recording, with second tier human review guided by the AI flagging of behavioural anomalies. This type of online proctoring and authentication is offered by numerous companies. Additional features currently available include locking of the screen, and blacklisting of applications to reduce cheating behaviour. Systems currently available provide a high level of accuracy for remote proctoring, of approximately 99.5% in flagging behavioural abnormalities, and provide a high level of automation, with approximately 7% of candidates being manually reviewed. Given the current capability in delivering online written examination and the success of MRCS part A virtual examinations, the authors would suggest provision of remote virtual assessment for Intercollegiate Fellowship examinations including FRCS (Tr&Orth) section 1 in the future. Additionally, the data from such examinations may be used to add to the small body of literature which exists surrounding summative post-graduate virtual examination within medical professions.

While there has been good uptake of virtual online written examination, remote patient- and case-based vivas were not the default scenario-based structured interviews in the FRCS (Tr & Orth) section 2 unless absolutely required, and remote OSCEs were not used in the Oct 2020 MRCS part B. These assessment methods have some evidence promoting their validity in virtual examination, with the exception of patient physical examination. However, for the November 2020 sitting of FRCS (Tr & Orth) clinical examination, measures were taken to eliminate direct interaction with patients. The authors would promote the uptake of virtual case-based and patient-based vivas in the FRCS (Tr & Orth) section 2, and the subsequent utilization of these data to further validate virtual vivas and OSCEs. Furthermore, it may be speculated that the utilization of virtual and augmented reality may enable remote virtual clinical examination in a merged reality space, which has been trialled for postoperative follow-up, or the examination of a completely virtual patient, similar to anatomical assessments trialled in dentistry examinations.57,58

In conclusion, online assessment provides numerous benefits to both trainee and assessor in the field of written and viva voce examinations, while retaining validity, but there are challenges underlying its implementation. Online assessment is not well suited to clinical assessment involving patient examination. However, the paradigm shift in the current working environment as a result of the COVID-19 pandemic has indeed provided us the opportunity to adopt and validate innovative ways of assessment in medical training. Virtual assessment is providing a path forward to overcome the obstacle posed by the current pandemic, which may in fact be the impetus for long-term change.

Correspondence should be sent to Vikas Khanduja. E-mail: ;


1. No authors listed . Specialist training in trauma and orthopaedics . 2020 . https://www.iscp.ac.uk/static/public/syllabus/syllabus_to_2017.pdf (date last accessed 12 October 2020 ). Google Scholar

2. Marston A . The final surgical examination: then and now . Bulletin . 2011 ; 93 ( 3 ): 86 88 . Google Scholar

3. Al-Amri S , Ali Z . Systematic review of computer based assessments in medical education . Saudi J Med Med Sci . 2016 ; 4 ( 2 ): 79 88 . Crossref PubMed Google Scholar

4. Norcini J , Anderson B , Bollela V , et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 conference . Med Teach . 2011 ; 33 ( 3 ): 206 214 . Crossref PubMed Google Scholar

5. Cook DA , Brydges R , Ginsburg S , Hatala R . A contemporary approach to validity arguments: a practical guide to Kane's framework . Med Educ . 2015 ; 49 ( 6 ): 560 575 . Crossref PubMed Google Scholar

6. No authors listed . England Royal College of Surgeons of international fellowship examination (JSCFE) . 2020 . https://www.rcseng.ac.uk/education-and-exams/exams/search/international-fellowship-examination-jscfe/ (date last accessed 23 May 2020 ). Google Scholar

7. Boevé AJ , Meijer RR , Albers CJ , Beetsma Y , Bosker RJ . Introducing computer-based testing in high-stakes exams in higher education: results of a field experiment . PLoS One . 2015 ; 10 ( 12 ): e0143616 13 . Crossref PubMed Google Scholar

8. Fallatah HI , Tekian A , Park YS , Al Shawa L , Al SL . The validity and reliability of the sixth-year internal medical examination administered at the King Abdulaziz University medical college . BMC Med Educ . 2015 ; 15 ( 1 ): 1 6 . Crossref PubMed Google Scholar

9. Iqbal IZ , Naqvi S , Abeysundara L , Narula AA . The value of oral assessments: a review . Bulletin . 2010 ; 92 ( 7 ): 1 6 . Google Scholar

10. Boissin C , Blom L , Wallis L , Laflamme L . Image-Based teleconsultation using smartphones or tablets: qualitative assessment of medical experts . Emerg Med J . 2017 ; 34 ( 2 ): 95 99 . Crossref PubMed Google Scholar

11. Deldar K , Bahaadinbeigy K , Tara SM . Teleconsultation and clinical decision making: a systematic review . Acta Inform Med . 2016 ; 24 ( 4 ): 286 292 . Crossref PubMed Google Scholar

12. Greenhalgh T , Vijayaraghavan S , Wherton J , et al. Virtual online consultations: advantages and limitations (vocal) study . BMJ Open . 2016 ; 6 ( 1 ): e009388 13 . Crossref PubMed Google Scholar

13. Mooney CJ , Peyre SE , Clark NS , Nofziger AC . Rapid transition to online assessment: practical steps and unanticipated advantages . Med Educ . 2020 ; 54 ( 9 ): 857 858 . Crossref PubMed Google Scholar

14. Kononowicz AA , Woodham LA , Edelbring S , et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration . J Med Internet Res . 2019 ; 21 ( 7 ): e14676 20 . Crossref PubMed Google Scholar

15. Ward RC , Muckle TJ , Kremer MJ , Krogh MA . Computer-Based case simulations for assessment in health care: a literature review of validity evidence . Eval Health Prof . 2019 ; 42 ( 1 ): 82 102 . Crossref PubMed Google Scholar

16. Lara S , Foster CW , Hawks M , Montgomery M . Remote assessment of clinical skills during COVID-19: a virtual, High-Stakes, Summative pediatric objective structured clinical examination . Acad Pediatr . 2020 ; 20 ( 6 ): 760 761 . Crossref PubMed Google Scholar

17. Hulsman RL , Mollema ED , Hoos AM , de Haes JCJM , Donnison-Speijer JD . Assessment of medical communication skills by computer: assessment method and student experiences . Med Educ . 2004 ; 38 ( 8 ): 813 824 . Crossref PubMed Google Scholar

18. Kyaw BM , Posadzki P , Paddock S , et al. Effectiveness of digital education on communication skills among medical students: systematic review and meta-analysis by the digital health education collaboration . J Med Internet Res . 2019 ; 21 ( 8 ): e12967 . Crossref PubMed Google Scholar

19. O'Rourke SR , Branford KR , Brooks TL , et al. The emotional and behavioral impact of delivering bad news to virtual versus real standardized patients: a pilot study . Teach Learn Med . 2020 ; 32 ( 2 ): 139 149 . Crossref PubMed Google Scholar

20. Guiton G , Hodgson CS , Delandshere G , Wilkerson L . Communication skills in standardized-patient assessment of final-year medical students: a psychometric study . Adv Health Sci Educ Theory Pract . 2004 ; 9 ( 3 ): 179 187 . Crossref PubMed Google Scholar

21. Jabeen D . Use of simulated patients for assessment of communication skills in undergraduate medical education in obstetrics and gynaecology . J Coll Physicians Surg Pak . 2013 ; 23 ( 1 ): 16 19 . PubMed Google Scholar

22. Tates K , Antheunis ML , Kanters S , Nieboer TE , Gerritse MB . The effect of Screen-to-Screen versus face-to-face consultation on doctor-patient communication: an experimental study with simulated patients . J Med Internet Res . 2017 ; 19 ( 12 ): e421 e421 . Crossref PubMed Google Scholar

23. Muula AS . Successes and challenges of Malawi's only medical school . Croat Med J . 2009 ; 50 ( 2 ): 189 191 . Crossref PubMed Google Scholar

24. Smith PJW , Garden OJ , Wigmore SJ , Borgstein E , Dewhurst D , et al. The effectiveness of an online, distance-learning Master’s in Surgical Sciences programme in Malawi . Afr J Health Prof Educ . 2018 ; 10 ( 3 ): 159 165 . Google Scholar

25. Kuikka M , Kitola M , Laakso M-J . Challenges when introducing electronic exam . Research in Learning Technology . 2014 ; 22 : 22817 . Google Scholar

26. No authors listed . Annual report and accounts for the year ended 30 June . 2019 . https://register-of-charities.charitycommission.gov.uk/charity-search?p_p_id=uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet&p_p_lifecycle=2&p_p_state=maximized&p_p_mode=view&p_p_resource_id=%2Faccountsresource&p_p_cacheability=cacheLevelPage&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_fileName=0000212808_AC_20190630_E_C.pdf&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_objectiveId=A10229642&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_priv_r_p_mvcRenderCommandName=%2Faccounts-and-annual-returns&_uk_gov_ccew_onereg_charitydetails_web_portlet_CharityDetailsPortlet_priv_r_p_organisationNumber=212808 (date last accessed 16 October 2020 ). Google Scholar

27. Sam AH , Peleva E , Fung CY , et al. Very short answer questions: a novel approach to Summative assessments in pathology . Adv Med Educ Pract . 2019 ; 10 : 943 948 . Crossref PubMed Google Scholar

28. Nisbett RE , Wilson TD , DeCamp Wilson T . The halo effect: evidence for unconscious alteration of judgments . J Pers Soc Psychol . 1977 ; 35 ( 4 ): 250 256 . Google Scholar

29. Sukadarmika G , Hartati RS , Linawati L , et al. Proposed model for e-exam availability in WLAN environment . 2016 International Conference on smart green technology in electrical and information systems (ICSGTEIS . Denpasar : Udayana University , 2016 : 89 92 . Google Scholar

30. Evans MDR , Kelley P , Kelley J . Identifying the best times for cognitive functioning using new methods: matching university times to undergraduate chronotypes . Front Hum Neurosci . 2017 ; 11 : 188 . Crossref PubMed Google Scholar

31. Kluth D , Hill J , De'Ath P , Jaap A . Introduction of a new online platform for formative and summative assessments in the MBChB programme . 2020 . http://www.docs.hss.ed.ac.uk/iad/Learning_teaching/Academic_teaching/Events/LT_Conference/2018/s1/1C2_KluthD_LTConf2018.pdf (date last accessed 15 June 2020 ). Google Scholar

32. Sharma S , Sharma V , Sharma M , Awasthi B , Chaudhary S . Formative assessment in postgraduate medical education - Perceptions of students and teachers . Int J Appl Basic Med Res . 2015 ; 5 ( Suppl 1 ): 66 70 . Crossref PubMed Google Scholar

33. Arja SB , Arja SB , Ramey J , et al. Logistics behind the OSCE in undergraduate medical education and an experience at a Caribbean medical school . MedEdPublish . 2018 ; 7 ( 4 ). Google Scholar

34. Whitelock DM , Brasher A . Developing a Roadmap for e-Assessment: Which Way Now? Danson M . ed . Proceedings of the 10th CAA international computer assisted assessment conference . Loughborough : Loughborough University , 2006 : 487 501 . Google Scholar

35. Egarter S , Mutschler A , Tekian A , Norcini J , Brass K . Medical assessment in the age of digitalisation . BMC Med Educ . 2020 ; 20 ( 1 ): 101 . Crossref PubMed Google Scholar

36. Pagram J , Cooper M , Jin H , Campbell A . Tales from the exam room: Trialing an e-exam system for computer education and design and technology students . Educ Sci . 2018 ; 8 ( 4 ): 188 . Google Scholar

37. Hillier M . To type or handwrite: student’s experience across six e-Exam trials . In : Reiners T , von Konsky BR , Gibson D . eds . Globally connected, digitally enabled. Proceedings ascilite 2015 . Perth : Perth University , 2016 : 143 154 . Google Scholar

38. Arndt CB , Guly UM , McManus IC . Preclinical anxiety: the stress associated with a VIVA voce examination . Med Educ . 1986 ; 20 ( 4 ): 274 280 . Crossref PubMed Google Scholar

39. Imran M , Doshi C , Kharadi D . Structured and unstructured VIVA voce assessment: a double-blind, randomized, comparative evaluation of medical students . Int J Health Sci . 2019 ; 13 ( 2 ): 3 9 . PubMed Google Scholar

40. Sabbah Y , Saroit I , Kotb A . Synchronous authentication with bimodal biometrics for e-assessment: A theoretical model . 2012 6th International Conference on sciences of electronics, technologies of information and telecommunications (SETIT . New York : IEEE , 2012 : 139 145 . Google Scholar

41. Alotaibi SJ . Using biometrics authentication via fingerprint recognition in e-exams in e-learning environment . The 4th Saudi International Conference, Friday 30 and Saturday 31 July 2010 . Manchester : The University of Manchester , 2010 . Google Scholar

42. Flior E , Kowalski K . Continuous biometric user authentication in online examinations . 2010 seventh International Conference on information technology: new generations . New York : IEEE , 2010:4 : 88 492 . Google Scholar

43. Abisado MB , Gerardo BD , Fajardo AC . Towards Keystroke Analysis using Neural Network for Multi-Factor Authentication of Learner Recognition in On-Line Examination . Manila International Conference on “Trends in Engineering and Technology” . Manila : IAETR , 2017 : 71 74 . Google Scholar

44. Sirovich L , Kirby M . Low-Dimensional procedure for the characterization of human faces . J Opt Soc Am A . 1987 ; 4 ( 3 ): 519 524 . Crossref PubMed Google Scholar

45. Sukmandhani AA , Sutedja I . Face Recognition Method for Online Exams . Proceedings of 2019 International Conference on information management and technology . New : IEEE , 2019 : 175 179 . Google Scholar

46. Rudrapal D , Das S , Debbarma S , Kar N , Debbarma N , et al. Voice recognition and authentication as a proficient biometric tool and its application in online exam for p.h people . Int J Comput Appl . 2012 ; 39 ( 12 ): 6 12 . Google Scholar

47. Curran J , Curran K . Biometric Authentication Techniques in Online Learning Environments Kumar AV . ed . Biometric authentication in online learning environments . Hershey : IGI Global , 2019 : 266 278 . Google Scholar

48. Bawarith R , Basuhail A , Fattouh A . Gamalel-Din S. E-exam Cheating Detection System . Int J Adv Comput Sci Appl . 2017 ; 8 : 176 181 . Google Scholar

49. Alfaries A , Mengash H , Yasar A , Shakshuki E . Advances in data science, cyber security and it applications . Cham : Springer Nature , 2019 . Google Scholar

50. Cluskey GR , Ehlen CR , Raiborn MH . Thwarting online exam cheating without proctor supervision . J Acad Bus Ethics . 2011 ; 4 : 1 8 . Google Scholar

51. Haney WM , Clarke MJ , Tests Con . Prevalence, Detection, and Implications for Online Testing . In : Anderman EM , Murdock TB . Psychology of academic cheating . London : Elsevier , 2007 : 255 287 . Google Scholar

52. Rosmansyah Y , Ritonga MH , Hardi AB . An Attack-Defence Tree of e-Exam System. . iJET . 2019 ; 14 : 251 259 . Google Scholar

53. Luo H , Chen Z , Li J , Vasilakos A V . Preventing distributed Denial-of-Service flooding attacks with dynamic path identifiers . IEEE Trans Inf Forensics Secur . 2017 . Google Scholar

54. Naik N , Jenkins P , Cooke R , et al. Augmented windows fuzzy firewall for preventing denial of service attack . Proceedings of the 2017 IEEE International Conference on fuzzy systems (FUZZ-IEEE . Red Hook : Curran Associates , 2017 : 1 6 . Google Scholar

55. Djuraev S , Choi J-G , Sohn K-S , Nam SY . Channel hopping scheme to mitigate jamming attacks in wireless LANs . EURASIP J Wirel Commun Netw . 2017 ; 2017 ( 1 ). Google Scholar

56. No authors listed . The Examplify testing application from ExamSoft . 2020 . https://examsoft.com/resources/examplify-testing-application-examsoft (date last accessed 20 September 2020 ). Google Scholar

57. Ponce BA , Brabston EW , Shin Zu ­ , et al. Telemedicine with mobile devices and augmented reality for early postoperative care . Annu Int Conf IEEE Eng Med Biol Soc . 2016 ; 2016 : 4411 4414 . Crossref PubMed Google Scholar

58. Kim-Berman H , Karl E , Sherbel J , Sytek L , Ramaswamy V . Validity and user experience in an augmented reality virtual tooth identification test . J Dent Educ . 2019 ; 83 ( 11 ): 1345 1352 . Crossref PubMed Google Scholar

Author contributions

M. Pettit: Conducted the literature search, Wrote the manuscript.

S. Shukla: Designed the study, Conducted the literature search, Wrote the manuscript.

J. Zhang: Conducted the literature search, Wrote the manuscript.

K. H. Sunil Kumar: Conceptualized and designed the study, Conducted the literature search, Wrote the manuscript.

V. Khanduja: Conceptualized and designed the study, Revised the manuscript.

Funding statement

No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article.


Follow K. H. Sunil Kumar @kumarkhs

Follow Vikas Khanduja @CambridgeHipDoc

© 2021 Author(s) et al. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives (CC BY-NC-ND 4.0) licence, which permits the copying and redistribution of the work only, and provided the original author and source are credited. See https://creativecommons.org/licenses/by-nc-nd/4.0/