Advertisement for orthosearch.org.uk
Bone & Joint Research Logo

Receive monthly Table of Contents alerts from Bone & Joint Research

Comprehensive article alerts can be set up and managed through your account settings

View my account settings

Visit Bone & Joint Research at:

Loading...

Loading...

Open Access

Research

Hip and knee replacement patients prefer pen-and-paper questionnaires

Implications for future patient-reported outcome measure studies



Download PDF

Abstract

Objectives

Electronic forms of data collection have gained interest in recent years. In orthopaedics, little is known about patient preference regarding pen-and-paper or electronic questionnaires. We aimed to determine whether patients undergoing total hip (THR) or total knee replacement (TKR) prefer pen-and-paper or electronic questionnaires and to identify variables that predict preference for electronic questionnaires.

Methods

We asked patients who participated in a multi-centre cohort study investigating improvement in health-related quality of life (HRQoL) after THR and TKR using pen-and-paper questionnaires, which mode of questionnaire they preferred. Patient age, gender, highest completed level of schooling, body mass index (BMI), comorbidities, indication for joint replacement and pre-operative HRQoL were compared between the groups preferring different modes of questionnaire. We then performed logistic regression analyses to investigate which variables independently predicted preference of electronic questionnaires.

Results

A total of 565 THR patients and 387 TKR patients completed the preference question. Of the THR patients, 81.8% (95% confidence interval (CI) 78.4 to 84.7) preferred pen-and-paper questionnaires to electronic questionnaires, as did 86.8% (95% CI 83.1 to 89.8) of TKR patients. Younger age, male gender, higher completed level of schooling and higher BMI independently predicted preference of electronic questionnaires in THR patients. Younger age and higher completed level of schooling independently predicted preference of electronic questionnaires in TKR patients.

Conclusions

The majority of THR and TKR patients prefer pen-and-paper questionnaires. Patients who preferred electronic questionnaires differed from patients who preferred pen-and-paper questionnaires. Restricting the mode of patient-reported outcome measures to electronic questionnaires might introduce selection bias.

Cite this article: Bone Joint Res 2013;2:238–44.

Article focus

Based on the recent literature, we hypothesised that patients would prefer electronic questionnaires to pen-and-paper questionnaires after total hip and total knee replacement

We hypothesised that elderly patients would prefer pen-and-paper questionnaires

Key messages

The vast majority of patients prefer pen-and-paper questionnaires

Patients who prefer electronic questionnaires are generally younger and have completed a higher level of schooling

Strengths and limitations

Strengths: a large sample size and a limited extent of optimism in our statistical models

Limitations: This study was performed using pen-and-paper questionnaires, which might lead to an overestimation of the actual preference of the pen-and-paper questionnaires

Introduction

Traditionally, the assessment of outcome in orthopaedics has focussed on technical aspects. In total hip (THR) or knee replacement (TKR), the cumulative incidence of revision surgery is often used to compare the outcome of different implants or surgical techniques.1 The underlying assumption of the traditional orthopaedic approach is that the technical aspects are the most important determinants of clinical success. However, a technically well-performed joint replacement does not guarantee clinical success, as no information is provided on functional status and pain. Additionally, the indication for revision surgery varies widely between orthopaedic surgeons.2 Patient-reported outcome measures (PROMs), defined as questionnaires that are completed by patients, provide complementary information as they give an impression of a patient’s experience of the surgical procedure and their concerns with regard to health status, health-related quality of life (HRQoL) and the results of the treatment received.3

PROMs can be measured using traditional pen-and-paper questionnaires or various electronic counterparts including touch screens,4 personal digital assistants,5,6 tablets or mobile phones.7 Expected advantages of electronic questionnaires include more complete data capturing, immediate availability of results and lower costs of administrating and entering data.6,8

On the other hand, electronic questionnaires may induce selection bias. A meta-analysis performed in 2008 showed that mail surveys had higher response rates than those based online.9 A recent randomised controlled trial, in which 2400 patients were randomised to receive either a pen-and-paper questionnaire or an internet-based questionnaire at four years after THR, revealed an enormous difference in response rate: 92% for the pen-and-paper group versus 49% for the internet-based group.10 Selection bias can occur if the association between exposure and outcome differs between participants and all eligible patients.8

To our knowledge, no study has investigated patient preference for electric questionnaires after THR and TKR. The majority of members of a senior citizens club prefers electronic to pen-and-paper questionnaires.11 Given the similar age of THR/TKR patients, we would expect a preference for electronic questionnaires. We aimed to estimate the proportion of patients who prefer pen-and-paper questionnaires to electronic questionnaires and to estimate predictors of electronic questionnaire preference.

Materials and Methods

The current study is part of a multi-centre cohort study of HRQoL after THR/TKR (NTR2190), performed from August 2010 to August 2011.12-15 Institutional review board approval was obtained from all participating centres and all patients gave written informed consent (CCMO-Nr:NL29018.058.09;MEC-Nr:P09.189). The data used in this report constitutes a subset of patients who underwent primary THR or TKR and who completed pre-operative HRQoL questionnaires along with a question regarding their preference for a mode of questionnaire at a mean of three years (1.5 to 6) after surgery.

We performed this study in order to investigate the preference for a mode of questionnaire for future studies in HRQoL after THR or TKR in a Dutch population. A prerequisite for such future studies is that patients can participate without outpatient department visits, thereby facilitating participation and forestalling the occurrence of selection bias. We selected a web-based questionnaire as the most feasible electronic option. At follow-up, we asked all THR and TKR patients which mode of questionnaire they preferred: pen-and-paper questionnaires or web-based electronic questionnaires, each completed at home.

In order to judge whether patients who preferred pen-and-paper questionnaires differed from patients who preferred electronic questionnaires, we compared age, gender, highest completed level of schooling, body mass index (BMI) categories (< 25 kg/m2, 25 to 30 kg/m2, 30 to 35 kg/m2, > 35 kg/m2), comorbidity, indication for joint replacement (osteoarthritis vs other indications) and pre-operative HRQoL between both groups.

We have aggregated the levels of schooling into an approximation of the social classes, on the assumption that level of schooling indexes the type of qualifications obtained, which in turn indicates the type of occupations available to the subject and hence their own adult social class.13 Thus: ‘university, higher vocational education and preparatory higher vocational and scientific education’ have been aggregated as indicating the professional and managerial social classes; ‘middle vocational education and preparatory middle vocational education’ have been aggregated as indicating the skilled non-manual and manual social classes; and ‘lower vocational education, elementary schooling and no formal education’ have been aggregated as indicating the semi- and unskilled manual social classes.

Comorbidity was measured using a patient-reported Charnley classification (A, patients in which the index operated hip or knee are affected only; B, patients in which the other hip or knee is affected as well; and C, patients with a hip or knee replacement and other affected joints and/or a medical condition which affects the patients’ ability to ambulate).16,17

HRQoL was measured two weeks before TKR/THR, using the Dutch version of the Short-Form 36 (SF-36).18,19 This questionnaire comprises 36 items covering eight domains (physical function, role physical, bodily pain, general health, vitality, social function, role emotional and mental health), for each of which a subscale score is calculated (100 indicating no symptoms and 0 indicating extreme symptoms). Additionally, these scales are incorporated into two summary measures: a physical component summary (PCS) and a mental component summary (MCS). Missing items were imputed whenever possible according to Ware et al.19 We compared pre-operative PCS and MCS between both preference groups.

Statistical analysis

We performed all analyses separately for THR and TKR patients, as clinically important differences vary considerably between these patient groups.20 We performed descriptive analyses of baseline patient characteristics. In order to predict which factors increased the probability of preference for electronic questionnaires, we performed multivariate mixed model logistic regression analyses. We considered the following potential predictors: age, gender, highest completed level of schooling category, BMI category, Charnley classification of comorbidity, indication for joint replacement and pre-operative PCS and MCS scores. In the mixed model regression analyses, patient preference was the dependent variable, all potential predictors were included as fixed effects and centre was included as a random effect. The explained variation was estimated using Nagelkerke’s generalised r2 and the discriminative ability was estimated using the area under the receiver operating characteristic (ROC) curve (AUC).21 The extent of optimism in the r2 and AUC estimates was estimated using bootstrap resampling (n = 1000 bootstrap samples).22-24 All analyses were performed using R v2.15.2 (R Development Core Team, Vienna, Austria).25

Results

Patient characteristics are shown in Table I. A total of 565 THR patients and 387 TKR patients completed the preference question. Pen-and-paper questionnaires were preferred by 462 THR patients (81.8% (95% confidence interval (CI) 78.4 to 84.7) and by 336 TKR patients (86.8% (95% CI 83.1 to 89.8)) (Table II).

Table I

Patient characteristics (THR, total hip replacement; TKR, total knee replacement)

Characteristic THR (n = 565) TKR (n = 387)
Mean (sd) age at THR/TKR (yrs)  65.9 (10.6)  68.9 (9.7)
Male (n, %) 198 (35.0) 126 (32.6)
Osteoarthritis (n, %) 486 (86.0) 346 (89.4)
Mean (sd) pre-operative Short-Form 36
  Physical component summary  38.9 (9.61)  40.6 (9.53)
  Mental component summary  51.8 (10.8)  51.5 (10.2)
Mean follow-up (yrs) (sd; range)  3.20 (1.13; 1.5 to 6.0)  3.14 (1.12; 1.3 to 6.0)
Body mass index at follow-up (n, %)
  < 25 kg/m2 194 (34.3)  69 (17.8)
  25 to 30 kg/m2 242 (42.9) 171 (44.1)
  30 to 35 kg/m2  97 (17.1)  92 (23.8)
  > 35 kg/m2  32 (5.7)  55 (14.2)
Charnley comorbidity classification (n, %)
  A 123 (23.3)  54 (14.6)
  B  75 (14.2)  39 (10.5)
  C 331 (62.6) 278 (74.9)
Highest completed level of education (n, %)
  University, higher vocational education and preparatory higher vocational & scientific education 115 (22.6)  52 (15.4)
  Middle vocational education and preparatory middle vocational education 186 (36.6) 120 (35.5)
  Lower vocational education, elementary schooling and no formal education 207 (40.7) 166 (49.1)

Table II

Proportion of patients who prefer pen-and-paper questionnaires to electronic questionnaires (THR, total hip replacement; TKR, total knee replacement; CI, confidence interval)

THR (n = 565) TKR (n = 387)
Preferred questionnaire Patients (n) Proportion (95% CI) Patients (n) Proportion (95% CI)
Pen-and-paper 462 81.8 (78.4 to 84.7) 336 86.8 (83.1 to 89.8)
Electronic 103 18.2  51 13.2

Patient characteristics per preference group are shown in Table III and Table IV for THR and TKR patients, respectively. THR patients who preferred electronic questionnaires tended to be younger, more often male, more often obese, less comorbid, more often highly educated and had worse pre-operative physical health. Age, gender and highest completed level of education remained associated with mode of questionnaire preference while adjusting for age and gender (Table III). TKR patients who preferred electronic questionnaires were younger, more often male, less often morbidly obese, less often Charnley class B and more often Charnley class C, more often highly educated and had worse pre-operative physical health. Age and highest completed level of education remained associated with mode of questionnaire preference while adjusting for age and gender (Table IV).

Table III

Comparison of total hip replacement (THR) patients preferring pen-and-paper versus electronic questionnaires (OR, odds ratio; CI, confidence interval)

Characteristic Pen-and-paper Electronic Age- and gender- adjusted OR (95% CI)
Patients (n) 462 103
Mean (sd) age at THR (yrs)  67.5 (9.5)  58.5 (12.2) 0.93 (0.91 to 0.95)
Male (n, %) 140 (30.3)  58 (56.3) 0.35 (0.22 to 0.56)
Osteoarthritis (n, %) 406 (88.5)  77 (75.5) 0.75 (0.40 to 1.43)
Mean (sd) pre-operative Short-Form 36
  Physical component summary  39.5 (8.9)  35.8 (11.9) 1.00 (0.97 to 1.03)
  Mental component summary  51.2 (10.6)  54.8 (10.9) 1.01 (0.99 to 1.04)
Mean (sd) follow-up (yrs)  3.17 (1.13)  3.31 (1.11) -
Body mass index at follow-up (n, %)
  < 25 kg/m2 157 (35.4)  29 (29.3) Reference
  25 to 30 kg/m2 187 (42.1)  46 (46.5) 1.32 (0.75 to 2.32)
  30 to 35 kg/m2  78 (17.6)  15 (15.2) 0.88 (0.42 to 1.83)
  > 35 kg/m2  22 (5.0)  9 (9.1) 2.18 (0.84 to 5.69)
Charnley comorbidity classification (n, %)
  A 100 (23.2)  23 (23.5) Reference
  B  57 (13.2)  18 (18.4) 1.28 (0.59 to 2.79)
  C 274 (63.6)  57 (58.2) 1.10 (0.61 to 1.98)
Highest completed level of education (n, %)
  University, higher vocational education and preparatory higher vocational & scientific education  78 (19.1)  37 (37.0) Reference
  Middle vocational education and preparatory middle vocational education 140 (34.3)  46 (46.0) 0.82 (0.47 to 1.45)
  Lower vocational education, elementary schooling and no formal education 190 (46.6)  17 (17.0) 0.24 (0.12 to 0.47)

Table IV

Comparison of total knee replacement (TKR) patients preferring pen-and-paper versus electronic questionnaires (OR, odds ratio; CI, confidence interval)

Characteristic Pen-and-paper Electronic Age- and gender- adjusted OR (95% CI)
Patients (n) 336 51
Mean (sd) age at TKR (yrs)  70.3 (8.9) 59.9 (9.7) 0.90 (0.86 to 0.93)
Male (n, %) 103 (30.7) 23 (45.1) 0.61 (0.31 to 1.18)
Osteoarthritis (n, %) 302 (90.4) 40 (81.6) 1.29 (0.48 to 3.45)
Mean (sd) pre-operative Short-Form 36
  Physical component summary  41.1 (8.8) 36.7 (12.9) 1.02 (0.98 to 1.06)
  Mental component summary  51.2 (10.4) 53.5 (8.9) 1.02 (0.99 to 1.06)
Mean (sd) follow-up (yrs)  3.10 (1.09)  3.44 (1.32) -
Body mass index at follow-up (n, %)
  < 25 kg/m2  55 (17.5) 10 (19.6) Reference
  25 to 30 kg/m2 136 (43.3) 25 (49.0) 0.86 (0.34 to 2.20)
  30 to 35 kg/m2  75 (23.9) 12 (23.5) 0.65 (0.23 to 1.83)
  > 35 kg/m2  48 (15.3)  4 (7.8) 0.41 (0.11 to 1.57)
Charnley comorbidity classification (n, %)
  A  48 (14.9)  6 (12.2) Reference
  B  36 (11.2)  3 (6.1) 0.83 (0.18 to 3.83)
  C 238 (73.9) 40 (81.6) 1.35 (0.52 to 3.52)
Highest completed level of education (n, %)
  University, higher vocational education and preparatory higher vocational & scientific education  34 (11.7) 18 (37.5) Reference
  Middle vocational education and preparatory middle vocational education  96 (33.1) 24 (50.0) 0.55 (0.24 to 1.26)
  Lower vocational education, elementary schooling and no formal education 160 (55.2)  6 (12.5) 0.08 (0.03 to 0.25)

Multivariate prediction of electronic questionnaire preference showed that lower age (p < 0.001), male gender (p < 0.001), higher completed level of schooling (p < 0.001) and higher BMI (p = 0.004) independently predicted preference of electronic questionnaires in THR patients (Table V). In TKR patients, multivariate prediction of electronic questionnaire preference showed that lower age (p < 0.001) and higher completed level of schooling (p < 0.001) independently predicted preference of electronic questionnaires (Table V). The prediction model for preference of electronic questionnaires in THR patients had an r2 of 0.31 with an optimism estimate of 0.04, yielding an optimism-corrected r2 estimate of 0.27. The AUC was 0.81, with an optimism estimate of -0.02, indicating absence of optimism. The prediction model for preference of electronic questionnaires in TKR patients had an r2 of 0.41 with an optimism estimate of -0.24, indicating absence of optimism. The AUC was 0.88, with an optimism estimate of -0.004, indicating absence of optimism.

Table V

Multivariate prediction for preference of electronic questionnaires in total hip and knee replacement patients. Odds ratios > 1 indicate higher odds of preferring an electronic questionnaire, per increasing predictor unit (CI, confidence interval; AUC, area under curve)

Characteristic* Odds ratio (95% CI) p-value
TOTAL HIP REPLACEMENT
r2 = 0.31; AUC = 0.81
Age at operation 0.93 (0.90 to 0.96) < 0.001
Male vs female gender 0.31 (0.17 to 0.56) < 0.001
Body mass index
  25 to 30 kg/m2 vs < 25 kg/m2 2.06 (1.03 to 4.11) 0.04
  30 to 35 kg/m2 vs < 25 kg/m2 1.17 (0.48 to 2.81) 0.73
  > 35 kg/m2 vs < 25 kg/m2 5.49 (1.74 to 17.3) 0.004
Other indications vs osteoarthritis 0.59 (0.28 to 1.26) 0.17
Charnley comorbidity classification
  A vs B 0.99 (0.40 to 2.42) 0.98
  A vs C 0.87 (0.43 to 1.78) 0.70
Education
  U+HVE+PHVSE vs MVE+PMVE 0.89 (0.45 to 1.77) 0.74
  U+HVE+PHVSE vs LVE+ES+NFE 0.27 (0.12 to 0.59) < 0.001
Short-Form 36 score
  Physical component summary 1.00 (0.97 to 1.03) 0.80
  Mental component summary 1.00 (0.98 to 1.03) 0.87
TOTAL KNEE REPLACEMENT
r2 = 0.41; AUC = 0.88
Age at operation 0.89 (0.84 to 0.94) < 0.001
Male vs female gender 0.53 (0.21 to 1.34) 0.18
Body mass index
  25 to 30 kg/m2 vs < 25 kg/m2 1.05 (0.26 to 4.28) 0.94
  30 to 35 kg/m2 vs < 25 kg/m2 1.27 (0.30 to 5.38) 0.75
  > 35 kg/m2 vs < 25 kg/m2 1.59 (0.28 to 8.91) 0.60
Other indications vs osteoarthritis 2.05 (0.53 to 7.89) 0.30
Charnley comorbidity classification
  A vs B 1.40 (0.23 to 8.58) 0.72
  A vs C 2.07 (0.58 to 7.31) 0.26
Education
  U+HVE+PHVSE vs MVE+PMVE 0.33 (0.13 to 0.85) 0.02
  U+HVE+PHVSE vs LVE+ES+NFE 0.04 (0.01 to 0.15) < 0.001
Short-Form 36 score
  Physical component summary 1.01 (0.96 to 1.06) 0.63
  Mental component summary 0.99 (0.95 to 1.04) 0.75
  1. * U+HVE+PHVSE, university, higher vocational education and preparatory higher vocational and scientific education; MVE+PMVE, middle vocational education and preparatory middle vocational education; LVE+ES+NFE, lower vocational education, elementary schooling and no formal education

Discussion

The vast majority of THR and TKR patients prefer pen-and-paper questionnaires. THR patients who prefer electronic questionnaires are younger, more often male, have completed higher levels of schooling and are more often obese. TKR patients who prefer electronic questionnaires are younger and have completed higher levels of schooling.

A limitation of our study is the mode of questionnaire used to capture the data. In this study, we invited patients to participate by conventional mail. Additionally, all questionnaires consisted of pen-and-paper questionnaires. Patients willing to participate in this study might be more inclined to prefer pen-and-paper questionnaires than THR and TKR patients in general, thus leading to an overestimation of the proportion of patients preferring pen-and-paper questionnaires. However, we consider it unlikely that the entire preference for pen-and-paper questionnaires is based on such selection bias. Additionally, the identified predictors for electronic questionnaire preference, such as age and completed level of schooling, are plausible, thereby indirectly validating our results.

Strengths of our study include the large sample size, which allows precise estimation and multivariate prediction of patient preference. Although the low r2 values indicate that not all variance is explained by the predictors, the high AUC values indicate that the prediction models have a high discriminatory ability. The limited extent of optimism in r2 and AUC estimates indicate that overfitting did not play a role in our study.24 In other words, it is unlikely that the prediction models in this study have captured the peculiarities in this data set; conversely, it is likely that predictions, based on this data, will be generalisable to other, similar populations.

Unfortunately, we do not have any information on the availability of internet access of our patients. Although The Netherlands is rated as one of the most mature internet markets,26 recent evidence suggests that non-users of the internet are more likely to be elderly,27 which could explain the preference for pen-and-paper questionnaires. Practical advantages of electronic questionnaires are stressed in the current orthopaedic literature.6,28 Patients are sometimes considered to prefer electronic questionnaires without any evidence supporting this claim.28 Although electronic questionnaires certainly appear more efficient, our results reveal limitations in line with the findings of Rolfson et al.10 Future studies, which only measure PROMs using electronic questionnaires, might suffer from limited generalisability, as elderly and less educated patients are less likely to participate. Moreover, selection bias might occur if the association of interest is related to age or social class.

When planning a study in which PROMs will be completed by THR and TKR patients at home, we recommend using pen-and-paper questionnaires, despite their logistic limitations. Such studies should at least provide the option of pen-and-paper questionnaires, in order to prevent selection bias by questionnaire mode.


Correspondence should be sent to Dr J. C. Keurentjes; e-mail:

1 Keurentjes JC , FioccoM, SchreursBW, et al.Revision surgery is overestimated in hip replacement. Bone Joint Res2012;1:258262.CrossrefPubMed Google Scholar

2 Nelissen RG , BrandR, RozingPM. Survivorship analysis in total condylar knee arthroplasty: a statistical review. J Bone Joint Surg [Am]1992;74-A:383389. Google Scholar

3 Fitzpatrick R , DaveyC, BuxtonMJ, JonesDR. Evaluating patient-based outcome measures for use in clinical trials. Health Technol Assess1998;2:174.PubMed Google Scholar

4 Rolfson O , KärrholmJ, DahlbergLE, GarellickG. Patient-reported outcomes in the Swedish Hip Arthroplasty Register: results of a nationwide prospective observational study. J Bone Joint Surg [Br]2011;93-B:867875.CrossrefPubMed Google Scholar

5 Kvien TK , MowinckelP, HeibergT, et al.Performance of health status measures with a pen based personal digital assistant. Ann Rheum Dis2005;64:14801484.CrossrefPubMed Google Scholar

6 Saleh KJ , RadosevichDM, KassimRA, et al.Comparison of commonly used orthopaedic outcome measures using palm-top computers and paper surveys. J Orthop Res2002;20:11461151.CrossrefPubMed Google Scholar

7 Bellamy N , WilsonC, HendrikzJ, et al.Osteoarthritis Index delivered by mobile phone (m-WOMAC) is valid, reliable, and responsive. J Clin Epidemiol2011;64:182190.CrossrefPubMed Google Scholar

8 van Gelder MMHJ , PijpeA. E-epidemiology: a comprehensive update. OA Epidemiology2013;1:5. Google Scholar

9 Manfreda K , BosnjakM, BerzelakJ, HaasI, VehovarV. Web surveys versus other survey modes: a meta-analysis comparing response rates. Int J Market Res2008;50:79104. Google Scholar

10 Rolfson O , SalomonssonR, DahlbergLE, GarellickG. Internet-based follow-up questionnaire for measuring patient-reported outcome after total hip replacement surgery-reliability and response rate. Value Health2011;14:316321.CrossrefPubMed Google Scholar

11 Ryan JM , CorryJR, AttewellR, SmithsonMJ. A comparison of an electronic version of the SF-36 General Health Questionnaire to the standard paper version. Qual Life Res2002;11:1926.CrossrefPubMed Google Scholar

12 Keurentjes JC , FioccoM, So-OsmanC, et al.Patients with severe radiographic osteoarthritis have a better prognosis in physical functioning after hip and knee replacement: a cohort-study. PLoS One2013;8:59500.CrossrefPubMed Google Scholar

13 Keurentjes JC , BlaneD, BartleyM, et al.Socio-economic position has no effect on improvement in health-related quality of life and patient satisfaction in total hip and knee replacement: a cohort study. PLoS One2013;8:56785.CrossrefPubMed Google Scholar

14 Keurentjes JC, Fiocco M, Nelissen RG. Willingness to undergo surgery again validated clinically important differences in health-related quality of life after total hip replacement or total knee replacement surgery. J Clin Epidemiol 2013:Epub. Google Scholar

15 Evangelou E, Kerkhof HJ, Styrkarsdottir U, et al. A meta-analysis of genome-wide association studies identifies novel variants associated with osteoarthritis of the hip. Ann Rheum Dis 2013:Epub. Google Scholar

16 Charnley J . The long-term results of low-friction arthroplasty of the hip performed as a primary intervention. J Bone Joint Surg [Br]1972;54-B:6176.PubMed Google Scholar

17 Dunbar MJ , RobertssonO, RydL. What’s all that noise?: the effect of co-morbidity on health outcome questionnaire results after knee arthroplasty. Acta Orthop Scand2004;75:119126. Google Scholar

18 Aaronson NK , MullerM, CohenPD, et al.Translation, validation, and norming of the Dutch language version of the SF-36 Health Survey in community and chronic disease populations. J Clin Epidemiol1998;51:10551068.CrossrefPubMed Google Scholar

19 Ware JE, Snow KK, Kosinski M, Gandek B. SF-36 health survey: manual and interpretation guide. Boston: New England Medical Center, 1993. Google Scholar

20 Keurentjes JC , Van TolFR, FioccoM, SchoonesJW, NelissenRG. Minimal clinically important differences in health-related quality of life after total hip or knee replacement: a systematic review. Bone Joint Res2012;1:7177. Google Scholar

21 Nagelkerke NJD . A note on a general definition of the coefficient of determination. Biometrika1991;78:691692. Google Scholar

22 Steyerberg EW , EijkemansMJ, HarrellFE, HabbemaJD. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med2000;19:10591079.CrossrefPubMed Google Scholar

23 Steyerberg EW , BleekerSE, MollHA, GrobbeeDE, MoonsKG. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol2003;56:441447.CrossrefPubMed Google Scholar

24 Steyerberg EW. Clinical prediction models: a practical approach to development, validation and updating. New York: Springer, 2009. Google Scholar

25 No authors listed. R: a language and environment for statistical computing. Vol. 1. Vienna: R Foundation for Statistical Computing, 2008. Google Scholar

26 Deutskens E , de RuyterK, WetzelsM, OosterveldP. Response rate and response quality of internet-based surveys: an experimental study. Marketing Letters2004;15:2136. Google Scholar

27 Brandtzæg PB , HeimJ, KarahasanovicA. Understanding the new digital divide: a typology of internet users in Europe. Int J Human-Computer Studies2011;69:123138. Google Scholar

28 Lee SJ , KavanaughA, LenertL. Electronic and computer-generated patient questionnaires in standard care. Best Pract Res Clin Rheumatol2007;21:637647.CrossrefPubMed Google Scholar

Funding statement:

Ongoing research grant of the Dutch Arthritis Association to Professor R. G. H. H Nelissen, Department of Orthopaedic Surgery, Leiden University Medical Center: grant number LLP-013 (http://www.reumafonds.nl/informatie-voor-doelgroepen/professionals/ongoing-research). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author contributions:

J. C. Keurentjes: Study conception and design, Data collection, Data analysis, Drafting of the article

M. Fiocco: Study conception and design, Data analysis, Critical revision of the article

C. So-Osman: Study conception and design, Data collection, Drafting of the article, Critical revision of the article

R. Onstenk: Data collection, Drafting of the article, Critical revision of the article

A. W. M. M. Koopman-Van Gemert: Data collection, Drafting of the article, Critical revision of the article

R. G. Pöll: Data collection, Drafting of the article, Critical revision of the article

R. G. H. H. Nelissen: Study conception and design, Data collection, Drafting of the article, Critical revision of the article

ICMJE Conflict of Interest:

None declared

©2013 The British Editorial Society of Bone & Joint Surgery. This is an open-access article distributed under the terms of the Creative Commons Attributions licence, which permits unrestricted use, distribution, and reproduction in any medium, but not for commercial gain, provided the original author and source are credited.