Backgroud: Allogeneic transfusion rates after primary hip and knee arthroplasty are used as quality indicators for hospitals, but hospital comparisons may be hampered by low event rates. Extended hospital stay is often used and may be more suitable as an alternative. This study aims to assess whether transfusion rates and extended hospital stay can be used to reliably rank hospitals. Methods. We used the baseline data from the LISBOA implementation trial, where data on patient characteristics and outcomes were collected in a sample of approximately 100 patients undergoing elective primary total hip or knee arthroplasty for each of the 23 participating hospitals. We calculated the reliability of ranking (Rankability) of transfusion rates and extended hospital stay (> 4 postoperative days), using fixed and random effects logistic regression analysis, by dividing the between-hospital variation to the sum of within and between-hospital variation. Rankability thus shows which part of the hospital differences are true differences and not due to random variation. Results. 1163 total hip and 986 total knee procedures were assessed. After adjustment for patient characteristics the odds ratio (OR) of receiving a transfusion in a hospital after total hip ranged from 0.72 to 1.38 and from 0.30 to 3.30 in total knee. Rankability was 17% for hip and 36% for knee arthroplasty, meaning that only 17% and 36% are true hospital differences. Larger hospital variation was found for extended hospital stay (OR range [0.28–3.51] for hip and [0.10–9.95] for knee arthroplasty), and better rankability. Conclusion. Although allogeneic transfusion rates are useful for monitoring quality within hospitals, they should not be used for ranking hospitals. A large proportion of differences in transfusion rates between hospitals is due to random variation, suggesting that this outcome is not suitable for ranking hospitals contrary to extended hospital stay. Level of evidence. Level 2. Financial disclosure. This study was funded by a grant from The Netherlands Organisation for Health Research and Development and by a grant from Sanquin Blood Supply. Conflict of interest. The authors declare that there are no conflicts of interest. Approval. The Medical Ethical Committee of the Leiden University Medical Center decided that ethical approval was not required under Dutch National law for this type of study