Surveys can be a valuable way of gathering feedback from patients, but by asking the wrong questions you may not be making the most of this resource
Patient surveys are now an essential tool for general practices to assess the level of patient satisfaction and experience of their services, and is the one consistent dimension across a range of domains considered essential in the measurement of the quality of care provided. Despite this, it is not uncommon for the task of developing the practice questionnaire to become the responsibility of the practice manager who may have little or no knowledge of questionnaire design or best practice.
Survey and questionnaire design is a skill-based process, and as the aim is to obtain reliable and valid information from patients for the practice to act upon, awareness of the key principles of questionnaire design is essential.1
To examine the quality of in-house designed patient survey questionnaires, we conducted a review of patient surveys available on the websites of 12 randomly-selected general practices in a clinical commissioning group (CCG) area using recognised best practice as criteria.
The aim of this article is to illustrate a few of the typical design errors identified in the review and provide some guidance and tips for practice managers as to how these errors can be avoided.
Findings
Of the 12 questionnaires reviewed, a key finding was that none of the questionnaires addressed the needs and experiences of patient groups with special needs or long-term illness. There was also considerable variation in the information collected which ranged from a few questions on opening hours and access, to more comprehensive coverage including access, staff friendliness, making appointments etc.
Of the 12 questionnaires reviewed, eight were identified as having one or more significant design faults which could impact on the validity of data collected. These included ambiguity in the wording of questions, insufficient or inappropriate answer options, questions being cognitively demanding for patients to complete, overuse of skip patterns and poor layout. A selection of these faults in design are illustrated in the following section.
Addressing the issues
Ensuring the extent and relevance of the questionnaire’s content is the primary focus in establishing its validity, and patient groups should be a part of this process. However, to gain real insight into the group’s experiences, needs and recommendations, it is best to gather information with face-to-face group sessions rather than virtually.
This is because group sessions enable participant interaction and for the moderator to modify the topics to be discussed as the discussions unfold.2
Three examples identified in the review of the type of design issues often found in questionnaire design are illustrated below with suggested revisions.
Example 1
On a scale of 1-5 with 1 being not satisfied and 5 being very satisfied could you please rate the following.
Friendliness of GPs and nurses
This type of question is known as a ‘double-barrel question’ which is where two questions being asked in one. The problem for the patient is not being provided with an opportunity to respond should the patient’s experiences of friendliness be different for GPs and nurses. If some nurses or GPs are less friendly than others, or the patient has only ever seen one GP or nurse, or has never seen a nurse, they do not have the opportunity to differentiate. The question should be rewritten as two questions along the lines shown below:
Revised question: How satisfied or dissatisfied are you overall with the friendliness of the doctor(s) who have seen you?
– Very satisfied
– Quite satisfied
– Neither satisfied or dissatisfied
– Quite dissatisfied
– Very dissatisfied
– Have not been seen by a doctor
Revised question: How satisfied or dissatisfied are you overall with the friendliness of the nurse(s) who have seen you?
– Very satisfied
– Quite satisfied
– Neither satisfied or dissatisfied
– Quite dissatisfied
– Very dissatisfied
– Have not been seen by a nurse
Example 2
On a scale of 1-10 (where 1 is very poor and 10 is excellent) how satisfied are you with the quality of care received from our nurses?
(If you are not satisfied, what would improve things for you?)
Scales with a 10 point range without adjective anchors can be cognitively demanding for the patient to complete. Based on much field and academic research, a 10-point scale is likely to yield ratings concentrated between scores of 7 and 9, with little of the lower half of the scale ever being used. The complexity in answering the question is further increased by the patient having to judge at what point from 1-10 is “not satisfied” in order to comment. The question also assumes the patient has seen a practice nurse at some time.
Revised question: In general how would you rate the quality of care you have received from any of the practice nurses?
– Poor
– Fair
– Good
– Very Good
– Excellent
– Have not received care from a practice nurse
If you rated the quality of care you have received either as fair or poor please explain why.
Example 3
About the practice. Satisfaction of opening hours.
– Poor
– Fair
– Good
– Very good
– Excellent
The errors in this question are clearly apparent. First, the lack of any contextual wording as to what the patient needs to do, but primarily the response options do not relate to the measured concept, which is satisfaction.
Revised question: How satisfied or dissatisfied are you with the practice opening hours?
– Very satisfied
– Quite satisfied
– Neither satisfied or dissatisfied
– Quite dissatisfied
– Very dissatisfied
Summary
This brief paper has illustrated some of the errors found
win a review of patient questionnaires available on the websites of 12 randomly selected general practices in one CCG area.
While it is acknowledged that a review of 12 practices maybe considered as unrepresentative, practices were randomly selected, and based on experience, the errors illustrated here are typical of those when questionnaires are developed with little or no experience of survey design.
It is not possible to discuss each type of question in detail in this article, but it is hoped that the examples shown here offer some guidance for the practice manager to design a questionnaire that will collect both reliable and valid information to act on.
References
1. Meadows KA. So you want to do research? 5: Questionnaire design. Br J Community Nurs. 2003 Dec;8(12):562-70.
2. Meadows KA. So you want to do research? 3. An introduction to qualitative methods. Br J Community Nurs. 2003 Oct;8(10):464-9.
