I've been taking part in a university study into the effectiveness of recipe boxes - such as Hello Fresh and Gousto - in improving people's nutritional intake and kitchen confidence and competence. Spoiler alert - it's a very expensive way of buying chicken thighs.
As part of the study, participants have questionnaires to complete, talking about how confident they feel in their kitchen skills doing things like chopping vegetables, steaming potatoes, frying meat etc, and how they feel emotionally about certain cooking activities.
Now, believe it or not, I do try to have a little humility when it comes to other people and the jobs they are supposed to be experts in. I'd like to be humble and think that university researchers are good at research. But...
...I hate to say this, I think it's a terrible survey. Any friends and colleagues who especially know my sometimes colourful way of expressing an opinion in the spoken medium will no doubt at this point be picturing me declaring it to be an utter abomination, or if I'm in a good mood that day, merely classifying it to be a total catastrophe.
Over-nuance
In the section about Surveys and Equalities Monitoring in my Content and User Experience Strategy for Local Government, I refer to the problem of nuance when it comes to trying to get people to tell you how they feel about things on a webform. You're trying to find out what people think, so don't make them think too much in the process of doing that.
Nuancing the answers to survey questions on a scale of 1 to 10 is to my mind over-nuancing it, but at least a scale of 1 to 10 is a common scale which just about every human on planet Earth will have at least a conceptual grasp of. Software engineers will have a conceptual grasp of a scale from 0 to F (or if it’s late at night and they’re underfed, 0 to 1), and ancient Babylonians will have a similar grasp of a scale of 1 to 16, but srsly, what is the conceptual framework of a scale of 1 to 7? A fascination with the Seven Wonders of the Ancient World?
At most, a granularity of 1 to 5 — strongly agree, agree, no opinion, disagree, strongly disagree — should be enough to determine an accurate picture of what your subjects think. But for the most part, in most circumstances people will either be confident or nervous about chopping vegetables, they’ll either agree or disagree with the sentiment expressed, or they’ll have no opinion, and a granularity of 1 to 3 will be the easiest way you can get them to tell you what they think without making them think too hard about the matter.
Consistency
It also suffers from the problem that it's not consistent in its application of the questions throughout the form.
First of all, what in the names of Omega and Rassilon is with the option 0 to denote Don't Know followed by 1 being Very Poor?
But on the first page shown, the user is asked (with 0 being Don't Know) whether they're Very Poor at the actions to Very Good at them, followed by on the second page the tone shifts to the user being asked to Strongly Disagree to Strongly Agree with a series of statements. Which curiously don't have a Don't Know option - it seems odd to me that somebody might not know how good they are at chopping vegetables but they'll always be able to confidently agree or disagree with a statement about how confident they are at dealing with the unexpected.
By having to change my mental model of a series of questions in one format to a different mental model on the next page, I was somewhat mentally thrown - I lost my focus on what was being asked of me, and my confidence that I answered subsequent questions accurately was compromised.
A possible value
So having castigated this questionnaire in its position as an online form, is there a place for recording opinions to a higher level of granularity?
Possibly, yes.
Back when online pothole report forms started asking people about the size of a pothole in comparison with the size of a football or a tennis ball, just about everybody I know laughed at how ridiculous that was. Stepping aside from the underlying reason for those questions in the first place, I theorised that these questions which felt interminable on a webform might not feel so interminable when you're on the phone to somebody. When you're talking to a contact centre agent, there's a human-to-human rapport being built, the agent is empathising with your plight, if you're not in an entirely bad mood you and the agent might even be exchanging some friendly but professional banter about the state of the roads these days, so questions which might seem pointless when given human context will feel more natural, less intrusive on your very busy day.
Now I'll never think of a defence for a scale of 1 to 7 for a graduation of a sentiment, but I can see how a nuance level of the common scale of 1 to 10 might be useful in the hands of a skilled and expert researcher who might be in the room or over the phone asking more open qualitative questions of the subject, then applying their own established criteria to interpret the answers the subject has given in a quantitative way.
But if that's not how you're conducting your research, if you're asking casual users rather than expert users to complete an online form in their own time, then just ask if they're confident, OK, or nervous about steaming potatoes, and just ask them if they found the form easy to complete or difficult to complete. Give them a free text box to add any further notes that may be helpful to note.