Over in the LocalGovDigital Slack group (what actually is the proper word for a Slack group? Team? Organisation? Wibbler?) a member asked:
'Hi everyone, does anyone have much experience or strong feelings about AI chat on council websites? We're looking at what user problems an AI chatbot could solve here, but before we seriously start working on anything I'm just doing some light research to get a feel for AI in local government'
Now, as it happens I don't have any particularly strong feelings and my experience is several years out of date, so I responded thus:
I've no particularly strong feelings myself, but I'd turn the question around by asking what any of us's feelings might be from our own lived experience of using them as users of other websites, such as eg the lecky board website or Virgin Media? Have we had positive or negative experiences? Is there reliable data which can demonstrate real Return on Investment for the organisations which have implemented them - which represents a worthwhile payoff from any consequent loss of customer satisfaction? And by reliable data, I do mean independently verified data, rather than what we, quite frankly, seem to see with too many projects where the people who commissioned the project use selectively chosen data out of context in order to overstate the success of the project to the people who sanctioned the budget for the project.
There were further comments to the effect that many of the chatbots elsewhere people had experienced actually amounted to little more than glorified decision trees, and indeed, my experience of the pilot we did a few years ago was that for any given conversation opener, typing that query into the site's internal search box brought the relevant page at the top of the search results page, whilst typing the same query into the chatbot went through quite a tedious series of this or that questions before offering the same page at the end of it with the same answer.
Now that pilot project was before the step-change in technology that ChatGPT has shown us, but GPT-based chatbots, whilst they might be functionally superior to the previous generation of chat, are not themselves without their own problems. If we use a generalised version of the bot which has been trained on the content of the wider internet - or on Reddit - then there's a high risk that some of the answers may be wrong. The worst that could happen to a student handing in an essay which claimed the influence of Led Zeppellin's music can be clearly seen in the soaring heights of an Anish Kapoor sculpture is the student will be failed for that essay. But if the chatbot on the council website has told you you don't have to pay your full council tax bill because you're in dispute over their cuts to arts and youth service funding - the only winners there are m'learned friends. The other option might be a GPT chatbot trained only on your own council content - but if that's the case, what would the chatbot be doing that your internal search engine isn't doing, apart from starting off with "Hi there, I'm Bill and I'm your virtual assistant - how can I help you?" and finishing with "I'm glad to be of service - before we go, is there anything else I can help you with today?".
To be clear, I'm not arguing against trying out new things for the online service; I've spent the last many years writing about new things it would be good if we tried, and indeed part of the point of me building this very site is to, over time, be a platform for me to try out some of the ideas I've had and demonstrate them because I've not had the platform of a real council website to try them on for some years.
It was a question about chatbots which prompted me to write this, but it could apply just as easily be any new innovation or technology; if the idea is something which might be cheap, quick, and/or reputationally low risk if it goes wrong, then absolutely we need to walk away from the risk-averse, innovation-averse, fun-averse culture which has resulted in average citizen satisfaction with our work going down and down and down every year for the last 10+ years.
But if our clients are asking us to introduce into our council service delivery toolboxes something that's currently a subject of media hype, or that their service director has seen on something, the first question we need ask back by way of kicking the tyres on the request is - OK, where have you seen this yourself? Have you yourself used one of these as a customer? Did you feel you had a good experience as a customer using it elsewhere? Whether you had a good or bad experience of using it elsewhere, how confident can we be that our users might have a better experience than you had elsewhere?