I’m at LocalGovDrupal Camp, going to sessions, finding out about things that I can pass on to you, the localgov digital strategist-practitioner. I’m in a session being run by Fintan Galvin of the digital services company Invuse talking about AI search, and this is what I’ve taken away from it:
The core difference between AI search results and traditional search results is in traditional search the user has the responsibility of determining which results in the list are the relevant ones to the query and extracting the meaning from those results to determine the correct answer, but with AI search, the AI does all that too. Which may or may not reflect the intent of the person who is asking the question - chances are, it may end up answering a completely different question that it was confused by because of the words used.
Apparently when Medway Council went live with an AI search, it straight away gave the wrong answer about who was running the council because it was trained on old information. Another council’s chatbot, when a 23 year old single mother asked for advice about being about to be made homeless said it couldn’t suggest any help, but when an elderly man asked for advice it gave an extensive answer. It’s possible the reason for the different responses is because of the different language usages from the two requesters - the young woman using her natural language of saying she was about to be kicked out of her home, whilst the older man using the council language of the keyword ‘homeless’.
A good Agentic AI search uses the AI to generate the results, give an introductory paragraph to summarise what it thinks the the intent behind the question was, to then list the results which it thinks are relevant to the question it thinks was asked, with the opportunity for the user to correct it if that was not their intention. A good AgAI does not give an answer to a question directly, because that may be misleading.
Key to being able to generate good AI search results is the same as traditional search - good content design with sensible information architecture and taxonomy that has been ensured is continually checked for accuracy by people who take ownership for the accuracy of the content and can be held accountable for it, with sensible metadata (not just keywords and description) applied to each piece of content. The search tool must not just be left sitting there, it needs continual monitoring - both at the front end by simply running regular checks that answers it gives to common questions continue to be accurate, but also by analysing data in the backend pertaining to actual recorded user behaviour.
