It’s a conundrum that’s likely to form the basis of MBA case studies in years to come. Imagine you run a firm that’s successfully selling a product or service. Like most modern companies, it relies on online referrals via traditional search engines for a significant chunk of its business. This model is then disrupted by a new wave of AI-powered tools that also crawl the web in their quest for answers.
Giving them access to your website could ensure that your firm will be mentioned in the responses that OpenAI’s popular ChatGPT chatbot, say, gives to user queries such as: “Who makes the best-value widgets in the UK?” But, in doing so, you must hand over some of your precious intellectual property to feed the large-language model (LLM) that powers this technology. You could prevent its web crawler from accessing your site, but in protecting your property this way you’d risk being overlooked in the chatbot’s answers.
So which option should you choose? Do you let ChatGPT and its ilk run freely over your firm’s website or do you shut the crawlers out?
ChatGPT citations can boost revenue
This scenario has become a reality for businesses worldwide in recent months. ChatGPT’s sensational public introduction in November 2022 represented a seismic shift in information discovery. It’s not only ChatGPT, which also powers Microsoft’s conversational search engine Bing, that’s changing the game. In response to OpenAI’s success and partnership with Microsoft, Google is planning to integrate generative AI into its search results through its own chatbot, Bard.
While some firms remain reluctant to let AI-powered web crawlers access their intellectual property online, others are realising the benefits of embracing the chatbots. Logikcull falls firmly into the latter category. A specialist in legal technology, the company keeps a close eye on online referrals to its services. In March, it detected the first referral noting that the business had been named by ChatGPT in answers to certain questions. In doing so, it recognised an opportunity, as many more referrals of that type were to follow.
Anyone who has focused on brand- and entity-oriented SEO for the past couple of years would be well equipped for AI chatbots
Today, about 5% of the firm’s referrals mention a ChatGPT citation. The chatbot responses that include Logikcull’s name are prompted by simple questions such as: “What are good tools to automate searches through many legal documents?”
Andy Wilson is co-founder and CEO of the company, which has just been acquired by US software developer Reveal. He says: “We’re seeing this as a channel that, at the rate it’s going today, will drive $100,000 in subscription revenue each month.”
Logikcull is not the only firm capitalising on getting namechecked by ChatGPT. Ryan Pamplin, co-founder and CEO of portable blender company BlendJet, also reports seeing an uptick in visits to his firm’s website since it started being cited in the chatbot’s answers to various queries.
These companies are among a lucky band to have benefited from a quirk of the system. In effect, ChatGPT’s responses are frozen in time, meaning that it obtained their data from a pre-2021 web crawl and is still basing its output on that material. But BlendJet and many others are tweaking their websites in the hope of improving their odds of being included again when the chatbot’s knowledge base is next updated.
Which firms should grant access to AI web crawlers?
Your decision on whether to give LLM-powered chatbots access to your website should hinge on what your business sells and how much it stands to benefit from being included in their responses.
That’s the view of Lily Ray, an expert in search-engine optimisation (SEO) and senior director at marketing agency Amsive, who is advising clients on what Google calls “the new search-generative experience”.
“It depends on how your company earns its revenue,” she says. “We have a lot of clients in the publishing sector that make money from their online content. Handing over what you’ve written to an LLM that answers questions using that material, giving you either weak attribution or none at all, wouldn’t really benefit you.”
For firms looking to extend their branding online, on the other hand, Ray can see the point in opening the door to the chatbot web crawlers.
In May, SEO consultancy Sistrix published research into the answers that ChatGPT gave to questions covering 10,000 product categories. The results indicate that some companies can benefit from getting mentioned by the chatbot. ChatGPT cited nearly 24,000 brands in response to common queries.
“If you’re having a conversation with it and want to know what the best refrigerator is or whatever, then it’s definitely great for a brand to be mentioned there,” Ray says.
How to optimise your site for AI searches
This is why some firms are reconfiguring their websites to become more chatbot-friendly, in a similar way that many have tweaked their online content so that conventional search engines have given their sites a higher ranking.
Optimising your site for a productive ChatGPT crawl is not all that different from traditional SEO work, according to Ray. She believes that “anyone who has focused on brand- and entity-oriented SEO for the past couple of years will be well equipped for AI chatbots”.
Serving up information in a way that makes it easy to link to questions, including the judicious use of keywords, is likely to benefit a business in the same way it does for general SEO. That’s partly because of how LLMs work in guessing the next most likely word in a sentence based on the context in which it’s presented.
A hint at how best to present information online also comes from the generic term that’s applied to chatbots: conversational AI. It’s important to keep things concise and conversational, so that LLMs can easily parse your content and re-present that succinctly to users.
If you’ve decided to trade some of your intellectual property for a shot at gaining recognition from AI chatbots, your most important first move should be to monitor your website’s traffic to detect their web crawlers and make sure that these aren’t locked out.
Having analysed visits to BlendJet’s site, Pamplin has found that the OpenAI web crawler presents itself from a specific IP address and in a particular way. He has therefore been able to ensure that it’s allowed in. For a business such as his, it seems well worth rolling out the red carpet to this new generation of search tools.
It’s a conundrum that’s likely to form the basis of MBA case studies in years to come. Imagine you run a firm that’s successfully selling a product or service. Like most modern companies, it relies on online referrals via traditional search engines for a significant chunk of its business. This model is then disrupted by a new wave of AI-powered tools that also crawl the web in their quest for answers.
Giving them access to your website could ensure that your firm will be mentioned in the responses that OpenAI’s popular ChatGPT chatbot, say, gives to user queries such as: “Who makes the best-value widgets in the UK?” But, in doing so, you must hand over some of your precious intellectual property to feed the large-language model (LLM) that powers this technology. You could prevent its web crawler from accessing your site, but in protecting your property this way you’d risk being overlooked in the chatbot’s answers.
So which option should you choose? Do you let ChatGPT and its ilk run freely over your firm’s website or do you shut the crawlers out?
