At the website of one of the largest consumer electronics e-retailers, a search for “beer cooler” returns a bounty of more than 500 relevant results, but a search for “beer chiller” delivers not even one. Type in “something to cool beer” and the only result you get is a Lego set.
Conduct the same exercise on Google or Bing and the experience is quite different. The two most popular engines seem to understand that “cooler” and “chiller” are synonymous, and it even performs pretty well on the “something to cool” test.
What do the search engine giants know that e-commerce sites don’t? The difference is “vector search,” a technology rooted in artificial intelligence research that represents information as numbers rather than text.
Once content is converted to search factors (which are essentially strings of numbers), machine learning algorithms can find similar content by comparing the distances between vectors to understand how different words relate to each other. They can also analyze surrounding content to understand the context of search queries, so that “songs by bad company” returns results about tunes by the 1980s supergroup and not the wailings of unwanted guests. If you want to dig into the technology of vector search, this post on the Google Cloud blog should satisfy your inner geek.
When search needs a human touch
That’s not how most e-commerce search engines work today, though. “Great search is actually a data and machine learning game, but none of the main search technologies available today do this directly,” said Hamish Ogilvy, CEO of Search.io, which makes a search engine for e-sellers based on vector technology. The result is that “the quality of search is fundamentally driven by the skill of humans at configuring and connecting to other systems.”
In other words, the search engines on most commercial sites are only as good as the human beings behind them. Giants like Amazon.com have been able to outsource the hacks needed to deliver relevant results to teams of data science over a period of years, but most retailers are stuck with whatever is the default search engine of the service provider they happen to use.
Most are not served well by that. A recent survey of the search performance of the top 50 grossing e-commerce sites in the U.S. by Baymard Institute declared the state of e-commerce search to be “broken,” noting that just 34% of sites could handle queries that utilize themes, features, or symptoms rather than specific product names. “A whopping 70% of the search engines are unable to return relevant results for product type synonyms—requiring users to search using the exact same jargon as the site,” the company asserted.
That’s costing sellers a lot of money. A recent Google report estimated that e-commerce firms lose $300 billion per year in the U.S. alone because visitors can’t find what they’re searching for.
Tweaks and unintended consequences
Traditional search relies on matching text strings, Ogilvy explained. As a result, a search on “crewnecks” won’t return a result related to T-shirts unless the relationship is defined by rules that are hard-coded into the index. To handle a search for a mobile computer, for example, the engine must be told that the words “portable,” “laptop,” “notebook,” and “MacBook” are functionally the same. The manual effort of coding those relationships multiplied by thousands of products that can each be referred to in multiple ways is almost unimaginably complex.
And hand-coding creates its own problems as the number of rules pile up. Ogilvy cites the example of one company that had programmed a workaround that reformatted searches for “USB C” into “USB-C,” which was the syntax it used in its catalog. The unintended result was that when visitors searched for “USB cable” the hyphen was automatically added to the text string and the resulting query—“USB-cable”—came up empty.
“It's very hard to write thousands of these things and not cause issues,” Ogilvy said.
These limitations had prompted most e-commerce site operators to optimize for the highest-volume queries and effectively give up on the 70% of requests that constitute the “long tail” of search terms that are rarely used.
The good news is that the situation will improve in the not-too-distant future. Makers of e-commerce search engines “are all scrambling toward vector,” Ogilvy said. “That is the way search will be done going forward.”
The question isn’t whether vector search will go mainstream but when. “I expect just about everybody will go in this direction,” he said. The transition won’t necessarily be smooth. As website operators swap out their heavily patched search utilities, many rules will need to be disposed of and some changed, since machine learning isn’t magic and can’t anticipate the nuances of every use case. However, in the long run everyone will be better off. I’ll bet a case of cold beer on it.
Next, Read This:
