Last week, it finally launched its much-hyped search product, ChatGPT Search, to usurp Google. Industry executives have been preparing for the eventuality for months, motivating Google to inject AI-generated answers into its core product earlier this year; it did so and with some embarrassing hallucinations into the bargain. The episode left many people believing that OpenAI's search engine was seriously going to be a "Google killer."

But after using ChatGPT Search as my default search engine-you can, too, with OpenAI's extension-for roughly a day, I quickly switched back to Google. OpenAI's search product was impressive in some ways and offered a glimpse of what an AI-search interface could one day look like. But for now, it's still too impractical to use as my daily driver.

ChatGPT Search sometimes came in handy in answering live queries for things that would otherwise have required going through a sea of irrelevant advertisements and SEO-optimized write-ups. It's pretty much similar to its peers in the AI search space: Perplexity and You.com-which gives you the answer but puts it in a nice clean format: Links on the side help you access information sources; the headlines and little excerpts let you know how fake the AI text actually is.

However, it was not often a useful tool to employ in one's daily life.

To date, versions of ChatGPT Search remain unreliable for the types of uses people have grown to rely upon Google for most: short, navigational queries. Queries under four words account for searches on Google; more frequently than not, that is a pair of search terms to get you to the right web page, the one the user knows and does not want to bother spelling out. They're the type of search most people hardly even have a clue they are making through the day, and it is what Google tends to do relatively well.
I mean "Celtics score", "cotton socks", "library hours", "San Francisco weather", "cafe close to me," and every other query which makes google the front door for billions over the internet.

My test run with ChatGPT Search was often frustrating, and it made me realize just how much keyword searching I do on a daily basis. It couldn't find information reliably using short queries, and for the first time in years, I found myself actually missing Google Search.

Of course, don't misunderstand: Google has gotten terrible over the last ten or so years basically because of a sea of ads and SEO manipulation and rather spotty AI summary blocks. However, while reviewing, I couldn't help but open a window to Google every other click because ChatGPT Search could never give me a right answer or webpage.

Which would win? ChatGPT Search or short queries?
I tried "Nuggets score" to see how the game was going between the Denver Nuggets and the Minnesota Timberwolves. ChatGPT let me know that the Nuggets were ahead, even though I certainly knew they hadn't won; it also showed me the actual Timberwolves score was 10 points more than what it was saying, from a Google result at the time.

Another time, I tried "earnings today," to find out which companies are reporting quarterly results that may affect the stocks' prices on Friday. ChatGPT said that Apple and Amazon reported their results on Friday even though both companies reported the previous day. It's just that it hallucinated and created some false information.

I also ran an experiment inputting the name of a tech executive in order to retrieve contact information. ChatGPT returned me a summary of the person's Facebook page and then generated a fictional link to the person's LinkedIn page, which, upon trying to click on, resulted in an error message.

Another day, another search: I input "baggy denim jeans" hoping to buy them. The ChatGPT Search told me what baggy denim jeans were-first off, all definitions were redundant and did tell me to pick a lovely pair at Amazon.com.

I could continue along on and on, but I will get the idea of broken links, hallucinations, and a general array of oddball responses defining my first day to using ChatGPT Search.

Maybe one day that "Google killer" it may become, but until then.
This was not an inconsequential launch for OpenAI. Sam Altman praised the feature for being "really good," even though he's known for downplaying his startup's AI capabilities. The reason this time is different may have something to do with search being one of the biggest businesses on the internet, and OpenAI's version could be a real threat to its biggest competitor, Google.

To be fair, Google Search is a 25-year-old product and ChatGPT Search is brand new. In a blog post, OpenAI says it plans to improve the feature based on user feedback in the coming months, and it seems more than likely this could be a significant area of investment for the startup.
 
To its credit, ChatGPT Search is rather good at answering long, written-out research questions. Something like, "What is the most diverse American professional sports league?" isn't a thing you'd likely search Google for an answer to in two seconds, but ChatGPT Search is pretty good at scraping multiple websites and getting you a decent answer in just a couple of seconds.

(Perplexity is also pretty good at these questions, and its search product has been around for well over a year.)

Compared to the old version of ChatGPT, which already had web access, the search feature feels like a better interface for browsing the web. There are now clearer links to the sources where ChatGPT gets its information — for news stories, for example, ChatGPT will be tapping into the media companies that it's been striking all those licensing deals with.

The problem is that most of the searches in Google are not such long questions. In order to completely replace Google, OpenAI should enhance these more practical and short searches that people are already making through their day.

OpenAI does not seem modest about the fact that the ChatGPT Search still has trouble with short queries.

With ChatGPT search, we have seen users asking questions in much more natural ways than with other search tools they've used before, according to OpenAI spokesperson Niko Felix, who responded by email to TechCrunch. At the same time-web navigational queries-which tend to be short are pretty common. We intend to make the experience better for those kinds of queries over time.

However, the short keyword queries have made Google indispensable and, till OpenAI gets it right, Google is still going to be the mainstay for many people.

There could be two reasons why OpenAI might be struggling with these short queries. The first one is that ChatGPT relies on Microsoft Bing, which is generally considered an inferior engine compared to Google. The second is that LLMs are, as a general rule, simply not very well-suited to these short prompts. LLMs normally require fully written-out questions to generate good answers because that helps build up a strong statistical pattern for them to complete: it doesn't know that people searching for "cotton socks" are virtually all trying to shop for them, not learn where the garment comes from.

Perhaps there needs to be some re-prompting — running short queries through an LLM as a longer question — before ChatGPT Search can do such searches well.

Though OpenAI has only now released its search product, Perplexity's own AI search tool is already serving 100 million search queries a week. Perplexity has also been touted as a "Google killer," but it runs into the same problems with short queries. Aravind Srinivas, Perplexity's chief executive officer, told TechCrunch Disrupt how his users were using his product to get something that they might not find with Google Search. "The median number of words in a Google query is somewhere between two and three," he said at the event. "In Perplexity, it's around 10 to 11 words. So clearly, more of the usage in Perplexity is people coming and directly being able to ask a question.".

At Google, on the other hand, you type in a few keywords and instantly reach a specific link.

I think what it means that people are not using these products for web navigation presents a much bigger problem than what OpenAI or Perplexity are letting on. It means that ChatGPT Search and Perplexity are not replacing Google Search for the task that it's best at: web navigation.

Instead, these AI products are filling a new niche, surfacing information that gets buried in traditional search. Don't get me wrong, that's valuable in its own right. OpenAI and Perplexity both claim to work on getting better in these short queries. Before that, I don't think one of these products can replace the Google. If OpenAI wants to replace the door to the internet, they have to create a better front door.