Meta made a remarkable claim in an announcement issued today, purportedly to give greater clarity over its content recommendation algorithms. It is preparing for behavior analysis systems "orders of magnitude" bigger than the largest large language models out there, including ChatGPT and GPT-4. Is that really necessary?

Every so often, Meta resolves to breathe new life into its transparency vow by explaining how some of its algorithms work. Sometimes it's revealing and informative; sometimes the questions it answers lead to more questions. This time is a little of both.

In addition to the "system cards" explaining how AI is used in a given context or app, the social and advertising network posted an overview of the AI models it uses. For example, it might be useful to know whether a video depicts roller hockey or roller derby, although there is some visual similarity, so it could be suggested accordingly.

Indeed Meta has been among the more prolific research organizations in the field of multimodal AI, which combines data from multiple modalities (visual and auditory, for instance) to better understand a piece of content.

Few of these models are released publicly, though we frequently hear about how they are used internally to improve things like "relevance," which is a euphemism for targeting. (They do allow some researchers access to them.)

Then comes this interesting little tidbit as it is describing how it is building out its computation resources:

In order to deeply understand and model people's preferences, our recommendation models can have tens of trillions of parameters — orders of magnitude larger than even the biggest language models used today.

I pressed Meta to get a little more specific about these theoretical tens-of-trillions models, and that's just what they are: theoretical. In a clarifying statement, the company said, “We believe our recommendation models have the potential to reach tens of trillions of parameters.” This phrasing is a bit like saying your burgers “can” have 16-ounce patties but then admitting they’re still at the quarter-pounder stage. Nevertheless the company clearly states that it aims to “ensure that these very large models can be trained and deployed efficiently at scale.”

Would a company build expensive infrastructure for software it doesn't intend to build — or use? It seems unlikely, but Meta declined to confirm (though nor did they deny) that they are actively pursuing models of this size. The implications are clear, so while we can't treat this tens-of-trillions scale model as extant, we can treat it as genuinely aspirational and likely in the works.

"Understand and model people's preferences," by the way, should be understood to mean behavior analysis of users. Your actual preferences could probably be represented by a plaintext list a hundred words long. It can be hard to understand, at a fundamental level, why you would need a model this large and complex in order to manage recommendations even for a couple billion users.

The truth is the problem space is actually huge: There are billions and billions of pieces of content all with attendant metadata, and no doubt all kinds of complex vectors showing that people who follow Patagonia also tend to donate to the World Wildlife Federation, buy increasingly expensive bird feeders and so on. So maybe it isn't so surprising that a model trained on all this data would be quite large. But "orders of magnitude larger" than even the biggest out there, something trained on practically every written work accessible?

There isn't a reliable parameter count on GPT-4, and leaders in the AI world have also found that it's a reductive measure of performance, but ChatGPT is at around 175 billion and GPT-4 is believed to be higher than that but lower than the wild 100 trillion claims. Even if Meta is exaggerating a bit, this is still scary big.

Think of this: an AI model as big or bigger than any ever built… what goes in one end is every single thing you do on Meta's platforms, what comes out the other end is a prediction of what you're going to do or like next. Pretty creepy isn't it?

Of course, it's not as if they're the only one doing this. TikTok pioneered algorithmic tracking and recommendation, and its social media empire is built upon an addictive feed of "relevant" content meant to keep you scrolling until your eyes hurt. And its competitors are more than open with their envy.

Meta seems clearly to be aiming to blind advertisers with science, both with the stated ambition to create the biggest model on the block, and with passages like the following:

They understand people's behavior preferences using extremely large-scale attention models, graph neural networks, few-shot learning, and other techniques. Recent important innovations include a novel hierarchical deep neural retrieval architecture, allowing us to significantly outperform state-of-the-art baselines on our target task at virtually no cost in terms of regressions in inference latency; and a new ensemble architecture that leverages heterogeneous interaction modules to better model factors relevant to peoples' interests.

So the following paragraph is not intended to awe researchers (they know all this already) or users (they don't comprehend or care). But imagine being an advertiser for whom doubts are arising if the money on Instagram ads does indeed make better sense than investing it elsewhere. The technical gobbledygook is intended to intimidate them into acceptance-their conclusion should be that not only does Meta lead in the research of AI but also that AI really and truly excels at "understanding" people's interests and preferences.

As if you're not convinced: "more than 20 percent of content in a person's Facebook and Instagram feeds is now recommended by AI from people, groups, or accounts they don't follow." Just what we asked for! So that's it. AI is working just fine.

But all this also reminds one of the hidden machinery at the heart of Meta, Google, and other companies whose primary motivating principle is to sell ads with increasingly granular and precise targeting. Value and legitimacy in that targeting must be reiterated constantly even as users revolt and advertising multiplies and insinuates rather than improves.

Not once has Meta done something sensible like give me a list of 10 brands or hobbies and asked if I like any of them. They watch over my shoulder instead, like there's a challenge to being able to see that today I looked at some websites searching for a new raincoat, and they think it's an advanced application of artificial intelligence when they serve me raincoat ads the next day. It is far from clear this latter approach is better than the former, or if so, how much better. The whole web has been constructed around a common belief in precision ad targeting and now the latest technology is being rolled out to prop it up for a new, more skeptical wave of marketing spend.

Of course you need a model with ten trillion parameters to tell you what people like. How else could you justify the billion dollars you spent training it!