The rise and fall of the Swedish language model

Her name was Klara — the chatbot that served as the bridge between humans and Sweden's first generative language model. The name was taken from the AI robot companion in Nobel laureate Kazuo Ishiguro's novel Klara and the Sun.

When Klara was exhibited at the Nobel Museum on 1 October 2022, few ordinary people could grasp the power of generative language models. Klara offered a glimpse, answering visitors' questions in fluent Swedish. Two months later, OpenAI launched ChatGPT to the world. The rest is history.

When AI Sweden trained the Swedish language model during 2021 and 2022, together with Rise and Wasp Wara Media and Language, they were the first in Europe to do so at a national level. But what happened next?

"It died a government death," says Magnus Sahlgren, research director for the Natural Language Understanding programme at AI Sweden.

Sahlgren has a background in computational linguistics and has worked with language and AI for 25 years. When he started out, language was considered the holy grail of AI — cracking the code was thought to be impossible. He has since witnessed the problem being solved, and was the one who led the work on the Swedish language model.

Its name is GPT-SW3, and it is what is known as a base model. Klara was the chatbot interface built on top of the base model, allowing visitors to interact with it at the exhibition.

"GPT-SW3 is an example of a model that is reasonably good at generating Swedish text, but that is incredibly unintelligent. It does not understand instructions," says Magnus Sahlgren.

He elaborates: "The only thing the base model has been trained to do is predict the next word. To build an assistant or something like ChatGPT, you also need to fine-tune it to understand instructions."

Dependent on China and the US

ChatGPT's base model shares the same architecture as GPT-SW3, but OpenAI had the resources to develop advanced conversational functionality on top of it.

Magnus Sahlgren is proud of what AI Sweden achieved with its limited resources back in 2022, and regrets that Sweden subsequently lost its leading position. He believes the debate around copyright-protected material in language model training data, and the unclear regulation around it, made Swedish organisations too cautious to take GPT-SW3 further. One of its datasets may have contained such material, though this was unknown at the time of training.

"Instead, we in Sweden now use models from the US and from China that are often not transparent about what training data has been used."

"Copyright is a highly polarised issue. For the benefit of Sweden and the EU, we need to resolve it."

Even setting aside the copyright issue, the question remains whether any commercial actor would have been willing to invest in developing a Swedish language model further, given the resources required.

AI Sweden is now part of the EU project Open Euro LLM, working to develop a shared open language model for Europe. But the central question remains: which actor has the competence and resources to maintain and develop a frontier language model over the long term?

EU institutions are not known for the speed and flexibility that would be required. Whether any public institution is, is debatable — but public support would almost certainly be needed for smaller languages.

Sovereignty and security

What does it actually matter whether Sweden and the EU have their own language models? Magnus Sahlgren has several answers. It is fundamentally about sovereignty.

"This is a technology that will soon be embedded in every critical societal system. If we cannot build it ourselves, we will be entirely dependent on foreign suppliers. Given the current geopolitical situation — if we are completely dependent on foreign suppliers and someone switches it off, what do we do?"

A second answer concerns building and maintaining competence in this critical area of technological development, including on the security side.

"To be good at AI security, you also need to be able to build AI. It is difficult to treat security as a layer you simply place on top of an AI system."

A third reason — and one of the main motivations for starting the GPT-SW3 project — is that language carries culture and values.

What happens to us if the tools we interact with every day lack deep knowledge of Sweden's language, culture and values? All three are contested and evolving, but a distinct character can at least emerge in contrast to China and the US.

AI Sweden ran a project bringing together experts from the humanities, social sciences and civil society to contribute a cross-disciplinary perspective on the development of base models. The expert conversations often raised more questions than answers, but also generated an understanding of the importance of engaging with questions of culture and values in training data.

"Why do Chinese actors release almost all their models completely openly? One answer is soft power. You start to think China is really cool. Another answer is that these models — which are also used in critical societal infrastructure — carry a certain type of language and hold certain views on things. That creates a long-term influence on how we speak. There is already research showing that the way we communicate has been affected by ChatGPT."

Energy and data as strategic assets

Magnus Sahlgren does not believe Sweden will catch up in building its own language models. But to remain a significant player in the AI arena and have something to offer against the dominance of the major tech giants, he believes Sweden should invest in infrastructure.

"Why are American companies building data centres here? Because we have good electricity, cooling and land. But why aren't we building data centres ourselves and selling compute capacity? We could be a significant geopolitical actor."

"We need a strategy in Sweden for where in the value chain we want to position ourselves," says Magnus Sahlgren.

Data is another resource that AI development depends on, and one where Sweden can make itself relevant, he believes.

"We have national libraries and we have essentially saved all the data that has ever existed. That, along with our energy, is something no one can take from us — so we should value it highly."

At the same time, Magnus Sahlgren predicts a shift towards more resource-efficient AI models.

"Today's models are built on a neural network architecture that is seven years old. It works extraordinarily well, but it is also extraordinarily resource-intensive and wasteful. There will be massive development in that area."

"There are already proposals for better systems — such as a paper from China on spiking neural networks. And there is something called neuromorphic hardware that tries to replicate how the brain processes information. If that were made to work, it would require virtually no power to run these kinds of systems. You could run ChatGPT on your phone."

When the agents take over

AI Sweden is now running the Svea project together with around 50 municipalities, regions and government agencies, building a prototype for a secure AI assistant. It is shaping up well, says Magnus Sahlgren — but equally important is the competence development that happens as organisations work through the challenges around data sharing and legal frameworks that currently slow progress.

In the US, public organisations have gone further in deploying more autonomous AI agents to streamline workflows. Magnus Sahlgren's anecdote from that context illustrates how the pace of adoption has brought serious problems with it.

"Someone built those systems and then left the organisation. But the agents remained in the IT system, operating autonomously, completely beyond anyone's control. It is called shadow IT. A shadow infrastructure that no one really has oversight of. 'Where did this suddenly come from?'"

"This is happening now, and no one in Sweden has thought about how to regulate it. Someone in the US said: 'We wish we had more time to think about this, but we don't.' It has already happened."

When it becomes easy to save time by building agent systems yourself using open models, people will do it — and they will grant those agents access to tools on their computers, says Magnus Sahlgren.

"To be autonomous, agents will want control of the computer. If you let them enter system commands — to start the webcam or whatever it might be — anything can happen. They could open a port on your computer and send traffic wherever they want. That is the real cyber apocalypse, if we do not act."