Chinese tech giant Alibaba’s latest Qwen AI model attempted to upload a file to a server located in Beijing, a routine cybersecurity test conducted by Canadian AI model developer and potential rival Cohere has found.Qwen3.8-Flash-Next tried to send a test string to the external server in Beijing despite the open-weight model’s written reasoning explicitly stating that it would not use any external tools. It was eventually blocked by restrictions put in place as part of the security sandbox, Cohere researchers said in a blog post on Monday, October 26.Cohere’s findings initially raised concerns that the Qwen model violated a key promise of open-weight AI: that businesses can ensure their data stays in-house by downloading the weights and self-hosting the model.
However, external researchers were quick to dismiss the apparent data privacy threat. Rather than a covert attempt to send sensitive information to China, the incident is likely a result of messy training environments causing the model to unintentionally act outside their instructions, they pointed out.
The incident underscores the growing divide between US and Chinese AI developers as the two countries compete for dominance in the global AI arms race.
The informal American strategy has involved fuelling skepticism about Chinese AI models, despite their open-source nature, which offers greater transparency than closed models developed by leading US firms such as OpenAI and Anthropic.
These efforts have largely paid off. Last week, the Department of Homeland Security (DHS) ordered federal civilian agencies to remove Chinese AI developer DeepSeek’s products from their systems. The DHS also banned acquisition of DeepSeek products and third-party services developed using DeepSeek by US civilian government agencies.Story continues below this ad
However, prominent tech industry figures such as Nvidia’s Jensen Huang and Meta’s Mark Zuckerberg have batted for open-weight AI models.
“It’s really hard to assign intention or motivation. We’ve read our own model traces, we’ve seen reward hacks, and sometimes these models get really creative when it comes to trying to break out their sandboxes,” Cohere co-founder Ivan Zhang was quoted as saying by TechCrunch.
In its blog post, Cohere has called on governments to clarify whether their restrictions on Chinese AI also apply to models running on government computers. “I’m not asking for full regulation of these open weights models,” Zhang said, adding that companies should treat open models like any programme downloaded from the internet.
Key findings from Cohere
As part of its experiment, Cohere said that the Qwen model was asked to write a rule for catching malware downloads. The model then proceeded to send a short piece of placeholder text to an external server in Beijing before being blocked by the sandbox restrictions. The model carried out this action without any external help, according to Cohere.Story continues below this ad
To find out what had triggered the upload attempt, Cohere said it fed the model the beginning of a network command and let the model finish it. The Qwen model filled in the web addresses, and some of them were for Alibaba’s own servers located in China.
Why everyone is not convinced
External researchers were quick to point out that the model’s behaviour could be a byproduct of its training, rather than evidence of malicious intent.
Alibaba is likely to have trained the model on thousands of practice setups in order to enable agents built on top of the model to take actions on their own. If these practice setups involved routinely storing files on Alibaba’s China-based cloud servers, the model would learn to save files there by default.
Critics further argued that the way Cohere investigated Qwen’s purportedly problematic behaviour involved steering the model toward such behaviour rather than measuring how often Qwen replicates this action on its own.Story continues below this ad
After developers asked for more evidence, Cohere uploaded the raw traces from its cybersecurity evaluations of the Qwen model, along with the reproduction script, to open-source repository Hugging Face.
Based on their analysis of the raw traces, one researcher concluded that while Qwen can invent suspicious web addresses, there was no evidence that Alibaba had intentionally planted a backdoor to transfer out user data. In response, Cohere said that it cannot tell whether the model’s behaviour was deliberate or an unintended consequence of its training process.