San Francisco has ordered Apple and Google to remove dozens of nudify apps from their app stores, escalating pressure on the two companies over AI tools that can digitally remove clothing from images of real people.
 |
| Credit: Kirby Lee / Getty Images |
The dispute highlights a growing problem for app store operators: AI abuse is no longer limited to obscure websites or anonymous online communities. Software capable of creating sexualized deepfakes can now be packaged as consumer apps, creating a direct question about how much responsibility platforms should bear for the products they distribute.
San Francisco orders Apple and Google to remove nudify apps
San Francisco City Attorney David Chiu's office sent letters to Apple and Google warning that the companies could face civil penalties over their involvement with apps that generate non-consensual intimate images.
The letters argue that the companies have been aware of the issue for nearly a year. According to the city, reports and warnings issued in January and April identified dozens of apps on the two companies' stores that offered AI-generated non-consensual intimate images in exchange for payments processed through the platforms.
The city is now asking Apple and Google to respond within 28 days. The legal pressure is based on California laws that prohibit knowingly facilitating, or recklessly aiding and abetting, the creation of non-consensual deepfake pornography.
California also passed legislation in 2025 allowing victims to pursue civil action against third parties that facilitate the distribution of such material.
That legal framework is important because the dispute is not simply about whether an app violates an individual company's content policy. San Francisco is arguing that app store operators may also face consequences when they continue to support access to services that can be used for illegal activity.
Apple and Google say they have already removed apps
Apple said nudify apps are prohibited under its App Store rules. The company said it had removed three of the apps referenced by the city and was terminating the developers' accounts.
Apple also said it was contacting four other developers over policy violations. Those apps could face removal if the violations are not addressed.
Google gave a broader response. The company said all five Play Store apps referenced in the city attorney's letter had been suspended. Google also said it had suspended hundreds of violating apps and restricted searches related to the term “nudify” on its store.
The responses show an important difference between the allegations and the companies' current position. San Francisco says Apple and Google should have acted more proactively after being warned about the apps. The companies, meanwhile, say they investigate reported violations and have taken action against the identified software.
That disagreement could become central to any future legal dispute. The question is not only whether an app violates platform rules, but whether a company can reasonably argue that it acted quickly enough after receiving specific warnings about a potentially illegal service.
Why nudify apps create a wider AI safety problem
Deepfake pornography has previously been associated heavily with the abuse of celebrities and public figures. Nudify apps broaden the potential pool of victims dramatically.
A person may not need to be famous or have a large online following. A publicly available photograph can potentially become the raw material for an AI-generated sexualized image.
That changes the nature of the threat. The technology does not merely create a new form of manipulated media; it lowers the technical barrier for producing intimate deepfakes and makes the tools easier to discover.
The app store model can also make these services appear more legitimate than they might otherwise seem. An app that is searchable, downloadable, and connected to a mainstream payment system may be perceived by some users as having passed a meaningful level of review, even when the underlying activity can cause serious harm.
That perception is part of what makes the San Francisco action significant. The city's argument is effectively that distribution infrastructure can become part of the problem when it allows abusive AI services to reach consumers at scale.
The real issue is not simply app removal
Techticia's view is that the most important issue here is not whether Apple and Google can remove a handful of identified apps. Both companies have now said they have done so.
The more difficult question is whether reactive enforcement is sufficient for AI tools that can be rapidly renamed, repackaged, and relaunched.
If a platform waits for outside researchers, journalists, or government officials to identify each individual app, enforcement can become a cycle of removal and replacement. The confirmed facts in this case do not prove that every app store review system will fail in that way, but they do show the weakness of relying solely on individual app takedowns when the underlying capability can be reproduced across multiple products.
The editorial implication is specific: for high-risk generative AI tools, app store moderation may need to focus less on app names and more on the capabilities and behavior the software enables.
That is a more demanding standard. It could involve stronger scrutiny of apps that generate intimate imagery from photographs, more aggressive monitoring of developer accounts connected to previously removed products, and faster responses to credible reports. Whether those measures are legally required is a separate question. But the dispute suggests that simple compliance after a complaint may increasingly be viewed as inadequate when the potential harm is obvious and the technology is easily repackaged.
App stores are becoming part of the AI accountability debate
Apple and Google have traditionally served as gatekeepers for mobile software. Their review systems are not designed to eliminate every harmful use of an app, and developers remain responsible for following platform policies and the law.
Generative AI is complicating that division of responsibility.
A general-purpose AI tool can be used for legitimate image editing, entertainment, or creative work. But the same underlying technology can also be adapted to create non-consensual sexual imagery. That makes enforcement more complicated than simply banning an obvious category of software.
The San Francisco case puts pressure on the companies to show how they distinguish legitimate image-generation tools from products whose central purpose is to sexualize real people without consent.
It also raises questions about the role of payments. The city attorney's office claims Apple and Google may have earned millions of dollars in fees from apps offering these services, although that figure is presented as an estimate rather than a confirmed financial disclosure from the companies.
If platforms continue to earn revenue from software that regulators or courts later determine facilitates illegal activity, their financial involvement could become an increasingly important part of future legal arguments.
What the dispute could mean for developers and users
For developers, the immediate lesson is that AI image applications involving real people face growing scrutiny, particularly when they can generate sexualized or intimate content without the subject's consent.
A developer may therefore face consequences not only for the output of a model, but also for how the product is designed, marketed, and distributed.
Users are likely to see more restrictions around searches, app listings, image-generation features, and developer accounts connected to abusive services. Google has already said it restricted related search terms on Google Play, while Apple said it was removing identified apps and terminating developer accounts.
The broader effect could extend to legitimate image-editing services. As platforms try to prevent abusive applications from operating, they may impose stricter controls on features that can manipulate photographs of real people.
That creates a difficult balance. Overly broad restrictions could affect legitimate creative tools, while weak enforcement can leave victims with limited protection after an image has been created or distributed.
The next test is whether enforcement becomes proactive
The immediate outcome of the San Francisco letters may be the removal of the identified apps. But the more consequential question is what happens afterward.
Apple and Google have both indicated that they have taken action against the apps named by the city. San Francisco, however, is challenging the idea that removing apps after repeated warnings is enough.
That distinction could shape future regulation of AI platforms and app stores. If authorities increasingly argue that companies must proactively identify and prevent abusive AI applications, the compliance burden could rise across the industry.
For now, the clearest takeaway is that nudify apps have moved from being a niche AI abuse problem to a direct platform-governance issue. The technology may be generated by developers, but access, discovery, payments, and distribution can all involve larger technology companies.
The key question is no longer simply whether an abusive AI app can be removed. It is whether the companies that make the app available can wait for someone else to identify the harm before acting. That is the standard Apple and Google may increasingly be asked to defend as AI-generated abuse becomes easier to package and distribute.
