OpenAI will honor a shared signal allowing website operators to communicate their preferences about using content for AI training to multiple participating providers, the company said on Oct. 30.
The change would reduce the need for website operators to maintain separate instructions for each provider, the company said. OpenAI, an AI developer, announced the commitment in “Making it easier for websites to express AI training preferences,” published Oct. 30.
The company said it aims to implement the signal in the coming months and will provide details when it does so. It gave no specific implementation date.
OpenAI said the shared signal will sit alongside its existing website controls. Those controls will continue to apply, allowing publishers to retain the choices they already communicate to the company.
The signal will appear in robots.txt, a file websites use to communicate instructions to automated visitors, the company said. It will convey preferences about the use of website content to train general-purpose generative AI models.
OpenAI described the effort as voluntary and said website operators will be able to communicate the same preference to several participating providers at once. Its announcement did not name the other providers.
The company said website operators previously had to identify individual crawlers, the automated systems that visit web pages, and maintain separate rules for every provider. A common signal would reduce that work, it said.
OpenAI said its existing crawler controls give publishers flexibility over their pages and will remain in force. The commitment to respect those controls will continue after the company introduces the shared signal.
The company said the initiative draws on work by the Internet Engineering Task Force’s AI Preferences working group. OpenAI joined other participants in the group in announcing the signal.
OpenAI said it believes the commitment strengthens its participation in the group’s standards discussions. It also said it will continue to adjust its practices as those discussions develop, taking account of appropriate controls and fairness.
The company described simpler communication of website owners’ choices as part of its aim for an open web where people can publish and discover information. It encouraged other AI providers to honor the shared signal.
OpenAI presented the commitment as a way to make preferences easier to communicate across providers. The timetable and implementation details remain to be provided in the further update the company promised.
