Adopt an AI policy

[bouweandela]

Using AI to do research and write code is becoming more and more popular. It would be good to have a policy on what we expect from contributions that have been created with (the help of) AI.

Opinions anyone? @ESMValGroup/esmvaltool-developmentteam

[bouweandela]

My proposal would be to do something similar to NumPy: https://numpy.org/doc/stable/dev/ai_policy.html.

[katjaweigel]

I like the NumPy policy in general, only one point I would modify a bit: It says "If so, you must document which tool(s) have been used, how they were used, and specify what code or text is AI generated." We kind of have the same rule in an other context an I found it difficult in praxis, because I remembered that I used an AI tool sometimes (DeepL, for translation), but I did not use it on whole text blocks, rather on short phrases within the the text - but I did not remember where exactly. The same issue one could have when copilot is used.
Therefore maybe something like: "If so, you must document which tool(s) have been used, how they were used, and specify what code or text generated by AI or with AI support." would be easier to follow, because in this way one could say "within this text this an AI tool was used to help with translation/improve grammar/improve code/whatever" although not the whole text is AI generated?

[alistairsellar]

I also think that the NumPy policy is broadly appropriate for ESMValTool, particular that at it's foundation is the human "responsibility" to understand and explain the code and to actually interact with the reviewers.

At the same time, it's important that everyone in the community is respectful in GitHub comments, even when talking to a bot or agent, because the tone of our language on GitHub reflects (and influences) the culture of our community. So I'd suggest that we add some guidance/tips for reviewers on approaches to take when they suspect that the policy has not been followed. And for similar reasons I would recommend that we replace the term "AI slop" with "AI generated and of little substantive value".

[bouweandela]

Example of an AI generated contribution: ESMValGroup/ESMValCore#3189

[rswamina]

Raising a few points that come to mind (mostly because I don't know the answers):

1. If a scientist rewrites their old code, say in a different programming language to Python using AI, would that be AI-generated? What about Vibe coding? Would they all come under AI-generated but acceptable if a human is responsible for the code?
2. Is it possible that AI generated code can be less efficient and put a burden on tech reviewers or in automated checks run during subsequent releases?
3. Is there a possibility that bits of code generated by AI could be copyrighted elsewhere? Would we be able to check that at all or leave it to be flagged by someone in the community and hold the contributor responsible if flagged?
4. Do we need a statement by contributors acknowledging AI was used in the code and short text saying how?

[rswamina]

Raising a few points that come to mind (mostly because I don't know the answers):

1. If a scientist rewrites their old code, say in a different programming language to Python using AI, would that be AI-generated? What about Vibe coding? Would they all come under AI-generated but acceptable if a human is responsible for the code?
2. Is it possible that AI generated code can be less efficient and put a burden on tech reviewers or in automated checks run during subsequent releases?
3. Is there a possibility that bits of code generated by AI could be copyrighted elsewhere? Would we be able to check that at all or leave it to be flagged by someone in the community and hold the contributor responsible if flagged?
4. Do we need a statement by contributors acknowledging AI was used in the code and short text saying how?
5. Should we allow AI fixes to code post-merging? (By original contributor/others)

[dr-ko]

Hi all, fantastic comments and links here. We are confronted with the same AI-powered development that no one seems to (or is willing to) deal with. One thing that may be relevant for you is also the use of AI in understanding the code that is there, which is relevant for the private/dev repo you have. For e.g., a private code scanned by the AI tools. Any plan on how that can be put under the guidelines? I will keep an eye-open in this post.
Cheers,
Sujan

[valeriupredoi]

Hi all, fantastic comments and links here. We are confronted with the same AI-powered development that no one seems to (or is willing to) deal with. One thing that may be relevant for you is also the use of AI in understanding the code that is there, which is relevant for the private/dev repo you have. For e.g., a private code scanned by the AI tools. Any plan on how that can be put under the guidelines? I will keep an eye-open in this post. Cheers, Sujan

Sujan, if the AI agent is not given access to the private repo, then it won't be able to access it. The problem starts rearing its ugly head when a user that has access to the private repo fires up a bunch of AI agents. ChatGPT breaks it down nicely:

If a human has legitimate access to a private GitHub repository and then runs AI agents using that user's credentials/session, the agents can potentially access the private repo because they are operating with the user's authority.

For example:

Private GitHub repo
       │
       ▼
Human's GitHub credentials / token
       │
       ▼
AI agent
       │
       ├── Agent 1
       ├── Agent 2
       ├── Agent 3
       └── Agent 4

If those agents can use the credential, GitHub generally sees the authenticated user/token, not "AI agent #3".

The really important distinction

There are two separate questions:

1. Can the agents technically access it?

Yes, if the human gives the agents access to credentials capable of reading the repository. They could potentially clone it, read files, search the history, etc.

2. Is the human authorised to let them do that?

That's where things get interesting.

A person's permission to access a private repo doesn't necessarily mean they have permission to give an external AI service the contents of that repo.

For example, an organisation might say:

Alice may access the source code.

but separately prohibit:

Alice may upload company source code to third-party AI services.

Those are different permissions.

And multiple agents can make this much more significant

Suppose someone gives an AI coding system access to a repository and says:

"Spin up 20 agents and have them investigate the code."

You could end up with:

• 20 agents reading the source;
• agents sending portions of source to different model/API providers;
• agents creating temporary copies;
• logs containing source-code fragments;
• embeddings/vector databases containing code;
• generated patches being stored elsewhere;
• agents accessing things beyond the original task if the credential is overly broad.

And if the agents are autonomous, the human may not necessarily know exactly which files were accessed or where pieces of the repository were transmitted.

GitHub itself doesn't necessarily know they're AI agents

This is an important security point.

If an agent uses:

Alice's GitHub PAT

then GitHub may simply see:

Alice → API request → private repository

It doesn't inherently mean:

Alice → AI agent → private repository

You can therefore have a perfectly legitimate GitHub audit trail showing that an authorised user accessed the repository while the actual downstream disclosure happened to an AI provider.

That's why organisations increasingly treat AI-agent access as a separate security/data-governance issue, rather than assuming normal GitHub repository permissions solve it.

If you're asking because you're considering what happens when an employee with access to a private GitHub repo starts autonomous agents (particularly coding agents) against it, I can also explain the security model and where the actual leakage risks are — it's quite an interesting problem.

[dr-ko]

Thanks V.
My question was on an even lower level. If someone clones a private repo to their computer and then uses AI to understand that code. This is irrespective of whether the agent has access to the GitHub remote at all. There are settings that may control whether commercial AI agents learn from your code or not, but the trust level there is unclear. So having a clear guideline is critical as well.

[valeriupredoi]

ah heck yeah - that's yet another layer of AI-induced BS uncertainty indeed

[valeriupredoi]

@rswamina re:

Is there a possibility that bits of code generated by AI could be copyrighted elsewhere? Would we be able to check that at all or leave it to be flagged by someone in the community and hold the contributor responsible if flagged?

An excellent point! And this is why I strongly recommend us prompting the AI agent to sign the CLA - at this point in time, and it'll be even more difficult in the future, it's hard to disentangle a human firing up AI agents vs an AI agent contributing on its own accord and will. @alistairsellar do you concur? Firing up AI agents doesn't mean you are no longer responsible for their actions; it'll be harder to account for a piece of code being an intellectual property thief, but hopefully, there'll be laws for that in the future.

[jfrost-mo]

There is an argument for the opposite. The agent should not be signing the CLA, as it doesn't have legal personhood, and if a person's coding agent signs for them, is that really consent?

Regardless of how much of the code/other content is AI generated, the signature should be done manually, as otherwise it loses its meaning. However, any person driving the agent should.

[ghossh]

Adding one more scenario, and it's happening nowadays that to review a PR. People are using AI agents to get a detailed opinion on it.
So, it can happen that a PR is written by an AI agent and another AI agent is reviewing it (of course humans are in the loop).

[bouweandela]

I've opened a pull request to add a policy as discussed here in ESMValGroup/ESMValTool#4631. I've tried to take everyone's comments in, but please take a look and leave any suggestions for improvement you may have.