In reply to a developer on one of the Linux kernel mailing lists, Linux creator Linus Torvalds firmly put a foot down to push back against anti-AI comments.
LLMs don’t “create”. Under the hood, they’re tokenizing the queries, looking for “clouds” of tokens that are similar to the query, then returning a sequence of tokens (with some random noise thrown in) that match what their training data says the answer should be.
In short: all LLM code is an amalgamation of their training data by definition. If there’s nothing similar in there, it’s literally not possible for it to be part of any response.
You’re exactly right. I should have used „generate“ instead of „create“.The point is I don’t think LLMs normally use copyrighted code in a way that would hurt open source projects.
Under the hood, they’re tokenizing the queries, looking for “clouds” of tokens that are similar to the query, then returning a sequence of tokens (with some random noise thrown in) that match what their training data says the answer should be.
Human programmers at least can tell you where they got a snippet they copied, whether it was in the docs, stack overflow or elsewhere, and you can try to keep attribution if you care about compliance. Not only that, but most of our skills are related to designing stuff and recognizing which pattern to use, the specific implementation isn’t necessary the same unless we go look for whatever we saw in the past, as our memories don’t just record everything and repeat it word by word. And after picking up a new language or framework I only need to look around when using a third party library or some API I’m less familiar with, or when something breaks.
The point is I don’t think LLMs normally use copyrighted code in a way that would hurt open source projects.
I don’t know. I’m not a lawyer, and copyright for code was a hot mess even before LLMs got involved. With how many opportunistic copyright/patent trolls there are and how easily convinced judges have been in the past, it could go either way.
Lol, so how do humans code in comparison?
The good programmers normally code by breaking down the problem into constituent parts and logically working through the problem, step by step. What differentiates this from tokenization is that instead of just looking for code that is similar for a similar problem, programmers can usually understand the effects of each line of code, visualize what the state of each variable will be in that step (or dump out the variables to look directly if unsure), and then move on to the next step. This logical problem-solving approach is fundamentally different from a tokenization+noise looking for a similar-looking problem approach. For one thing, you can solve problems that haven’t been solved before.
They use everything for everything, that’s the big issue. Also gpl code. Anything they can trawl through they use. And replicate, in part or in full.
They take code snippets and copy and paste them? Or do they create own code based on what they’ve learned by trawling?
LLMs don’t “create”. Under the hood, they’re tokenizing the queries, looking for “clouds” of tokens that are similar to the query, then returning a sequence of tokens (with some random noise thrown in) that match what their training data says the answer should be.
In short: all LLM code is an amalgamation of their training data by definition. If there’s nothing similar in there, it’s literally not possible for it to be part of any response.
You’re exactly right. I should have used „generate“ instead of „create“.The point is I don’t think LLMs normally use copyrighted code in a way that would hurt open source projects.
Lol, so how do humans code in comparison?
By copy pasting from Stack Overflow
Did you purposely respond like an AI?
Human programmers at least can tell you where they got a snippet they copied, whether it was in the docs, stack overflow or elsewhere, and you can try to keep attribution if you care about compliance. Not only that, but most of our skills are related to designing stuff and recognizing which pattern to use, the specific implementation isn’t necessary the same unless we go look for whatever we saw in the past, as our memories don’t just record everything and repeat it word by word. And after picking up a new language or framework I only need to look around when using a third party library or some API I’m less familiar with, or when something breaks.
I don’t know. I’m not a lawyer, and copyright for code was a hot mess even before LLMs got involved. With how many opportunistic copyright/patent trolls there are and how easily convinced judges have been in the past, it could go either way.
The good programmers normally code by breaking down the problem into constituent parts and logically working through the problem, step by step. What differentiates this from tokenization is that instead of just looking for code that is similar for a similar problem, programmers can usually understand the effects of each line of code, visualize what the state of each variable will be in that step (or dump out the variables to look directly if unsure), and then move on to the next step. This logical problem-solving approach is fundamentally different from a tokenization+noise looking for a similar-looking problem approach. For one thing, you can solve problems that haven’t been solved before.