AI is finally killing IT education for good

Code assignments are a common class of exercise in programming education. They serve two purposes, and LLMs mess with both differently:

1. Coding is a way to get a grade – they help teachers assessing the outcome of the learning process. Here using LLMs is just cheating – they give you an easy, low-effort way to complete the assignment and get a good grade.
2. When you struggle through engineering a piece of code, this is how you actually learn coding and elements of software engineering. LLMs are reasonably good in writing short programs, usually even better than programming newbies. So if you take a shortcut and let it generate code for you, or use LLM heavily for assistance, you won't struggle and you won't develop the same kind and depth of understanding as if you were doing it by yourself.

Cheating is an old problem, and LLM just scales it to absurd proportions. I do not believe cheating is the root cause itself – it usually signifies that students do not want, or can not learn the way teachers expect them. Solving cheating requires looking at the institutions, teachers, students, state and other stakeholders as a whole.

Shortcuts on the formative paths are different, and every new tool makes us revisit this problem – high level languages, StackOverflow, compilers, no-code and so on. I think given a proper education process LLMs are manageable, and even beneficial as they make realistic to teach system architecture practically, something software engineering education always struggled with. I will come back to it in the next post.

LLMs make cheaters rejoice, and teachers trying to grade them are desperate. Here is a rant like many others. Sometimes the students have hard feelings about AI too.

As for me, I was teaching programming from 2009 until recently. I've written textbooks, built a couple of classes from scratch. I started studying programming in 2008 myself so for a long time I was on the same boat as my students – we went through the same classes, had similar struggles, and I was often more of an experienced buddy to them rather than a distant figure of power. I've then led a project of building a 4 years long curriculum for bachelors – an ambitious blend of system engineering, advanced applied mathematics and software architecture.

As a teacher, I've experienced all kinds of cheating. When there is a new class, the first thing students concentrate on is to collect solutions for all assignments in one place. If assignments are generated algorithmically, they will eventually write automatic solvers. Once this is done, everyone gets a choice: do it by yourself, fairly, or just copy from someone.

As a student, I also cheated. It is hard not to, when you came to study programming, and what you're fed is a multiple choice questionnaires like "What is the maximal allowed length of a coaxial cable?". What kind of understanding does it breed in me? I could memorize answers to 1000 questions like that and I still would not write a single line of network code. I had my share of classes that were either out of place, or badly structured, or badly taught, or obsolete. Sometimes, I was just so exhausted and overworked, or my ADHD (undiagnosed back then) just took over and I burned out, so copying someone else's assignment was the only way not to get expelled. That's not my excuse – I did an unethical thing, but I did it for a reason, and so did every other student who ever cheated.

Besides solidarity and cooperation, there were also stolen solutions, and some code copied from StackOverflow verbatim, without thinking. Some solutions for assignments are just bought from other students. I've witnessed one student automating the process by putting a solver online as a service and charging a small fee for using it. I kid you not, he even registered himself as an entrepreneur and supposedly paid taxes from it.

But why are students cheating? Isn't it irrational in some sense? They willingly entered an institution to learn a subject, and instead of following their tutors they cheat. Are they just doing worse to themselves?

In many cases cheating is actually the most rational thing to do – it is just optimizing for a different goal or in a different context. Most students do not cheat just because they can. They cheat to pass the class they do not want to, or can not, pass otherwise. Let's unpack it.

So, students who genuinely have some interest in programming still cheat because:

• they do not understand their goals clearly enough;
• they do not see how a specific class helps them in achieving these goals;
• the class is just bad;
• they do not have resources to pass fairly.

These are systemic faults and failures involving teachers, universities, governments, not just underperforming students.

System engineers know that fixing systems involves analyzing stakeholders, figuring out their requirements and metrics to validate whether requirements are satisfied. For the students, goal of studying is understanding, yet from my experience teachers usually measure the learning (through memorization or outcomes of the assignments) because it is easier. It is managerial – we attempt to access an integral, poorly formalizable understanding through a proxy metric. Our grades, our test results, our assignments are not our understanding, or an ability to use the knowledge we are learning. It is the same as trying to estimate code complexity through the number of lines of code – it may sometimes be a sign of excessive complexity, but generally is not a useful metric by itself. How do we assess understanding? There is a way: an expert talks to the student and asks them questions. This was always the ultimate test, provided the teacher is unbiased and functioning. Understanding is about connections between concepts and applying knowledge in practice – both epistêmê and technê. Even if one has to produce an artifact to prove his mastery, it is usually necessary to discuss how it was done and follow the decision making process together.

Administering it requires a large number of teachers for relatively few students. But the system makes it impossible by scaling everything out of proportion, which forces teachers into classrooms of sometimes 500 students at a time. It is clearly impossible to teach them in the same ways as small focused 10-something groups, and it is questionable if students may even learn much in this way.

For decades we were fooling ourselves that we can somehow assess understanding through a number of a cheaper metrics, so we could deliver a worse education on a larger scale. LLMs made generation so cheap that these cheaper metrics collapsed into complete and obvious inadequacy.

We also know that traditional one-way lectures in a passive audience are not efficient even if students might prefer them for the low-effort experience. Active, engaged learning requires interactivity, constantly making students think, repeat what they've understood, test their comprehension, identify flaws and blind spots and so on.

Seminars work better for this reason, but how many people can I productively engage in a single class? Assuming a class of a non-standard, longer duration of 100 minutes, having more than 10 students at a time leaves very little actual time per student. Besides, I can only talk to so many students per day before I start hating my life. Looking for flaws in their understanding is mentally draining. Again, too many students, too few teachers.

Teachers also lack resources. At times we feel that our classes are not good enough but we can't improve them, because we have no time or energy, or because the context makes it impossible. I've seen a balanced but voluminous class sharing semester with a hard and poorly designed one. The students focused on the poorly organized class because they were scared of failing it. As a result many actually had problems with the easier class. So teachers also go through the motions, forgetting why they teach. The mouse keeps cheesing the game, the cat forgets its nature and ceases to exist.

So, proper studying requires doing so in small groups. The expert attention is the real scarce resource in education. Otherwise we are just impeding most people from productively using their time. Many students never really learned from the classes they have attended – they have memorized facts, but did not draw connections; they could not answer questions, they did not understand what is it all about. They went through the motions and wasted their time, even when they stayed ethical.

Today, the expert attention is not only scarce, but routinely misallocated – instead of spending two hours delivering a lecture to 500 people (who could more comfortably watch a recording) I could tutor some students in that time, but properly and efficiently.

Why would we be so delusional then and try to put more and more people through the programming education without properly scaling the education system? I understand the interest of industry and government: they want more specialists. I have a feeling that institutions sometimes take it as a hint to get more and more students, for which they are paid. But the outcome is more people with diplomas, and worse quality of education for everyone.

And the system keeps contributing other problems on a large scale. For example, they may distribute finances between faculties based on how many students graduate. Expelling students for bad performance then contradicts the survival of the said faculties, and may lead to their starvation and disbanding.

Look how the instutitions adapted to selling assignments, cheating, StackOverflow, past cheating situation, pandemic. It was either an attempt to solve problems through violence and fear of being expelled, or a proctoring theater.

I believe we should not use the overall scarce resources in education to force the unwilling to study. Especially the attention of experts, which is already the bottleneck. Noone has ever been forced into understanding something anyway.

The whole thing only makes sense if the actual goal of the universities is different from the stated goal of teaching. This is not unreasonable to assume – schools, for example, contribute to socializing kids, teaching them how to exist in the society, building state-wide mythology that adults incorporate into their national identity, preventing teenagers from shooting drugs on the streets, and so on. Even a school with incompetent teachers somehow manages with these goals. Maybe universities of today, too, are more about socializing, networking, connecting to the future employers and, above all, getting paid for their services.

This is why I welcome LLMs with open arms. To the education system, LLMs are not poison – they are medicine, a bitter pill making it impossible to keep ignoring the old problems. Hopefully, we will not be able to fool ourselves anymore. It is too cheap to cheat, and no patch can justify keeping the system as before.

The attacker has an edge in this arms race. Students will be able to cheat at scale and virtually everyone will be able to get some degree, or get a certificate for passing a course, or pass a certification exam. Consequently, the value of having any diploma or certificate will plummet because for employers it becomes pure noise. Eventually, less and less people will be interested in getting a degree and the community will develop other ways of assessing competence for hiring, for example, reviewable contributions to open source, referrals, community reputation registry, or just discussing their work with a candidate. Those who seek to "get some diploma so they could get a job" will lose interest.

But economy is ruthless. Few students mean fewer money, unless they pay premium. Will the proper apprenticeship become a privilege of the few talented and interested who also were lucky to be born rich?

The expert attention is the scarcest resource in education. Can we supplement it with LLMs? I have doubts.

First, LLM hallucinations are a well known thing. They are a minor issue as long as the subject is simple enough, and is represented in the training data reasonably well. But they appear everywhere, the flaws in reasoning are tenacious and take more and more expertise to catch as the models become better. Is it OK to train students using explanations provided by such unreliable tool? I do not know. Are most current teachers better? I do not know either.

Second, the primary commercial drive for models like Claude Opus seems to be coding. Code has stricter verification loops than, say, complex system theory (which is at times borderline philosophy), as well as an abundant public training data. Implicitly specializing huge general purpose models for coding, I believe, may make general reasoning, ambiguous problem-solving, or novel, non-repetitive tasks less robust by comparison. So we might need to train specific LLMs to be more accurate in our domains of interest.

I have had mixed experience studying philosophy and occasionally asking LLMs to clarify things for me. Their answers often seem legit on the surface, but once I start unpacking them, I often see the leaps in logic, disconnected sentences, ungrounded conclusions and so on. High-level reasoning might be still too fragile and demanding as of today – not to mention LLMs are fundamentally a wrong tool for precise reasoning.

Is augmenting teachers with LLMs sensible? Reading through a text generated by LLM and finding flaws in reasoning and other errors is a huge effort, just like grading an essay properly. Perhaps, as one of the lines of defense that you need to pass before you get to the expert with a limited attention pool.

There is no shortage of domains requiring delicate reasoning in software engineering – anything that involves somehow modeling reality and verifying whether the model fits reality well enough is a risky application without a closed verification loop.

We ran out of experts and decided not to notice. Cheating is a symptom.

Understanding lives in the connections a student draws between ideas, and only one instrument has ever measured it: a competent person who asks a question, listens to the answer, and asks a better one. That instrument does not scale. So we built proxies – tests, autograders, assignments, grades – and assured ourselves they were close enough, encouraged by market demands. For decades they were close enough, because faking them usually took nearly as much work as doing them. Now faking them takes seconds.

Employers are about to learn the same lesson. Every proposed replacement for the diploma – the portfolio, the open source record, the take-home project – is another proxy, and a generative model eats proxies for breakfast. One item on that list survives: an engineer sitting down with a candidate and talking about the work. Hiring will rediscover what teaching forgot. The bottleneck does not disappear when the credential does; it moves.

The remedy is unwelcome because it is arithmetic and possibly disciminating. Small groups, more teachers, and money spent on the people doing the teaching rather than on renovations and new computer classrooms. Cost per student will skyrocket. Universities compete for teachers against an industry offering more money for less grief, and they mostly lose. Industry can help pay, but only when it sees what it buys, which it hopefully will soon enough, once it starts interviewing even worse graduates with completely meaningless diplomas.

There is an unfair mercy in this. Push the unwilling out of the queue and the ratio fixes itself. Nobody has ever been forced into understanding, and the student enrolled for a deferment, a parent, or a salary is not learning anyway – he is occupying attention that a curious person needed. When the diploma stops opening doors, he will stop coming. That is not a loss. It is the arithmetic finally working in our favour.

LLMs hallucinate, and catching the flaw in a plausible paragraph costs an expert as much, or more attention as writing it properly. Let the linter and the test suite reject the broken submission before a human ever opens it. Let the model draft the exercise, the summary, the first pass. Then spend the hour you saved on the thing only an hour with a person can buy: the question, the answer, the better question.

So the universities I know are, hopefully, ending, and I find I do not mourn them. Their real product was certification at scale; the teaching was the pretext, and it showed. What remains is either apprenticeship – small, expensive, slow, and the only thing that ever worked – or nothing at all.

I would rather teach ten people something true than certify five hundred in nothing. Even when somehow, for some of them, I managed to make it work, somehow, I could have given them infinitely more and much faster. We are about to find out whether anyone will pay for that. In the next post I will argue that if they do, these same models turn out to be the best gift software engineering education has ever received.