A PLC program that compiles can still be wrong in the only way that matters: when the machine is running. SemaPLC, a new open-source agent harness from Midea AIRC and collaborators, makes that failure visible by refusing to call a task finished until external checks pass. The idea is simple enough to steal for other coding agents: the model does not get to grade its own homework.

SemaPLC turns natural language into a running PLC program

The paper, published on August 19, tested the harness on 117 independent PLC program organization unit tasks across seven models. It reports a 72.6% mean strict verified pass rate. That number is useful, but it is not the interesting part. The more important result appears when generated logic has to live inside an existing project and respond correctly at runtime. Static scores bunch together. Runtime behavior does not.

SemaPLC is built around Structured Text, the textual IEC 61131-3 language used for PLC control logic. Its repository packages an agent-driven IDE, a command-line toolchain, and an OpenPLC Runtime v4 Docker environment. The toolchain can check syntax, compile a program, upload and run it, force input variables, read outputs, trace variables over time, and execute a declarative verification plan. The paper calls this a verification gate. In practical terms, it is a rule that turns “the model says it is done” into “the compiler and the running controller produced the expected evidence.”

What the numbers actually prove

The headline 72.6% result comes from the function track, where the tasks resemble existing independent-POU benchmarks. That is a reasonable test of whether models can produce a self-contained unit of PLC logic. It is not a safety certificate for a factory project. A POU that works in isolation may still use the wrong variable, ignore an initialization convention, or conflict with the surrounding state machine.

The project-context track is closer to the problem engineers face. It uses 65 tasks over ten industrial plants. Generated logic has to compile inside a real project, pass static behavior checks, and behave correctly when deployed to a live PLC runtime. The paper separates those layers instead of collapsing them into a single score. That choice matters because a single pass rate can hide where the system is failing.

The dynamic result is the sharpest warning. Baseline methods land between 22.4 and 31.4 on the dynamic behavior score, while SemaPLC reaches 52.2. The static scores sit within ten points of each other. If you only evaluate syntax, compilation, or a formal verdict, you can conclude that the methods are roughly equivalent. Once the programs run and their traces are compared with reference behavior, the gap gets much larger.

That does not mean SemaPLC solved industrial code generation. The evaluation is still a paper benchmark, and the runtime is OpenPLC rather than every vendor's safety-certified controller. The paper also measures generated control behavior, not whether an AI-generated program should be allowed to modify a live production line without a human review. The result supports a narrower claim: execution traces expose failures that static checks miss, and a harness can make those checks part of the agent's stopping condition.

There is a useful lesson here for software agents outside PLCs. “It compiles” is a property of an artifact. “It behaves correctly under the inputs that matter” is a property of a system. Those are different tests. Coding agents routinely stop after the first one.

A safe local deployment recipe

The repository is unusually concrete about the path between model output and a running controller. You need Node.js >= 18, npm, Docker, and an API key for a supported language model. Clone the repository, build sema-plc-tools, build the pinned MatIEC-era runtime base image, install the web app, and configure .env. The first runtime image build can take several minutes, so do not mistake a quiet Docker build for a hung agent.

The project then exposes two useful modes. The web IDE lets an agent generate Structured Text while showing a ladder view, live variables, a process simulation, and tool-call logs. The standalone CLI is better for a repeatable check in CI or a shell script. The README lists 16 tools, including plc_check, plc_compile, plc_buildAndRun, plc_readVariables, plc_forceVariables, plc_trace, and verify.

A sensible acceptance loop looks like this:

compile program.st
buildAndRun program.st
force --set start_btn=true
trace --names led --durationMs 3000
verify plan.json

The exact command syntax comes from the repository, but the order is the point. First reject malformed or uncompilable logic. Then start it in the runtime. Apply the input transitions that exercise the requirement, rather than leaving every input at its default value. Trace the outputs and internal state for long enough to catch timers, resets, and state transitions. Finally, run a declarative verification plan that records what was checked.

For a motor latch, that might mean forcing the start input, waiting through the timer interval, checking that the motor output becomes true, forcing the stop input, and checking that the output drops. For a conveyor interlock, it means testing the blocked path as well as the happy path. A generated program that only passes the happy path has not been verified; it has been demonstrated once.

The repository's local architecture helps with privacy. The browser, backend, PLC tools, and OpenPLC runtime run on localhost, while the configured model provider receives the prompts and code-generation requests. That does not make the setup safe by itself. Keep the runtime isolated from production networks, use simulated or bench inputs, inspect the generated Structured Text, and treat every model API key and Docker volume as part of the test environment. The README also notes that the runtime exposes a REST API on port 8443 with a self-signed certificate, which is another reason to keep it local rather than publishing it casually.

The workflow is also a good filter for deciding when not to trust the result. Stop if the project context is incomplete, if the generated code changes after a verification result without invalidating that result, if a trace does not cover the stated requirement, or if the test depends on a hidden manual step. SemaPLC's strongest design choice is not a new model or a clever prompt. It is the discipline that edits void prior verdicts and that incomplete or indeterminate checks block completion.

The part worth copying

SemaPLC is an industrial case study, but its useful abstraction is broader: put an external, logged, repeatable test between the model and the word “done.” For web code, that can be an integration test against a disposable database. For infrastructure, it can be a plan-and-apply check in a sandbox. For an agent editing documents, it can be a round-trip parser and a schema validator. The test has to exercise the thing people care about, not merely inspect the text the model produced.

The 52.2 dynamic score is not a reason to deploy an agent on a factory floor. It is a reason to stop treating static evaluation as the finish line. If the generated logic controls anything with a timer, reset, interlock, or state transition, run it. Read the trace. Keep the evidence. Then decide whether the code deserves a human review, a bench test, or a hard no.

Sources