This is first post in a series about the design and implementation of Bytebot. Give us a star on our open source repo.

We’re still in the early innings of AI agents. There are hundreds of companies building wrappers around LLMs, trying to make them more useful; more tool-aware, more stateful, more capable of completing tasks across applications. But most of them are barking up the same tree: they’re building agents that work by connecting APIs and tools in structured ways.

Bytebot was born out of a fundamentally different belief: that the simplest and most universal abstraction for agent control already exists, and we’ve been using it for decades.

The Agent as Remote Worker

Here’s the core idea: give an LLM access to a keyboard, a mouse, and a screen. Nothing more.

That’s it. That’s the interface. That’s what a human remote worker uses. And it’s the only interface you need to approximate the vast majority of digital work.

Why does this work? Because nearly all software, all workflows, and all enterprise tooling has been designed (whether explicitly or implicitly) for a human sitting at a computer. If we can simulate the inputs of a human worker and read the same outputs (screen pixels), we can plug into the same workflows. No custom integrations required.

This approach isn’t just simpler - it’s more robust, more generalizable, and more future-proof.

We Tried the Other Way First

Before the current version of Bytebot, we built it as a browser agent.

It started innocently enough: add hooks for prompting into Playwright scripts, letting LLMs handle finding selectors and xpaths:

(The first version of Bytebot, a framework for browser automation)

Then we started building agents that could remote-control browsers using models. Then we built a full-on RPA-style orchestration layer for browser-based tasks.

And we hit every wall imaginable:

• No reliable drag-and-drop support

• Workarounds for downloading files that broke every other week

• Customers who wanted 2FA or password managers like 1Password or Okta

• People wanting to fill a PDF offline, then upload it back

• Apps without APIs but crucial to their workflow

It turns out, browsers are only a fraction of what people actually use. The DOM is not the whole world. So we had a choice: build 1,000 integrations… or zoom out.

The Bitter Lesson

In 2019, Rich Sutton published what’s now known as “The Bitter Lesson”. The gist is: the biggest gains in AI haven’t come from building complex logic or specialized systems. They’ve come from simple methods scaled with compute.

We saw this lesson firsthand.

Every time we built a clever workaround (a DOM compressor, a canvas interpreter, a proprietary multi-step planner), a new model would come out and blow it away. We’d have to rip it all up and start over. Again.

So instead of fighting the models, we decided to get out of their way.

We stopped designing our agent abstractions to match the limitations of current LLMs. We started designing them around the realities of human-computer interaction: assuming models will get better at dealing with screens, input events, and sequential planning. Which they are, rapidly.

Horseless Carriages vs. Autonomous Agents

There’s a natural criticism of our approach: “Aren’t you just recreating a horseless carriage?” - a legacy interface for a new kind of intelligence?

Yes. Intentionally.

We’re not pretending every task is best solved with a desktop agent. There are agent architectures that thrive on abstraction:

• Research agents benefit from tool-rich environments with structured APIs: search tools, calculators, translators, embeddings, and more.

• Code agents can digest an entire codebase, refactor modules, or write libraries using recursive self-feedback and powerful context windows.

• Multi-agent planners can orchestrate workflows where no human-like interaction is needed at all.

But that’s not the whole world.

There’s a massive category of work that lives in the no-man’s land between APIs and deep internal logic. It’s the gnarly, hard-to-automate workflows that involve:

• Jumping between apps

• Copy-pasting between SaaS tools

• Downloading PDFs, renaming them, and uploading them to different portals

• Typing into legacy desktop software that has no public API

• Handling authentication, file dialogs, drag-and-drop interactions

This is the unglamorous work: the stuff that still eats up countless hours across legal, ops, HR, finance, logistics, compliance, and enterprise support. And it’s not going away. It’s deeply embedded in real companies doing real work.

That’s the gap we’re trying to bridge. Not by replacing every workflow with a smarter agent, but by giving models a body; a way to operate inside existing systems without asking the systems to change.

Why This Abstraction Wins

So what do you get when you build from first principles—keyboard, mouse, screen?

• Universality: It works across every app, every OS, every website.

• Fidelity: You can click, scroll, drag, type, trace. Anything a human can do, the agent can do.

• Composability: Actions can be learned, chained, recorded, reasoned over.

• Observability: Screenshots are the source of truth, making agent decisions traceable.

• Extensibility: Add audio, notifications, or sensors later. The core stays stable.

It’s not minimalism for its own sake. It’s choosing primitives that are stable across all computing environments and robust to change.

What This Enables for Companies

Most companies don’t want to build their own agents. They want something that works out of the box and doesn’t require reorganizing their workflows.

That’s what this architecture enables:

• No APIs to maintain

• No special integrations to build

• No model-specific logic to write

Just tasks, tools, and screens.

Give the agent access to the same computing environment your remote team uses. Assign tasks. Let it click, type, scroll, and think. It works with what you already have.

And because the intelligence is model-agnostic and externalized, it gets better automatically as models get better. The scaffolding doesn’t change. The agent just gets smarter.

Building for the Plateau, Not the Peak

Everyone’s building for GPT-5. We’re building for what comes after.

The goal of Bytebot isn’t to chase state-of-the-art LLM performance with narrow integrations. It’s to build an environment where agents can operate reliably no matter how the intelligence improves: an interface that’s durable, general, and designed for the long game.

Next up: how we built a containerized Linux OS from scratch to support that vision.