DjenisAiAgent

Experimental agent / Windows + browser

Computer automation you can inspect.

DjenisAiAgent reads the current interface, chooses one bounded action, and checks the result before moving on. It is a practical study in making tool-calling agents easier to observe and harder to over-trust.

Run the walkthrough Read the setup guide
operator / local

01objectiveOpen Calculator and evaluate 12 × 8

02observeCalculator window detected

03actpress_keys(["12", "*", "8", "="])

04verifyDisplay reads 96

05finishResult verified ✓

01One toolper reasoning step
02Three tiersobserve, interact, system
03Two runtimesnative Windows, remote browser

01 / System

See the state.
Choose one action.
Check the result.

The loop stays deliberately legible. Screenshots and accessibility data provide context; Gemini returns a structured function call; the tool layer executes only what the active runtime and permission tier expose.

Architecture

A short path from intent to evidence.

  1. 01
    Observe

    Capture the screen and a structured UI snapshot.

  2. 02
    Reason

    Ask Gemini for exactly one declared function call.

  3. 03
    Act

    Run a tool allowed by the runtime and operator policy.

  4. 04
    Verify

    Feed the outcome into the next turn before claiming success.

02 / Guided demo

A complete turn,
without the black box.

This deterministic walkthrough runs in your browser. It demonstrates the orchestration contract; it is not connected to a live desktop or Gemini account.

Operator objective01 / 04

“Open Calculator and evaluate 12 × 8.”

The operator defines one bounded outcome. The task starts with no implied permission to write files or run shell commands.

Capability matrix

The runtime determines the tools.

CapabilityWindows nativeDocker / remote browser
Desktop UI automationWindows nativeSupportedDocker / remote browserUnavailable
Browser DOM toolsWindows nativeLocal debuggerDocker / remote browserRemote Selenium
Host screen captureWindows nativeSupportedDocker / remote browserUnavailable
Authenticated web consoleWindows nativeSupportedDocker / remote browserSupported

03 / Security model

Useful by choice.
Restricted by default.

The default tier can inspect approved files and reason about the current state. Interaction and system control are separate opt-ins. The web console binds to localhost unless you change it, and web mode refuses to start without an operator token.

Default

Observe

Read within approved paths, inspect runtime status, and reason. No desktop mutation.

Opt in

Interact

Use approved UI and browser controls without unlocking shell or file writes.

Double opt in

System

Shell, file writes, app launch, and saved screenshots require the system tier plus explicit dangerous-action confirmation.

04 / Build it locally

Read the code.
Set the boundaries.
Then run it.

git clone https://github.com/ejupi-djenis30/DjenisAiAgent.git
cd DjenisAiAgent
uv sync --frozen --extra dev --extra full
uv run --frozen --no-sync djenis

This is experimental software with powerful optional tools. Review the security model and .env.example before enabling interaction or system access.

Open the repository