What changed when we gave the team an internal AI agent

Semih Unaldi – Staff Software Engineer · Yomly – 16 May 2026 – 8 min read

A fortnight ago we put a Slack-resident AI agent into the working day at Yomly. Quiet launch. No press, no demo day, no rebrand. Just a bot in the channels people already use and a short message in #all-yomly listing the commands.

What followed surprised us — not because the technology was new, but because of what changed: the unit of work on a given Tuesday.

The smallest interesting change

Before, the default move on a problem looked like this: open the alert, open New Relic, open Jira, grep the repo, pull the DB credentials, find the right teammate, wait, then write the answer.

Now a single Slack message does most of that assembly. A human still verifies and decides. But the gap between intent و first draft has collapsed.

That sounds small. The compounding effect is not.

A real week (anonymised where appropriate)

In the first week of public use, the agent took part in:

• A production incident on a customer’s payroll run. A bulk requery had crashed the majority of pods in the service. The agent traced the cascade across three services, identified two latent code paths the original change had quietly activated, and proposed the fix. The first PR was merged the same evening. The agent then drafted the post-mortem in our template, with timeline, root cause, and corrective actions, ready for review.
• A null-pointer error blocking permissions across eleven services. From one Slack message (“check this error on prod, understand why, pinpoint the service, fix it, then create the PR”), the agent isolated the silent startup failure, opened a single-file PR adding error logging and graceful shutdown handling, and posted the operational steps to bring the pod back.

• A supply chain alert (Mini Shai-Hulud, npm and PyPI, May 2026). Our security team had manually verified Yomly was unaffected. The agent confirmed it across every repository, manifest, lock file, and CI workflow in the organisation, with per-file proof. The same skill now ships as a scheduled /cve audit running each day against the GitHub Advisory database, posting only when something needs attention.
• An expense page performance complaint from three clients. The agent walked through the Angular component, the Apollo cache configuration, and the GraphQL resolver layer; opened a PR with surgical changes; then cycled the reviewer’s comments on a second pass and replied to each inline.
• A “what does our insurance cover?” question in a DM. Answered against the employee handbook, with citations, in seconds. (HR has had this self-served many times before.)
• A board-level question about daily active users. The agent built a NRQL-backed New Relic dashboard end to end, including a separate watchdog service to feed it. Time from question to dashboard: under an afternoon.

These are not curated highlights. They are typical of the runs.

What the architecture is not

We did not write a model. We did not host one. The novelty is not in the LLM. Anthropic supplies that.

What sits between Slack and the model is the part worth describing:

• A thin Lambda function that does one job: verify the Slack signature and dispatch a repository_dispatch to GitHub Actions. It has no opinions about skills, models, tools, or prompts.
• A thick GitHub workflow that does everything else: classifies the task (a cheap Haiku pass), selects the skill, installs the right tool integrations, builds the prompt with the right persona and the right guardrails, runs the agent, watches the stream, and posts the result.
• A single configuration file (agent-config.json) that defines every skill, command, persona, tool, sub-agent, model alias, and spend limit. Most changes ship without touching the Lambda.

That structure has one practical consequence: new capabilities arrive as small markdown playbooks. To add a skill, you write a file. The team has been doing exactly that, with contributions from across engineering, support, and platform.

Three decisions we made deliberately

Security, before convenience.

The agent never sees production PII unless the task requires it, and even then only via read-only credentials, scoped tools, and audited paths. UAE PDPL and ISO 27001 obligations were the brief, not the afterthought. A formal Terms of Use was published before the slash commands were rolled out broadly. Security leadership has confirmed that the agent is the only AI integration in the company currently approved against the production estate.

Human in the loop on anything scheduled.

Engineers can set up agents to run on a cron — say “every morning summarise the overnight error patterns” or “every Monday check the CVE feed against our dependencies.” Scheduled runs are read-only by default. If the agent finds something that needs a write, it stops and waits for a human to say “do it.” Validation of each schedule is itself an agent run, gated by Sonnet, with rejection criteria for vague tasks, PII concerns, or unbounded scope.

Budgets as code.

The agent has a hard monthly spend cap, set in the same configuration file as everything else. The cap is enforced before each run, and a per-run cap kills runaway agents in flight. Total spend across this first fortnight has been roughly the cost of a single experienced engineer’s afternoon. The output ratio is the interesting number.

Why this matters more for a GCC HR platform

Yomly serves clients across the GCC. HR and payroll are not domains where speed without judgement is acceptable. Mistakes affect salaries, statutory filings, and people’s lives.

An internal agent suits this category of work because of those stakes, not despite them. It removes the toil around investigation, documentation, and routine triage. It does not remove judgement. Reviewers, engineers, and operators still own every shipped change. What they have lost is the cost of getting to a usable first draft.

In HR SaaS, the gap between a customer’s question and a defensible answer is now small enough to compete on.

The honest claim is narrower than “it changes everything.” In HR SaaS, the gap between a customer’s question and a defensible answer is now small enough to compete on. And the cost of producing internal documentation has fallen by an order of magnitude, which compounds over a long product roadmap.

What it does not do, and what it never should

It does not approve a salary change. It does not merge a PR alone. It does not decide what a corrective action plan looks like, only what the evidence supports. It does not see, and must not see, sensitive employee data outside of the explicit, audited access paths.

It is also not always right. The most useful pattern we have seen is the second prompt: an engineer asks the agent to validate its previous answer against the source data, and it surfaces the gap. Senior judgement is still where the value lives. The agent compresses the path to it.

What is next

We have a small backlog of skills that the team itself has requested. PagerDuty incident first-responder. Slack release notes from merged PRs. Pre-meeting briefings. A second region of the same architecture for our European clients. None of these need a redeploy. They need a playbook.

The category to watch is not “AI in HR.” It is “judgement under compressed cycle time.” The companies that learn how to keep the judgement, while removing the toil, will do disproportionately well. We intend to be one of them.