Senior Backend / Distributed Systems Engineer

Aivar Innovations is an AI-native services company and AWS Preferred Partner building governed, production-grade agentic AI systems. We partner with enterprises to deploy intelligent agents that automate complex business processes—from intelligent customer interactions to enterprise knowledge systems. Experience: 5–9 years | 4+ years building product services, or enterprise APIs

: You build the control-plane backend behind Kubogent. Kubogent manages organizations, users, teams, projects, clusters, resource allocations, datastores, experiments, models, training and inference workloads, credentials, webhooks, events, alerts, and long-running infrastru cture operations. Many of those actions ultimately affect Kubernetes, but the system around them is a conventional distributed control plane with durable state, asynchronous workflows, authorization boundaries, event delivery, retries, and strong API semantics. You will own the backend architecture that keeps those pieces coherent.

Own core control-plane services. Design and implement APIs and domain services for organizations, projects, clusters, resources, workloads, models, experiments, webhooks, credentials, events, and other platform entities. Design durable workflow state. Model long-running operations such as cluster onboarding, project provisioning, capability enablement, model deployment, upgrades, and deletion as explicit, recoverable state machines. Build reliable asynchronous systems. Jobs, queues, retries, idempotency, deduplication, leases, backoff, cancellation, timeout handling, and failure recovery are fundamental parts of the product. Own API design. Build clean, versionable REST APIs with predictable resource semantics, filtering, pagination, validation, error models, concurrency behaviour, and compatibility guarantees. Define the domain model. Decide what belongs in PostgreSQL, what belongs in Kubernetes, what is cached, what is derived, and which system is authoritative for each piece of state. Build multi-tenant authorization boundaries. Implement organization-, tenant-, project-, and resource-scoped permissions with clear server-side enforcement and auditable decisions. Build the event backbone. Design how product events, Kubernetes events, operational state changes, notifications, alerts, audit records, and downstream consumers interact. Own webhook infrastructure. Event selection, signing, delivery attempts, retry policy, dead-letter behaviour, observability, and customer-facing delivery history. Build communication with cluster-side systems. Define command, acknowledgement, state-sync, and event protocols between the Kubogent control plane and workload-cluster agents. Design for HA and recovery. Multiple API instances, worker failover, duplicate delivery, leader election where necessary, database failure, stale cache state, and process restarts should not compromise correctness. Own persistence quality. Schema design, indexing, migrations, transactions, optimistic concurrency, query performance, retention, archival, and operational safety. Build platform observability. Structured logs, metrics, traces, request correlation, job execution traces, auditability, and tools that make production incidents diagnosable. Partner closely with frontend and Kubernetes engineers. Product UX, API semantics, controller behaviour, and persistence models should evolve together rather than as separate layers. Set the bar. Review architecture, mentor engineers, and establish backend standards for correctness, testing, operability, and maintainability.