The Ops Community ⚙️

Cover image for Cloud in the AI Era
Eyal Estrin
Eyal Estrin

Posted on • Originally published at Medium

Cloud in the AI Era

I have been working with public cloud services of the major cloud providers for a decade, and for a while I've been asking myself – what has changed from an engineering, architecture, or cybersecurity point of view when organizations began to embrace generative AI for building modern applications.

In this blog post, I will review various domains (or pillars) and try to figure out if the changes are that drastic, or simply a minor change.

Physical Infrastructure

The most fundamental component in any architecture (whether cloud-native or AI) is the infrastructure, i.e., compute (to run workloads), storage (to store data temporarily or persistently), and network (for connectivity between components).

We still use standard compute, object storage, and VPCs. However, hardware selection and backplane throughput now dictate performance limits.

What to Offload to Neo-Clouds

Neo-clouds specialize strictly in high-density GPU infrastructure without the overhead or costs of traditional hyperscalers' managed services.

  • Why Offload: Hyperscalers often face GPU quota limitations and charge heavy premiums for high-end accelerators (e.g., NVIDIA H100/H200/GB200 clusters). Neo-clouds offer bare-metal or near-bare-metal GPU clusters with specialized InfiniBand fabrics at 30%–60% lower compute rates.

  • What Remains in Public Cloud: Web apps, API gateways, microservices, and general orchestration engines.

What to Offload to On-Premises Environments

On-premises deployment (using platforms like NVIDIA DGX SuperPODs, Dell/HPE GPU servers, and local Kubernetes clusters) makes sense for baseline capacity, extreme latency, and strict data sovereignty.

  • Why Offload: Industrial automation, smart manufacturing, and medical imaging require real-time inference latency (<5ms) that cannot tolerate round-trip travel to a public cloud region. Small Language Models (SLMs) run directly on local edge hardware or private racks.

Bottom line: The shift of cloud to AI workloads is a minor change (evolutionary) from an architecture or engineering point of view. We may have to make some adjustments in terms of selecting the hardware, but it is not considered a major change.

Sustainability and Energy Efficiency

Mature organizations and the hyper-scale cloud providers are already prioritizing sustainability and energy efficiency when building or modernizing their data centers around the world. This is not something we can ignore – both in terms of power consumption requirements and the environmental impact.

AI training and high-volume inference generate significant power spikes, requiring explicit region selection and model right-sizing to hit corporate ESG targets.

What to Offload to On-Premises Environments

To hit corporate sustainability targets under heavy AI demand, architects must actively route compute to regions offering clean energy and high thermal efficiency.

  • Why Offload: Specialized providers locate GPU data centers directly at stranded energy sources (like flared gas or remote renewables), allowing enterprises to run heavy training runs with lower carbon impact.

Bottom line: The shift of cloud to AI workloads is a major paradigm shift (revolutionary) from an architecture or engineering point of view. AI has turned power grid capacity and thermal limits into non-negotiable architectural constraints rather than routine background operational details.

FinOps and Cost Optimization

With the high cost of the public cloud, and now with the demand for expensive hardware and token costs, any organization designing a modern application must embed cost as part of any architecture decision.

The unit of cost is no longer fixed hourly vCPUs; it's token generation rates, context lengths, and idle GPU capacity. Unmonitored recursive loops can exhaust monthly budgets in hours.

What to Offload to Neo-Clouds

Moving terabyte-scale datasets between environments to feed hungry GPU pipelines makes legacy cloud egress fees an immediate architectural liability.

  • Why Offload: Many neo-clouds offer zero-egress or minimal data transfer fees, preventing the severe bandwidth penalties associated with shifting massive dataset checkpoints into public clouds.

What to Offload to On-Premises Environments

When inference workloads run continuously around the clock, public cloud pay-per-token pricing models trigger a cost inversion where owning on-premises hardware becomes vastly cheaper than renting cloud GPUs.

  • Why Offload: Cloud AI pricing scales with usage (per-token or per-instance-hour); on-premises hardware amortizes over time. If an enterprise runs steady 24/7 baseline inference workloads, local GPU hardware often achieves a lower total cost of ownership (TCO) within 12–18 months.

Bottom line: The shift of cloud to AI workloads is a major paradigm shift (revolutionary) from an architecture or engineering point of view. AI shifts cloud economics from predictable infrastructure hours to volatile, context-dependent token generation, turning unmonitored execution loops into instant financial liabilities.

Reliability and Resiliency

Any mature architecture must take reliability and resiliency under consideration. No production workload would be built without embedding those principles as part of an architecture design.

The core availability patterns (multi-AZ, retry mechanisms) are unchanged, but we now apply them to stateful training jobs and external LLM rate limits.

Bottom line: The shift of cloud to AI workloads is considered a minor change (Incremental/Evolutionary). The underlying architectural patterns like redundant regions, retries, and circuit breakers remain identical; we are simply applying those same foundational rules to training checkpoints and LLM rate limits.

Data and Database Layer

As with any other production workload, the data layer must be taken into architectural consideration. Relational and NoSQL platforms remain standard, but native high-dimensional vector search and low-latency RAG pipelines are now fundamental requirements.

What to Offload to On-Premises Environments

For highly regulated industries handling sensitive records, keeping data storage and Retrieval-Augmented Generation context pipelines on-premises is essential to guarantee absolute data sovereignty and eliminate cloud privacy risks.

  • Why Offload: Keeping vector stores, embeddings, and fine-tuning datasets inside private on-premises databases keeps sensitive data entirely within the internal security perimeter.

Bottom line: The shift of cloud to AI workloads is considered a mixed shift (Evolutionary Storage, Major Vector Shift). Core storage engines and relational databases simply added features to support vector embeddings, but the real-time context streaming required for RAG pipelines completely transformed how data feeds into applications.

Operational Excellence and Observability

Mature organizations look far beyond deployment, monitoring, and maintenance, and embed operational excellence and observability. CPU/Memory charts are insufficient. Platform engineers must now monitor token generation speeds, prompt drift, and output accuracy.

Bottom line: The shift of cloud to AI workloads is considered a major paradigm shift (Revolutionary). Observability has evolved from tracking deterministic system health metrics like CPU and memory to measuring non-deterministic application behaviors like token speeds, hallucination rates, and model drift.

Security, Identity and Application Governance

We cannot consider any workload containing customer or business data without thinking about the various security or regulatory topics in every aspect of the workload. It is considered the highest operational risk layer. Security teams shift from securing static HTTP endpoints to governing autonomous, multi-step AI agents and mitigating prompt injection attacks.

What to Offload to On-Premises Environments

Running autonomous AI agents and identity guardrails on-premises gives security teams total control over credential delegation, prompt filtering, and governance boundaries without exposing internal logic to third-party cloud endpoints.

  • Why Offload: Regulated industries processing sensitive customer PII or proprietary IP cannot allow prompt payloads or agent execution chains to traverse public networks. Running identity controls, content safety guardrails, and open-weight models on-premises ensures that autonomous agent actions and credential delegation remain entirely within the internal security perimeter.

Bottom line: The shift of cloud to AI workloads is considered a major paradigm shift (Revolutionary). Security teams must shift from defending static code perimeters to governing non-deterministic outputs, autonomous multi-step agents, and prompt payloads entering model context windows.

Summary

In this blog post, I reviewed many aspects common to any architect and how they shifted in the context of cloud services when organizations began to develop, build, and maintain AI workloads.

Due to cost, hardware availability, and data sovereignty, organizations should consider designing hybrid architectures that include the use of neo-clouds or on-premises, and this blog post highlights reasons to offload some of the workloads from the public cloud.

As technology evolves, organizations should keep questioning their existing architectures and make architecture and technology decisions accordingly – no "all-or-nothing" decision.

About the Author

Eyal Estrin is a cloud and information security architect with more than 25 years in the industry. He is the author of Cloud Security Handbook and Security for Cloud Native Applications.

The views expressed are his own.

Top comments (3)

Collapse
 
unknownstranger profile image
unknown stranger •

Backlink research plays an important role in understanding a website's external link profile. Searching for a majestic group buy * can help you explore different ways to access relevant SEO features. Before selecting a service, review its current pricing and permitted usage. This makes it easier to determine whether the arrangement matches your research requirements.

Collapse
 
screen_white_893c1e68e867 profile image
Screen White •

The data layer part caught my attention. I think retrieval quality is still a bit overlooked when people talk about monitoring RAG systems.

You can have perfectly normal latency and token usage while the system keeps retrieving the wrong documents. Error codes, config keys and technical identifiers are good examples where semantic similarity alone can be unreliable.

I'd probably track retrieval accuracy on a small set of known queries alongside the model evaluation metrics you mentioned.

Do you think retrieval quality will eventually become a standard part of observability for AI applications?

Collapse
 
horald_wilsom_c7df5577ed2 profile image
Horald wilsom •

A free Morse code audio translator supports WAV, MP3, and OGG files plus live microphone input, using adaptive speed detection for fast, accurate results today. You can decode Morse code audio from WAV, MP3, and OGG files instantly, use live microphone input, and convert Morse audio to text online for free. Adaptive speed detection automatically adjusts to different sending speeds, ensuring accurate decoding whether you upload recordings or use a microphone. This makes it ideal for hobbyists, students, and radio operators who need quick, reliable Morse-to-text conversion without installing software or paying fees.