Writing ·

LLM Features in Rails and .NET With Background Jobs

By Sovaid Shah, Software Architect · AI · LLM · Rails · .NET · Background jobs · Architecture

TL;DR: Treat LLM calls like any slow, unreliable, metered external API: never call them inside a web request. Enqueue a background job (Sidekiq in Rails, Hangfire in .NET), retry on rate limits with backoff, push results to the browser over ActionCable or SignalR, cache repeat prompts, and log tokens and cost per feature. The job patterns teams already use for payments and imports apply directly.

Every product team is being asked to “add AI”. The first version usually looks like this: a controller action that calls an LLM API, waits for the answer and returns it.

It works in the demo. In production it falls over in predictable ways. Requests take many seconds, the provider rate-limits you at the busiest hour, a timeout leaves a user staring at a spinner, and the bill at the end of the month is a surprise.

None of this is new. An LLM API is a slow, unreliable, metered external service. Rails and .NET teams already know how to handle those, because that’s what background jobs are for. I’ve built job systems on both stacks, Hangfire on ATC in ASP.NET Core and Sidekiq on Go Green Together in Rails, and the same patterns carry straight over.

Rule 1: never call the model inside a web request

A web request should finish in well under a second. LLM calls often take several seconds, and sometimes much longer for big outputs.

Instead:

  1. The request creates a record (say, Summary with status pending) and enqueues a job with its ID.
  2. The job calls the model, saves the result and sets status done or failed.
  3. The browser gets the result pushed to it, or polls.

This keeps web workers free, and it gives you a record of every AI call, which you’ll want later for debugging and cost.

In Rails that’s a Sidekiq job (or ActiveJob on Sidekiq or Solid Queue). In ASP.NET Core it’s BackgroundJob.Enqueue with Hangfire. The comparison is in Hangfire vs Sidekiq.

Rule 2: retry the right failures, the right way

LLM providers fail in a few distinct ways, and each needs different handling:

  • Rate limits (HTTP 429): retry with exponential backoff, and respect the Retry-After header if the provider sends one.
  • Server errors and timeouts (5xx): retry a few times with backoff.
  • Bad requests (4xx other than 429): don’t retry. The prompt or input is wrong, and retrying burns money for the same error.

Both Sidekiq and Hangfire retry failed jobs automatically, so the main job is to raise the right errors and stop retrying on the ones that will never succeed. Set a timeout on the HTTP call itself so a hung connection can’t block a worker indefinitely.

And make the job idempotent. If it saves the result and then crashes before finishing, the retry shouldn’t create a duplicate or charge you twice for the same output. Check the record’s status first.

Rule 3: stream or push results to the user

Users will wait for an AI feature if they can see it working.

  • Rails: broadcast the result (or partial chunks) over ActionCable when the job saves it. Turbo Streams make this a few lines.
  • ASP.NET Core: push through SignalR from the job.

If the provider supports streaming responses, the job can forward chunks as they arrive, so text appears word by word instead of all at once after a long pause.

Rule 4: put limits and caching in front of the model

  • Cache by input. Hash the prompt, model and settings. If the same request comes in again, return the stored answer. Summaries of the same document don’t need to be generated twice.
  • Rate-limit per user so one user or a script can’t run up your bill.
  • Use separate queues. AI jobs shouldn’t delay password resets and payment webhooks. Give them their own queue and concurrency limit, which also caps how hard you hit the provider’s rate limits.

Rule 5: track tokens and cost per feature

Providers return token counts with each response. Save them on the record along with the model name and the feature that made the call.

Then you can answer the questions that come up within a month of launch: which feature costs the most, which users drive it, and whether a cheaper model would be good enough for a specific task.

Also save which prompt version produced each result. When someone changes a prompt and quality drops, you’ll want to know which outputs came from which version.

A minimal checklist

  1. LLM calls only in background jobs, never in requests.
  2. Retry 429 and 5xx with backoff. Don’t retry other 4xx.
  3. Timeouts on every provider call.
  4. Idempotent jobs keyed on a record ID.
  5. Results pushed over ActionCable or SignalR.
  6. Cache by prompt hash, and rate-limit per user.
  7. A dedicated queue for AI jobs.
  8. Tokens, cost, model and prompt version logged per call.

FAQ

Should LLM API calls run in background jobs?

Yes, in almost all cases. They’re slow, rate-limited and can fail, which is exactly what background jobs handle well. Keep web requests fast and push results when they’re ready.

How do you handle LLM rate limits?

Retry HTTP 429 responses with exponential backoff, respect Retry-After, and cap concurrency with a dedicated job queue so you don’t exceed the provider’s limits.

How do you control LLM costs in production?

Cache repeat requests, rate-limit per user, log tokens and cost for each call by feature, and test whether smaller models are good enough for specific tasks.

Planning AI features on a Rails or .NET product? See software architecture.