Step Functions vs Lambda: when you need a state machine

Reach for AWS Step Functions the moment your Lambda functions start orchestrating each other: synchronous function-to-function calls, execution progress tracked by hand in DynamoDB, retry loops copy-pasted into every handler. A state machine moves retries, branching, timeouts and human waits into a declarative definition you can read, test and replay. And since Distributed Map, the same service covers large-scale batch fan-out with up to 10,000 parallel child workflows, a job that used to require an SQS consumer fleet.

When does Lambda orchestration need a state machine?

The reliable smell is a Lambda function whose main job is calling other Lambda functions. When handlers invoke each other synchronously, persist step progress in a homemade DynamoDB table, and each carries its own copy of the retry loop, you are already running a state machine: an invisible one, scattered across your codebase.

Synchronous chaining has concrete costs. The caller pays for the callee's whole duration, timeouts stack up toward Lambda's 15-minute ceiling, and a crash mid-chain leaves a status column that nobody can safely replay. Retry behavior drifts because every handler implements it slightly differently. In the serverless codebases we audit, this is where partial failures hide: an order marked PAID whose shipping step silently never ran.

What does a state machine give you that handlers cannot?

Declarative control flow: per-state Retry with exponential backoff and jitter, Catch routes to recovery states, per-state timeouts and heartbeats, and a visual execution history that shows every state's input and output. Handlers shrink back to pure business logic; the orchestration lives in one reviewable JSON document.

Two capabilities go further than tidier plumbing. With a task token (.waitForTaskToken), a Standard workflow pauses, for hours or months if needed, until a human or an external system calls back: approval steps without polling loops or cron jobs. And SDK integrations let a state call DynamoDB, SQS or Bedrock directly, across more than two hundred AWS services (AWS documentation, 2026), removing glue functions you would otherwise write, patch and pay for.

Standard or Express: which workflow type fits?

Pick Standard for business processes: exactly-once execution semantics, runs up to one year, task token support, and 90 days of queryable history. Pick Express for short, high-volume, idempotent event processing: a five-minute cap, at-least-once semantics, and history through CloudWatch Logs. The type is immutable after creation, so decide per machine.

Cost shapes differ more than the headline model suggests. Standard bills per state transition, so a chatty machine made of many small states costs more than a compact one. Express bills on execution count, duration and memory, which stays reasonable for short bursts but deserves a calculation before it lands on a hot request path. The official comparison lists the full trade-offs.

What does Distributed Map change for batch processing?

Distributed Map turns a single Map state into a fan-out engine: it reads items directly from an S3 prefix, a JSON or CSV file, or an S3 inventory, then runs each item or batch as its own child execution, up to 10,000 in parallel (documented quota, 2026).

Before it existed, the standard recipe for "process a million objects" was an SQS queue plus a Lambda consumer fleet, with hand-built batching, retries and progress tracking. Distributed Map replaces that plumbing for batch workloads: each child execution has its own retry policy and history, ItemBatcher groups items to amortize invocation overhead, ToleratedFailurePercentage keeps one poison item from failing the whole run, and a ResultWriter can aggregate outcomes to S3. Children can run as Express workflows for cost, while the Standard parent keeps the audit trail:

{
  "Comment": "Order pipeline: retry with backoff, compensation on failure, distributed fan-out",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "validate-order" },
      "Retry": [{
        "ErrorEquals": ["Lambda.ServiceException", "States.Timeout"],
        "IntervalSeconds": 2, "MaxAttempts": 4,
        "BackoffRate": 2.0, "JitterStrategy": "FULL"
      }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "CompensateOrder" }],
      "Next": "ProcessItems"
    },
    "ProcessItems": {
      "Comment": "Distributed Map: one Express child execution per batch of 100 objects",
      "Type": "Map",
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": { "Bucket": "orders-inbox", "Prefix": "2026/08/" }
      },
      "ItemBatcher": { "MaxItemsPerBatch": 100 },
      "MaxConcurrency": 1000,
      "ToleratedFailurePercentage": 1,
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "DISTRIBUTED", "ExecutionType": "EXPRESS" },
        "StartAt": "HandleBatch",
        "States": {
          "HandleBatch": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "process-order-batch" },
            "End": true
          }
        }
      },
      "End": true
    },
    "CompensateOrder": {
      "Comment": "Undo side effects, park the input for replay, then fail loudly",
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "cancel-order" },
      "Next": "OrderFailed"
    },
    "OrderFailed": { "Type": "Fail", "Error": "OrderProcessingFailed" }
  }
}

When is Step Functions the wrong tool?

Skip it in three cases. A simple linear flow (one function, one API call, one write) gains nothing from a state machine except latency and another deployment artifact. A high-volume synchronous request path deserves scrutiny: even Express billing on every execution can lose against a single well-built Lambda. And events crossing team boundaries are choreography, not orchestration.

The rule we apply: orchestrate a process you own end to end; choreograph events between domains. If the order team only needs to announce OrderPlaced and let billing and shipping react, that is an EventBridge topology, which we detailed in our EventBridge, SQS and Lambda pattern. The two compose well: an event triggers a state machine that owns one bounded process.

How do you design the failure paths?

Give every state machine an explicit failure branch, the equivalent of a dead-letter queue. Route States.ALL from a Catch into a compensation state that undoes side effects (release the stock, void the authorization), then persist the original input somewhere replayable, S3 or SQS, before ending in a named Fail state.

What we refuse to ship: a machine whose only failure handling is the default abort, because it recreates the "order stuck in DynamoDB" problem the migration was meant to solve. Alarm on the ExecutionsFailed metric, keep compensation states idempotent (they will be retried too), and rehearse a replay from the parked inputs. This failure-path review is usually where our AWS architecture work starts.

Adoption checklist

• List the Lambda functions whose main job is invoking other functions: they are your first candidates.
• Choose the workflow type per machine before creating it; it cannot be changed afterwards.
• Move every hand-written retry into declarative Retry blocks with backoff and jitter, then delete the handler-side copies.
• Replace glue functions with direct SDK integrations wherever a state only calls one AWS API.
• For batch workloads, prototype Distributed Map with ItemBatcher before building another queue and consumer fleet.
• Give every Catch a compensation path that parks the failed input somewhere replayable.
• Keep cross-domain events on EventBridge; orchestrate only processes you own end to end.