The AWS CloudFormation documentation provides a good overview of what StackSets are and how to use them, including some best practices on updating StackSets progressively across a large number of stack instances. This article provides some tips on how to progressively roll out changes where your StackSets cover a mixture of controlled and uncontrolled environments, for example accounts for Development, Test, UAT (User Acceptance Test) and Production.

Good change control in CloudFormation needs to extend across both template code and template parameters. In a previous article I wrote in depth about how template parameters work in StackSets: it's worth a quick look to make sure you have a full understanding of the impact of parameter value changes.

The appeal of StackSets is being able to standardise a CloudFormation template across many accounts and regions. In many cases you'd want to extend this standardisation across your deployment tiers, from Dev through to Prod. The trouble with this though is the tiers should be updated sequentially, with verification performed in each tier before proceeding to the next, and with formal change control and approvals in the controlled environments. In contrast, when a change is made to a StackSet in the AWS Console the change is applied across all of its Stack instances.

Another issue is that if the StackSet covers a large number of accounts and regions then an initial change roll-out to just the Dev tier is still too broad - scope of the initial roll-out should generally be very restricted, possibly even a single account and region.

The StackSets best practices documentation covers this situation, saying "we recommend that to test the updated version of a template, you selectively update the stack instances in a few test accounts before updating all stack instances". This can't be done from the console so that leaves a couple of options:

For example here's a CLI command to update a StackSet "Example-StackSet" that applies the updates only to Stacks in the ap-southeast-2 region (specified by the "--regions" parameter, not "--region") and only in accounts 111111111111 and 222222222222:

Note that

A template change should be applied separately to the different deployment tiers - first Dev, then Test, UAT and finally Prod. The StackSets best practices documentation says "To get more granular control over updating individual stacks within your stack set, plan to create multiple stack sets"; probably advice intended for this situation but what's the best way to manage this and is it the best option?

Taking this option means creating separate StackSets for each tier, so four StackSets for Dev, Test, UAT and Prod all with the same template. This dilutes a StackSet's standardisation benefit and might seem like ugly fragmentation but it also aligns well to the way a typical stand-alone Stack is managed. In both cases the tiers are kept separate with their own copies of the standard template code.

The four StackSets could all be in the same StackSet Administrator account, or set up four StackSet Administrator accounts, one in each tier, with the StackSet in each tier containing the Stack instances for that tier. This aligns even more closely to the way stand-alone Stacks are handled. Also it makes it much less likely you'll hit the limit AWS has of 100 StackSets in an administrator account. (Note: Prior to Aug-19 this limit was 20, so I recommended always setting up four StackSet Administrator accounts. It's really not needed in a lot of cases now though.)

Depending on the type of resources and number of Stack instances your StackSet is managing, there's another option to consider. The "Initial Roll-Out" approach above of applying a StackSet update to specific Stack instances can be used to roll out updates progressively from Dev to Prod all under a single StackSet.

For example using the CLI command above, if a StackSet covers two accounts in each tier it can be applied to just Dev like this:

Then it can be applied to Test, still listing the lower tier's accounts:

Re-applying the update to the Dev accounts like this doesn't actually cause any updates to the Dev Stacks. If you don't do this though the Dev stacks will be flagged as "Outdated" in the StackSet.

Then the update can be applied to UAT in a similar way:

Finally the update can be applied to Prod by listing all the StackSet's target accounts, or by simply doing a StackSet update in the AWS Console.

A StackSet can cover a mixture of controlled and uncontrolled environments but the AWS Console doesn't help you perform a limited roll-out for initial testing, so the CloudFormation API (e.g. via the AWS CLI) will need to be used.

A similar approach can roll out StackSet changes progressively across Dev, Test, UAT and Prod, but for many situations it will be better to fragment the StackSet into four - one for each tier. If you do that then establishing four separate StackSet Administrator accounts for Dev, Test, UAT and Prod may be needed in complex environments but is probably overkill in most situations now that an Administrator account can have up to 100 StackSets.