Migrating old AWS accounts into Control Tower (Enrollment) – My Journey

I wrote this blog post during my adventure today whilst I was undertaking it. So it may be a little fractious as a result 🙂

Today AWS announced you can now use AWS control tower to set up new multi-account AWS environments in AWS Organisations.

This change:

“.. allows cloud administrators and architects to set up an AWS Control Tower landing zone with an existing Organization.”

Exciting!

My first question is “What about those master accounts that don’t have control tower enabled? What happens to them?”.

I wrote a blog regarding the uplift of a pre-existing master account to become Control Tower Enabled. I had to re-create my entire organisation to even be able to use Control Tower. So does this announcement mean this isn’t a pre-req anymore?

Well it turns out that I can’t fully test this but I can tell you that I recieved the following error when I tried step 1 of doing so in a master without a control tower:

“Your AWS Environment is not ready for AWS Control Tower to be set up. AWS Control Tower detected issues with your AWS account environment that prevent successful setup. You must unsubscribe your organization from AWS config so that AWS Control Tower can proceed.”

So in the least you’re going to have to clean up config recorders before trying this on a pre-existing master. It is possible you will have to do all the things mentioned in my previous blog except the following:

• Removing all the accounts fro the old org
• Creating a new org

I don’t know. As I can’t fully test this end to end any more sorry. Feel free to comment if you go through this process 🙂 I would however hope this was the case. I.e. not having to re-create the org as that was nasty.

The announcement also says:

For existing accounts currently managed in Organizations, you can enroll them into new OUs created using AWS Control Tower individually or via script.

Note: New OUs. Not old.

So, you need a working Control Tower and Control Tower created & managed OUs to enroll accounts that were not created by the control tower vending machine.

I happen to want everything to be managed by Control Tower. So lets go for a journey and see if I can enroll the accounts in my new org that are not managed by Control Tower to date.

First step… read the docs. The enroll link from the announcement takes me here: https://docs.aws.amazon.com/controltower/latest/userguide/enroll-account.html

Which says I “can extend AWS Control Tower governance to individual, existing AWS accounts from unregistered OUs that are part of the same AWS Organizations organization by enrolling them into an AWS Control Tower organizational unit (OU).”.

So far so good. this is what I am after.

“Before you can enroll an existing AWS account into AWS Control Tower you must give permission for AWS Control Tower to manage, or govern, the account. “

Okie dokie. Reason and specificity is stated as: “AWS Control Tower requires permission to establish trusted access between AWS CloudFormation and AWS Organizations on your behalf, so that AWS CloudFormation can deploy your stack automatically to the accounts in your selected organization.”

There are two aspects to this:

1. Enabling an AWS Organisations setting to have AWS CloudFormation StackSets at the org level enabled. https://docs.aws.amazon.com/organizations/latest/userguide/services-that-can-integrate-cloudformation.html
2. Deploying a role into the account to be enrolled for use by AWS Control Tower

Note: The console lists it as “AWS CloudFormation StackSets“ not “AWS Cloudformation” as the doc says under the org settings.

I have 4 accounts to deploy a role into. A script would be handy for larger deployments and fortunately AWS provided one as documented in the blog post here.

Ok. Just for the hell of it, I am not going to do this just yet to any of my accounts. I want to see what happens if I just try to enroll one of my pre-existing accounts using the console without a role to see what errors are produced.

The process for manual enrollment as specified in the docs is:

To enroll an individual account in AWS Control Tower

• Navigate to the AWS Control Tower Account Factory page and select Enroll account.
• Specify the current email address of the existing account you’d like to enroll in AWS Control Tower.
• Specify the first and last name of the account owner.
• Specify the organizational unit (OU) in which you’d like to enroll the account.
• Choose Enroll account.

The first thing I note in the Enroll account form is that the drop down list of OUs are ones the control tower manages / knows about. Hence the “New OU requirement I guess :).

At this point I feel as if there are some steps missing here? ( I guess I could have missed them). Those being:

• Create the OU using Control Tower
• Move the account under the new OU using AWS Orgs

Fortunately I had a new OU created by Control Tower and can move an account under it for testing. So I’m good on that front.

Trying to enroll an account hits an immediate snag!

I federate to Azure AD with AWS SSO! I don’t have a “local AWS SSO user Id and password”! Lets see how this goes! 🙁

Fail. Using my primary Azure AD user account email address has resulted in the above error. Why?? It does say “designate an SSO user” which means an EXISTING one.

Jumping across to have a read of the blog about the script that automates enrollment I note no AWS SSO Email mentioned in the options:

So not sure whats going on there. Assumption drawn at this point is that the process is going to create a new AWS SSO User account (even though I am federating) even though “designate an SSO user” means an EXISTING one.

Hacking in a new email and clicking enroll account then gives me the message:

Ohhhkkkaaayyy.. Not quite what I was expecting 🙂

Checking Service Catalog:

This’ll be interesting.

Ok Cool.. After a minute or so I received the error I was expecting about the lack of a role. “

“AWS Control Tower is unable to assume the AWSControlTowerExecution role in the account. Be sure the role is present in the account, or add it.”

So this definitely confirms my need to create one.

What happened in AWS SSO land? Lets check..

I have a non federated SSO User account hanging around as a result. At least it confirmed my suspicions that it is creating a new, non federated, SSO user. Fortunately it was simple to remove also.

Ok! Whats next!

Now I’ve delved into the script blog post I figured what the hell, lets use it!

It looks like it may handle this federated AWS SSO user issue a little better. (I hope)

Also, checking into the execution example of the script elucidates an interesting variation here.

1. The script will do bulk accounts
2. The script can take into account the unregistered OUs not seen by Control Tower.

as evidenced by the example:

python3 enroll_account.py -o MigrateToRegisteredOU -u FromUnregisteredOU

Ok so moving on to reading the scripts blog post in detail shows up a few more interesting things it does for you:

• It will automate the AWSControlTowerExecution  role creation for you
• If you have a default VPC in the account, the enrollment process tries to delete it. If any resources are present in the VPC, the account enrollment fails.

However! It also states:

If AWS Config was ever enabled on the account you enroll, a default config recorder and delivery channel were created. Delete the configuration-recorder and delivery channel for the account enrollment to work.

This is a real pain. Why isn’t this in the primary documentation page about manually enrolling an account!?!? You should Plan for this activity accordingly folks!

As a result I would enhance the steps to migrate old accounts into control as such:

1. Create using AWS Organisations an OU called “AccountsToMigrate”
2. Create using AWS Control Tower an OU called “AccountsMigrated” and do not enable any additional guardrails until you enroll in the accounts.
3. Move the accounts you want to migrate into the OU AccountsToMigrate
4. For each account you are going to prep the account for migration:
   1. Delete the config recorders in all regions
   2. Delete the Default VPCs in all regions (while you’re there? (you don’t have too))
   3. Enable STS endpoints for us-east-2, eu-west-1, us-east-1, us-west-2, ap-southeast-2
5. Increase your session timeout
6. Execute the migration script
7. Create a new Organizational Unit in AWS Control Tower for account(s) and enable additional guardrails.
8. Move the respective accounts into their respective OUs one by one to check the impact of the guardrails in your environment.

As per AWS recommendation “Start with enrolling the dev/staging accounts (using the script) to get a better understanding of any dependencies or impact of enrolling the accounts in your environment.”

Note: re: Session Timeout. As mentioned in the scripts blog post you will need to up your session time out. Max is 12 hours. The post also states that it takes around 30 minutes for each account to be enrolled. The script polls the service catalog provisioning before moving onto another account. So maximum accounts you can move at one time is 24 accounts. 22 if you wanted to be safer.

Moving on…

I note the script blog states:

The email address associated with the AWS account is used as AWS SSO user name with default First Name Admin and Last Name User.

That will prove interesting! I already had an error when I tried that in the console!

The instructions state:

In your AWS Control Tower environment, access an Amazon EC2 instance running in the master account of the AWS Control Tower home Region.

Get temporary credentials for AWSAdministratorAccess from AWS SSO login screen

Ok; so I have to break best practices and create resources in my master account?? Not good. Admin privilege use case probably should be added to the infosec policy also I guess 🙂

Download and execute the enroll_script.py script

Assumption here is that an EC2 instance in a master account has access to the internet! Bad assumption !! Naughty AWS.

Why can’t I just run this script on my work device I lament whilst reading this. Fortunately, further down it states:

Optionally you may use your own compute environment where the session timeouts can be handled. If you go with your own environment, make sure you have python3, screen and a latest version of boto3 installed.

Not really an “optional” if you ask me. Should be the approach as VPC resources in master should be discouraged.

Moving on….

It is interesting to note that the script doesn’t do the AWSControlTowerExecution role creation itself persay, rather its an OU feature of the stack set as mentioned in the blog :

The script creates the AWSControlTowerExecution role on the target account using Automatic Deployments for a Stack Set feature.

Noted.

Next few steps I took on a windows machine:

• Downloaded script
• Upgraded boto3
• Skipped section about screen (as I’m on a windows machine)
• Copied my AWS SSO Admin creds and set them as env vars
• Set up my OUs

• Moved my accounts (only 2 of the 4 for now) into AccountsToMigrate OU
• Ran the script.

python enroll_account.py -o AccountsMigrated -u AccountsToMigrate

“An error occurred (ValidationError) when calling the CreateStackSet operation: You must enable organizations access to operate a service managed stack set”

Wait. What?! I thought I did that. Recheck docs:

Skip reading fail. Not just the org settings page.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-enable-trusted-access.html Says this is done from the master account CloudFormation console also as shown in the screenshot below:

Ok. Done! Retry script…

Running! Quick check shows now SSO user yet.

Time passes:

Whats going on in my account.. no sso user yet.. no account moved yet.. service catalog has failed!

“InvalidParametersException Parent organizational unit ‘AccountsMigrated’ is not managed by AWS Control Tower.”

How can that be! It says its registered! I used the Control Tower OU Create Process!! I have however.. Renamed it from another name to AccountsMigrated using AWS Orgs not Control Tower. Perhaps under the hood stuff doesn’t like that!

Ironically this error did not bomb the entire script and it just carried on to the next account. Whoops. This should have been fatal.

Ok. Renamed the OU to something else. Made a fresh OU using control tower interfaces. Lets see now…

Well this python script isn’t great with errors then is it. Didn’t clean up after itself.. Also it wasn’t a stack it’s a service catalog provisioned product. So terminated those and tried again.

Re-running script seems to work. So morale of story.. don’t rename your managed OUs from AWS Orgs.. 😀

The AWS SSO user was created fine as a new user as the root account email didn’t clash with any pre-existing SSO user accounts. I already have permission sets set up on these accounts I am migrating as part of them being part of the ORG and AWS SSO set up previously so these new SSO accounts could be deleted just fine. Check before you do yours tho.

Ok new error when provisioning the service catalog product:

“AWS STS isn’t activated in some regions for the following accounts: abcdefghijk. Ask your account administrator to activate AWS STS from the IAM console in each account for the following regions: us-east-2, eu-west-1, us-east-1, us-west-2, ap-southeast-2”

They are on 🙁

Did my tokens expire? Were they being used somehow by the account factory?? To test this theory I’ve extended my session on the admin role. Redeployed the role to my accounts. Logged back into AWS SSO with a new session. Got new temp creds.. And started again (clean up SSO users / service catalog error’d products / moved account back to AccountsToMigrate OU)

Note: This is turning into my most epic longest post by far. If you’re still here reading this you’re a trooper.

SAME ERROR!! RE: STS!! FFS!!

us-west-1 is off so turning that on and seeing if eu-west-1 was a mistake.

Nope. same error. Hmmm. This showed up under the Control Tower / Accounts:

Arg! its not the master account STS 😀 Its the account I am moving! OF COURSE! (Updated the process steps above :))

Ok. Turned on STS in the member accounts. Cleaned up the failures. Trying again.

Success!

The clean up process if the script fails is:

1. Terminate the provisioned service catalog product that has failed
2. Delete the AWS SSO user
3. Move the AWS Account from the root back into the AccountsToMigrate OU
4. Restart the script

Things I’ve learnt:

• You need a working Control Tower
• You cannot rename a managed Control Tower OU via AWS Orgs and expect it to work with this process
• It is better to use the script when enrolling accounts
• You need to prep your member accounts properly (including STS)
• The script doesn’t trap errors well
• AWS SSO gets new users to clean up
• AWS sometimes recommend a process that isn’t best practice
• Config recorder management in 2020 still sucks

The one thing I am unsure about is if my member account had a config recorder or not. I forgot to check. Will update this post when I move the other two accounts I have.

Anyway I now have two service catalog provisioned

And under Control Tower / Accounts:

Not sure what Self means.. will do a post when I find out.

Hurrah! Only took 3 – 4 hours but I got there in the end.

Thanks for tuning in for a decent read. Hope this helps.

Paul.