Skip to content

Setup Walkthrough

Getting Started

To create a chat server, you must first create the universe.

Using this cold start procedure you can go from nothing to a deployed Keanu backend quickly.

Assumptions

  1. You are starting with a clean, fresh AWS account
  2. You have admin credentials for this AWS account
  3. You have installed all the tools you need on your Linux workstation (follow the tools setup docs)
  4. You are familiar with basic usage of PGP (GnuPG)
  5. You are familiar with basic usage of git

Starting Checklist

  • You have registered a fresh AWS account, added a CC and done SMS verification (when creating, name the account 'root')
  • Before you can create new AWS accounts under your organization, you must verify your email address.
  • Open a support ticket to request the limit of AWS accounts be increased for your organization (the default is 1). This could take a few days.
  • Clone the keanuapp/keanu-reference-deployment repo on your workstation.
  • Register a domain to use for your Keanu service
  • Add the domain to Cloudflare, we are using it only for DNS management so a free account is sufficient.
  • You have the PGP key of all administrators who will have (1) AWS user accounts (2) read access to encrypted server secrets
  • Decide which AWS region you want to deploy your Keanu Backend to

Handling credentials securely

TODO: talk about password manager

1. Account Provisioning

This is a manual process (we have no automation for it).

  1. Login to your fresh AWS account Note: if you did not name your account 'root' then you can update that on the account settings page
  2. Enable Organizations for your root account, perform the email verification.
  3. Click the "Add account" button, then click "Create Account"
  4. Name the account 'production', use a new email address, and enter 'OrganizationAccountAccessRole' for the IAM role name field.
    Note: If your email provider supports it, use plus addresses to reuse the same account, but have 3 distinct emails. Example: [email protected], [email protected], [email protected].
  5. Repeat steps 2 and 3 but for an account named 'development'
    Note: if you ever want to add more accounts (additional prod, dev or staging environments) then just follow this same procedure.
  6. From the Organizations page, note down the Account Ids of each account

You should now have three AWS accounts all together under one organization:

  • root
  • production
  • development

2. Setup Workspace

Since we are managing our infrastructure as code, we need a place to commit and store that code. This guide assumes you will be using git to do so.

Throughout this guide we will refer to several directories:

  • the workspace directory - the umbrella folder that will contain the several repos you need. It should not contain anything unrelated to keanu.
  • the operations repo - the folder (under the workspace dir) where your primary operations git repository lives, 99% of your admin will be run out of this repo

2.1 Create the workspace folder and operations repo

  1. Create the workspace directory on your computer, in this guide we will use ~/projects/keanu
  2. Clone the keanuapp/keanu-reference-deployment repo to your workspace directory giving it the name of your project/organization. We will use keanu-ops. 
    cd ~/projects/keanu
git clone https://gitlab.com/keanuapp/keanu-reference-deployment.git keanu-ops
  3. Enter the directory and remove the existing .git directory, initialize a fresh repository, add everything in the folder, and commit it. 
    cd keanu-ops
rm -rf .git
git init
git add .
git commit -m "Starting fresh with keanu-ops"

  4. Push this repository to a new private git{lab,hub,server} repo

    Warning

    Do not make the keanu-ops repo public. This ops repo will contain confidential information (in the form of encrypted secrets).

  1. From your workspace directory, clone the keanuapp/keanu-terraform-modules repo.

    cd ~/projects/keanu
git clone https://gitlab.com/keanuapp/keanu-terraform-modules.git
    This repo contains the terraform modules for the keanu backend platform. We release tagged versions of the modules as updates and bug fixes are made. The master branch is always the latest stable release of the modules.

  2. From the operations repo, rename the template environments

    cd ~/projects/keanu/keanu-ops
mv template-root root
cp -r template-keanu development
cp -r template-keanu production

3. Bootstrapping Admin User

At this point you have three fresh AWS accounts that are empty and one user (the one you are logged into the root account with). You also have setup your workspace directory.

As a strict rule we want to avoid using the root user of any account. This root user has ultimate control over the account and all infrastructure, so using it for day-to-day operations is an incredible security risk. Even for this bootstrapping process we want to avoid the use of the root user as much as possible.

In this section we will provision an admin user

3.1 The bootstrap admin user

Login to the root account with the root user credentials and do the following:

  1. From the IAM control panel, create new IAM group admin
  2. Assign the AdministratorAccess policy to the group
  3. Create an IAM user with the name admin
    • Only Programmatic Access
  4. Add the admin user to the admin group
  5. Securely Note the Access Key ID and Secret Access Key for the user, you’ll need them in step 7.
  6. From the IAM control panel, browser to Users, select the admin user and Enable MFA for the user
    • It's under Security Credentials tab when viewing the user, click Manage
    • Securely Note the MFA device ARN (looks like arn:aws:iam::XXXXXXXX:mfa/admin), you will need it in step 8.

  7. In a terminal run aws-vault add keanu-root-admin, and add the key id and access key from step 5. It will look like this:

    $ aws-vault add keanu-root-admin
Enter Access Key ID: XXXXXXXXXXXXX
Enter Secret Access Key: XXXXXXXXXXXXX
Added credentials to profile "keanu-root-admin" in vault

  8. Edit ~/.aws/config and add the following config.

    Change the region to be the region you want to host your Keanu Backend infrastructure.

    [profile keanu-root-admin]
mfa_serial=THE MFA ARN FROM STEP 6 ABOVE
region=eu-central-1

  9. Now test that the user and credentials are working as expected using aws-vault login keanu-root-admin. It will prompt you for your MFA code, and then open an AWS console in your browser where you will be logged in as the admin user.

    $ aws-vault login keanu-root-admin
Enter token for arn:aws:iam::xxxxxxxx:mfa/admin: 123456

    If your browser opened to a functioning AWS console logged in as the admin user, then you can proceed. Otherwise you need to troubleshoot your aws-vault setup.

4. Initialize Root Environment

In this section we will setup the root account. This entails:

  • Bootstrap the Terraform state bucket so that the state can be stored remotely.
  • Create a set of IAM groups and policies that will allow us to define admins, billing users, and require MFA.

4.1 Bootstrap terraform state for root account

We store Terraform state in an S3 bucket and use a DynamoDB table for state locking (allowing many users to work on the same project without affecting each other and corrupting the state).

But, because this is a fresh empty account, the state bucket and DynamoDB table are not provisioned yet, so we can’t store Terraform state in the bucket yet.

We will create the bucket and table using local state, and then import the state into the bucket.

Execute these steps:

  1. From the operations repo, enter the environments/root/tfstate-backend sub-folder
  2. Edit the terraform.tfvars and change the namespace and region variables.

The namespace should be your organization name. The default is keanu.

  1. aws-vault exec keanu-root-admin -- make init

This command will create the S3 bucket and DynamoDB table that terraform will use to manage state throughout all the keanu terraform modules.

When asked Do you want to copy existing state to the new backend?, answer yes. This will import the terraform state created so far into the remote bucket.

The module will output a file in ../env that contains required variables and initialization for all the other modules. 4. Commit this ../env to git

Now we have the S3 bucket and DynamoDB table provisioned, and Terraform state for the root environment stored in the bucket itself.

4.2 Provision the root IAM roles and groups

As was mentioned in the introduction, we require that all users assume roles to access the AWS sub-accounts. This lets us manage users, groups, and permissions at the root account level.

So far we have been executing terraform using the admin user we created previously. In this step we will create the admin roles and groups.

Execute these steps:

  1. From the operations repo, enter the environments/root/root-iam sub-folder
  2. source ../env
  3. Edit main.tf, comment out the line that starts with role_arn = ....
  4. aws-vault exec keanu-root-admin -- make init
  5. aws-vault exec keanu-root-admin -- make plan
  6. aws-vault exec keanu-root-admin -- make apply
  7. Edit main.tf, uncomment the line that starts with role_arn = ....

In the Outputs you should see role_admin_arn and role_readonly_arn, these are the unique ids for the roles we created.

Now that we have the root role created, update the root AWS profile in ~/.aws/config. Do this by adding the line role_arn=<THE ROLE ADMIN ARN> to the keanu-root-admin profile block you added before:

[profile keanu-root-admin]
mfa_serial=arn:xxxxxx
region=eu-central-1
## The following line is added
role_arn=arn:aws:iam::XXXXXXXXX:role/keanu-root-admin

You can check that your role is being used by running:

$ check_aws_role keanu-root-admin
Checking AWS Role for profile keanu-root-admin
arn:aws:iam::XXXXXX:role/keanu-root-admin

Great! Next time you use aws-vault, the assumed role will be used.

4.3 Provision root account settings

This will setup a password policy and create a user-friendly account alias for the root account.

Execute these steps:

  1. From the operations repo, enter the environments/root/account-settings sub-folder
  2. source ../env
  3. aws-vault exec keanu-root-admin -- make init
  4. aws-vault exec keanu-root-admin -- make plan
  5. aws-vault exec keanu-root-admin -- make apply

4.4 Provision sub-accounts access config

Note

Remember the word "account" in this context is not a user's account, but an AWS account (root, production, development) in your Organization.

Every user that we provision (in the next step) will be a user added to the root account, and using roles they will be able to access the subaccounts.

Why? Well, it would be very unwieldy if Jane had an IAM user in the root account, another in the production IAM account, and yet another in the development IAM account. Managing all those keys and passwords is error prone and confusing.

The solution? All users will be created in the root account, that is, they will have just one IAM user. In order to access resources and infrastructure in the sub-accounts, we use IAM roles.

Read more about how this works on the AWS docs. But the TL;DR is that we have configured the sub-accounts to trust the root account. The root account then will have a special group per sub-account called keanu-XX-admin, where XX is the sub-account. We will configure a policy such that members of this group have administrator access to the corresponding sub-account.

For example, the production sub-account will have a corresponding keanu-production-admin group in the root account. Every user in this group will have the ability to "assume role" into the production account

This step will setup those IAM roles and policies across the sub accounts.

Before you execute these steps, grab the account arns and ids from the Organizations page when logged into the root account. You will also need to decide which accounts you want to provision at this step.

The default assumes you will have two sub accounts, production and development.

Execute these steps:

  1. From the operations repo, enter the environments/root/accounts sub-folder
  2. Edit terraform.tfvars and edit the account arns and ids as necessary. You can find these values on the organizations account list page.
  3. source ../env
  4. aws-vault exec keanu-root-admin -- make init
  5. aws-vault exec keanu-root-admin -- make plan
  6. aws-vault exec keanu-root-admin -- make apply

Later if you want to provision a new sub-account, you can come back here, add the appropriate config and copy/rename the production.tf file (don't forget to s/production/new_account_name/ in the file too.

Finally, in the output of the apply command you will see some strings you need to save to your notes to use later:

  • *_organization_account_access_role
  • *_switchrole_url

4.5 Create actual user accounts

Up until now we have been operating with the admin user we created in the beginning. While this is certainly better than operating as the root user, we want to create our day-to-day admin users which each person on the team (who needs access) will get.

  1. From the operations repo, enter the environments/root/accounts sub-folder
  2. source ../env
  3. Read and follow the instructions in the README.md in that folder

Once you have users created, login and access the AWS console.

You will notice when you login to the root account as a user, you will not have permission to do much of anything besides access billing and change your password and MFA. This is intentional as we do not want to create resources in the root account.

Now, test the production_switchrole_urls you noted down from the previous step while logged in as your new user. You will be prompted to assume the role in the production account. Click the switch role button, wazaam! You have now assumed the role of an administrator in the production account!

Screenshot of the AWS console showing the user abel logged into the root account, but having the role of admin in the production account

4.6 Setup aws-vault config for users to access sub accounts

Thus far in our commands we have been using the keanu-root-admin profile we created in ~/.aws/config. This profile allows access to the root account.

In order to access the development and production accounts, we need to add an additional profile in ~/.aws/config:

[profile keanu-development-admin]
mfa_serial=arn:aws:iam::xxxx:mfa/admin
region=eu-central-1
role_arn=arn:aws:iam::xxxx:role/OrganizationAccountAccessRole
source=keanu-root-admin

[profile keanu-production-admin]
mfa_serial=arn:aws:iam::xxxx:mfa/admin
region=eu-central-1
role_arn=arn:aws:iam::xxxx:role/OrganizationAccountAccessRole
source=keanu-root-admin

The mfa_serial is copied from the keanu-root-admin profile.

The role_arn is copied from your notes and was output in step 4.4 as *_organization_account_access_role, where * is one of development or production. Make sure to specify the correct role_arn for the respective sub-account.

4.7 Remove the admin account

Now that you have created your own user account, we can remove the admin account.

TODO

Let's come up for air...

Let's review what we have completed thus far:

  1. Register a root AWS account and setup Organizations

  2. Setup your local workspace with the terraform modules and operations directory.

  3. Register sub-accounts for production and development under the root account in the Organization.

  4. Created an admin user to complete the bootstrap process and avoid using the root users.

  5. Bootstrap the S3 bucket for the root account to store the terraform state.

  6. Provision IAM policies and account settings for the root account that setup password requirements, MFA requirements, etc.

  7. Provision an IAM role for the production and development accounts that allows user in the root account to access their resources as admins

  8. Create the actual daily-driver users for our team members.

We've learned a lot in this process. In the future when we need to add more production or staging accounts, or add new users, we can come back here and follow the same procedure.

However we are not done yet. The AWS account setup is complete, and we are ready to begin deploying actual servers to run our Matrix powered messaging infrastructure.

The next steps are more automated, so it will go faster.

5. Deploying the development environment

TODO ontology TODO terraformtvfars explain sample

5.1 Understand the matrix root modules structure

Inside the environments/development/ folder exists the terraform root modules for deploying all the various components of the matrix stack.

The order that the modules are executed in is important as there are dependencies among them.

Here they are explained in execution order:

  1. tfstate-backend: creates the S3 bucket and DynamoDB table for storing terraform state. It must be executed manually first (see next section).
  2. vpc: creates the VPC with public and private LANs, creates security groups, and the internet gateway to allow internal instances to access the internet via NAT
  3. session-manager: sets up the AWS Session Manager so admins can get shell access on EC2 instances
  4. rds: provisions the AWS RDS Postgresql database that is the primary data store for synapse and mxisd
  5. letsencrypt-accounts: registers the Let's Encrypt accounts used to provision certificates. Any domain you authorize or certificate you request is associated with a specific account. Accounts allow you to provide your contact details for expiration notices and other communication from Let’s Encrypt. They also allow you to revoke certificates if you’ve lost access to a certificate’s private key.
  6. letsencrypt-certs: requests and stores the TLS certificates from Let's Encrypt for the needed domains.
  7. log-bucket: creates some IAM policies and an S3 bucket that is used to store logs from various AWS services (RDS, ALB, etc)
  8. matrix-stack: creates and configures the matrix server components (synapse, sygnal, mxisd), the ALB, TLS certs, and all the necessary configuration settings
  9. cloudfront-iam-cert: installs the TLS cert for serving the Riot web application from Cloudfront
  10. riot-web-gitlab: creates the configuration for hosting the Riot web application on S3 and Cloudfront. It also configures a Git Lab repository's CI settings with credentials to deploy to an S3 bucket.
The _data directory

There is one additional directory that is not a terraform root module, _data. This directory holds the configuration and other settings needed to deploy the system.

The settings in this directory are stored in encrypted YAML files. You must use sops to edit the files, so make sure you have set sops up. In the next step we will edit all the settings.

5.2 Edit the settings

Up until now we have deployed pieces of infrastructure that don't involve the specific matrix components directly, so not much configuration was needed. However, before we can move on to deploying these components we must edit the configuration details that are unique for each organization and deployment environment.

When possible, we have provided defaults for many of the configuration settings. The defaults are sensible but many of them must be changed to suit your circumstances.

Included in the reference deployment are sample files (_data/*.sample.yaml). However the root modules expect them to be encrypted and named in the format *.enc.yaml.

You can easily achieve this by renaming them then using sops --in-place --encrypt filename.enc.yaml.

Example:

cd environments/_data
mv common.sample.yaml common.enc.yaml
sops --in-place --encrypt filename.enc.yaml
# repeat with each sample file

The settings are documented extensively. Before continuing to the next section, make sure you have edited all the settings.

5.3 Bootstrap terraform state for development sub-account

For each sub-account (root, production, development, etc) used we store the terraform state in an account-specific bucket.

We will create the bucket and lock table using local state, and then import the state into the bucket.

This is very similiar to the state bucket we created in the previous section for the root account.

Warning

You must have configured at minimum the _data/common.enc.yml settings file to continue.

Execute these steps:

  1. From the operations repo, enter the environments/development/tfstate-backend sub-folder

  2. Edit the terraform.tfvars and set the tf_bucket_region varaible.

  3. aws-vault exec keanu-development-admin -- make init

This command will create the S3 bucket and DynamoDB table that terraform will use to manage state throughout all the keanu terraform modules.

When asked Do you want to copy existing state to the new backend?, answer yes. This will import the terraform state created so far into the remote bucket.

The module will output a file in ../env that contains required variables and initialization for all the other modules.

  1. Commit this ../env to git

Now we have the S3 bucket and DynamoDB table provisioned, and Terraform state for the development environment stored in the bucket itself.

5.4 Deploy the stack

With the tfstate backend deployed for this development environment, you are ready to deploy the entire stack.

Assuming your settings are all ready to go you can deploy the stack in one fell swoop by changing to the environments/development directory and executing make.

You can also deploy the stack one unit at a time using make <root name>. Refer to section 5.1 for details on execution order.

For example, the first root module executed is vpc. So from the development dir execute make vpc.

Troubleshooting

Symptom: 

aws-vault: error: Failed to get credentials for XX (source profile for
XX-YY-ZZ): Invalid data in keyring: unexpected end of JSON input

Cause: The profile XX in ~/.aws/config is not properly setup

Solution: Edit ~/.aws/config and ensure it matches the examples given in this documentation. Check it for typos or extraneous lines.

Common tasks

List aws-vault profiles

aws-vault list

Open AWS console for a profile

aws-vault login XX

List S3 buckets

aws-vault exec XX -- aws s3 ls s3://

What account am I executing under?

aws-vault exec XX -- aws iam list-account-aliases
aws-vault exec XX -- aws sts get-caller-identity