Provisioning a edge device in a private network with Ansible via AWS Session Manager

aws
Provisioning a edge device in a private network with Ansible via AWS Session Manager
Page content

In this article, I will show you how to access devices in a private network in a remote location to manage the device configuration by AWS System Manager Session Manager and Ansible.

Device provisioning in a private network

I would like to run a configuration management script from my development machine at home on a remotely located device. The image is as follows.

purpose

In most cases, the remotely located devices are probably in a private network, so it is not that easy to run Ansible.

We can consider using AWS Greengrass for simple module updates to edge devices, but it is not suitable for serious configuration management, so we will look for a way to run Ansible.

Environment

Setup procedure

Configuring AWS System Manager Session Manager

The first step is to configure the AWS System Manager Session Manager settings.

AWS System Manager Session Manager is often used as a replacement for a stepping server, but it can be used not only to access virtual servers (EC2), but can also be configured to put up sessions to on-premises servers. This means that if the AWS-provided module supports the OS distribution, it can be installed on edge devices as well.

Refer to the official documentation for the setup procedure. It’s very well written.

I’ll just give you a rough outline of the process.

  1. Create an IAM Instance Profile that will be attached to the Device (the host where Ansible will run)
  2. Configure the System Manager.
  3. Install the AWS Session Manager Plugin on your PC (local machine).
  4. Install the AWS Session Manager modules on the Device (the host where Ansible will run) and register the device to System Manager as a on-premise instance.

Open a port forwarding session in AWS Session Manager

Execute the following command on your PC to enable port forwarding from your PC to the device via Session Manager. The --target option should be the Instance ID of the device registered in Session Manager. And set forwarding the 9090 port on the your PC to the 22 port on the device.

1aws ssm start-session \
2 --target mi-xxxxxxxx \
3 --document-name AWS-StartPortForwardingSession \
4 --parameters '{"portNumber":["22"],"localPortNumber":["9090"]}'

Create an Ansible Playbook file and run it

The next step is to create a set of Ansible Playbook files. The content of the tasks can be anything you want. The important thing here is how you specify the inventory. We have set the provisioning target to the 9090 port on localhost.

1[jetson]
2localhost
3
4[jetson:vars]
5ansible_port=9090
6ansible_user=XXXXXXX
7ansible_ssh_pass=XXXXXX
8ansible_become_password=XXXXXX
9ansible_python_interpreter=/usr/bin/python3

You can run the ansible-playbook command from your PC to the device via port forwarding.

Why port forwarding?

This section explains why you should open a port forwarding session first.

AWS Session Manager document introduces the configuration of ~/.ssh/config by ProxyCommand.

This setting is very useful because it allows you to specify an Instance ID for SSH login, such as ssh user@mi-xxxxxxx.

1Host i-_ mi-_
2ProxyCommand sh -c "aws ssm start-session \
3 --target %h \
4 --document-name AWS-StartSSHSession \
5 --parameters 'portNumber=%p'"

However, the AWS-StartSSHSession specified here will only open a new session (Local Machine -> Session Manager -> Device) every time it is called. The session or session-worker will remain to maintain the SSH connection until you explicitly call aws ssm terminate-session or the session idle times out.

For example, with the ProxyCommand setting in place, if you specify the connection host by Instance ID, a Connection reset by peer error will occur in the middle of execution for relatively long Ansible tasks.

1[jetson]
2mi-xxxxxxxx
3
4[jetson:vars]
5ansible_user=XXXXXXX
6ansible_ssh_pass=XXXXXX
7ansible_become_password=XXXXXX
8ansible_python_interpreter=/usr/bin/python3

If you check the /var/log/amazon/ssm/errors.log file on the Ansible host, you can see that too many open files is output. You can also see that the number of sshd processes on the Ansible host is increasing while Ansible is running.

1ERROR [NewFileWatcherChannel @ filechannel.go.79] [ssm-session-worker] [xxxxxxx-07679fe2fa5825aa8] \
2filewatcher listener encountered error when start watcher: too many open files
3ERROR [createFileChannelAndExecutePlugin @ main.go.105] [ssm-session-worker] \
4[xxxxxxx-07679fe2fa5825aa8] failed to create channel: too many open files
5ERROR [handleSSHDPortError @ port.go.292] [ssm-session-worker] \
6[xxxxxxx-0590c3ea63794d55f] [DataBackend] [pluginName=Port] \
7Failed to read from port: read tcp 127.0.0.1:41294->127.0.0.1:22: use of closed network connection

Conclusion

When you run Ansible commands via Session Manager, you should open a port forwarding session and then run them against localhost.