Skip to main content

Setting Up AI-Safe in a GPU-Enabled VM

Prerequisites​

  • A Virtual Machine (VM) with minimum configuration
    • 4vCPU
    • 16 GB Ram
    • 1 NVIDIA T4 Tensor Core GPU with 16 GiB of memory
    • 64GB Storage
  • NVIDIA driver installed on the VM.
  • Docker and Docker Compose installed on the VM.
  • nvidia-container-toolkit installed and configured to manage GPU devices.
  • A ZIP file containing necessary setup files.

Installation Steps​

Install Docker & Docker Compose​

Ensure Docker and Docker Compose are installed. Here’s the installation guide for Ubuntu:

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add Docker repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update and install Docker and Docker Compose
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

For installation on other OS, refer to Docker Installation Documentation.

Install NVIDIA Driver​

If you don’t already have the NVIDIA driver installed, you can install it using:

sudo apt update
sudo apt install -y nvidia-driver-470  # Replace with the appropriate driver version for your GPU
# After installing please reboot the system
sudo reboot

Check if NVIDIA Driver is installed:

nvidia-smi

Install NVIDIA Container Toolkit​

The NVIDIA Container Toolkit allows Docker to access GPU devices:

# Add NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
    | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
    | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
    | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update and install NVIDIA Container Toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Configure NVIDIA Runtime for Docker​

After installing the NVIDIA Container Toolkit, configure Docker to use the NVIDIA runtime:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Install NGINX and Required Tools​

Additionally, install NGINX, ZIP, and UNZIP:

sudo apt install -y nginx zip unzip

Setting Up the AI-Safe Container​

Copy the ZIP File to the VM​

Transfer the provided ZIP file into the VM’s home directory:

Create Necessary Directories​

Create the main directory structure /opt/maya and subdirectories setup, script, and logs:

sudo mkdir -p /opt/maya/setup
sudo mkdir -p /opt/maya/script
sudo mkdir -p /opt/maya/logs
sudo chown -R root:docker /opt/maya
sudo chmod -R 775 /opt/maya

Unzip the ZIP File​

Unzip the ZIP file inside /opt/maya/setup:

sudo unzip $HOME/ai_safe.zip -d /opt/maya/setup/

Docker Login to Private Azure Container Registry (ACR)​

Login to the private Azure Container Registry (ACR):

echo -n <DOCKER_LOGIN_KEY> | sudo docker login cafmdp.azurecr.io -u pull-token --password-stdin

You’ll be provided the DOCKER_LOGIN_KEY detail seperately

Pull Docker Images​

Pull the necessary images using Docker Compose:

sudo docker compose -f /opt/maya/setup/docker-compose.yml pull

Run Docker Containers​

Start the Docker containers:

sudo docker compose -f /opt/maya/setup/docker-compose.yml up -d

Pull the local AI model​

sudo docker exec -it OLLAMA bash -c 'ollama pull <Local AI MODEL>'

Nginx Configuration​

Modify Nginx Default Configuration​

Delete the default Nginx configuration file:

sudo rm /etc/nginx/sites-available/default

Create New Nginx Configuration​

Create a new default configuration file for Nginx with the necessary routes:

sudo vim /etc/nginx/sites-available/default

Insert the following configuration:

server {
    listen 80;
    server_name _;
    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    location /test {
        proxy_pass http://localhost:8001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    location /mdp {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Restart Nginx​

Restart Nginx to apply the changes:

sudo systemctl restart nginx

Create Scripts for Starting and Stopping Containers​

Start Containers Script​

Create a script named start-containers.sh in /opt/maya/script/:

sudo vim /opt/maya/script/start-containers.sh

And paste below code:

#!/bin/bash

LOG_FILE="/opt/maya/logs/start-containers.log"

sudo docker compose -f /opt/maya/setup/docker-compose.yml pull >> "$LOG_FILE" 2>&1
sudo docker compose -f /opt/maya/setup/docker-compose.yml up -d >> "$LOG_FILE" 2>&1

Stop Containers Script​

Create a script named stop-containers.sh in /opt/maya/script/:

sudo vim /opt/maya/script/stop-containers.sh

And paste below code:

#!/bin/bash
LOG_FILE="/opt/maya/logs/stop-containers.log"
sudo docker compose -f /opt/maya/setup/docker-compose.yml down >> "$LOG_FILE" 2>&1

Make scripts executable​

sudo chmod +x /opt/maya/script/start-containers.sh
sudo chmod +x /opt/maya/script/stop-containers.sh

Create Systemd Service to Start and Stop Containers​

Create Systemd Service File​

sudo vim /etc/systemd/system/docker-containers.service

Insert the following content:

[Unit]
Description=Docker Containers Service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/maya/script/start-containers.sh
ExecStop=/opt/maya/script/stop-containers.sh

[Install]
WantedBy=multi-user.target

Install the Service File​

sudo systemctl daemon-reload
sudo systemctl enable docker-containers.service

Running the Service​

To start the containers:

sudo systemctl start docker-containers.service

To stop the containers:

sudo systemctl stop docker-containers.service

Access AI-Safe​

Once the containers are running, the AI-Safe service can be accessed via the assigned public IP within your network:

http://<PUBLIC_IP_INSIDE_CLIENT_NETWORK>