June 16th, 2026

After some weeks of research, trials and errors, I managed to successfully install Llama.cpp and have it working with my AMD GPU using vulkan instead of RoCm on Linux Mint v22.3 OS. Learnt many lessons from the experience, and happy to share that here.

The next thing now would be to install Hermes AI Desktop to use these optimised AI models curated via Llama.cpp, all hosted on my own desktop PC, all local, all private within my own network space.

Llama.cpp Install for AMD GPU on Linux Mint

This guide is tailored for AMD desktop PC hardware: Ryzen CPU, 32GB DRAM, a AMD GPU with 16GB VRAM (RDNA 3), running Linux Mint v22.3 (Ubuntu 24.04 base). We will be bypassing the buggy ROCm installations in favor of a robust, direct build process with vulkan.

Phase 1: System Preparation & ROCm Development Tools

(Ensuring the system has the compiler needed to build the software for your GPU)

1. Deep System Clean (Crucial for stability):
Before installing anything new, we must ensure no old, conflicting package configurations are interfering with the new AMD repositories.

bash

# 1. Wipe out all previous AMD repository and pinning configurations
sudo rm -f /etc/apt/keyrings/rocm.gpg
sudo rm -f /etc/apt/sources.list.d/rocm.list
sudo rm -f /etc/apt/preferences.d/rocm-pin-600
sudo apt clean && sudo apt update

2. Install ROCm Development SDK (The Compiler):
We will use the unified rocm-hip-sdk package, as this is the most reliable way to ensure the hipcc compiler and necessary headers are available for building RDNA 3 applications.

bash

# Install the complete, unified ROCm SDK meta-package
sudo apt update
sudo apt install -y rocm-hip-sdk

3. Configure Hardware Permissions (Essential for GPU Access):
Your user account must be explicitly granted permission to talk to the GPU’s compute engine.

bash

# Add your user ($USER) to the video and render groups
sudo usermod -aG video,render $USER

⚠️ MANDATORY ACTION: You must fully Log Out of Linux Mint and log back in now for these group permissions to take effect.

4. Verify ROCm Installation:
After logging back in, open a new terminal and run the check.

bash

/opt/rocm/bin/rocminfo

• SUCCESS: If you see a detailed list of hardware statistics including your AMD GPU, proceed to Phase 2.
• FAILURE: If this command fails or you see an error, please re-run the logout/login step.

---

Phase 2: Compiling llama.cpp with HIP/ROCm Support

(Converting the open-source code into a high-performance executable for your GPU)

1. Clone the Official Codebase:

bash

# Navigate to your home folder
cd ~

# Clone the repository from the official ggml-org organization
git clone https://github.com/ggerganov/llama.cpp

# Enter the directory
cd llama.cpp

2. Clean and Configure the Build:
We use the modern -B build flag to keep all compilation artifacts neatly organized. We point directly to the compiler tools provided by the ROCm SDK.

bash

# 1. Erase any old, failed build artifacts
rm -rf build

# 2. Configure the build container (This is the 'magic' step)
# We map the compilers directly and specify the target architecture (gfx1100 for RDNA 3)
cmake -B build \
-DCMAKE_CXX_COMPILER=hipcc \
-DGGML_HIP=ON \
-DAMDGPU_TARGETS=gfx1100 \
-DCMAKE_BUILD_TYPE=Release

3. Compile the Application:
This step uses all cores of your Ryzen CPU to compile the source code into an executable.

bash

# Compile the binaries using all CPU cores
cmake --build build --config Release -- -j6

• Wait: This process will take a few minutes. Wait until you see a final success message.

4. Apply Performance Overrides (Permanent Setup):
To ensure your AMD GPU handles text generation instantly without lagging during AI processing, we permanently inject the RDNA 3 translation layer override into your shell startup file.

bash

# Inject the performance override permanently into ~/.bashrc
echo 'export HSA_OVERRIDE_GFX_VERSION=11.0.0' >> ~/.bashrc
source ~/.bashrc

# Verify the override is active in your current terminal
echo $HSA_OVERRIDE_GFX_VERSION

---

Phase 3: Testing and Integration with Hermes AI

(Verifying the engine and preparing it for your desktop application)

1. Hardware Integration Test (The Final Proof):
Run the built llama-cli tool to confirm that the compiler successfully mapped the math to your AMD GPU.

bash

# Execute the help command to force the GPU to do the math
./build/bin/llama-cli --help

• SUCCESS: If the manual text payload scrolls cleanly and quickly across your screen, Llama.cpp is successfully compiled and running on your AMD GPU.

2. Preparing the API Server:
The Hermes AI Desktop App requires a running API server. You must launch this in a terminal window. Do not close this terminal while using Hermes.

bash

# Download a small, efficient test model (e.g., Llama 3 8B Q4_K_M)
wget https://huggingface.co/bartowski/Llama-3-8B-Instruct-GGUF/resolve/main/Llama-3-8B-Instruct-Q4_K_M.gguf

# Run the llama.cpp server using the necessary flags for GPU offloading and API exposure
HSA_OVERRIDE_GFX_VERSION=11.0.0 ./build/bin/llama-server \
-m Llama-3-8B-Instruct-Q4_K_M.gguf \
--port 8080 \
--ngl 33 
# --ngl 33 tells it to offload 33 layers to your AMD GPU VRAM (eg. 8GB)

3. Connecting Hermes AI Desktop App:
Once the server is running in the terminal above:

1. Open the Hermes AI Desktop App.
2. When prompted for a provider, choose “Local LLM / Custom Endpoint.”
3. Set the Base URL to: http://localhost:8080/v1
4. Leave the API Key blank.
5. Select the model you used in the server command (Llama-3-8B-Instruct-Q4_K_M.gguf).

You are now fully set up!

Note: This guide was generated using local AI –

AI model: gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf

Generated Tokens: 1,624 tokens

Generation time: 19s

Generation Speed: 81.29 t/s