Can I run a local LLM on an old server?
Old enterprise servers often have firmware restrictions that stop the motherboard from recognizing generic GPUs, requiring you to buy special branded hardware. If you try to run local LLMs without a GPU, any dense model over 10B parameters will be too slow. While big MoE models can perform faster despite long loading times, these hardware limitations often make buying new equipment the better choice.
What people tried
Every workaround mentioned in the threads below. We haven’t tested any of them — and nobody here is claiming they worked.
Found another way around this?
Share your steps and trade-offs so fellow engineers don't have to reinvent the wheel. No account required.
Can I run a local LLM on an old server?
- 1Use existing legacy hardware like Dell T420 or HP DL380p G8 servers for a proof of concept before scaling up.Did this help you?
- 2Pay for cloud AI subscriptions or API services like Anthropic, OpenAI, or OpenRouter.Did this help you?
- 3Run the GPU in a spare desktop with a newer processor instead of old server platforms.Did this help you?
- 4Using Mixture of Experts (MoE) models instead of dense models, utilizing tools like numactl to pin processes to specific CPU nodes for better memory bandwidth, or sourcing proprietary refurbished GPUs to bypass vendor firmware restrictions.Did this help you?
In their words
Unedited, grouped by where they were said, most upvoted first within each place, each linked to the thread it came from.
“One thing you are going to run into (at least we did with C240-M7SX) is that Cisco require special Cisco branded GPUs. It's a firmware level restriction that will stop the motherboard from recognizing generic GPUs.”source ↗
“Edit: as you can see anything dense more than 10B is too slow. Big moe model will be faster even though the first loading time is very long. This is using ik_llama and numactl.”source ↗
“So shoul i go for one of those or its better to tell them to buy a new hardware?!”source ↗
“I have an old Cisco UCS C240 M4 server running ESXi 6.7 (Intel Xeon E5-2630) with 64 GB RAM and around 1.7 TB of free storage, but no GPU. This one was sitting dust in server room so i had installed ubuntu server and kali linux in this. So, i was thinking of hosting a local llm in that server and ya i haven't made any mind for what to use so as fellow IT guys which ai do you host in server and what do you use it for making your work easier .”source ↗
Where this came up
People with this problem also raised
- 2How do you find the owner of an abandoned cloud server?
- 5How to speed up a slow aging computer
- 5Why are enterprise servers and hardware so expensive now?
- 6How to handle enterprise hardware hitting end-of-life on a tight budget
- 46How to find replacement parts for discontinued hardware
- 2How to manage unmanaged Linux workstations across an organization