LLM Development Stack on Windows

Created With Bing Image Creator (WALL-E Powered)

A lot of the tutorials I’ve seen for getting a basic LLM stack locally make the assumption the user has a Mac or Linux machine. Maybe the assumption makes sense given the prevalence of these two platforms in tech circles. However, a big swath of the world runs on Windows. In this post I will show you how to get setup quickly and painlessly (assuming you have Windows 10 or 11, and have admin privileges).

And to be sure, you could try using something like Google Colab or Noteable and though use of %pip install line magic get to the same outcome. But, if you don’t want to worry about resource quotas (or code privacy in the case of company-owned code) and just want to play around locally you do so in Windows in two different ways.

Created With Bing Image Creator (WALL-E Powered)

Both approaches require these steps:

• First install WSL on your machine with these instructions.
• Because WSL installs with a somewhat old version of Ubuntu, let’s upgrade it with sudo do-release-upgrade command. Confirm when prompted and then confirm again to delete old files.
• Exit/close the current WSL terminal window.
• Download and install VS Code. Although not strictly necessary, it also has WSL extensions for a better experience.
• Install git sudo apt-get install git in a WSL terminal window (note it’s possible it’s already installed through ubuntu, but it won’t hurt)

Approach 1: WSL + Python virtualenvs

Created With Bing Image Creator (WALL-E Powered)

• In a WSL terminal run pip install vex (vex is a lightweight and easy to use python virtualenv manager)
• Then run vex -m <ENV_NAME> where ‘<ENV_NAME>` is up to you, say vex -m openai.
• Then once vex creates the virtualenv run:

git clone git@github.com:rdodev/windows-llm.git

• cd into the windows-llm directory
• Run pip install -r requirements.txt
• Then jupyter notebook and you’re off to the races. Refer to langchain or OpenAI docs to go from here.

Approach 2: WSL + Docker

Created With Bing Image Creator (WALL-E Powered)

• Assuming you do not have it installed already, download and install Docker Desktop for Windows.
• While it’s finishing setting up, it will detect you have WSL installed and will ask if you want to set up WSL extensions and command. If it doesn’t for some reason, you can find this option under settings as shown below

• Open a WSL terminal
• Run git clone git@github.com:rdodev/windows-llm.git
• cd into windows-llm directory
• Run docker build . -t <my_repo>/<image_name>:<tag> (replace those values as appropriate, say, rdodev/llm-playground:v1 for example).
• Run docker run -p 8888:8888 -v $(pwd):/app rdodev/llm-playground:v1 and presto! Refer to langchain or OpenAI docs to go from here.

Note that we could run the above command without -v $(pwd):/app and would still work, however, any changes or new files you add will be lost whenever the container is terminated.

• Once you’re done, make sure to exit the container CTRL+C in the WSL terminal.

Hope this is helpful in getting you started with LLMs on Windows. Till next post.