blog
Lawrence Murray on 22 November 2022
Having tired of mashing together Nvidia drivers and Linux kernels on a laptop with discrete graphics, I started looking for alternatives: how about a laptop with integrated graphics only and a GPU in the cloud? A trial of a few cloud service providers became a roundup of many. My particular use case is CUDA programming on Nvidia GPUs—machine learning, sometimes PyTorch, but mostly my own thing. This commentary is flavored by such, but can be adapted more broadly.
Here are my findings, a how-to for you.
I recommend Paperspace. Getting started is a breeze, and they offer a good mix of instance types, including cheaper options for development and testing, and cutting edge hardware for production runs. An added bonus is the ability to pause and resume a running instance, and only be billed for storage in between.
You will need to choose a cloud service provider. Factors to consider include cost, ease of getting started, and the location of data centers.
For me, quota increase requests were resolved in three minutes for Google Cloud, two days for Microsoft Azure, and four days for Amazon Web Services. I already had instances running within a few minutes on all of Paperspace, Lambda and Genesis Cloud.
You may want to consider one of the majors over one of the bespokes if they offer better support to your region. All the majors have a global footprint, while the bespokes have more limited coverage, albeit still wide.
Provider | Difficulty | GPU instances? | CPU instances? | Region |
---|---|---|---|---|
Paperspace | Easy | Yes | Yes | North America, Europe |
Lambda | Easy | Yes | No | North America, Europe, Asia-Pacific, Middle East |
Genesis Cloud | Medium | Yes | Yes | Europe |
Linode | Hard | On request | Yes | North America, Europe, Asia-Pacific |
Amazon Web Services | Hard | On request | Yes | Global |
Google Cloud Services | Hard | On request | Yes | Global |
Microsoft Azure | Painful | On request | Yes | Global |
Providers not in this roundup include Digital Ocean (which, anyway, does not offer GPU instances), Alibaba Cloud and IBM Cloud.
Paperspace offers a complete range of instance types, from affordable older generation hardware to the latest generation. It strives for simplicity—and obtains it—with a straightforward sign up process that gets new users up and running in minutes. There are several machine images available based on Ubuntu.
Lambda focuses on state-of-the-art, having mostly latest generation hardware, but there are some cheaper instances with older hardware in limited quantities. Lambda specializes in GPU instances and does not provide CPU instances. All instances run Ubuntu and have LambdaStack pre-installed—Lambda’s software stack for deep learning that includes e.g. CUDA, TensorFlow and PyTorch. Getting started is frictionless and takes only minutes.
Genesis Cloud is unique in two ways: providing hardware from Nvidia’s GeForce consumer line, and operating from Iceland and Norway. GeForce are great for many applications, and certainly for development, but look elsewhere for Nvidia’s data center GPUs if double precision performance is important. All instances run Ubuntu. A good selection of instance types are available by default, with access to the others on request. Essential software, such as CUDA, must be self-installed.
Linode offers a limited GPU service. All instance types use the Nvidia Quadro RTX 6000 which, while affordable for production, is a pricier floor for development than other providers. Access requires a support ticket, a description of the use case, and at least $100 USD of spend (which can be pre-purchased as credits). Once granted, instances are easy to setup with a simple web interface, but only base operating system images are provided, so that CUDA and other essential software must be self-installed.
Amazon Web Services (AWS) offers an extended range of instance types spanning multiple generations of hardware. There are a wide variety of machine images with GPU support. Access to GPU instances requires a support ticket, as initial quotas are zero. Frustratingly, you will likely find out the first time you try and fail to launch an instance, and you must repeat the process.
Google Cloud Services offers an extended range of instance types covering multiple generations of hardware. There are a wide variety of machine images with GPU support. Access to GPU instances is on request, as initial quotas are zero. Frustratingly, you will likely find out the first time you try and fail to launch an instance—but, unlike the other majors, quota increase requests seem to be processed quickly, and there is a convenient retry button to avoid repeating the whole process again. The web interface is much more helpful than the other majors, presenting instance types in a human readable form rather than obscure codes, and making better default suggestions for machine images.
Microsoft Azure offers an extended range of instance types covering multiple generations of hardware. There are a wide variety of machine images with GPU support. Access to GPU instances requires a support ticket, as initial quotas are zero. Frustratingly, you will likely find out the first time you try and fail to launch an instance—and good luck from there, as you navigate a quagmire of product SKUs, regional availability, and interface nuisances, while getting rejected for quota increases.
Access to an instance requires an SSH key pair, consisting of a public key and a private key. The public key is shared with your cloud service provider to manage access to your instances, while the private key is only for you. The public key is the lock that only the private key can open.
You may already have an SSH key. Check for the files ~/.ssh/id_rsa.pub
and ~/.ssh/id_rsa
. If they exist, you can use them. If not, create them by running ssh-keygen
in your terminal. The first file contains your public key, the second your private key.
You will need the contents of
~/.ssh/id_rsa.pub
below. We will refer to it as your 🔑key
.
The next step is to launch an instance via the web interface of your chosen provider. As part of this you will need to provide your 🔑key
. Some providers require additional steps such as upgrading from a free trial to a paid plan, and submitting a support ticket to enable GPU access. Detailed instructions for each provider are below.
By the end of this section you should have an instance running on your chosen cloud service provider, with a username and hostname (or IP address) that can be used to access it. We will refer to these as your 👤user
and 🏠host
.
Work from the console.
key
key
.user
and 🏠host
ssh [email protected]
. The 👤user
is paperspace
and the 🏠host
some IP address.export PATH=/usr/local/cuda/bin:$PATH
export CPATH=/usr/local/cuda/include:$CPATH
export LIBRARY_PATH=/usr/local/cuda/lib64:$LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
These can be added to the end of ~/.profile
to avoid entering them on each login. Run:
nano ~/.profile
Copy them into the bottom of the file, then hit Ctrl+O
followed by Enter
to save, and Ctrl+X
to exit.
Work from the dashboard.
key
. Once complete you will be taken back to the Instances page. The new instance will have a status of Booting, which will progress to Running after a few minutes.user
and 🏠host
ssh [email protected]
.Work from the dashboard.
key
. Click Create instance. The Instances page will appear with the new instance listed. Its status will progress from Enqueued to Creating… to Active.user
and 🏠host
ssh [email protected]
.sudo apt update
sudo apt install nvidia-cuda-toolkit
Work from the AWS Management Console.
key
key
into the large text area. Click the Import key pair button.gpu
, and under the Quickstart AMIs tab choose Ubuntu’s Deep Learning AMI GPU PyTorch. For the instance type select box choose g4dn-xlarge
(the economical option—more information on instance types available). Under Key Pair (login) select the name of the key pair that you created in step 2. Click the Launch instance button. Once the instance starts click View All Instances. If it fails to start due to lack of quota, follow the instructions given to submit a support ticket, and try again once approveduser
and 🏠host
host
. Your 👤user
is ubuntu
. To confirm these details, or if you chose a different machine image than that suggested above, click the instance name, then the Connect button, then the SSH client tab for detailed instructions. This will give an SSH command of the form ssh -i key.pem [email protected]
; you will only need the 👤user
and 🏠host
parts of that.Work from the Google Cloud Console.
key
into the box and hit Save.quotas
in the search box at the top and selecting All Quotas. However, this firehose of regions and instance types assumes that you already know what you are doing. Easier is to try to start an instance, let it fail, and use the prompt to increase the relevant quota.user
and 🏠host
host
. For 👤user
you will need the username associated with your SSH key pair, which is likely just the username on the laptop or desktop where you create it. If that does not work, you can always generate a new SSH key pair using the instructions above.My Azure experience was frustrating. I found the user interface tedious to work with, and was caught in a cycle of rejected quota requests for a single GPU instance with no reason provided. Finally, a helpful support engineer found a GPU for me a continent away. To get that far took two days. Some prior study may help: perhaps start with the documentation on GPU instance types and cross reference with regional availability. You may still find it necessary to contact support as I did, however, and be flexible with regions: the availability of an instance type in a given region does not guarantee that a quota request is approved (demand management, presumably).
Work from the Azure Portal.
key
ssh keys
in the search box at the top of the dashboard. Click the SSH keys item. Click the Create SSH key button. Under Resource group select an existing group or create a new one. Under Key pair name enter a name to give the key. Under SSH public key source select Upload existing public key and copy your 🔑key
into the text area. Click the Review + Create button. A validation screen appears, check the details and click the Create button.quotas
in the search box at the top of the dashboard. Select Quotas then Compute. At the top, click on Region and select your preferred region to narrow the search. Search for a GPU instance type. Click Request quota increase > Enter a new limit in the top left. Enter a new limit equal to the number of vCPUs for the instance type and continue (emphasis: don’t just enter 1, enter the number of vCPUs). A response takes several minutes, and you need to keep the web page open while you wait for it (🤷). After waiting, you may find that the request is rejected, and that you must file a support ticket instead. There is a link to do so.virtual machines
in the search box at the top of the dashboard. Select the Virtual Machines item. Click the Create button and in the popup menu, Azure Virtual Machine.T4
, A100
or A10
. You may find an instance for which you do not have sufficient quota—there should be a link next to it to request, and you can try again once (if!) approved.nvidia
to find the extension. Click on it, hit Next, Review + Create, Create. Wait for it to deploy—it takes several minutes.user
and 🏠host
host
is under Public IP Address. Your 👤user
is azureuser
.sudo apt update
sudo apt install nvidia-cuda-toolkit
Open a terminal and enter:
where 👤user
is the username and 🏠host
the hostname for your instance. You will be prompted with a message “The authenticity of host… can’t be established… are you sure you want to continue connecting?”; this is normal, respond yes
.
You can now work with your instance via the terminal. If additional preparation of the instance is required according to the instructions above, you can do that preparation now.
See Develop in the Cloud with Visual Studio Code.
See Profile in the Cloud with Nsight Systems.
Step by step through the HTML, ImageMagick and Ruby. Works with Jekyll 4.
Lawrence Murray
30 Oct 22
Zero-stride catch and a custom CUDA kernel.
Lawrence Murray
16 Mar 23
Gnome Authenticator for Desktop, Aegis Authenticator for Android, import and export between.
Lawrence Murray
10 Nov 22