Lawrence Murray on 22 November 2022
updated on 3 September 2023
I updated my GPU development setup from a large laptop with discrete GPU, to a portable laptop with integrated graphics only, now using the cloud for GPU access. During the process I surveyed a good number number of cloud service providers to find the best for my needs, spinning up an instance on each, accessing it with SSH, getting CUDA installed, running my test programs, etc. This article documents my findings. It includes a round-up of cloud service providers, advice on choosing one for your needs, and how-to guides for each. My particular use case is C, C++ and CUDA programming on Nvidia GPUs: machine learning, scientific computing, the occasional PyTorch, but mostly bespoke code. The commentary is flavored by such.
You will need to choose a cloud service provider. Factors to consider include cost, ease of getting started, and the location of data centers.
I recommend Paperspace, now owned by Digital Ocean. Getting started is a breeze, and they offer a good mix of instance types, including cheaper options for development and testing, and cutting edge hardware for production runs. An added bonus is the ability to pause and resume a running instance, and only be billed for storage in between.
For me, quota increase requests were resolved in three minutes for Google Cloud, two days for Microsoft Azure, and four days for Amazon Web Services. I already had instances running within a few minutes on all of Paperspace, Lambda and Genesis Cloud.
You may want to consider one of the majors over one of the bespokes if they offer better support to your region. All the majors have a global footprint, while the bespokes have more limited coverage, albeit still wide.
|Provider||Difficulty||GPU instances?||CPU instances?||Region|
|Paperspace||Easy||Yes||Yes||North America, Europe|
|Lambda||Easy||Yes||No||North America, Europe, Asia-Pacific, Middle East|
|Linode||Hard||On request||Yes||North America, Europe, Asia-Pacific|
|Amazon Web Services||Hard||On request||Yes||Global|
|Google Cloud Services||Hard||On request||Yes||Global|
|Microsoft Azure||Painful||On request||Yes||Global|
Paperspace offers a complete range of instance types, from affordable older generation hardware to the latest generation. It strives for simplicity—and obtains it—with a straightforward sign up process that gets new users up and running in minutes. A particular advantage is the ability to pause a running instance and resume it later while paying only for storage in between, offering convenience and cost effectiveness over competitors that only offer complete shutdown and reconfiguration. There are several machine images available based on Ubuntu. Paperspace was acquired by Digital Ocean in August 2023 but, as of September 2023, still runs as a separate service.
Lambda focuses on state-of-the-art, having mostly latest generation hardware, but there are some cheaper instances with older hardware in limited quantities. Lambda specializes in GPU instances and does not provide CPU instances. All instances run Ubuntu and have LambdaStack pre-installed—Lambda’s software stack for deep learning that includes e.g. CUDA, TensorFlow and PyTorch. Getting started is frictionless and takes only minutes.
Genesis Cloud is unique in two ways: providing hardware from Nvidia’s GeForce consumer line, and operating from Iceland and Norway. GeForce are great for many applications, and certainly for development, but look elsewhere for Nvidia’s data center GPUs if double precision performance is important. All instances run Ubuntu. A good selection of instance types are available by default, with access to the others on request. Essential software, such as CUDA, must be self-installed.
Linode offers a limited GPU service. All instance types use the Nvidia Quadro RTX 6000 which, while affordable for production, is a pricier floor for development than other providers. Access requires a support ticket, a description of the use case, and at least $100 USD of spend (which can be pre-purchased as credits). Once granted, instances are easy to setup with a simple web interface, but only base operating system images are provided, so that CUDA and other essential software must be self-installed.
Amazon Web Services (AWS) offers an extended range of instance types spanning multiple generations of hardware. There are a wide variety of machine images with GPU support. Access to GPU instances requires a support ticket, as initial quotas are zero. Frustratingly, you will likely find out the first time you try and fail to launch an instance, and you must repeat the process.
Google Cloud Services offers an extended range of instance types covering multiple generations of hardware. There are a wide variety of machine images with GPU support. Access to GPU instances is on request, as initial quotas are zero. Frustratingly, you will likely find out the first time you try and fail to launch an instance—but, unlike the other majors, quota increase requests seem to be processed quickly, and there is a convenient retry button to avoid repeating the whole process again. The web interface is much more helpful than the other majors, presenting instance types in a human readable form rather than obscure codes, and making better default suggestions for machine images.
Microsoft Azure offers an extended range of instance types covering multiple generations of hardware. There are a wide variety of machine images with GPU support. Access to GPU instances requires a support ticket, as initial quotas are zero. Frustratingly, you will likely find out the first time you try and fail to launch an instance—and good luck from there, as you navigate a quagmire of product SKUs, regional availability, and interface nuisances, while getting rejected for quota increases.
Access to an instance requires an SSH key pair, consisting of a public key and a private key. The public key is shared with your cloud service provider to manage access to your instances, while the private key is only for you. The public key is the lock that only the private key can open.
You may already have an SSH key. Check for the files
~/.ssh/id_rsa. If they exist, you can use them. If not, create them by running
ssh-keygen in your terminal. The first file contains your public key, the second your private key.
You will need the contents of
~/.ssh/id_rsa.pubbelow. We will refer to it as your 🔑
The next step is to launch an instance via the web interface of your chosen provider. As part of this you will need to provide your 🔑
key. Some providers require additional steps such as upgrading from a free trial to a paid plan, and submitting a support ticket to enable GPU access. Detailed instructions for each provider are below.
By the end of this section you should have an instance running on your chosen cloud service provider, with a username and hostname (or IP address) that can be used to access it. We will refer to these as your 👤
user and 🏠
Work from the console.
ssh user@host. The 👤
paperspaceand the 🏠
hostsome IP address.
export PATH=/usr/local/cuda/bin:$PATH export CPATH=/usr/local/cuda/include:$CPATH export LIBRARY_PATH=/usr/local/cuda/lib64:$LIBRARY_PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
These can be added to the end of
~/.profile to avoid entering them on each login. Run:
Copy them into the bottom of the file, then hit
Ctrl+O followed by
Enter to save, and
Ctrl+X to exit.
Work from the dashboard.
key. Once complete you will be taken back to the Instances page. The new instance will have a status of Booting, which will progress to Running after a few minutes.
Work from the dashboard.
key. Click Create instance. The Instances page will appear with the new instance listed. Its status will progress from Enqueued to Creating… to Active.
sudo apt update sudo apt install nvidia-cuda-toolkit
Work from the AWS Management Console.
keyinto the large text area. Click the Import key pair button.
gpu, and under the Quickstart AMIs tab choose Ubuntu’s Deep Learning AMI GPU PyTorch. For the instance type select box choose
g4dn-xlarge(the economical option—more information on instance types available). Under Key Pair (login) select the name of the key pair that you created in step 2. Click the Launch instance button. Once the instance starts click View All Instances. If it fails to start due to lack of quota, follow the instructions given to submit a support ticket, and try again once approved
host. Your 👤
ubuntu. To confirm these details, or if you chose a different machine image than that suggested above, click the instance name, then the Connect button, then the SSH client tab for detailed instructions. This will give an SSH command of the form
ssh -i key.pem user@host; you will only need the 👤
hostparts of that.
Work from the Google Cloud Console.
keyinto the box and hit Save.
quotasin the search box at the top and selecting All Quotas. However, this firehose of regions and instance types assumes that you already know what you are doing. Easier is to try to start an instance, let it fail, and use the prompt to increase the relevant quota.
host. For 👤
useryou will need the username associated with your SSH key pair, which is likely just the username on the laptop or desktop where you create it. If that does not work, you can always generate a new SSH key pair using the instructions above.
My Azure experience was frustrating. I found the user interface tedious to work with, and was caught in a cycle of rejected quota requests for a single GPU instance with no reason provided. Finally, a helpful support engineer found a GPU for me a continent away. To get that far took two days. Some prior study may help: perhaps start with the documentation on GPU instance types and cross reference with regional availability. You may still find it necessary to contact support as I did, however, and be flexible with regions: the availability of an instance type in a given region does not guarantee that a quota request is approved (demand management, presumably).
Work from the Azure Portal.
ssh keysin the search box at the top of the dashboard. Click the SSH keys item. Click the Create SSH key button. Under Resource group select an existing group or create a new one. Under Key pair name enter a name to give the key. Under SSH public key source select Upload existing public key and copy your 🔑
keyinto the text area. Click the Review + Create button. A validation screen appears, check the details and click the Create button.
quotasin the search box at the top of the dashboard. Select Quotas then Compute. At the top, click on Region and select your preferred region to narrow the search. Search for a GPU instance type. Click Request quota increase > Enter a new limit in the top left. Enter a new limit equal to the number of vCPUs for the instance type and continue (emphasis: don’t just enter 1, enter the number of vCPUs). A response takes several minutes, and you need to keep the web page open while you wait for it (🤷). After waiting, you may find that the request is rejected, and that you must file a support ticket instead. There is a link to do so.
virtual machinesin the search box at the top of the dashboard. Select the Virtual Machines item. Click the Create button and in the popup menu, Azure Virtual Machine.
A10. You may find an instance for which you do not have sufficient quota—there should be a link next to it to request, and you can try again once (if!) approved.
nvidiato find the extension. Click on it, hit Next, Review + Create, Create. Wait for it to deploy—it takes several minutes.
hostis under Public IP Address. Your 👤
sudo apt update sudo apt install nvidia-cuda-toolkit
Open a terminal and enter:
user is the username and 🏠
host the hostname for your instance. You will be prompted with a message “The authenticity of host… can’t be established… are you sure you want to continue connecting?”; this is normal, respond
You can now work with your instance via the terminal. If additional preparation of the instance is required according to the instructions above, you can do that preparation now.
Step by step through the HTML, ImageMagick and Ruby. Works with Jekyll 4.
30 Oct 22
Zero-stride catch and a custom CUDA kernel.
16 Mar 23
Gnome Authenticator for Desktop, Aegis Authenticator for Android, import and export between.
10 Nov 22