Below I will step through how I setup a Virtual Machine on Google Compute Engine with standard requirements as well as Python and related packages. In a previous post, I went through how to setup a VM and access to it from my laptop CLI.

As mentioned, I like accessing the remote server through the CLI and below I walk through commands to setup the remote instance with basic packages for Linux instance and common Python machine learning packages. Also, the instance I worked with was based on the Debian/GNU Linux 9 image. If you use a different instance, the commands may change but the general steps are similar.

Setup the VM

For the initial setup, I installed common packages and the good old Google Cloud SDK.

Basics

When you first login to your instance, there are a couple basics to get the instance ready to work with. I ran the following commands to get the Linux server setup.

sudo apt-get update   
sudo apt-get --assume-yes upgrade  
sudo apt-get --assume-yes install tmux build-essential gcc g++ make binutils   
sudo apt-get --assume-yes install software-properties-common  
sudo apt-get install htop  
sudo apt-get --assume-yes install git-all wget curl llvm python-openssl unzip

These are additional packages I installed for my project esp. to get pyenv to work and will vary based on what you are working with.

sudo apt-get --assume-yes install libssl-dev zlib1g-dev libbz2-dev \  
libreadline-dev libsqlite3-dev libncurses5-dev libncursesw5-dev \  
xz-utils tk-dev libffi-dev liblzma-dev libpng-dev

Google Cloud SDK

Next up, I installed the Google Cloud SDK so I could run commands like gsutil to connect with other Cloud services in my project. Note if you read my post before this then this will look redundant and it is with the difference being that I’m installing the SDK on my remote server.

Make a download directory and change directory into it. Download the latest SDK software.

mkdir downloads && cd downloads  
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-280.0.0-linux-x86_64.tar.gz

This is the latest version as of the date of the post but you should check here to find the latest version for your operating system and use that.

Untar/unzip and install the SDK on the remote VM.

tar -xf google-cloud-sdk-280.0.0-linux-x86_64.tar.gz  
gcloud init

When it asks chose the account you would like to use to perform operations, chose new account to configure and then login with credentials you use for your project. This is necessary for the SDK configuration and permissions.

Install Python & Required Libraries

For the project I’m hacking on, I’m working in Python and have a number of requirements that I need to fill which include installing Pyenv, TensorFlow and other required libraries. The following gives an example of using pip primarily to setup all Python requirements.

Pyenv

Pyenv is a Python version management software that makes it easy to switch between multiple Python versions. If you need to develop in multiple versions of a language or want to work with libraries that are only compatible with certain versions, something like Pyenv to keep them separate and easy to switch between.

Use the installer which covers the following core steps to setup Pyenv.

cd ~/downloads  
curl https://pyenv.run | bash

Open and add to the ~/.bashrc file at the end the following:

export PATH="/home/[path]/.pyenv/bin:$PATH"  
eval "$(pyenv init -)"  
eval "$(pyenv virtualenv-init -)"

Basic commands to get started with are to checkout the versions that are available to install.

pyenv install --list

Install a couple different versions to work with. The following are examples of installation. Choose what you need.

pyenv install 2.7.17  
pyenv install 3.6.10  
pyenv install 3.7.6  
pyenv install 3.8.1

Note, if there are any operating system or Python libraries you find you need to install after installing the Python versions, then come back to the above step an reinstall the Python versions.

Review what versions are installed and what version is in use which is noted with a *.

pyenv versions

Switch between a version that is in use with the global command.

pyenv global 3.6.10

And like that you have the ability to easily move between and work with different versions of Python.

Common ML Python Libraries

These are requirements I needed for my project and are common packages to use when working with machine learning.

Always start by upgrading pip. Granted when installing any package, it will warn if pip is out of date.

pip install --upgrade pip

Pip install pandas, matplotlib, sklearn, networkx and seaborn. Note, Pandas required pylzma for it to be fully functional.

pip install pylzma  
pip install pandas  
pip install matplotlib  
pip install sklearn  
pip install networkx  
pip install seaborn

A good practice is to create a requirements text of the libraries and run that with pip when setting up the environment. Note, you will have to install all of these packages in each Python version installed in Pyenv which makes it all the more useful to use a requirements doc to simplify installation.

TensorFlow

TensorFlow is an open-sourced end-to-end machine learning platform that I plan to use to run models on the VM.

Install and upgrade TensorFlow.

pip install tensorflow-cpu  
pip install --upgrade tensorflow-cpu

The machine I’ve configured does not have a GPU and using a straight pip install TensorFlow was installing the GPU version and throwing dependency errors for packages that wouldn’t work on my machine. Also, install while using Python 3.7.6 because it is not working with 3.8 as of this post.

Note, if you need an older version of TensorFlow like 1.14 then set install under the python version that works with it like 3.6.10.

pip install tensorflow==1.14

Wrap up

These steps are a more manual approach in comparison to many options that automate or pseudo automate the setup for your like using serverless solutions, Docker images or other types of packages. Still prepackage solutions can get outdated quickly if someone isn’t maintaining them especially when different software versions are constantly coming out and not always playing nicely together. Also, you may need to work with a version or some special configuration that hasn’t been put into one of these packaged solutions. So manual setup is still a very real thing.

What we walked through above is the process of setting up a Debian Linux GCE with different Python versions and common machine learning packages like TensorFlow. That is it for this post. More to come.