Tech Session: Intro to Remote Workflow

This document will provide you with tools for comfortably using our remote environment (the course server) to develop and test your team's pipeline.

Here's the agenda for today and a cheatsheet.

We will be covering three things today:

Using the linux command line to navigate directories and files on the course server (and your project directory).
Installing VSCode for coding (if you have not already) and setting up an ssh connection so you can edit code on the course server directly.
Making sure you can access the project data on the course database using psql and dbeaver.

Basic tools for common tasks

Basic Project Workflow

We're providing setup instructions and support for "good enough" tools for each of the common tasks in the workflow for this class but if you're comfortable with other tools, feel free to use them.

Writing code:

Python: This tutorial introduces VSCode, an editor with good Python support, and some tools that make remote development easy.
However, feel free to use any editor you want (vim, emacs, sublime, pycharm).
SQL: In other tutorials, we've introduced psql (for writing sql on the server) and DBeaver or DBVisualizer (on your laptop).

Jupyter notebooks:

For now, the easiest way to use jupyter notebooks is through ``VSCode''
Many Python IDEs (such as, Pycharm) have good Jupyter support - feel free to use one of these!

Share code with your team:

Use the git command line interface to push to your team github repository. We'll do a github refresher next week.
Many IDEs (including VSCode) have git integration.

Run code:

Run Python code manually in an SSH terminal, either by pasting code into a Python REPL, or running a Python script.
Some IDEs (such as VSCode) support remote interpreters, allowing you to run scripts in a python instance on a remote machine (here, the course server).

Agenda for today

Repeat what we did last week: ssh to server, test psql, test dbeaver to connect to the database to make sure we remember.
Navigating the course server using the linux command line including the very important screen command.
Using VSCode for remote development
Remote development concepts - how exactly does all of this work?

Recap from last week:

Let's try repeating what we did last week to get started:

1. Make sure you can SSH to the class server

Using WSL (on Windows) or terminal (on Mac/Linux), connect to the server via the command below (replacing the parameters in curly brackets ({...}) with your info):

ssh {andrew_id}@server.mlpolicylab.dssg.io

Once there, confirm that you're in the right place with the command: echo "$USER@$HOSTNAME"

This should return your andrew id at the server hostname (mlpolicylab-94889)

:warning: If you get something else, let us know.

2. Make sure you can reach the class database via DBeaver

Using DBeaver (or DBVisualizer), connect to the class database and run:

SELECT
'Hello, my name is '||CURRENT_USER||', and I''m connected to '||current_database()||' via '||application_name
FROM pg_stat_activity
WHERE usename=CURRENT_USER
AND state='active';

This should output a friendly message identifying you on the database.

:WARNING: If you get something else, let us know.

3. Initial setup of VSCode

We'll be setting up VSCode as an editor to work with files remotely over SSH during the session. As a first step beforehand, please install VSCode and the Remote-SSH and Microsoft's python extensions using the instructions below.

Download and install VSCode
Install the Remote - SSH extension:
1. Press ctrl+shift+x (Linux/Windows) or ⌘+shift+x (MacOS) to open the extensions menu
2. Search for and install Remote - SSH
1. At this time, also search for and install the microsoft Python extension.

Living the command line

Getting started: SSH to the server (if you're not already connected)

Open up a wsl/*nix/gitbash terminal and connect to the server with:

ssh {andrew_id}@server.mlpolicylab.dssg.io

Some key Linux concepts

Linux Paths

Absolute paths:

An absolute path is a path that starts at a system's root directory.

For example, the command pwd will print the absolute path to your current directory:

To refer to a location using an absolute path, specify your path starting with a /

Absolute paths are as unambiguous as possible. However, they're not as convenient as...

Relative paths

A relative path specifies the path to some folder or file, relative to the current location.

To use a relative path, specify a path not starting with a /

An example:

I start in /class/rayid/
I go up one level by typing cd .. (get to /class)
I then use cd groups (note: doesn't start with /) to go to the project groups directory
I've changed directories to /class/groups/

The home directory

In Linux, each user has a "home directory". This is the default directory a user enters upon login.

You can access your home directory with the command cd ~.

You can also specify absolute paths in a similar way:

My home directory is /class/rayid
I can access the folder mlforpublicpolicylab stored in my home directory with cd ~/mlforpublicpolicylab

Anatomy of a Linux command

Linux commands share a basic syntax. Let's take a look at one to see how it works:

ls -l --human-readable ./mlforpublicpolicy

This command contains four parts:

ls: This is the name of the command we're running. ls is a utility that lists the files and folders present in a directory. The command name is always the part that comes first.

-l & --human-readable: Both of these are options. Options are used to change the behavior of a command. Options usually start with one or two dashes (one dash for single-character options, two for longer options).

-l tells ls to give detailed descriptions of all the files it lists (including size and permissions). --human-readable is self-explanatory: it tells ls to make its output easy to read.

./mlforpublicpolicylab: This is the argument. Here, it's a relative path to the folder that we're telling ls to list the contents of. Most Linux commands take an argument - often text, or a file or folder to operate on.

Getting help

Linux makes it easy to get help with a command:

man {command}

Opens the manual page for the command in question. Many commands also offer a help menu accessible with {comand} --help

Some key command line tools

At first, it can be tough to do basic things like browsing folders or editing text in the command line. But Linux includes a lot of helpful tools for these kinds of tasks. In this section, we'll show how to use some of these tools to get around the terminal.

Follow along by executing the commands on the numbered lines.

Connect to the course server with SSH (if you aren't already)

Getting oriented:

Let's start by getting our bearings inside of the filesystem.

First, let's figure out where we are, with pwd:

pwd prints the absolute path of the current working directory.

Print your current working directory: pwd

Next, let's find out what's in our current directory, with ls:

ls {some_folder (by default, the working directory)}

lists the files in a directory.

List the files in your home directory: ls

Making files

Let's start doing some work. Start by using mkdir to make a new directory:

mkdir {folder_name}

Creates a new folder

Make a new directory: mkdir my_test_dir

Now, let's change into our new directory to do some work, with cd:

cd {some path}

Changes the working directory

Move to your new directory: cd my_test_dir

Make a new empty file with touch:

touch {file_name}

Create a new file

Make a new (empty) file: touch a_test_file

Editing text in the command line

Nano is a barebones text editor available on most Linux computers. While it's not as nice to use as something like VSCode, it's still quite convenient for making quick edits from the command line.

Start Nano like any other commandline tool:

nano filename

You should see something like this. The options along the bottom are keyboard shortcuts for controlling Nano. Here, ^ means ctrl. For example ctrl+x exits Nano, and ctrl+w searches the open file.

The top part of the screen is the editor. You can move your flashing cursor with your arrow keys.

If you make changes and exit, Nano will display the following message, asking if you'd like to save. Press y to save, n to exit without saving, or ctrl+c to cancel and continue editing.

Let's try it out:

Open the file you created in step 6 with nano, and put some text in it:
1. nano a_test_file
2. Type something you learned in this tech session
3. press ctrl+x, then y, then enter to save and exit

Let's use cat to make sure our changes worked:

cat {filename}

Prints the contents of a file (works best with text-based files)

Print the contents: cat a_test_file

Moving files

Let's learn some tools for manipulating existing files.

Let's start by copying our text file, with cp:

cp {source} {destination}

Copies the file at source to destination.

Make a copy of your file, named "another_one": cp a_test_file another_one

Now, let's move that new file, with mv:

mv {source} {destination}

Moves the file or folder at source to destination.

Move the copy to your home directory: mv another_one ~/

Let's check that the file made it, using ls:

Confirm the copy is in your home directory: ls ~/another_one

Finally, let's delete that file with rm (turns out we didn't need it after all)

rm {file}

Remove (delete!) a file

Remove the copy file: rm ~/another_one

Background tasks with screen

In this class, you'll often want to run long-running jobs in the terminal. However, by default, any tasks left running when you log out of ssh will be closed.

We can get around this with a Linux utility called screen. Screen is a "terminal multiplexer". That is, it allows you to keep run multiple terminal sessions, and keep them active even after you've logged off.

Screen allows us to start a process (like a long-running python script), put it in the background, and log off without cancelling the script

Running screen

Log into the course server with ssh
Open a new screen session:

$ screen

You should see a screen with information about screen (licensing, a plea for free beer, etc). Press enter to bypass this. This will open a fresh terminal session, with your terminal history should be cleared out.

Verify that you're in a screen session by listing the open sessions owned by your account:

$ screen -ls
>There is a screen on:
>        18855.pts-44.ip-10-0-1-213      (09/30/20 18:32:05)     (Attached)
>1 Socket in /run/screen/S-adunmore.

One session is listed. It's labeled as (Attached), which means you're logged into it.

Let's give our system some work to do. Run the following command, which will start a useless but friendly infinite loop:

$ while :; do echo "howdy do!"; sleep 1; done

Note that at this point, you could safely log off of ssh. Your loop would still be here when you logged back on.

Now that your screen session is busy, let's go back to our default session to get some work done.

pres ctrl+a, release those keys, and press d.

You should return to your original terminal prompt.

Check that your screen session is still there: run screen -ls to list open sessions again. This time, the single open session should be labeled as (Detached), which means that you're not viewing it.

Note the 5-digit number printed at the beginning of the line referring to your screen session. We'll use that number to log back into that session.

Let's return to our session and kill that loop - we don't need it anymore.

We'll use screen -r. This reattaches the named screen. Use the 5-digit number from step 6 to refer to that session:

screen -r {screen session number}

You should now be back in your old terminal session, where that loop has been "howdy"-ing away.

Press ctrl-c to close that loop.

Now we can close this screen session. Simply type exit in the command line.

This should kill our session and return us to the command prompt. If you'd like, confirm that your session is closed with screen -ls.

Some notes:

You can name your session, with the -S flag:

$ screen -S some_name

Once you've assigned a name, you can use it to reattach your screen sessions, which is easier than remembering/looking up a number.

You can use screen (and any of the utilities introduced here) in your VSCode terminal. Just press ctrl+c to exit your python session (if you're in one), and you'll be able to enter these commands just like a regular terminal session.

tmux

screen is the class default and everything above works. tmux is the same idea with a nicer split-pane model. You can choose to use either one.

tmux new -s dengue           # start a named session
# ... start your long job ...
# Ctrl-b then d              # detach; the job keeps running
tmux ls                      # list sessions
tmux attach -t dengue        # come back
tmux kill-session -t dengue  # done

Ctrl-b is the prefix: press and release, then press the next key.

Task	screen	tmux
Start named session	`screen -S name`	`tmux new -s name`
Detach	`Ctrl-a` `d`	`Ctrl-b` `d`
List	`screen -ls`	`tmux ls`
Reattach	`screen -r name`	`tmux attach -t name`
New window	`Ctrl-a` `c`	`Ctrl-b` `c`
Next window	`Ctrl-a` `n`	`Ctrl-b` `n`
Split vertically	`Ctrl-a` `\|`	`Ctrl-b` `%`
Split horizontally	`Ctrl-a` `S`	`Ctrl-b` `"`
Scroll back	`Ctrl-a` `Esc`	`Ctrl-b` `[` (`q` to exit)
End session	`exit`	`exit`

Everyone in the class is on mlpolicylab-94889. Before launching something big, check htop. At the end of the week, run screen -ls (or tmux ls) and kill what you're not using. Close VS Code windows you've finished with — each connection leaves a ~/.vscode-server process running.

Remote development with VSCode

Why VSCode over SSH?

This section will introduce one convenient workflow for developing code on the remote server.

VSCode is an IDE that provides a lot of useful tools for developing Python, including autocomplete, syntax highlighting, support for virtual environments, and shortcuts to run python files.

With the VSCode SSH extension, VSCode can access code and other files stored on a remote computer. Furthermore, it can run any code stored on the remote machine.

This has several advantages:

You don't have to keep any code stored on your local computer - you only need one copy, stored on the course server
You don't have to copy code between your computer and the course server. Instead, VSCode lets you edit files where they're stored on the course server.
VSCode makes it convenient to run code stored on the course server. When you're developing this way, you'll always have access to the database and your group's virtual environment.

Note: This workflow isn't required - it's just one "good enough" approach that we think many of you will find convenient. Please feel free to use other workflows if you're already set up and comfortable with them.

Configuring VSCode SSH

Download and install VSCode
Install the Remote - SSH extension:
1. Press ctrl+shift+x (Linux/Windows) or ⌘+shift+x (MacOS) to open the extensions menu
2. Search for and install Remote - SSH
1. At this time, also search for and install the microsoft Python extension.
Configure our course server as an SSH host:

With the SSH plugin installed, we can tell VSCode how to log into the server. In this step we'll be entering our connection string and saving it in a file, making it easy to connect in the future.
1. Press ctrl+shift+p (Linux/Windows) or ⌘+shift+p (MacOS) to open the command palette, and select Remote-SSH: Connect to Host
1. Select Add New SSH Host...
1. Enter ssh {andrewid}@server.mlpolicylab.dssg.io
1. Select the first option to store your login config:
Connect VSCode to the course server:
1. Connect to the CMU VPN (if you're not on CMU-Secure wifi network)
2. Press ctrl+shift+p (Linux/Windows) or ⌘+shift+p (MacOS) to open the command palette, and select Remote-SSH: Connect to Host
1. Select the ssh config we just created: server.mlpolicylab.dssg.io
1. Enter your private key passcode if VSCode prompts you to (it will open a box at the top of the screen).
2. You should be connected to the course server. This should be indicated in the bottom of your VSCode window:
Install the Python extension on the course server:

VSCode installs extensions separately for each remote host, so the Python extension you installed locally won't be available in your SSH session until you install it there too.
1. Press ctrl+shift+x (Linux/Windows) or ⌘+shift+x (MacOS) to open the extensions menu
2. Search for the microsoft Python extension and click Install in SSH: server.mlpolicylab.dssg.io
3. Check that the Jupyter extension is also installed on the host, and install it the same way if it isn't.
Open a workspace folder:

Now that VSCode is connected via SSH, you can browse all of the files and folders on the course server. In this step, we select a folder containing some code to edit and test.
1. Select the folder menu button
1. Select Open Folder
2. Select a folder to work in
Select your python virtual environment:

VSCode can be configured to automatically run python code in a virtual environment. Here, we'll select and activate our group virtual environments.
1. Press ctrl+shift+p (Linux/Windows) or ⌘+shift+p (MacOS) to open the command pallette, and select Python: Select Interpreter
1. Select Enter interpreter path
1. Select Find...
1. Enter the path to the python executable in your virtual environment: /path/to/your/environment/bin/python (/class/{andrew_id}/.pyenv/versions/mlpolicyclass/bin/python).
1. After a moment, your selected python interpreter should be activated. This should be indicated in the status bar at the bottom of your VSCode window, which will show the name of the virtual environment, mlpolicyclass (note, your version of VSCode may look different, so don't worry if you don't see the environment at the bottom at this point):
Run python!
1. mlpolicyclass is an empty virtual environment, so you'll need to install packages before importing them. Open a terminal in VSCode (Terminal > New Terminal) and install the version of pandas that triage uses, along with ipykernel:
```
pip install pandas==2.0.3 ipykernel
```
ipykernel lets VSCode use mlpolicyclass as the kernel for Jupyter notebooks.
1. Open the folder menu and select a python file (or press ctrl+n (Linux/Windows) or ⌘+n (MacOS) to create a new one)
1. Click the green "play" button at the top of your window. This starts a new terminal session, activates your virtual environment, and runs your python code.

Understanding the class remote workflow

Basic Project Workflow

The architecture above exists to make sure that the confidential project data stays inside the secure network. Every tool choice in this session follows from that.

Your machine is a client

You can think of your machine "client" in our system. This is because it doesn't do much of the computational heavy lifting. Rather, it views data stored on the database, uses utilities running on the server, and edits and runs code in the server's environment.

SSH tunnelling

Since our projects involve sensitive, personal data, we keep the course server and database inside of a secure network hosted by Amazon Web Services. The course database and server are the only computers on the network. They cannot talk to computers outside of the network, with two exceptions:

The course server can access the web (ie to download files from a website or query the census.gov api)
The course server accepts SSH connections through the CMU VPN

External computers cannot connect directly to the course server.

We can use SSH to get inside this network. We use SSH in two main ways:

We use SSH to access the course server terminal. We can use this to access files stored on the server, and run programs like python, psql, nano, etc.
We use SSH to open tunnels through the course server, to the course database. An SSH tunnel allows a client computer (ex: your laptop) to connect securely to any application accessible from a remote server (ex: our course server). For example:
- The course server can connect to the course database. We can use an SSH tunnel to allow local applications like DBeaver to connect to the course server, via the course server.

Interested in a deeper dive? Here's an article on SSH tunneling.

A diagram illustrating the class architecture.

Workflow Cheatsheet

Here's a typical workflow to get you started:

ssh to the server
start a screen session or reconnect to an existing one screen or screen -r
go to your project directory cd /class/groups/{group_name}
go to your own directory inside where you cloned the github repo cd name_of_your_directory
do a git pull to get updates git pull
open VSCode on your laptop and make sure to connect via the ssh connection to the server
write/edit code
go back to the ssh connection you have open in wsl or termianl. run the code with python (make sure you're in a screen session if it's a long run)
if everything looks good, do a git pull (in case anyone has pushed code while you were working), git commit and push