Quantifyinguncertainty.GitHub.io

Table of Contents

1. Introduction
2. Setting up for Amazon Web Services
3. Using Generalized Metropolis-Hastings on AWS
4. Using Generalized Metropolis-Hastings on JuliaBox
5. Using Generalized Metropolis-Hastings locally
6. Preparing an AWS multi-machine cluster
7. Configuring Amazon Machine Images with Generalized Metropolis-Hastings
8. References

1. Introduction

The Quantifying Uncertainty project develops code, examples and documentation for the Generalized Metropolis-Hastings (GMH) algorithm proposed in (Calderhead, 2014). It forms part of the Oxford/UCL/Microsoft Research 2020 Science project. GMH implements a class of parallelized Monte-Carlo Markov Chains methods (MCMC) that can be used to estimate model parameters of complex physical and biological processes. Details of the GMH algorithm can be found in the scientific publications listed in the References.

The GMH MCMC library is written in Julia, a language designed for fast and flexible mathematical and technical computing. It combines aspects of R, MATLAB and Python and can be easily parallelized to take advantage of multi-processor machines and clusters. It has an active developer community making it available on Windows, Linux and Mac, and via cloud computing services such as JuliaBox or Amazon Web Services (AWS).

There are 2 different, potentially overlapping, situations in which to use the GMH library:

1. To replicate MCMC experiments from published scientific papers. In this case, we recommend to run the experiments from a pre-compiled machine image published on AWS. The machine images contain all the required code, libraries and data to run the MCMC experiments for a particular article, which reduces your set-up time. By following the AWS setup steps in Section 2 and 3 below, you will be able to run these experiments via a web browser such as Firefox or Chrome.
2. To use the GMH library to define your own MCMC experiments. In addition to using GMH on AWS, alternative ways are to download the GMH repository from GitHub to your local machine or to work via JuliaBox.

The GMH code and documentation resides on GitHub in GeneralizedMetropolisHastings.jl. Example models, scripts and notebooks to help users develop their own MCMC simulations are included in GMHExamples.jl.

The remainder of this documentation outlines how to setup GMH on different platforms. It explains how to sign up for AWS (Section 2), how to use GMH on AWS (Section 3), on JuliaBox (Section 4) or on your local machine (Section 5), how to prepare AWS to run GMH on a cluster of machines (Section 6) and how to publish your own AMIs to accompany the publication of scientific articles that use the GMH repository (Section 7).

Table 1. Acronyms

2. Configuring AWS

Step 2.1: Sign up for an AWS account

This step requires a credit card to set up an account and a phone number to receive a security callback. By using Amazon's "Free Tier", it is possible to avoid being billed, but a credit card is still required. Follow the instructions in the AWS documentation:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html

You should complete the following steps:

1. Create an AWS account
2. Create an IAM user
3. Create a Key Pair
4. Create a Security Group

Table 2. Required Security Group Settings

When setting up a Security Group, specify the following inbound rules:

The “0.0.0.0/0” in the “Source” field means that incoming connections from any IP address are allowed. You can make the security group more secure by only allowing incoming connections from known IP addresses (e.g., "My IP" in the EC2 security group control panel). This means, however, that every time you change your local IP address you will have to update the Security Group. As long as you are careful in implementing the security steps outlined in Section 3, the above settings should be fine.

The entire process of setting up an AWS account is likely to take between 1 and 2 hours, depending on your familiarity with AWS. These steps need to be completed only once.

Step 2.2: Check the AWS region

This step typically needs to be performed only once - especially upon first signing up for AWS or if you cannot find the published AMIs listed in Section 3. Amazon regions (data centers) are independent of each other. In order to find the GMH AMIs, you need to be logged into the correct region. At this moment, the only region in which the GMH AMIs are published is Ireland. You can check and change the region in the top right corner of the AWS and EC2 dashboards, between your IAM username and “Support”.

3. Using GMH on AWS

Table 3: Published AMIs

Step 3.1: Launch a published AMI

Step 3.1.1: Choose an AMI

Once logged into the AWS dashboard, go to the EC2 console. Select Instances in the left-hand menu, and click the Launch button at the top of the panel. Select Community AMIs from the left-hand menu and in the search box that appears, copy-paste one of the published AMI names from Table 3. Click the Select button to launch the AMI.

Troubleshooting: if the AMI does not appear in the list, then check the AWS region.

Step 3.1.2: Choose an Instance Type

Here you select the physical hardware on which to run. Whereas the t2.micro is free to use under Amazon's free tier scheme, it is recommended that you select one of the larger machines (e.g., the t2.medium or higher) as the system needs sufficient memory to operate (>1GB).

When selected, click Next: Configure Instance Details.

Step 3.1.3: Configure Instance Details

From the "Subnet" drop-down list, select a subnet that corresponds to a particular availability zone, which is important if you will also attach an existing EBS-backed volume to contain your data (this is optional - see Step 3.3 below). Make a note of which availability zone you select (currently eu-west-1a, 1b or 1c). Choose Next: Add Storage at the bottom of the page.

Step 3.1.4: Add Storage

Optional: you are welcome - if required - to add an additional drive to contain your private data and code. This can either be done on this page and/or at a later stage, in which case see Add an EBS volume.) Click Next: Tag Instance.

Step 3.1.5: Tag Instance

Optional: add descriptive tags for your instance. Then click Next: Configure Security Group.

Step 3.1.6: Configure Security Group

Select the radio button Select an existing security group and select both the default VPC security group and the security group for Jupyter/IJulia access that you made in Step 2.1. Click Next: Review and Launch.

Step 3.1.7: Review and Launch

It is safe to ignore the warning that the Security Group is open to the world (if the security steps outlined in Step 3.4 are adhered to). Click Launch at the bottom of the page. A pop-up window comes up. If you previously made a Key Pair, select Choose an existing key pair. Otherwise select Create a new key pair and make sure to download the private SSH key to a safe location on your local machine.

Click Launch Instances. On the next window, Launch Status, click View Instances, which takes you to the Instances page. Once the Instance State becomes green and says running, you can connect via SSH.

Step 3.2: Connect to a running instance

The username for this machine is ubuntu. There is no password, but verification occurs with the SSH Key Pair you created previously.

Instructions on how to connect to the instance via SSH can be found here:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-connect-to-instance-linux.html

If you are using Windows and PuTTy, follow these instructions to transform your local key: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html#create-a-key-pair

Troubleshooting: if you cannot connect, then it may be because of the Security Group settings you launched the machine with. Check that they correspond to the ones in Table 2. If you opted to not allow access from all IP addresses (i.e., did not select 0.0.0.0/0 in Table 2), then make sure that the IP address in the Security Group is the same as the IP address of your local machine (visit http://myipaddress.com to find out).

Step 3.3: Add an EBS Volume

Optional: you can add additional storage to hold your data and experiments by following these instructions:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-add-volume-to-instance.html

When connected to the instance via SSH, mount the new volume in the home folder rather than in the location suggested in the AWS user guide, for example:

sudo mount /dev/xvdf ~/data

If you create an EBS volume, limit the size to a few GB as the Free Tier only comes with 30GB of free storage for all volumes combined.

Step 3.4: Configure the Notebook server

This step configures the Jupyter Notebook Server for access via a browser such as Chrome or Firefox.

Important: During the running of the configuration script, you will be asked for a password to login via a browser into the Jupyter Notebook Server. Ensure to select a secure password, consisting preferably of 3 unconnected words, e.g., "apple-spring-saxophone". If the password you select is not secure, then your AWS server is vulnerable to attack as the Notebook server allows any system command to be executed via a web browser.

Execute the configuration script:

GMHConfigure/gmh-server/config.sh

As part of the configuration, the script will ask you to enter a password, and create a self-signed SSL certificate to encrypt the communication to the server. It will ask for some information. You may press enter to accept all default values.

Step 3.5: Launch a Notebook server

On the SSH command line, you can start the Notebook server:

jupyter notebook &

The ampersand means that the notebook server is placed in the background. Once you see that the server is running on Port 8998, you can logout of the SSH session, and the server will continue to run in the background until the instance is stopped.

The Jupyter server prints status messages to the console. You can redirect this output to a log file:

jupyter notebook > ~/.jupyter/jupyter_output.log 2>&1 &

Step 3.6 Login to the Notebook server

To connect to the Notebook server via your browser, get the public IP address of your running instance from the EC2 console (the “Instances” page) and type it in the address bar of your web browser. Make sure to use https to connect (rather than plain http).

https://xxx.xxx.xxx.xxx:8998

The browser will complain that the connection is not private because the identity of the self-signed SSL certificate could not be checked. You can safely ignore these warnings.

You are now at the notebook server login page. Type the password you created above.

Step 3.7: Using the Notebook server

Under the Files tab in the notebook server, you will see repositories with example experiments in IJUlia Notebooks. Notebooks contain both code and documentation, so please refer to these notebooks for further information.

You can start individual notebooks by navigating to one of the folders containing a notebook, and double-clicking it. You can also start a command-line terminal by selecting “New/Terminal” from the right-hand side of the screen. For further information on the notebook server interface, see the Jupyter documentation.

Step 3.8: Stopping/Terminating the Notebook server and AWS machine

When finished, there are multiple levels of halting/terminating the Notebook server and AWS machine:

1. Logout of the notebook server by clicking the “Logout” button at the top right of the server screen. The server will continue to be accessible on Port 8998 for future login.
2. The Notebook server can be halted via the SSH terminal (e.g., using the Unix kill command).
3. In the AWS/EC2 dashboard, under the Instances tab, you can temporarily stop the running EC2 instance, by selecting the instance and clicking Actions/Instance State/Stop at the top of the screen. This will power down the computer, but it will remain available to restart later. Note that, unless you Stop an EC2 instance, the running time is chargeable, even if your computer is idle (unless you operate within Amazon’s “Free Tier”).
4. You can also Terminate an instance. This means that the resources associated with the instance will be deleted. In other words, an instance that has been terminated cannot be restarted, and you would need to setup the machine anew.

4. Using GMH on JuliaBox

Step 4.1: Sign in to JuliaBox

Go to https://juliabox.org and sign in with your Google credentials.

Step 4.2: Start a Julia session in the Terminal

At the top of the JuliaBox window, select Terminal. The currently preferred version of Julia to run the GMH module is v0.4.

Start a Julia session in the terminal window by executing:

juser@juliabox:~$ julia

Step 4.3: Install, update and test GeneralizedMetropolisHastings.jl

If this is the first time you are using the GMH module in your JuliaBox, you will need to install the required packages by executing at the Julia command prompt:

julia> Pkg.add("GeneralizedMetropolisHastings")

Run a package update to ensure the latest versions are installed:

julia> Pkg.update()

You can now test if GMH installed correctly and is compatible with this particular JuliaBox installation by running the GMH tests:

julia> Pkg.test("GeneralizedMetropolisHastings")

Step 4.4 Sync example repositories to JuliaBox

Sync repositories with examples from GitHub to your JuliaBox home folder. At the top of the JuliaBox window, select Sync. Add the following to the “Git Repositories”.

• https://github.com/QuantifyingUncertainty/GMHExamples.jl.git
• https://github.com/QuantifyingUncertainty/GMHPhotoReceptor.jl.git

Paste the above https URLs into the “Git Clone URL”. Accept “master” as branch, "GMHExamples.jl" (or "GMHPhotoReceptor.jl") as location, and press "+" to add the repository. This will now download the package to the specified location.

Please refer to the documentation of the GMHExamples.jl and GMHPhotoReceptor.jl repositories for details on how to use them.

Step 4.5 Set the Julia search path

Add the path to the GMHPhotoReceptor.jl module file to ~/.juliarc.jl by executing the following command in the terminal window.

juser@juliabox:~$ echo 'push!(LOAD_PATH,"/home/juser/GMHExamples.jl")' > .juliarc.jl
juser@juliabox:~$ echo 'push!(LOAD_PATH,"/home/juser/GMHPhotoReceptor.jl")' > .juliarc.jl

The command creates the file if it does not yet exist. You can check that it was correctly added by executing cat ~/.juliarc.jl in the terminal window.

5. Using GMH locally

Step 5.1: Install Julia + IDE

Install Julia from http://julialang.org/downloads/. You can also install a development environment. On Windows, Juno is recommended.

Step 5.2: Install the GMH package

Start Julia and follow the instructions from Step 4.3 to install and test the Generalized Metropolis-Hastings packages.

Step 5.3 Sync example repositories

Install the GMHExamples.jl and GMHPhotoReceptor.jl repositories from GitHub in a local folder. Instructions on how to install from GitHub can be found on GitHub.

Please refer to the documentation of the GMHExamples.jl and GMHPhotoReceptor.jl repositories on how to use them.

Step 5.4 Set Julia search path

Add the following line to the ~/.juliarc.jl file, inserting the correct path to the GMHPhotoReceptor.jl folder.

push!(LOAD_PATH,"/path/to/GMHExamples.jl")
push!(LOAD_PATH,"/path/to/GMHPhotoReceptor.jl")

.juliarc.jl should reside in your home folder, which is system dependent. On Windows, this is usually C:\users\<your name>.

6. Preparing an AWS cluster

6.1 Get AWS Access Keys

The master server in the cluster needs to be configured with AWS Access Keys. Please follow the steps outlined in "To create, modify, or delete a user's access keys" in:

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey

Make sure to download the AWS Access Key ID and Secret Access Key to your local machine and store them in a secure location. Typically, this step needs to be executed only once.

Note: if you see in your security credentials that you have an AWS Access Key ID, but did not create this yourself, did not download the Secret Access Key, or lost the Secret Access Key, you will have to create a new one as outlined in the documentation above. In that case, you can also delete the existing AWS Access Key as you will not be able to use it.

6.2 Start the Master server

Follow the steps in Section 3 to set up your own Jupyter/IJulia Notebook server. Connect to this machine using SSH, as explained in Step 3.2.

This machine will be the only one in your cluster which can be accessed from outside AWS, which is determined by the security group specified in Step 2.1.

6.3 Run the cluster configuration script

To configure SSH access to the additional worker machines in the cluster, run the following script:

GMHConfigure/gmh-cluster/config.sh

The script asks for your AWS Access Keys and region. The default region is currently eu-west-1.

The script creates a Key Pair for password-less login over SSH, saves the private key to a secure location on the master server, and adds the public key to your AWS console. The name of the key is julia-cluster-access-xxx.xxx.xxx.xxx with xxx standing for the private IP address of the cluster.

6.4 Start additional Worker machines

Return to the Instances tab in the AWS console, click Launch Instance and launch 1 or more Worker instances in a similar manner to the Launch Instance procedure outlined in Step 3.1. Launch the same AMI as your master server using the My AMIs section, specifying the desired number of instances in option 3. Configure Instance, and ensure to launch it into the same subnet (Step 3.1.3). For the security group (Setp 3.1.6), this time only select the default VPC security group.

When selecting the key pair (Step 3.1.7), select the julia-cluster-access-xxx.xxx.xxx.xxx Key Pair you created in the previous step.

Go to the instances page, select the launching instance, and in the Description tab in the bottom half of the page, note down the Private IPs.

You should test that passwordless SSH access to this worker is set up correctly by executing in your existing SSH terminal session.

ssh ubuntu@xxx.xxx.xxx.xxx

where xxx.xxx.xxx.xxx is the private IP address you obtained from the AWS console.

If you have followed all steps, this command should log into the slave machine. This step also results in the SSH key being added to the list of known hosts and is required before automatic SSH login can work inside Julia.

Type exit to log out from your internal SSH session and return to the original SSH session.

Step 6.5 Check addprocs() in Julia

To test that you can now launch Julia worker processes on the additional worker machines in the cluster, open a Julia REPL (by typing julia) and execute the following:

julia> machvec = [("ubuntu@xxx.xxx.xxx.xxx",:auto),("ubuntu@xxx.xxx.xxx.xxx",:auto)]

Where xxx.xxx.xxx.xxx are the private IP addresses you obtained in Step 6.4.

As shown above, you can add multiple (IP address,:auto) pairs to the vector, separated by commas. The :auto symbol ensures that each worker machine launches as many processes as it has cores.

Launch additional workers on the remote machine by executing the following command:

julia> addprocs(machvec,topology=:master_slave)

The function call will return a list of the new process IDs.

You can run the tests of the GeneralizedMetropolisHastings package in such a way that it will utilize these additional workers:

julia> cd(Pkg.dir("GeneralizedMetropolisHastings"))
julia> include("test/runtests.jl")

If you have more than 1 worker running, in the last printout of the tests you will notice a line that says that the test ran with more than 1 remote segment:

RemoteSegments with 4 segments and 13 proposals per segment.

For a cluster with 1 master process and 4 worker processes.

Close all remote workers by executing:

julia> rmprocs(workers())

Close the julia session by typing exit().

You can now launch the Jupyter/IJulia server as explained in Step 3.4 and Step 3.5. You are now ready to run Julia scripts and notebooks on the cluster.

7. Configuring GMH AMIs

7.1 Configure an AMI with Julia and GMH

Step 7.1.1 Launch and connect to a fresh EC2 instance

Starting from a fresh Ubuntu Server 14.04LTS or similar, follow the instructions to launch and connect to an AMI over SSH.

Step 7.1.2 Install Git

In the SSH terminal, execute the following commands to install the Git repository software:

sudo apt-get update 
sudo apt-get -y install git

Step 7.1.3 Checkout the configuration scripts

Install the configuration scripts from the GMHConfigure repository:

git clone https://github.com/QuantifyingUncertainty/GMHConfigure

Step 7.1.4 Run the AMI configuration script

GMHConfigure/gmh-ami/config.sh

The script will install Julia, the GeneralizedMetropolisHastings code, the example repositories and all required packages.

It will also run the tests of the GeneralizedMetropolisHastings package in Julia. These tests should normally pass, but the installation is allowed to continue in case they fail.

The last line printed by the script will indicate whether the installation process was successful.

7.2 Prepare an AMI for publishing

To remove all private keys, command-line history and other settings from the AMI prepared using the steps from Section 7.1, run the script:

GMHConfigure/gmh-ami/prepare-for-publication.sh

After this script has been executed, you will not be able to log in again to the same instance. Logout from the instance, stop the instance from the EC2 console, and follow these steps to create the AMI:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html#how-to-create-ebs-ami

Further security guidelines can be found in: https://aws.amazon.com/articles/0155828273219400

Here are general guidelines to share a public AMI:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sharing-amis.html

7.3 Develop GMHConfigure

If you have made changes to the GMHConfigure repository, you can submit them to the GMHConfigure repository. If you do not have write permissions to the repository, fork, submit and submit a pull request, as explained in the GitHub documentation. If you do have write permissions, you can submit your changes via the following commands:

git commit -a

Check if the remote branch is set correctly by executing git remote -v. If not, add it with the following command:

git remote add origin https://github.com/QuantifyingUncertainty/GMHConfigure

Push the changes to the remote repository:

git push

You will be asked for your GitHub username and password.

8. References

Calderhead B. (2014), A general construction for parallelizing Metropolis-Hastings algorithms, PNAS, Vol: 111, Pages: 17408-17413