Technical

17 minute read

Note that this writeup is designed for CS411 Project.

Introduction

This tutorial aims to provide a hands-on tutorial for students with little prior web development experience. In this example, we chose back-end rendering for its simplicity. Many frameworks support alternative (and fancier) web development methods. We do not cover these here, but we hope that the following workshop allows you to explore these options.

The goal of this writeup hopes to deliver:

  1. How to setup Flask
  2. How to do back-end rendering and serve content to a client
  3. How to setup Ajax for the front-end to interact with the back-end
  4. How to connect MySQL GCP to the local flask server
  5. How to deploy the flask application to the GCP app engine.

This tutorial is developed on WSL 2 (Windows Linux subsystem, aka bash on windows). This means that it should be compatible with Mac and most Linux systems. There could be a slight difference in the output on the terminal but nothing else.

Objective:

We will complete a Todo demo App in this tutorial. The todo demo app can Create, Read, Update, Delete, tasks on this to-do list app. There is no under feature. Everyone on the same webpage shares the same view. In other words, it's a pretty useless app at this point. But it should not be hard to extend this app.

Video

Yes, I recorded a video. Note: this write-up is more of a complement to the video.

Part 1: Setting up Flask

Setting up environment and flask

The first thing is to set up a server locally. A server is a service that listens for requests from clients and returns a response. First, we want to ensure we are running in a python virtual environment. This is best practice and also helps us export our packages easily as the app moves forward.

mkdir demo # creating the demo folder
cd demo # moving into the folder
python -m venv .venv # creating the virtual env placed in the .venv folder
source .venv/bin/activate # activate the virtual env
# you can exit a virtual env by typing "deactivate"

Once this is done, we can install the packages and libraries needed for this application. Feel free to google the first two packages. They have comprehensive documentation for the development process.

pip install flask flask_sqlalchemy pymysql pyyaml

Running a barebone Flask application

With the environment completed, we set up the following file structure:

demo
├── app/
│   └── __init__.py
└── main.py

main.py: This is the app entry point. This means that the application can be invoked by python main.py.

from app import app

if __name__ == '__main__':
	app.run(debug = True)

__init__.py: A init file is a special python file. More details here. We want to encapsulate the app inside this module. Therefore, we define the following:

from flask import Flask

app = Flask(__name__)

This “app” is the one that we imported into main.py. Since this is an independent module, we can make use of flask to execute this module. In the terminal, we define:

export FLASK_APP=app # this tells the terminal that our flask app module is in the app folder
export FLASK_DEBUG=1 # this tells flask to reload server on change
flask run

Since flask now knows which module to run, it will spin up the server. You should now see the server running and nothing else. This is because the server does not know what to do. Let us add the following code to the __init__.py file:

# Add jsonify to import. It is part of the flask package

@app.route("/")
def homepage():
	return jsonify({"status": "OK"})

This is a very short example of how to write a route. You can define very complex routing in Flask. Examples can be found in the documentation here. We will not go into details here. At a high level, it tells the server to listen to requests coming from this route and complete the corresponding information. We bind the response data into a JSON file. Modern browsers are very smart in the sense that even though we did not return a “perfect” formatted file (i.e., HTML), it will render the JSON response as HTML for you. This implementation is more for constructing an API for another service to consume. By saving the file, your server should have restarted. Visiting the URL will show you exactly the JSON you returned. Feel free to play around with this code. It improves your understanding of how this all worked if you are new to web development.

Serving HTML files with Flask

Moving to the next level, we now want to return an HTML file.

First, we need to update the file structure:

demo
├── app/
│   ├── templates/
│   │   └── index.html
│   └── __init__.py
└── main.py

index.html

<h1>Hello Bob!</h1>

Let us also change the code in the homepage() function into the following:

# Let's change jsonify to render_template

@app.route("/")
def homepage():
	return render_template("index.html")

The render_template function helps render a template with variables and return the final HTML file to the client. We will see an example in the next step. These templates, by default, live in the templates folder. You can see this in the documentation. One of the variables under Flask defines template_folder = 'templates'. After the change, we can now refresh the change, and the view should now display the HTML file.

Serving Dynamic HTML file with Flask

The final stage is to serve HTML that supports dynamic information. We are going to update the index.html and __init__.py as the following:

index.html

<h1>Hello {{name}}!</h1>

Let us also change the code in the homepage() function into the following:

# Let's change jsonify to render_template

@app.route("/")
def homepage():
	return render_template("index.html", name="Alice")

We now change index.html into a jinja template format. Reference the documentation here. For example, double brackets mean a variable that the rendering engine would replace. Try refreshing the name, and the text should now show “Hello Alice.”

Mimicking a “database” (not in the video)

Database serves data. We have a server that can serve dynamic content. So we want to achieve the following logic:

Client sends a request to server → Server sends a query to database → database returns the data → server renders the html with the data → serves the final html to the front-end to the client

Now we can setup a function that mimics the “database returns the data part.”

Let us update the file structure once more:

demo
├── app/
│   ├── templates/
│   │   └── index.html
│   ├── routes.py
│   ├── database.py
│   └── __init__.py
└── main.py

database.py:

import random

def query_name():
	return random.choice(["Alice", "Bob", "Chris", "Dolly"])

We then move the routing section to the routes.py file:

from flask import render_template
from app import app
from app import database as db_helper

@app.route("/")
def homepage():
	return render_template("index.html", name=db_helper())

The entire __init__.py file now becomes:

from flask import Flask

app = Flask(__name__)

from app import routes

The last line needs to be there because we need to tell flask on initialization that the routes are stored in a separate file. This is a circular import since we imported app into the routes.py. Though the better implementation is to make use of a blueprint described in the flask documentation, we do this for simplicity and understandability at this level. Notice how the “database” returns data intended for template rendering. Try and refresh the webpage several times to observe the changes.

Now that we have everything set up, how do we transform this into a todo list app?

  1. We want to update the index.html file into a todo list html.
  2. We want routes.py to bridge JavaScript in the frontend to the backend URLs.
  3. We want to connect to the database so that the database.py file returns data for actual queries

Part 2: Establishing the front-end code

Serving a dummy todo list html

First, let us change the code in routes.py back to returning only the html. So we can see the changes to the html.

@app.route("/")
def homepage():
	return render_template("index.html")

We also need to change the file structure to add some supporting files

demo
├── app/
│   ├── static/
│   │   ├── script/
│   │   │   └── model.js
│   │   ├── styles/
│   │   │   └── custom.css
│   │   └── index.html
│   ├── templates/
│   │   └── index.html
│   ├── routes.py
│   ├── database.py
│   └── __init__.py
└── main.py

index.html

<!DOCTYPE html>

<html>
    <head>
        <title>Demo: TODO</title>

        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">

        <link rel="icon" href="{{ url_for('static', filename='img/cs.ico') }}">
        <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta2/dist/css/bootstrap.min.css" rel="stylesheet" 
        integrity="sha384-BmbxuPwQa2lc/FVzBcNJ7UAyJxM6wuqIj61tLrc4wSX0szH/Ev+nYRRuWlolflfl" crossorigin="anonymous">

        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.css">
        <link rel="stylesheet" href="{{ url_for('static', filename='styles/custom.css') }}">
    </head>

    <body>

    <main role="main" class="container extra-bottom">
      <h1 class="mt-5">TODO List Demo App</h1>
      <p><mark>Do it now.</mark></p>
    </main>
    

    <div class="container">    
        
        <!-- Button trigger modal -->
        <div style="text-align: right;">
        <button type="button" class="btn btn-outline-info btn-sm" data-bs-toggle="modal"
														data-bs-target="#task-modal" data-source="New Task">Add Task</button>
        </div>

        
        <div class="modal fade" id="task-modal" tabindex="-1" aria-labelledby="Label" aria-hidden="true">
        <div class="modal-dialog">
            <div class="modal-content">
            <div class="modal-header">
                <h5 class="modal-title" id="Label">Add a task</h5>
                <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="Close"></button>
            </div>

            <div class="modal-body">
                <div class="input-group mb-3">
                    <span class="input-group-text" id="task-form-display">Task</span>
                    <input type="text" class="form-control" placeholder="Description of task"
													aria-label="task-name" aria-describedby="basic-addon1">
                </div>
            </div>
            <div class="modal-footer">
                <button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
                <button id="submit-task" type="button" class="btn btn-primary">Save changes</button>
            </div>
            </div>
        </div>
        </div>

    </div>

    <!--Todo Table-->
    <div class="container table-responsive">          
    <table class="table">
        <thead>
        <tr>
            <th class="task-id">#</th>
            <th class="task">Task Name</th>
            <th class="status">Status</th>
            <th class="update">Edit</th>
            <th class="update">Remove</th>
        </tr>
        </thead>

        <tbody>
            <tr>
                <td>1</td>
                <td>task1</td>
								<td><button type="button" 
														class="btn btn-outline-warning btn-sm state" 
														data-source="1">Todo</button></td>                
                <td>
									<button type="button" class="btn btn-outline-info btn-sm"
													data-bs-toggle="modal" data-bs-target="#task-modal"
													data-source="1" data-content="task1">
										<i class="fa fa-pen fa-1" aria-hidden="true"></i>
									</button>
								</td>
                <td><button class="btn btn-outline-secondary btn-sm remove"
														data-source="1" type="button">
									<i class="fa fa-trash fa-1" aria-hidden="true"></i>
										</button>
								</td>
            </tr>
						<tr>
                <td>2</td>
                <td>task2</td>
								<td><button type="button" class="btn btn-outline-warning btn-sm state"
														data-source="2">Todo</button></td>                
                <td>
									<button type="button" class="btn btn-outline-info btn-sm"
												  data-bs-toggle="modal" data-bs-target="#task-modal"
													data-source="2" data-content="task2">
										<i class="fa fa-pen fa-1" aria-hidden="true"></i>
									</button>
								</td>
                <td><button class="btn btn-outline-secondary btn-sm remove" data-source="2" type="button">
									<i class="fa fa-trash fa-1" aria-hidden="true"></i>
										</button>
								</td>
            </tr>
        
        </tbody>
    </table>
    </div>

    <footer class="footer">
      <div class="container">
            <p class="text-muted"><small>App developed for CS411 UIUC by @tcheng10</small></p>
      </div>
    </footer>
  </body>

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta2/dist/js/bootstrap.bundle.min.js" 
        integrity="sha384-b5kHyXgcpbZJO/tY9Ul7kGkf1S0CWuKcCD38l8YkeH8z8QjE0GmW1gYU5S9FOnJ0" crossorigin="anonymous"></script>
    <script src="{{ url_for('static', filename='script/modal.js') }}"></script>

</html>

custom.css

html {
    position: relative;
    min-height: 100%;
}

body {
    /* Margin bottom by footer height */
    margin-bottom: 2em;
}

.footer {
    position: absolute;
    bottom: 0;
    width: 100%;
    height: 2em;
    /* Set the fixed height of the footer here */
    line-height: 1.5em;
    /* Vertically center the text there */
    background-color: #f5f5f5;
}

hr {
    width: 50%;
    margin: auto;
}

.table td {
    text-align: center;
    vertical-align: middle;
}

.table tr {
    text-align: center;
    vertical-align: middle;
}

/* bootstrap modifications */
.container {
    width: auto;
    max-width: 680px;
    padding: 0 15px;
}

.extra-bottom {
    padding-bottom: 2em;
}

.task {
    width: 50%;
}

.task-id {
    width: 10%;
}

.status {
    width: 20%;
}

.update {
    width: 10%;
}

The html and css files should be pretty straightforward. Notice how the entire file itself is actually static, as we did in the name example. Always start by building how you want the final product want to be like. Now we only need to fix two things:

  1. We want the todo list table to render dynamic tasks from the server.
  2. We want the buttons to work.

This is what we will do next.

Dynamic Todo HTML

To allow dynamic Todo list, we need to replace the table body with jinja stuffs:

<tbody>
        {% for item in items %}
            <tr>
                <td>{{item.id}}</td>
                <td>{{item.task}}</td>

                {% if item.status == "In Progress" %}
                    <td><button type="button" class="btn btn-outline-warning btn-sm state"
														data-source="{{item.id}}">{{item.status}}</button></td>
                {%endif%}
                {% if item.status == "Todo" %}
                    <td><button type="button" class="btn btn-outline-secondary btn-sm state"
														data-source="{{item.id}}">{{item.status}}</button></td>
                {%endif%}
                {% if item.status == "Complete" %}
                    <td><button type="button" class="btn btn-outline-success btn-sm state"
														data-source="{{item.id}}">{{item.status}}</button></td>
                {%endif%}
                
                <td><button type="button" class="btn btn-outline-info btn-sm" data-bs-toggle="modal"
														data-bs-target="#task-modal" data-source="{{item.id}}"
														data-content="{{item.task}}"><i class="fa fa-pen fa-1" aria-hidden="true"></i>
										</button></td>

                <td><button class="btn btn-outline-secondary btn-sm remove" data-source="{{item.id}}"
														type="button"><i class="fa fa-trash fa-1" aria-hidden="true"></i>
										</button></td>
            </tr>
        {% endfor %}  
</tbody>

We imagine passing a list of dictionaries called items and each of the dictionary would contain an id, task description, a status that is either “Todo”, “Complete”, or “In Progress.”

Thus, we update our routes.py as:

@app.route("/")
def homepage():
    items = db_helper.fetch_todo()
    return render_template("index.html", items=items)

And subsequently our database.py

def fetch_todo():
    todo_list = [
			{"id": 1, "task": "Task 1" , "status": "In Progress"},
			{"id": 2, "task": "Task 2", "status": "Todo"},\
		]
    return todo_list

Subsequent JavaScript code

Right now we only have one single route. We also want routes that support the Create, Update and Delete operations. Now we expand the routes.py file so that it becomes:

from flask import render_template, request, jsonify
from app import app
from app import database as db_helper

@app.route("/delete/<int:task_id>", methods=['POST'])
def delete(task_id):
     try:
        # db_helper.remove_task_by_id(task_id)
        result = {'success': True, 'response': 'Removed task'}
    except:
        result = {'success': False, 'response': 'Something went wrong'}

    return jsonify(result)

@app.route("/edit/<int:task_id>", methods=['POST'])
def update(task_id):
    data = request.get_json()
		print(data)
    try:
        if "status" in data:
            # db_helper.update_status_entry(task_id, data["status"])
            result = {'success': True, 'response': 'Status Updated'}
        elif "description" in data:
            # db_helper.update_task_entry(task_id, data["description"])
            result = {'success': True, 'response': 'Task Updated'}
        else:
            result = {'success': True, 'response': 'Nothing Updated'}
    except:
        result = {'success': False, 'response': 'Something went wrong'}

    return jsonify(result)

@app.route("/create", methods=['POST'])
def create():
    data = request.get_json()
    # db_helper.insert_new_task(data['description'])
    result = {'success': True, 'response': 'Done'}
    return jsonify(result)

@app.route("/")
def homepage():
    """ returns rendered homepage """
    items = db_helper.fetch_todo()
    return render_template("index.html", items=items)

Notice here I've already placed some comments on what we want to update for the database operations. Right now we focus on the other three operations. It should be rather straightforward. To be much more clear, I summarize the routes below:

[POST] /create : Payload contains description of the task
[POST] /delete/<int:task_id>
[POST] /edit/<int:task_id> : Payload contains either new task description or status
[GET] /: Returns homepage

Next, we can define the JavaScript such that it calls these URLs:

modal.js

$(document).ready(function () {
    $('#task-modal').on('show.bs.modal', function (event) {
        const button = $(event.relatedTarget); // Button that triggered the modal
        const taskID = button.data('source'); // Extract info from data-* attributes
        const content = button.data('content'); // Extract info from data-* attributes

        const modal = $(this);
        if (taskID === 'New Task') {
            modal.find('.modal-title').text(taskID);
            $('#task-form-display').removeAttr('taskID');
        } else {
            modal.find('.modal-title').text('Edit Task ' + taskID);
            $('#task-form-display').attr('taskID', taskID);
        }

        if (content) {
            modal.find('.form-control').val(content);
        } else {
            modal.find('.form-control').val('');
        }
    })

    $('#submit-task').click(function () {
        const tID = $('#task-form-display').attr('taskID');
        console.log($('#task-modal').find('.form-control').val());
        $.ajax({
            type: 'POST',
            url: tID ? '/edit/' + tID : '/create',
            contentType: 'application/json;charset=UTF-8',
            data: JSON.stringify({
                'description': $('#task-modal').find('.form-control').val()
            }),
            success: function (res) {
                console.log(res.response)
                location.reload();
            },
            error: function () {
                console.log('Error');
            }
        });
    });

    $('.remove').click(function () {
        const remove = $(this);
        $.ajax({
            type: 'POST',
            url: '/delete/' + remove.data('source'),
            success: function (res) {
                console.log(res.response)
                location.reload();
            },
            error: function () {
                console.log('Error');
            }
        });
    });

    $('.state').click(function () {
        const state = $(this);
        const tID = state.data('source');
        const new_state;
        if (state.text() === "In Progress") {
            new_state = "Complete";
        } else if (state.text() === "Complete") {
            new_state = "Todo";
        } else if (state.text() === "Todo") {
            new_state = "In Progress";
        }

        $.ajax({
            type: 'POST',
            url: '/edit/' + tID,
            contentType: 'application/json;charset=UTF-8',
            data: JSON.stringify({
                'status': new_state
            }),
            success: function (res) {
                console.log(res)
                location.reload();
            },
            error: function () {
                console.log('Error');
            }
        });
    });

});

I'm going to use the delete task snippet as an example:

    $('.remove').click(function () {
        const remove = $(this);
        $.ajax({
            type: 'POST',
            url: '/delete/' + remove.data('source'),
            success: function (res) {
                console.log(res.response);
                location.reload();
            },
            error: function () {console.log('Error');}
        });
    });

Here it used Ajax to interact with the backend server. This is somewhat old school and inefficient, but it gets the job done. In this example, if the button with ID submit-task is clicked, it collects the id (data-source in the html) of that entry. It fires a Post request to /delete/id where the id is that task id. If the operation is successful (which the server returns a 200, it logs the server's responses to the client console and then refreshes the page. This refresh triggers the webpage to pull from the database again (recall the homepage function). The rest of the JavaScript follows this concept (besides the first one, which does some auto-filling.)

These updates should allow you to interact with the website without error besides not having data changing (since we are not modifying the data.)

Part 3 Connecting to GCP MySQL

First, we need to set up the table in GCP. Assuming you already have your MySQL setup on GCP, we can use the shell in the console to login into the database. We can then create the database and define the tables:

CREATE DATABASE todo;
CREATE TABLE tasks (
    id int NOT NULL AUTO_INCREMENT,
    task varchar(255) NOT NULL,
    status char(30)
    PRIMARY KEY (id)
);
INSERT INTO tasks (task, status) VALUES ("task no.1" , "Todo");
INSERT INTO tasks (task, status) VALUES ("task no.2" , "Todo");
INSERT INTO tasks (task, status) VALUES ("task no.3" , "Todo");

Now we can set up GCP MySQL connection with the flask app.

First, we need to create an app.yaml file to store the database's credentials. This file should be a sibling of main.py. This file is the standard file that GCP supports for python applications. We also don't want to store these credentials onto GitHub and other public places.

runtime: python38 # or another supported version

instance_class: F1 # this is the cheapest option

env_variables:
  MYSQL_USER: <fill username>
  MYSQL_PASSWORD: <fill password>
  MYSQL_DB: 'todo'
  MYSQL_HOST: <fill mysql gcp ip>

handlers:

- url: /script
  static_dir: static/script

- url: /styles
  static_dir: static/styles

now, we can head back to out __init__.py to add the connection:

import os
import sqlalchemy
from yaml import load, Loader
from flask import Flask

def init_connect_engine():
    if os.environ.get('GAE_ENV') != 'standard':
        variables = load(open("app.yaml"), Loader=Loader)
        env_variables = variables['env_variables']
        for var in env_variables:
            os.environ[var] = env_variables[var]

    pool = sqlalchemy.create_engine(
            sqlalchemy.engine.url.URL(
                drivername="mysql+pymysql",
                username=os.environ.get('MYSQL_USER'), #username
                password=os.environ.get('MYSQL_PASSWORD'), #user password
                database=os.environ.get('MYSQL_DB'), #database name
                host=os.environ.get('MYSQL_HOST') #ip
            )
        )
    return pool

app = Flask(__name__)
db = init_connect_engine()

from app import routes

We declared a function called init_connect_engine where it starts a connection to the database. Notice we had a statement: if os.environ.get('GAE_ENV') != 'standard'. This is used to detect if the server is running on a GCP instance or on a local computer. This is defined here.

The nice thing is that for a GCP instance, it reads in the yaml file and sets the defined string right into its os environment. But we do not have that convenience on our local machine. Therefore, if flask realizes it is not on GCP, it should directly set these variables from the yaml file. Otherwise, directly consume these variables from the environment.

Finally, we call this function and capture this connection as db right below the app declaration.

Testing our connection

To test our connection, we can add the following snippet below the db declaration:

conn = db.connect()
results = conn.execute("Select * from tasks")
# we do this because results is an object, this is just a quick way to verify the content
print([x for x in results])
conn.close()

The code should be very self-explanatory. You should see the output of the database content in the terminal.

** Important **

Since in CS411 we disallow the use of ORM. I am demonstrating the use of the execute function so I can place it in raw SQL. There is a more advanced and elegant method to write using sqlalchemy. See more in the documentation.

Updating our database.py file

We can now finally update the database.py file with the following content. Also, don't forget to uncomment the function calls to db_help in the routes.py file. Again the following file should be pretty self-explanatory.

from app import db

def fetch_todo() -> dict:
    conn = db.connect()
    query_results = conn.execute("Select * from tasks;").fetchall()
    conn.close()
    todo_list = []
    for result in query_results:
        item = {
            "id": result[0],
            "task": result[1],
            "status": result[2]
        }
        todo_list.append(item)
    return todo_list

def update_task_entry(task_id: int, text: str) -> None:
    conn = db.connect()
    query = 'Update tasks set task = "{}" where id = {};'.format(text, task_id)
    conn.execute(query)
    conn.close()

def update_status_entry(task_id: int, text: str) -> None:
    conn = db.connect()
    query = 'Update tasks set status = "{}" where id = {};'.format(text, task_id)
    conn.execute(query)
    conn.close()

def insert_new_task(text: str) ->  int:
    conn = db.connect()
    query = 'Insert Into tasks (task, status) VALUES ("{}", "{}");'.format(
        text, "Todo")
    conn.execute(query)
    query_results = conn.execute("Select LAST_INSERT_ID();")
    query_results = [x for x in query_results]
    task_id = query_results[0][0]
    conn.close()

    return task_id

def remove_task_by_id(task_id: int) -> None:
    """ remove entries based on task ID """
    conn = db.connect()
    query = 'Delete From tasks where id={};'.format(task_id)
    conn.execute(query)
    conn.close()

Your todo app should be running smoothly on your local machine by now

Part 4: Setting up GCP app engine

Finally, we can deploy the entire application onto GCP.

Setting up local commands

In your terminal, we run each command:

# installs gcloud
curl https://sdk.cloud.google.com | bash

# installs components for python apps
gcloud components install app-engine-python

# set the folder to a project on GCP
gcloud config set project <project name>

# login to set credental on the device
gcloud auth login

Add .gcloudignore file and requirements.txt before deployment

.gcloudignore
.git
.gitignore

# Python pycache:
__pycache__/

# Ignored by the build system
/setup.cfg

#python venv
.venv

#system files
.vscode

This file is very similar to what .gitignore does. It prevents specific files from being uploaded to GCP app engine. Feel free to add directories and files that are not needed for the application.

We also need to tell GCP what packages are needed. We can run the following command on bash

pip freeze > requirements.txt

(make sure that you do not have pkg-resources==0.0.0 in the requirements.txt file)

Both of these files should live sibling to your main.py.

Deploy your app

gcloud app deploy

You can now follow the address to your application.

Conclusion

I hope this write-up is beneficial to understanding and building a very minimalistic application. I also provided the final code in this GitHub repository. Students in CS411 that source this repository should follow the Licensing and readme.

For all readers, comments and suggestions of how to improve this doc are welcomed!

comments powered by Disqus