Our goal with Workflows & Automations includes a standardization on Python as our backend language of choice. We wanted to standardize on a common web framework as well.

While Flask and FastAPI are the most popular for APIs, they tend to have a no-batteries-included approach leading to N+1 ways of building software. However, opsZero is building an opinionated stack and this requires standardization, and we have chosen Django for our Web Framework.

Our entire business is already reliant on Django as the primary framework so we have extensive experience using it, and absolutely love the built-in templating and ORM with migrations.

Lastly, with ASGI, a lot of extra features for WebSockets and Events are built into the framework using channels. The built-in functionality along with years of Django experience means we can provide Django expertise quickly.

Pre-Built Django Templates​

We want to meet our customers' needs regardless of the Cloud they are using and with the lowest cost possible, so we are making available pre-built templates to use Django in both Serverless as well as Kubernetes environments.

We are releasing three templates:

• Django for Helm – Configures Django to be deployed onto any Kubernetes environment using Helm charts powered by Tiphys.
• Django for AWS Lambda – Powered by the Serverless Framework; deploys on AWS Lambda.
• Django for Azure Functions – Powered by the Serverless Framework; deploys on Azure Functions.

These three templates allow us to deliver value to you faster.

Think what you want of Elon Musk, but he has achieved quite a bit in engineering novel solutions to complex problems. We’ve worked mainly on implementing the same process to great effect in what we do.

The principles are:

1. Fix dumb requirements. Each requirement has a specific owner.
2. Remove unnecessary parts
3. Simplify/Optimize
4. Speed up cycle time
5. Automate

You can watch him describe his process here:

Fix dumb requirements​

When solving a problem for a customer, the customer may not actually know what they need. So uncover the actual requirement behind the request.

Usually, a problem such as:

The production database is high CPU and clients can’t connect

may actually be a root cause issue:

The production database is being used to replicate data to a data warehouse, which is causing the issue.

This root cause analysis can be gleaned through a five whys analysis.

Second, with these requirements there needs to be a clear owner responsible for the issue. If there is not an owner for something, then that itself is an issue. Ownership of each component means that someone exists to optimize each piece.

Remove unnecessary parts​

Systems over time become complex. Pieces are added that don’t need to exist—or they were added, then forgotten about.

Systems should get less complex, not more so.

As we build things, we build to get the task done. This means we may add complexity to the system that didn’t need to exist, but because we are pathfinding our way to the solution, that complexity is needed.

Once we get to the point of the system working as needed, we go back and remove the pieces that are not needed.

Simplify/Optimize​

After removing unnecessary parts, there may still be complexity within the current components.

To simplify these components, we need to reduce variability and increase standardization.

For example:

• The use of multiple if-else blocks to account for variability can increase complexity.
• Simplification requires subjective decisions on the optimal approach.

It’s best to initially build with some variability, then refine through A/B testing over time toward the optimal solution.

Speed up cycle time​

Once an optimal approach is found:

• Remove variability
• Standardize the approach for deliverability

This leads to faster outcomes, with fewer branching paths, creating better flow.

Automate​

Lastly, automate the processes such that things happen without intervention.

Cloudflare D1 is a great way to quickly create and work with SQLite databases where a larger PostgreSQL or MySQL don’t make sense. These are some examples to quickly work with D1.

Create the Database and Table​

wrangler d1 create data-cloud-vendors
wrangler d1 execute data-cloud-vendors --command='CREATE TABLE Customers (CustomerID INT, CompanyName TEXT, ContactName TEXT, PRIMARY KEY (`CustomerID`));'

I like Excel for Windows. The Mac version is a joke compared to what the full-blown Windows version can do with data analysis and data finagling right from the app itself.

A lot of what I have been working on as of late has been trying to get data into Excel stored on OneDrive with data crawled using Playwright. The reason for this is that some of the data is small enough that building a full database isn’t necessary, and is not normalized enough to just use PowerQuery.

To achieve this outcome I have used GitHub Actions to trigger the run. GitHub Actions triggers on a schedule which sends the task to a GitHub Runner that starts a Python script. Since GitHub Actions has access to the root volume on the Mac Mini (don’t worry, the machine is dedicated to just GitHub Actions) I can use xlwings to launch Excel and update the workbook.

Once completed, it just copies the file into OneDrive or Dropbox for me to access elsewhere.

There is absolutely no difference between the hosted runner and the self-hosted runner for this example, other than that it just runs on a self-hosted instance that happens to have Excel on it:

name: Download and Upload
on:
  schedule:
    - cron: "0 1 * * *"
  push:
    branches:
      - main

jobs:
  build:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v3
      - name: Install Dependencies
        run: |
          pyenv global 3.11
          pip3 install -r ./requirements.txt
      - name: Combine
        run: |
          python ./main.py