Finn Greig

Running Honk on Alpine Linux

2022-11-14T00:00:00Z

With all the recent talk about the fediverse now that Twitter is on fire, I decided to give it a go by running my own instance. There's a lot of benefits to this - considering that you're joining a decentralised social networking system, you might as well own your data, and also start with a clean slate - you don't need to worry about other servers having already blocked the server that you're only just getting set up on.

I decided to go with an AWS t3a.nano instance running Alpine Linux for my server, and I'll be running Honk on it - this is a minimalist web application written in Go that interfaces with the rest of the fediverse using ActivityPub (the underlying standard that powers Mastodon, Pleroma, Akkoma, Misskey, etc.)

This is the AMI image I used - up to date ones can always be found on the Alpine website. Just ensure that your security group has TCP/80 and TCP/443 open to the world :)

The Gist with the scripts and configs can be found here

If all has gone well, when you load your domain you should get the purple Honk interface, and be able to log in. :)

Locking It Down? Easy Tools to Help You Develop Secure Software

2022-01-09T00:00:00Z

If you're like me, sometimes when you're working on bringing a great idea to life security can be a bit of an afterthought. You have already thought about the big stuff like TLS and implementing the right authentication and authorisation system, but there can still be blind spots - for example, how your application is handling user data flow, or how it's exposing that data to the rest of your application's code. Luckily there are tools that can help you with secure development, and I'll show you some of the big ones out there.

1. SonarQube

SonarQube is a hosted automatic static analysis tool that detects bugs, vulnerabilities and code smells in your code. It works great with your existing workflow - for example, when you make a pull request, it can automatically check for code smells in your work and produces a quality report which can appear in the conversation over on GitHub (or whichever version control system you use).

You're probably thinking "What's a code smell? Code doesn't smell!" 👃 While it isn't a fragrance, a code smell is a hint that there could be deeper problems in your code. By fixing code smells when they occur, this can help to prevent critical paths for vulnerabilities forming in your code that can be used for exploitation at some point. These are generally small things, like how you could be handling a string, or uncontrolled side effects of a function.

P.S. If anyone knows the term for some code that has a lot of smells, please let me know. Is the code stinky? Pungent? Who knows!

2. SonarLint

If you don't want to actually set up a SonarQube instance and you just need something simple that can work, SonarLint is great. It's made by the same team behind SonarQube but it's a text editor/IDE plugin that can run standalone without SonarQube and it highlights smells while you're writing code.

3. Dependabot

Enabled by default on GitHub repositories, Dependabot automatically scans dependencies in Ruby, JavaScript, Python, PHP, Elixir, Elm, Go, Rust, Java and .NET projects, and if any of them have been updated to address security vulnerabilities, it'll update your dependency definitions to use the new versions. You can even run it yourself.

We Need To Talk About Technical Debt

2022-01-05T00:00:00Z

Although I have had a short career in tech so far, the list of technical debt that Ive come across is not. Even though Ive only had a few odd solo jobs and internships before my current role, which is my first permanent one, there's been plenty of technical debt along the way.

Generally software businesses follow fairly standard Agile practices - in my current role, we have a mixture of Scrum and Kanban teams - and each of these teams have the usual rituals like standup, planning, refinement and retro.

In Agile, product owners are seen as the interface between executives/project sponsors, and the developers. They communicate to the developers the business requirements of a product they're building to make sure its made as the customer needs, and then the developers can communicate progress and technicalities that might come up along the way, and then they can communicate this to the project sponsors.

When developers mention technical debt that they need to deal with, its possible that the product owner can minimise the importance of it repeatedly, often because feature x needs to be developed or deadline y for something is coming up and the customer wants a million other things done by then.

Technical debt cannot be minimised. As it increases, the effects it has compound, and when future code relies on indebted code, the work required to finally deal with it really increases. If the debt is something that isn't feature specific and its effects are present on all new code being written, developers can become hesitant to work on something that could possibly fall over at any point in the future - it could be years away, or it could even happen tomorrow. It can also hinder DX (developer experience) - for example, a temporary stopgap that introduces few extra steps in the debugging and execution cycle could become permanent, and developer happiness can degrade significantly.

Ideally we should have a development cycle that focuses on constantly working down technical debt each day in some way, and occasionally slowing development on new features and stories in favour of this. Product owners can help a lot with this, as they can convince executives/project sponsors that taking care of some of the technical debt is needed in order to start or complete work on a new feature/story. If the product owner is unwilling to do this, then the developers on the team should band together and negotiate that they deal with technical debt before starting on new work.

Sometimes we have to pass up the opportunity to make another dollar in order to ensure long-term developer happiness and stability.

Review: GitHub Copilot Technical Preview

2021-11-24T00:00:00Z

About a month ago, I got invited to the GitHub Copilot Technical Preview. It is essentially code autocomplete on steroids for Visual Studio Code - behind the scenes it's built upon an OpenAI machine learning model named Codex, which was trained using public code from GitHub repositories and Stack Overflow answers. It works with a variety of languages - so far I've mainly used it with JavaScript code.

Usage is fairly simple - once you install the plugin from the VS Code Extensions Marketplace, you just need to login with your GitHub account - this is to ensure that you're eligible to access the technical preview. Since it collects usage data for improvement, a nice feature that it has is that it's automatically disabled when inside of anything that could be some sort of config or secrets file - for me, when I was in GitHub Actions yaml files and also .env files, it was disabled. Once you're in a file it's already running - as you write code documentation and function signatures it begins to suggest autocompletions:

Once you're happy with the suggestion, you can press Tab to accept it:

(I tried using Shift + [ and Shift + ] to select other suggestions but it wouldn't work for me at the time.)

I'm very happy with this - I see how it significantly speeds up development, especially considering that you could have one less window open as you wouldn't need to look at Stack Overflow or documentation when trying to figure out the best way to complete common tasks in code. However, I wouldn't say it's ready to be used as a daily driver at work yet, there is no way to run it offline and have it not send usage statistics and potentially your code to a third party server - I can imagine quite a few employers wouldn't be happy with this, especially when dealing with sensitive IP in a competitive or national security space. It's very handy for working on personal projects though.

Tech Culture Fatigue

2021-11-09T00:00:00Z

Having a bit of a knack with technology as a kid had me wanting to work in the field when I got older, and the future was looking so bright - we were meant to propel the world forward, solve big ticket problems and make things more accessible for everyone. However, being at that point now, I am left somewhat jaded (a bit of an understatement).

Web 1.0: the web is for sharing information!

Web 2.0: but what if we could make money too?

Web 3.0: the web is only for making money.

— foone (@Foone) November 2, 2021

Originally when what was dubbed "Web 1.0" came around, the world wide web was a bleeding edge concept. Only those lucky enough to have access to a computer with the right networking setup could use it. Pages were incredibly simple, consisting of mostly text and hyperlinks - ways to click around to get to other pages, whether they be your own or someone else's. There was no Google, and everyone was happy about just being able to get information out there to other people more or less instantaneously - there was no need to wait for a postie to deliver letters anymore, or to deal with fax machines. Your email address was the new address. Over time, as prices for commodity computing hardware decreased, more and more people were able to enjoy this.

Those were some pretty good times for me - naturally due to my age, I only got in on it at the end of the era. I can still fondly remember how the computer would need to scream before you could use the web (foreshadowing), or how instant messaging took off and became an instant hit. A decade and a half later, I can still hear the sound MSN Messenger would play when my brothers received new messages.

Then started "Web 2.0". JavaScript and Adobe Flash started gaining momentum, and websites now had interactive content. Then, the ads started to come. Much like classifieds in a newspaper, at first we thought "sure, why not - after all, they need an income to keep their site running and earn a living." But then the big corporations started to notice, and they got onto publishing content and building successful internet-facing businesses as well. Over time there were more and more flashy ads and less and less of the content you were actually consuming - "hey, wait a minute! what's going on here?"

This led us to where we are roughly now today, but we have a lot more issues on top of that. Data collection for targeting ads to specific people, addictive feedback loops from dark design patterns intended to keep people using something and chemically craving it so that ad revenue can be maximised, and collateral damage from these systems, for example, letting misinformation and disinformation run amok so that, once again, that juicy revenue can keep on flowing in.

Now we're onto "Web 3.0" - the idea that everything online has to be decentralised and kept on something called a "blockchain" (which is just a term to obfuscate the fact that it's just a very convoluted linked list), and it's all about making money with tech in any way. Images are now "non fungible tokens", exactly the same but now you get a receipt for owning something that can be copied infinite times at no cost. All this sort of stuff does is just chew electricity and wear out computer parts when it could be done another way, or in the case of non fungible tokens, just not be done in the first place. It's all just a big grift. In this stage of the game, even dishwashers have become iPhones. DRM (a.k.a. proprietary hardware lockouts) is in vogue now, reducing re-usability and repair-ability for things that would normally last consumers a very long time.

Stuff like this makes me think about the 2019 Annual Letter for Social Capital, specifically the section on talent hoarding. Why are grads out here wasting their time and talent working on ad targeting for Big Tech, when their skills could be used to solve climate change, improve healthcare or make agriculture more efficient?

We need to do better than this - while right now I am lucky enough to be in a role where I work on a variety of public transport solutions, these sorts of things sometimes make me feel embarrassed to be a technologist, especially when people ask me about them. We should be excited to share our developments with others, not anxious.

Gaining ADB Access on a Mobiwire Sakari

2021-10-14T00:00:00Z

Last year a technology retailer here in New Zealand, PB Tech, was running a Black Friday sale, and I came across their deal for a Vodafone Smart A9 - a rebranded Mobiwire Sakari "feature phone" - for NZ$13. I had previously encountered the phone a few more weeks before this at a display stand in a Harvey Norman store, and from briefly messing around with it, I realised it was running a really pruned back older version of Android. I figured at the time that I might as well buy one and see what I could get it to do.

Poking Around

The Sakari is a simple phone - it has a 240x320 TFT LCD screen, a VGA camera with flash, a 3x4 physical keyboard and 5-way navigator keypad, a 3.5mm headset jack, a microUSB connector, a micro SIM slot and a microSD slot. The manual references an optional second SIM - there must be a variant out there with dual SIM support, however, mine only has a single slot. The phone also comes with a removable 1000 mAh battery, a mono wired headset, and a charger (unfortunately the cable is integrated into the wall power supply, so you will need your own microUSB cable to connect it to a PC).

My model is a "VFD 120" in the grey slate colour - note the single SIM slot labelled as "SIM 1".

Screenshot of the home screen:

CPU-Z screenshot:

Gaining ADB Access

As SP Flash Tool is unsigned software whose origin I am not sure of (supposedly MediaTek but the site appears unofficial), I used a Windows VM to perform the steps below. An easy way to set up a temporary one is to use the VM images Microsoft provides for Edge browser testing. If you want you can download one from Microsoft and run it in your preferred virtualisation software. If you have trouble trying to get the VM to capture the USB device at the right time when plugging the phone in, you might need to just run it directly.
Download SP Flash Tool
Download the scatter file: (Updated link coming soon)
Download the modified boot image: (Updated link coming soon)
Open SP Flash Tool, select the "download" tab and select the scatter file as the "scatter-loading file". What this does is provide SP Flash Tool with the details about the partition table on the phone's flash memory.
Select the boot image file as the location of BOOTIMG by double clicking in the location cell of the BOOTIMG row. Ensure the dropdown is set to Download Only. Click the download button at the top.

With the phone powered off, plug it into the computer. Flashing should now begin. Once it is done, the phone should reboot - if not, perform a reboot manually by doing a reset.
If the phone didn't flash, you likely need to install drivers for the MediaTek preloader. There are web links for these within the Drivers folder of the SP Flash Tool folder - the VCOM driver is likely most relevant for this. After it's installed try again from step 5.
Access a shell on the phone through ADB by running adb shell in a command line terminal.
Enter the command settings put global install_non_market_apps 1. The phone will now allow you to install apps from APKs.

Many thanks to Kugelblitz from the Mobiwire Modding Discord server, who provided the scatter file and the modified boot image to get ADB working on the Sakari.

Simple Wireguard VPN Setup

2021-08-27T00:00:00Z

First, install Wireguard. For Ubuntu:

sudo apt install wireguard

If you want to forward all traffic from clients, enable IP forwarding in the kernel.

Next, create a config file at /etc/wireguard/wg0.conf containing the following (you can also choose to allocate v6 addresses to clients):

[Interface]
PrivateKey = <server_private_key_goes_here>
Address = 172.16.1.1/32
ListenPort = 51820
# Use the following for forwarding all traffic from clients:
# Change eth0 to your network interface if it differs
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; iptables -A INPUT -s 172.16.1.0/24 -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW -j ACCEPT; iptables -A INPUT -s 172.16.1.0/24 -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = <client_public_key_goes_here>
AllowedIPs = 172.16.1.2/32

Next, on the client, create a config file containing this:

[Interface]
PrivateKey = <client_private_key_goes_here>
Address = 172.16.1.2/32
# A DNS server can be specified for the client to use when the tunnel is active, optional
DNS = 172.16.1.1

[Peer]
PublicKey = <server_public_key_goes_here>
# AllowedIPs = 0.0.0.0/0, ::/0 for forwarding all traffic
AllowedIPs = 172.16.1.1/32
Endpoint = <server_ip>:51820
PersistentKeepalive = 25

For each client, add a new [Peer] block to the server config with their public key and their IP as the allowed IP.

To start the interface on the server and make it persistent on reboots, run:

systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0

To generate a new keypair use the following command:

wg genkey | tee x_private_key | wg pubkey > x_public_key

Turning a Git Repository into a Simple File Host with Cloudflare Workers

2021-08-27T00:00:00Z

(post originally hosted elsewhere)

Recently I used CloudFlare Workers for the first time to create the website you're reading right now (look at Fruition to learn how I did it), and I decided to try out writing some of my own workers to get an idea of what it could do.

I was in need of a simple static file host to serve various small files - stuff like HTML, presentations, PDFs etc - and thought about how Git repository hosts generally allow you to view a file within a repository in its raw format, served directly to you without modifications. I figured that I could use a CloudFlare Worker to proxy a request to a raw file path on my Git hosting service, and rewrite any necessary headers to ensure that files are served to browsers correctly.

Here's how I did it:

If you do not already have an account, sign up at CloudFlare and follow the step-by-step instructions provided to set up your domain to use their services.
Click the workers button at the top of the page
Set up your worker subdomain - this can be anything you like (if prompted), then click the manage workers button
Click the create worker button
In the script editor on the left half of the page, replace the example script with this:

addEventListener('fetch', event => {
    event.respondWith(handleRequest(event.request))
})

// add a link to the raw file content host pointed at your primary ref
const REPO_URL = 'https://raw.githubusercontent.com/exampleuser/examplerepo/main/'

const PATH_REGEX = /(?<=/*YOUR DOMAIN HERE*/\/).+/

const mimeTable = {
    'html': 'text/html',
    'css': 'text/css',
    'jpeg': 'image/jpeg',
    'jpg': 'image/jpeg',
    'js': 'text/javascript',
    'json': 'application/json',
    'png': 'image/png',
    'txt': 'text/plain'
}

/**
* Respond to the request
* @param {Request} request
*/
async function handleRequest(request) {
    const req = new Request(request);

    const path = req.url.match(PATH_REGEX)[0];
    const extension = path.split('.').pop();

    const file = await fetch(REPO_URL + path);
    if (!file.ok) return new Response(null, {status: file.status});

    const response = new Response(file.body, file);

    const contentDisposition = response.headers.get('content-disposition');
    if (contentDisposition) response.headers.delete('content-disposition');

    response.headers.set('content-type', mimeTable[extension]);
    return response;
}

Click the save and deploy button, and confirm that you want to save and deploy
Go back to your domain's dashboard, and click the workers button again like in step 2
Click the add route button
Specify the route that the worker function will serve git repository assets for (e.g. files.exampledomain.com/*), select your worker function that you just created in the worker dropdown and then click save
Click the DNS button at the top of the page
Add a CNAME record, with the name set to the domain used in the worker route, and the target can be set to a non-existent record (e.g. sinkhole.exampledomain.com)
Try and open a file in your git repository by visiting https://<WORKER_ROUTE_DOMAIN>/<FILENAME>.<EXT>

Your Git repository is now like a simple file hosting server - files can be loaded directly over HTTP now. Since the worker script removes the content-disposition header if it is present and corrects the content-type header, it can also be used as an asset source within HTML and CSS.

The importance of 'small sanitisation'

2021-03-14T00:00:00Z

I've been working on a Python script that takes the contents of Te Ara and puts them into a compressed archive, after being inspired by how the entirety of the English Wikipedia's text and the Simple English Wikipedia's text can be downloaded as 18GB and 201MB archives respectively (as at March 2021), and wanting to have my very own copy of Aotearoa's history.

There were a variety of bugs along the way - I was missing calls to .replace() to handle some Unicode character conversions in my file path sanitisation function, and my browser and IDE were confusing me by showing the same representations for an em dash and a minus, which wasn't helping.

I had implemented multiprocessing to take care of saving each article as a PDF once the sitemap had been scraped into a list and all was running well at the time - it was 4x faster and blazing its way through them.

However, a seemingly small bug came up in the tail end of execution. It was giving a FileNotFoundError for ./section_places/southland/stewart_island/rakiura.pdf

This confused me to no end for 6 hours - at first, I thought it was an issue with the list of scraped articles and there was an invalid item in it, and then I thought it was a race condition because of how the multiprocessing interacted with filesystem changes. I started putting guard conditions in to wait for directory creation before moving on to saving the PDF, and when that didn't work I tried using the Lock class that the multiprocessing library provided, but to no avail - and on top of that my print statements weren't showing and the debugger really wasn't helping much.

I tried filtering the scraped article list down to just the item for Stewart Island and my print statements also started working then (something to do with the processes is my guess) - and it turned out that the article title had a forward slash in it (Stewart Island/Rakiura). Since I was using pathlib so that my paths could be POSIX compliant regardless of OS, it was interpreting Stewart Island as the directory containing the file, while not actually being created before saving the PDF. I made one small change to my sanitisation function to replace the slash with an underscore and it was working again.

I feel like this highlighted the importance of 'small sanitisation' for me - that is, sanitisation that's not considered critical by usual good practice conventions - like correctly parameterising SQL statements that take user input on a web server, or escaping HTML to prevent XSS is. It's the small cases that can often be quite difficult - you could have a non-standard specification to conform to with no public libraries to help you, and while data in this case is most likely not user-submitted, you might need to update your specifications each time your dataset changes. Hell, you might even have the odd wild goose chase through your code every now and then. Garbage in, garbage out and all that.

Stay safe, and keep on keeping those strings nice and clean.

Flamingo Scooters API

2021-02-13T00:00:00Z

A while ago I was working on an open-source scooter map web application, and I wanted to add the locations of electric scooters that are offered for hire here in New Zealand by a company named Flamingo. I reached out to their support, and they helpfully provided me with details about their public API.

I figured someone else could also be looking for them, so here's how you can access it:

API Details

The Flamingo API returns data that follows the format of the GBFS standard (General Bikeshare Feed Specification). Each city has its own endpoint that contains information about the scooters available in the local area:

Wellington

https://api.flamingoscooters.com/gbfs/wellington/gbfs.json

Christchurch

https://api.flamingoscooters.com/gbfs/christchurch/gbfs.json

These endpoints provide information about the feeds that are available to consume. For example, if you wanted to see which scooters are available in Wellington, you would use the free_bike_status endpoint:

https://api.flamingoscooters.com/gbfs/wellington/free_bike_status.json

Which should return something like this:

{
  "success": true,
  "data": {
    "bikes": [
      {
        "bike_id": "4129",
        "lat": -41.276589,
        "lon": 174.77937,
        "current_range_meters": 27132,
        "last_reported": 1611545465
      },
      {
        "bike_id": "8025",
        "lat": -41.288907,
        "lon": 174.803936,
        "current_range_meters": 28917,
        "last_reported": 1611545501
      },
      {
        "bike_id": "8430",
        "lat": -41.21483,
        "lon": 174.804965,
        "current_range_meters": 25620,
        "last_reported": 1611545449
      },
      {
        "bike_id": "8240",
        "lat": -41.312017,
        "lon": 174.782053,
        "current_range_meters": 19635,
        "last_reported": 1611545412
      },
      {
        "bike_id": "8412",
        "lat": -41.291181,
        "lon": 174.79007,
        "current_range_meters": 24276,
        "last_reported": 1611545523
      },
      {
        "bike_id": "8252",
        "lat": -41.278834,
        "lon": 174.771693,
        "current_range_meters": 24990,
        "last_reported": 1611545464
      }
    ]
  }
}

Finn Greig

Running Honk on Alpine Linux

Locking It Down? Easy Tools to Help You Develop Secure Software

1. SonarQube #

2. SonarLint #

3. Dependabot #