fq.nz

Python Necromancy

Tue, 30 Dec 2025 12:23:23 +1300

This year I had deal with reviewing two old projects. Both projects had Python 2 support and Python 3 compatibility with six. Both had been using setup.py for installation and can not be installed on any of the modern & currently supported Python versions. But I was fortunate that both projects had tests. Here’s my action plan for Python necromancy.

If projects are on PyPI and test PyPI, make sure that you have maintainer/ owner access to the old project. If you don’t you will have to create separate projects on PyPI and test PyPI.
Convert old setup systems to pyproject.toml.
With the above step you will have to update the dependencies as well.
Drop support for any unsupported Python versions. This will make your life easier with dealing with dependencies and not having to support Python 2 or older versions.
When possible drop any dependencies. Less dependencies to worry about is always a plus.
When possible drop any features that doesn’t require. Try to get the MVP (Minimal Viable Product), you can always look to add any missing features later on.
When the package is installable, do some manual tests to see the intended features are working as intended.
Delete any special Python 2 checks and Python 2 specific logic. Lot of projects that have transitioned Python 3 will have checks for the Python version.
If you are lucky to have tests, use tox to run tests for all supported Python versions.
Work on fixing tests, you might have to make judgement calls about keeping some tests or making complex tests simpler.
Add tests as a CI/CD test.
Add PyPI Trusted Publisher to your projects.

Optional steps:

Remove six logic.
Use black or ruff to format code.

Lightning Talk - How docker best practices can haunt you. 👻

Sun, 01 Dec 2024 13:46:00 +1300

Here’s my first lightning talk from PyCon AU 2024 that was held in Narrm (Melbourne), Australia this year. This was given in the DevOops (not a typo) specialized track. I also gave another lightning talk (much more prepared than this) on very next day in the main conference. This speech is about the unforeseen issues caused by following best practices with docker in production deployments. This was an unprepared talk, I didn’t practice it all and it was spur of the moment thing to submit the talk. I didn’t have any presentation slides but since all other speakers had on I created landing slide to fill the space, just couple of talks before mine. :)

The PyCon AU 2024 had lightning talks threee different days (Friday on specialized DevOops tack, Saturday and Sunday). PyCon AU was highly educational and entreating. There were lots of interesting talks, I highly recommend watching the talks and attending future PyCon AUs.

(Source: “DevOops Lightning talks” - Evan Kohilas (Pycon AU 2024))

Other talks from PyCon XI 2024 can be found here.

Lightning Talk - How to create your own RFC

Sat, 30 Nov 2024 13:36:00 +1300

Here’s my lightning talk from PyCon AU 2024 that was held in Narrm (Melbourne), Australia this year. The speech is about writing your an Internet-Draft (I-D) and how RFCs are generated. This was my second lightning talk of the conference, I gave another talk a day before in DevOops (not a typo) specialized track. I was bit more prepared for this talk. I made the presentation and example draft in early of lighting talk day and did some adjustments up until the last few hours.

(Source: “Lightning Talks (Saturday)” - Christopher Neugebauer (Pycon AU 2024))

Presentation slides in PDF format.
Sample I-D: draft-rathnayake-lightning-talks

Other talks from PyCon XI 2024 can be found here.

Time travel with Python

Fri, 12 Jul 2024 12:00:00 +1200

Testing time is crucial in software development, especially for applications that rely on time-based functionality. Time-based tests ensure that features like scheduling, expiration, and logging work correctly across different time zones and daylight-saving changes.

To test that your time-based logic works correctly, you need to do time travel. This blog post lists a few ways to test time-based logic with Python in other words, How to time travel with Python. After all, Python comes with antigravity. Just import antigravity. :) So time travel should be a breeze.

For testing, let’s say we have a file called new_year.py with the following code. It is a simple function that returns whether the current day is the new year or not.

Source code for `new_year.py`:

from datetime import datetime


def is_new_year():
    """
    Returns True when it's new year
    """
    now = datetime.now()
    return now.month == 1 and now.day == 1

Our objective is to write tests to validate is_new_year() function.

Mocking datetime

This is one of the straightforward ways to change the time for your tests. All you need to do is patch datetime.datetime to do set desired datetime. Let’s say our tests are in test_new_year.py.

Source code for `test_new_year.py`:

from datetime import datetime
from unittest import TestCase
from unittest.mock import patch

from new_year import is_new_year


class TestNewYear(TestCase):
    @patch("new_year.datetime")
    def test_is_new_year_true(self, mock_datetime):
        # test if is_new_year() returns True for first of January
        mock_datetime.now.return_value = datetime(2024, 1, 1)
        self.assertTrue(is_new_year())

    @patch("new_year.datetime")
    def test_is_new_year_false(self, mock_datetime):
        # test if is_new_year() returns False for second of January
        mock_datetime.now.return_value = datetime(2025, 1, 2)
        self.assertFalse(is_new_year())

You can run tests with the following command:

python3 -m unittest test_new_year

In both of the test cases unittest.mock is used to change the datetime of the new_year module. But this can be cumbersome and limiting when you use this in complex applications because this doesn’t change the time across the whole application.

Freezegun

Freezegun is a third-party Python library that allows test code with time travelling. So, make sure that you install freezegun with

pip install freezegun

Source code for `test_new_year.py` with `freezegun`:

from freezegun import freeze_time
from unittest import TestCase

from new_year import is_new_year


class TestNewYear(TestCase):
    @freeze_time("2024-01-01")
    def test_is_new_year_true(self):
        # test if is_new_year() returns True for first of January
        self.assertTrue(is_new_year())

    @freeze_time("2024-01-02")
    def test_is_new_year_false(self):
        # test if is_new_year() returns False for second of January
        self.assertFalse(is_new_year())

You can run tests with the following command similar to the above example:

python3 -m unittest test_new_year

Conclusion

Testing time-based functionality in Python can be done in a few different ways. Using unittest.mock to patch datetime is a basic method but can be limiting for more complex applications. On the other hand, Freezegun provides a more robust solution for time travel in your tests.

Happy testing, and may your code be ever on time! :)

Lightning Talk - PyPI Supply Chain Attacks

Sat, 03 Feb 2024 22:00:00 +1300

Here’s my lightning talk from Kiwi PyCon XII that was held in Waihōpai (Invercargill) last year. The speech is about suplied chained attacks that affected PyPI and how they were mitigated.

(Source: “Lightning Talks” - (Kiwi Pycon XI))

All the talks from Kiwi PyCon XI can be found here.

Internet-Draft: draft-rathnayake-xml2rfc-unicode

Wed, 27 Dec 2023 23:23:23 +1300

xml2rfc is the tool used to generate RFC and Internet-Drafts (I-D). xml2rfc converts XML documents to various formats like HTML, PDF and text. The draft-rathnayake-xml2rfc-unicode is an experiment to see how various Unicode characters are represented in those various output formats.

I used a couple of Python scripts to generate a list of Unicode characters and the Unicode blocks that they belong to. Then these were embedded in my draft XML with another Python script.

The first revision I-D is available in various formats:

Lightning Talk - How Python can break the Internet

Sun, 23 Oct 2022 23:00:00 +1300

Here’s my lightning talk from Kiwi PyCon XI that was held in Ōtautahi (Christchurch) this year. The speech is about how Python is a vital part of the tools used in IETF.

(Source: “Lightning Talks” - (Kiwi Pycon XI))

All the talks from Kiwi PyCon XI can be found here.

Visualization of Internet-Drafts

Sat, 02 Oct 2021 15:02:02 +1300

IETF (Internet Engineering Task Force) is a non-profit standards organization that develops and promotes technical standards related to the internet. Visualize IETF project is an attempt to visually present how different areas have evolved and how authors have contributed to Internet-Drafts (IDs) over time. Visualizations were created using Gsource.

I have published all the visualizations that I created under this YouTube Playlist. The source code of the project is hosted in GitHub under Visualize IETF.

Here’s a sample visualization of how IDs were authored from January 2021 to July 2021, This is clustered by areas and working groups that belong to the area.

This project was created as a part of the IETF 111 hackathon. Here’s my presentation at the hackathon. Slices are available here.

(Source: IETF111-HACKATHON-20210723-1900)

Python Dependency Management

Sat, 12 Jun 2021 08:09:23 +1200

Dependency management in any language is hard, best way to avoid a dependency hell is to avoid any dependencies. :) But that’s never practical.

These are few suggestions to make dependency management easier. This blog post is mainly focused on dependency management in a web applications. Hope this will be useful.

TL;DR

Use venv and split out requirements similar to:

Requirements file Use

requirements.txt Direct dependencies of the application.

requirements.txt.lock Version locked list of direct dependncies and their dependencies

requirements.test.txt Dependencies for testing

requirements.sec.txt Dependencies for security audits and tests

requirements.dev.txt Development tools and libraries for the project

Requirements file	Use
`requirements.txt`	Direct dependencies of the application.
`requirements.txt.lock`	Version locked list of direct dependncies and their dependencies
`requirements.test.txt`	Dependencies for testing
`requirements.sec.txt`	Dependencies for security audits and tests
`requirements.dev.txt`	Development tools and libraries for the project

Always use venv

Python virtual environment (venv) module comes with your standard Python installation. venv helps you to keep your development environment clean without leaking Python libraries to other projects or environments. This also allows you to keep different versions of Python and dependencies for testing.

To create a new Python virtual environment use following command, this will create a Python venv in venv directory.

python -m venv venv

To activate your python venv:

. venv/bin/activate

If you want to deactivate your venv, use deactivate command.

I thought it’s good idea to mention about tox here. tox is very useful tool run tests for different Python environments. When you have a properly configured tox.ini file and all the required Python versions that you need to support installed. Just running tox command will take care of things.

Use requirements.txt file for direct dependencies

Use a requirements.txt file for your direct dependencies, depending on your project requirements you may add version information as well. But I like to keep the requirements.txt file without any version information. Let’s call this main requirements file because there’ll be more.

You can use pip inside venv to install your dependencies.

pip install -r requirements.txt

It’s important to keep only the libraries that required to run your application in this main requirements file. So don’t include libraries that’s required for testing, deployment etc in this one.

What about dependencies of direct dependencies?

Now it’s important to version lock dependencies for a production release. It’s a bad idea to let your production environment decide which version to install because that can lead to breaking changes or unintended side effects. For that use requirements.txt.lock file.

Create this file after installing all your main requirements. You can use pip freeze command for that.

pip freeze > requirements.txt.lock

This requirements.txt.lock must be used when you are deploying to a production or test server or running tests on a CI/CD pipeline. If you have to debug a production issue use this file to install dependencies, so you have same versions of libraries as your production environment has.

Note that, some libraries depends on operating system and other external ibraries, so this does not guarantee that your work environment will be 100% imilar to production.

imilar to python version, pip version matters too. So it’s important to make ure all environments use same pip version, and keep that up to date.

What about testing and other tools?

Keep your dependencies for testing in a seperate requirements file. Let’s call that requirements.test.txt. Your test dependencies like hypothesis, faker can go to that file.

Keep other tools that you use for development like ipython in requirements.dev.txt file.

I like to keep tools used for security vulnerability test in a seperate file called requirements.sec.txt, but this could be part of requirements.test.txt as well. I like to keep it seperate because that allows me to set up a seperate pipeline that only checks for security vulnerabilities and dependency audits daily.

Security frist!

Talking about security, it’s important audit your requirements.txt.lock file daily. You can use safety for that. When safety is installed it’s simple as running:

safety check -r requirements.txt.lock

Another good habit is to keep pip up to date. pip will let you know if there’s an update. You can upgrade pip with following command.

pip install --upgrade pip

Constraints files

I haven’t used Constraint files in my projects so far. These files allows you to main a certain version of a dependency through out.

If you have a constraint file, you can use that with pip install as follow.

pip install -c constraints.txt -r requirements.txt

Web Application Security on a Budget

Tue, 01 Dec 2020 12:54:23 +1300

For a small business, information security mishap can be fatal blow to the business. When you don’t have budget do annual penetration tests and keep a dedicated set of engineers to maintain and look after your systems it might be tough task to provide a secure online service to your customers.

In this blog post, I’m providing some tips on keeping your online services secured with on a very low budget.

Passwords

First things first use a password manager to keep your passwords for different services like emails, domain name, accounting software etc.

Make sure that you use good password for each service and it’s different from service to service. Never use the same password for two different services.

Other must do thing is to enable Two Factor Authentication (2FA) on all the services that provide it. These days most of major online services provide 2FA, if the service that you use doesn’t provide 2FA, it might be a good idea to move to a different provider that provide a better security. twofactorauth.org has a list of major online services that provide 2FA.

Subscribe for Have I been pwned? Notify Me service with all of your email addresses. This will provide you an alert if any of your email addresses are linked in existing data breaches or newly identified data breaches. Make sure to update your password if you got an alert.

Static Website

Ask your self whether you need a dynamic web application or a just a static website is enough. If you don’t update your content that often and your website is just place to provide information to visitors, then small static website will be enough.

In that case consider using a service that provide small static service or get a static website developed by credible developer or an agency. If you are tech savvy you could make use of services like GitHub Pages or GitLab Pages.

Not having dynamic content, and sections such as user comments etc. greatly minimizes some of the major attacks.

Dynamic Web Applications

If you provide a dynamic web application such as web shop, consider using a reputable web shop provider. Similarly if you are after blog, use a reputable blog service provider.

Some of the security advantages that you get by going with an online web application platform rather than getting a developer or agency to build you a bespoke web application are:

Ongoing maintenance and security upgrades.
Better monitoring (from platform provider side).
Low chance of existing vulnerabilities. (Usually these services get frequent penetration testing and has good bug bounty programs).

When using a web platform, make sure to be familiar with the security controls provided by those services and use a good password and enable 2FA.

Don’t install any “add-ons” / “plugins” unless your really have to. If you are installing them make sure to do proper research on them and update them promptly. Things to look out for in third party “add-ons” / “plugins” are:

Online reviews
Popularity
Plugin update frequency

Avoid using any plugins that hasn’t been updated in a while.

Domain Names

Having a good domain name is important to your business but it is very important that you keep your domain name with you without letting it expired. As discussed above, using a good password and 2FA with your domain name registrar or reseller is important. It is advisable to keep the domain name on auto renew. Put a annual event in your calendar to check that auto renewal is successful. Sometimes, a bad credit card, incorrect/old email address or failure at the provider might end up your domain not getting renewed.

Allowing your domain to expire and if a malicious party get hang of your domain name, that will open a plethora of attacks on you, your customers and providers.

Photograph: “The News - Suvarnabhumi Airport, Bangkok” by Kesara Rathnayake