Structuring & Package Initialization

There are a lot of ready-made templates for Python packages out there. You can simply clone them, make your changes, and get ready for deployment, but in this article, we're going to talk about the very basic files and routes you need to have to make a very basic Python package.

Our package is going to be used in two ways. You can either use it inside your CLI or import it within your Python source code. All it does is it shows a hello-world message so I'm going to name this project ‘Hello’. Keep in mind that if you want to publish your package to PyPI, you must choose a unique name meaning "pypi.org/project/<package-name>" has to lead you to a 404 page.

Open a fresh terminal tab and create a directory called Hello. In this directory, we'll keep the source code and metadata showcasing our package.

Each Python package has a setup.py file that contains all the metadata about our package including its version, name, entry points, etc. I'm going to write down this file based on a basic greeting package.

from setuptools import setup
setup(
name="hello",
entry_points={
"console_scripts": [
"hello = hello.cli:main"
]
}
)

In the first line, we've imported the main library that we need for package initialization which is setuptools. Then, we call the setup() function with the metadata of our package. Here I've filled up the required parameters only.

Keep in mind that some parameters and metadata keys are nice to have. I've included them here.

from setuptools import setup, find_packages    # new

setup(
name="hello",
version="0.1.0",    # new
packages=find_packages(),   # new
entry_points={
"console_scripts": [
"hello = hello.cli:main"
]
}
)

The find_packages() function selects the subdirectories of the package and counts them as the packages that need to be included within the distribution cleverly. One handy parameter of this function is nothing but the exclude param that allows you to ignore some subdirectories that you don't want your distribution to have such as docs/, tests/, or benchmarks/, and so on as they're not part of your actual practical package and your package does not require these to work perfectly fine.

We also have an entry_points parameter inside the setup() function that is responsible for adding CLI commands that point to some of the functions/methods of our package. As you can see, we only have a hello command that points to the main() function inside the cli.py file of the hello directory. A quick tree-like shape of our package would be like this so far.

hello
├── hello
│   └── cli.py
└── setup.py

Inside the cli.py file, we need a function called main() that plays the role of the CLI command.

import sys

def main():
try:
stdin_name = sys.argv[1]
print(f"Hello {stdin_name}!")
except IndexError:
sys.exit("Provide a name please!")

If you want to, you can create a virtual environment inside the root directory of your package and install the package in editable mode via the -e option. Otherwise, you can run the same command outside of any virtual environment.

pip install -e .

The reason that we've used the -e is because we want to see the changes that we make at every execution of our package. It helps us not to install the package on every change that we make.

To check if the package works fine, simply run the hello command:

$ hello Sadra
hello Sadra!

$ hello
Provide a name please!

Internal Usage

To implement the internally used function for our hello package, all we need to do is create a new internal.py file that contains a greet(name: str = "World") function that returns the greeting phrase. For that matter, I'll create it inside the hello/ directory right next to the main.py file.

def greet(name: str = "World"):
    return f"Hello {name}!"

A tree-look of our package would be as follows:

hello
├── hello
|   ├── internal.py
│   └── cli.py
└── setup.py

Easily open a Python REPL and use the greet function as follows:

from hello.internal import greet
greet("Sadra")
Hello Sadra
greet()
Hello World

Now, let's talk about the tools and best practices for creating a Python package.

Using Poetry

Poetry is an advanced Python environment manager that helps you manage your package dependencies and create Python packages as well. This tool automates almost all phases of your package development- creation, dependency management, and publishing.

In this example, we’ll install the Poetry package first, create a new package, and add some dependencies afterward.

$ pip install poetry

Once you have the tool installed on your machine, create a fresh package via the init command and keep prompting until your package skeleton is created.

$ poetry init

It creates a pyproject.toml file for you, which contains all the metadata of your package. (Everything that we had passed to the setup function from the setuptools library)

To add a new dependency, use the command- ‘add’.

$ poetry add requests==2.2.0

Although we’ve not talked about the publishing phase of your package, but keep in mind that the commands built and published using Poetry are responsible for doing so.

$ poetry build && poetry publish

The build command creates a new distribution file for you. It’s in the format of an archive file (source distribution or sdist) like .tar.gz or you could even specify whether you need a wheel distribution or the default one.

What is PyPI

Python Package Index (PyPI) is the place where almost all the public Python libraries are stored. It’s the server that pip tries reaching out to, by default, whenever you want to install a new package.

In this tutorial, we’re focusing on publishing your package on PyPI. You can publish your package to a private infrastructure, such as a private repository on GitHub.

Publish Your Package

If you want to publish your package to PyPI, you need to follow some steps.

• Make sure your package is healthy and installable.
• Make sure that it works as expected.
• Create an account on PyPI.
• Build your package and push.

The above points are the fundamental steps you need to consider to make your package available on PyPI. In this section, we’ll go through this subject in detail.

Head to pypi.org and create a new account. Log into your dashboard and go to your account settings. Scroll down and you’ll see the API tokens section. Generate a new token with all the accesses granted. Don’t forget to copy the token into your clipboard so that you can create a simple push configuration where it’s going to use your API token to push the changes.

Once you’ve copied the token, create the $HOME/.pypirc file and fill it with the following keys and values:

[pypi]
username = __token__
password = <the token value, including the `pypi-` prefix>

Make sure to paste the token as the value for the password key. Now, let’s publish our ‘hello’ package to PyPI. To do so, we need another third-party package called Twine. It helps us push our distribution file to PyPI so install it on your machine.

$ pip install twine

Since our package is working fine with no issues, let’s create a new distribution archive. We’re going with a simple tar.gz archive for now.

$ python setup.py sdist

It immediately creates a new directory called dist/ which contains your source-distribution archives.

$ twine upload dist/*

Notice that we’re trying to push all the distributions. PyPI is clever enough to not allow the duplicate releases to get overridden. So it’ll simply ignore the duplicate distributions.

If your .pypirc is filled with the correct values, your package must be up there by now. If so, you’ll see the success message.

Uploading distributions to https://upload.pypi.org/legacy/
Uploading hello-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 kB • 00:00 • ?
View at:
https://pypi.org/project/hello/0.1.0/

Now, if you run the following command, it’ll reach the hello==0.1.0 version and install the latest release on your machine.

$ pip install hello

Now, you can easily run the hello CLI command that we designed earlier.

$ hello Sadra
Hello Sadra!

No More PyPI

As a quick tip, you can privately store your entire project on platforms like GitHub. Throughout this process, we’ll no longer need PyPI or anything related to this service. Simply push your project to a private/public GitHub repository and install it with the following command:

$ pip install git+https://github.com/username/hello.git

Notice that you should have Git installed on your machine.

Versioning

Versioning your package is a crucial phase as you can publish thousands of releases in different numeric patterns. All you need to have is a proper strategy for keeping them. A good versioning strategy is Semantic Versioning which Python highly suggests for some reasons. If you’re using Poetry, you can use the poetry version command that helps you bump up your package’s version.

Otherwise, if you’re comfortable with the simple procedure, I highly suggest you use a third-party package as it’s safer. You might’ve stored the version number of your package in multiple files and those kinds of packages are smart enough to find them and update them at the same time.

There is a popular package called bump2version that does the same thing. It’s responsible for taking care of your package versioning. Make sure you’re using the LTS fork as a few people have tried working on a stable version of this package. (The original one is deprecated)

Therefore, I prefer this as a proper version-bumping tool in Python. It also accepts a custom config file that allows you to specify the files and patterns.

Publish Automatically

This section is for those who maintain their packages on GitHub. To clear this up, GitHub offers you a free tool called Actions. It’s a nice solution for CI automation and CD pipelines. Imagine you could hit the release button on GitHub, release a new version of your package and there was a task trying to publish your package to PyPI at the same time. Thus, you won’t need to do it on your own as it might not be quite delightful in some circumstances.

There is an open-source action that builds your package, creates a new sdist archive, gets your API token, and uses Twine to push your newly released distribution. The following steps are relevant and needed if you want to automate this procedure.

• Create your repository and push your package to GitHub.
• Create a new .yaml workflow file.
• Configure the workflow and make sure that it gets triggered on a release action.

Create a new .github/workflows/publish.yml file. It’ll contain your CD pipeline.

name: Upload Python Package
on:
release:
types: [published]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: Set up Python
  uses: actions/setup-python@v3
  with:
    python-version: '3.10'
- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install build
- name: Build package
  run: python -m build
- name: Publish new distributions to PyPI
  uses: pypa/gh-action-pypi-publish@release/v1
  with:
    password: ${{ secrets.PYPI_API_TOKEN }}</pre><p>In the last step, we will “publish new distributions to PyPI”. This step takes a parameter called password and as you can see, it’s trying to read the API token that we’ve been talking about. As you can see, it’s being stored as an action secret meaning you have to create a new secret for this workflow execution named PYPI_API_TOKEN and put the API token that you’ve copied from pypi.org as the value of this secret key.</p><p>When you’re creating a new API token on your PyPI dashboard, make sure not to set its scope to ALL as anyone who has access to this token will be able to modify or add new releases to the packages being maintained on the same account specifically when you’re working on a platform like GitHub whereas lots of maintainers might have access to the highly-secured parts of your repository.</p>

Conclusion
In this tutorial, we took a quick look over the structure of a Python package and immediately we dived through the best practices, tips, and tools that would simplify this procedure for you. We also talked about Poetry which is a popular tool that enables you to maintain your Python package from the very beginning step toward deployment and publishing to a third-party service such as PyPI.