Python Development 2023

Python development on Windows

For windows, you can use WinPython. I prefer to use this instead of the official distribution because:

1. It is a portable distro.
2. You can choose for a batteries included, or just the dot release which only contains Python and Pip.

This way I can have multiple versions available. This is particularly useful for me because of my OTC development which uses OpenStack SDK. In my Windows system, I was only able to make it work with Python v3.8.10 (due to some ancient dependancies). Because I usually don't have a compiler, I can only use binary distros and I am unable to find it for a newer Python version.

Distributing Python scripts as single EXE or Directory

When distributing I had good results with pyinstaller. You can create a single folder or a single exe distribution. The results work suprisingly well (if they work). Some hints when using pyinstaller:

• Cross packaging is not possible. If you need a Windows package you need to run it on Windows.
• Dependancies not always work correctly. You may need to use these options:
  • --hidden-import module
  • --collect-data module
  • --copy-metadata module
  • --collect-all package
• In some cases, you may need to force the inclusion of non-python files. Use:
  • set sitedir=%WINPYDIR%\Lib\site-packages
  • And in the command line:
  • --add-data %sitedir%\path\to\data\file;path\to\data
  • I use the %sitedir% variable to find things in the Python packages directory.
• It is best to create a batch file to issue the pyinstaller command.
• Because the command-line could become quite long, you can use the ^ escape. Example:

  pyinstaller %buildtype% ^
    --hidden-import keystoneauth1 ^
    --collect-data keystoneauth1 ^
    --copy-metadata keystoneauth1 ^
    --hidden-import os_service_types ^
    --collect-data os_service_types ^
    --copy-metadata os_service_types ^
    --collect-all openstacksdk ^
    --copy-metadata openstacksdk ^
    --add-data %sitedir%\openstack\config\defaults.json;openstack\config ^
    --hidden-import keystoneauth1.loading._plugins ^
    --hidden-import keystoneauth1.loading._plugins.identity ^
    --hidden-import keystoneauth1.loading._plugins.identity.generic ^
  urotc.py

Installing netifaces on Windows

NOTE: I tested this on Feb 2023. See Original article here

netifaces is a OpenStack SDK dependancy. Under version 3.8.10 I am able to install using --only-binary=netifaces option.

For newer versions it will fail with Microsoft Visual C++ 14.0 is required error message.

C:\RH>pip install netifaces
Collecting netifaces
  Downloading https://files.pythonhosted.org/packages/81/39/4e9a026265ba944ddf1fea176dbb29e0fe50c43717ba4fcf3646d099fe38/netifaces-0.10.7.tar.gz
Installing collected packages: netifaces
  Running setup.py install for netifaces ... error
    Complete output from command c:\users\rh\appdata\local\programs\python\python37\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\RH\\AppData\\Local\\Temp\\pip-install-wbfanly3\\netifaces\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\RONALD~1.HEI\AppData\Local\Temp\pip-record-m26yfbyt\install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'netifaces' extension
        error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

Since the suggested URL doesn't work. You need to do the following:

1. Go to the Microsoft-Repository Tools for Visual Studio 2017 or use the direct link to vs_buildtools.exe.
   • ... it’s about 1.2MB
2. run „vs_buildtools.exe“
   • it downloads ~ 70 MB
3. Select Workloads => Windows => [x] Visual C++ Build Tools“ => [Install]
   • it downloads 1.12 GB
   • and installs
4. Re-boot (I don't know if it is required, but I did it just in case)

Now netifaces can get installed:

C:\RH>pip install netifaces
Collecting netifaces
  Using cached https://files.pythonhosted.org/packages/81/39/4e9a026265ba944ddf1fea176dbb29e0fe50c43717ba4fcf3646d099fe38/netifaces-0.10.7.tar.gz
Installing collected packages: netifaces
  Running setup.py install for netifaces ... done
Successfully installed netifaces-0.10.7

Documentation generation

When programming documentation is important, allthough very often it takes a back seat.

To help keep it up to date, it is good to make it so it is easier to maintain and update. One way to do that is with keeping documentation and code together, and automating the way documentation is generated.

There is a number of solutions to do this. The one I looked at the most were:

• mkdocs with mkdocstrings : which is nice because it uses markdown, however, because it is essentially a static site generator, a lot of things needed to be done manually.
• sphinx : At the end, sphinx was the option that I liked the most. It uses RST for markup which is different from markdown, but it is close enough. Also, sphinx can also support markdown via some extensions but I did not try that.

Using sphinx for documentation

Prepare your environment:

pip install sphinx sphinx-argparse

In your project directory, I have two folders:

• docs : where the documentation source resides
• src : where the python code resides

Also, I ignore:

• public : where the generated documentation is created. This can then be added into a CI pipeline to publish documentation.

Run:

sphinx-quickstart

In the docs folder, to initialize things. This will create the files:

• conf.py
• Makefile
• index.rst

Modify conf.py to:

• include the source:

  sys.path.insert(0, os.path.abspath('../src'))

• Customize project meta data
• Include enable desired extensions. For max automation I enable:
  • sphinx.ext.autodoc
  • sphinx.ext.autosummary

Modify Makefile to run:

sphinx-apidoc -o apidoc ../src

This command extracts from python docstrings and creates the relevant rst files.

For command line arguments, I use arparse extension. Create a cli.rst like so:

CLI
===

.. argparse::
   :filename: ../src/cli.py
   :func: cli_args
   :prog: cli.py

Where cli.py contains a function cli_args that returns an ArgumentParser object.

Example docstring

This is an example doc string to add to your code, after the element declaration:

   '''
   Summary text

   Description of the function

   :param str argname: argument passed
   :returns bool: Returns a boolean True on success, False on failure
   '''

No Need to make it too complicated.

Passing Reserved Keywords as Keyword arguments

Very often when using wrapped APIs, that used functions with Keyword arguments that you would need to pass reserved keywords (such as class, or import) as function keywords.

Of course, this is NOT allowed in python. And you will get an error like:

SyntaxError: Invalid syntax

To work around that, you need to place those keywords in a dictionary and use ** notation. So instead of:

response = client.service.SendSMS( toNum = '0666666666666',
                pass = '123456'}
           )

you would:

response = client.service.SendSMS( toNum = '0666666666666',
                **{'pass': '123456'}
           )