0ink.net

Ansible Custom Action Plugins

2025-09-19T00:00:00+02:00

This is a continuation to my Ansible custom modules article.

Action plugins let you integrate local processing and local data with module functionality.

At the time of this writing, documentation I could find was Developing plugins - Action plugins which was somewhat out-of-date.

Introduction

Ansible action plugins are Python-based extensions that run on the control node, allowing you to intercept and modify the behavior of tasks before they're executed on remote hosts. Unlike regular modules, which run remotely, action plugins give you local control over task execution logic, argument manipulation, and result handling.

They’re especially useful when:

You need to preprocess arguments or include local data before calling a module.
You want to combine multiple modules into a single task.
You’re working in environments with limited remote capabilities, or need local-only logic.

Each plugin is a class that inherits from ActionBase, and typically overrides the run() method. Inside this method, you can use _execute_module() to call the actual module, or bypass it entirely for local-only tasks.

Motivation

While building an Ansible role for nftables, incorporating ideas from Frzk’s nftables role and kormat’s iptables-apply to handle recovery from new rulesets that would break or hang Ansible's control connection it became challeging to use YAML to manage control flow, error handling, and connection state.

By writing a custom action plugin, I could:

Run logic locally on the control node
Use Python to apply rules and handle failures gracefully
Keep playbooks declarative, while isolating recovery logic in code

This gives a cleaner, more robust way to experiment with firewall changes without compromising automation reliability.

Where to place Action Plugins

Custom action plugins can be placed in several locations depending on your project structure and scope:

Role or Collection scope

Inside a role:
Place the plugin in roles/<role_name>/action_plugins/.
This makes it available only when the role is used.
Inside a collection:
Use plugins/action/ within your collection directory.
This is ideal for distributing reusable plugins.

User or System Scoped

User-level plugin directory:
~/.ansible/plugins/action/
This makes the plugin available across all playbooks for the current user.
System-level plugin directory:
/usr/share/ansible/plugins/action/
Useful for global availability across users and projects.

Configuring Plugin Paths

To make Ansible recognize custom plugin locations, you can configure the plugin path:

Via ansible.cfg
Add or modify the following in your ansible.cfg file:
```
[defaults]
action_plugins = ./action_plugins:~/.ansible/plugins/action:/usr/share/ansible/plugins/action
```
- Paths are colon-separated.
- Relative paths (like ./action_plugins) are resolved from the playbook directory.
Via Environment Variable
Set the ANSIBLE_ACTION_PLUGINS environment variable:
```
export ANSIBLE_ACTION_PLUGINS=./action_plugins:~/.ansible/plugins/action
```
This overrides the config file setting and is useful for temporary or dynamic setups.

You can inspect your current plugin paths using ansible-config dump | grep ACTION_PLUGIN_PATH.

Basic layout

#!/usr/bin/python
# Make coding more python3-ish, this is required for contributions to Ansible
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

# Base class
from ansible.plugins.action import ActionBase

class ActionModule(ActionBase):
  def run(self, tmp=None, task_vars=None):
    if task_vars is None: task_vars = dict()
    result = super().run(tmp, task_vars)
    del tmp  # tmp no longer has any effect

    # Do something here...

    return result

This is the simplest action plugin. It declares itself as a sub-class of ActionBase, and populates the task_vars structure with values and initializes result with some defaults.

Error handling

Ansible leverages Python's exception system to handle errors cleanly. Within your custom action plugin, you can raise exceptions to signal failure conditions. Start by importing the relevant classes:

from ansible.errors import AnsibleError, AnsibleFileNotFound, AnsibleAction, AnsibleActionFail

You can also use Python’s built-in exceptions if preferred.

To raise an error:

raise AnsibleFileNotFound('thisfile.txt')

By default, Ansible displays a simplified error message. To see the full stack trace, run your playbook with increased verbosity:

ansible-playbook playbook.yml -vvv

This is especially useful when debugging plugin logic or tracing failures deep in the call stack.

Diagnostics and Verbose output

Custom action plugins often need to communicate status or debug information. Ansible provides the Display class for structured output:

from ansible.utils.display import Display
display = Display()

You can then emit messages based on verbosity level:

if display.verbosity > 1:
    display.display('Verbose message')

Or use the built-in verbosity helpers:

display.v('Level 1 verbosity')
display.vv('Level 2 verbosity')
display.vvv('Level 3 verbosity')

Control verbosity from the command line using -v, -vv, or -vvv. This lets users opt into more detailed diagnostics without cluttering standard output.

Run-time environment

When writing custom action plugins, it is often useful to inspect the execution context; whether you're debugging behavior, adapting to playbook settings, or resolving file paths. Ansible exposes several runtime variables and constants that help you do just that.

Start by importing the constants module:

import ansible.constants as C

Here are some useful runtime indicators and paths:

Execution Context Flags

self._task.diff
Indicates if the task is running in diff mode (--diff).
self._task.check_mode
True if the task is running in check mode (--check).
self._task.no_log
True if the task or play has enabled the no_log flag.
self._task.role
A string with the current role, otherwise None.

Path Resolution

self._task.get_search_path()
Returns the search path used to resolve relative file names.
C.DEFAULT_ROLES_PATH
Default path(s) where Ansible looks for roles.
C.COLLECTIONS_PATHS
Path(s) used to locate collections.
C.PLAYBOOK_DIR
Directory of the current playbook being executed.

Remote File Cleanup

C.DEFAULT_KEEP_REMOTE_FILES
Controls whether temporary files on remote hosts are cleaned up.
You can enable this by setting the environment variable:
```
export ANSIBLE_KEEP_REMOTE_FILES=1
```

These variables are especially helpful when building plugins that need to behave differently based on execution mode or when resolving paths dynamically.

Accessing task parameters

Action plugins receive task parameters as a dictionary via self._task.args. These are the arguments defined in the playbook for the task invoking your plugin.

params = self._task.args

You can interact with this dictionary using standard Python methods like .get() or direct key access.

Note: Any parameters containing Jinja2 templates are rendered before the plugin is executed. This means you’ll receive fully evaluated values, not raw template strings.

This makes it easy to write plugins that behave predictably, without needing to manually resolve templating logic.

More Task Context via `task_vars`

In addition to parameters passed directly to your action plugin via self._task.args, Ansible provides a broader context through the task_vars dictionary. This contains all variables available to the task at runtime, including:

Inventory & Host Variables
- Variables from host_vars and group_vars
- inventory_hostname, group_names, ansible_play_hosts
Facts
- Gathered facts like ansible_distribution, ansible_interfaces, ansible_facts
- Custom facts from setup or set_fact
Plabook & Role Context
- Variables defined in vars, vars_files, or vars_prompt
- Role defaults and vars
- Tags: ansible_run_tags, ansible_skip_tags
Execution Flags
- ansible_check_mode: Whether the task is running in check mode
- ansible_diff_mode: Whether diff mode is enabled
- ansible_dependent_role_names: List of dependent roles
User-defined Variables
- Variables passed via --extra-vars
- Any custom values defined earlier in the play

From your run method:

hostname = task_vars.get('inventory_hostname')
os_family = task_vars.get('ansible_os_family')
debug_flag = task_vars.get('my_custom_flag', False)
raw_value = task_vars.get('my_inventory_value')

Unlike self._task.args, which contains pre-rendered values, entries in task_vars may still include raw Jinja2 expressions, especially when variables are defined in inventory or playbooks using templating syntax.

To evaluate these expressions inside your plugin, use the templar:

my_value = self._templar.template(task_vars.get('my_inventory_value'))
os_family = self._templar.template(task_vars.get('ansible_os_family'))

This ensures your plugin works with fully resolved values, allowing it to behave consistently regardless of how variables were defined.

You can also use template() on dictionaries or lists to resolve nested structures.

Implementing check mode

Similar to custom modules, action plugins support check mode. When the user passes the --check flag, this will set the flag self._task.check_mode to True. This means the playbook is running in dry-run mode. See Ansible check mode.

When in check mode, the Action plugin must simulate changes without actually applying them to the system. Modules that support check mode, must report what would change or simply skip execution.

This is useful for:

Validating playbooks before deployment
Auditing potential changes
Testing custom modules or logic safely

Implementing diff mode

Check mode is often paired with the --diff command line argument. This will set the self._task.diff to True. When this is the case, Action plugins must show the before-and-after changes, especially those that modify files or configurations.

A quick example:


if self._task.diff and result['changed']:
  if not 'diff' in result: result['diff'] = list()
  result['diff'].append({
        'before_header': 'hdr1 (original)',
        'before': 'before\n',
        'after_header': 'hdr1 (updated)',
        'after': 'after\n',
        'prepared': '>>> Preparation',
  })

In this example, we are creating a list for the diff key. You can return multiple diff sets. If you are only returning one (1) diff set, you can simply return it directly (without wrapping it inside a list).

You can return at least before,after pair, or a prepared key. Returning all three is possible. before_header and after_header are optional.

Normally I would return before and after for when configuration files are being changed. I would use the prepared key for when we are running commands on the system. For example, running mkfs, lvcreate, etc.

Template Rendering

When rendering templates from files within an action plugin, the following pattern ensures correctness and compatibility with Ansible’s templating engine:

Import the required dependency:
```
from ansible.template import Templar
```
Locate the template file:
```
template_path = self._task.args.get('template')
srcfile = self._find_needle('templates', template_path)
```
This searches for the template in the configured templates/ directories and resolves the full path.

Adjust the loader's basedir to support {% include %} and relative paths:

old_basedir = self._loader.get_basedir()
self._loader.set_basedir(os.path.dirname(srcfile))

Render the template:

with open(srcfile, 'r') as fp:
    template = fp.read()

tp = Templar(loader=self._loader, variables=task_vars)
rendered = tp.template(template)

Restore the original basedir to avoid side effects:
```
self._loader.set_basedir(old_basedir)
```

Notes:

Always reset the loader's basedir after rendering to prevent unexpected behavior in subsequent tasks.
Using Templar directly gives you full control over variable resolution and template rendering, including support for Ansible-specific filters and syntax.

Calling Modules from Action Plugins

Action plugins often serve as orchestrators, combining logic, templating, and conditional behavior before delegating actual work to modules. To invoke a module from within an action plugin, use the _execute_module() method, which handles serialization, transport, and return parsing.

Example: Delegating to a Custom Module

mod_args = {
    'dest': dest,
    'files': '\n'.join(flist),
    'ro_check': ro_check if ro_check else '',
}

mod_return = self._execute_module(
    module_name='mab_prune',
    module_args=mod_args,
    task_vars=task_vars
)

Key Concepts

module_name: This should match the name of the module file (without .py) and be resolvable via the plugin loader. If you're calling a module from your own collection, use the fully qualified name (e.g. my_namespace.my_collection.mab_prune).
module_args: A dictionary of arguments passed to the module. These should be flat and serializable. Avoid nested structures unless your module explicitly supports them.
task_vars: Pass the current task variables to ensure proper templating and context resolution. This includes facts, host vars, and role defaults.

Return Value

The result of _execute_module() is a dictionary that mimics the return from a normal module execution. It includes keys like:

changed: Boolean indicating whether the module made changes.
failed: Boolean indicating failure.
msg: Optional message string.
Any custom return values defined by your module.

You can use this return to control flow, emit diagnostics, or raise errors:

if mod_return.get('failed'):
    raise AnsibleActionFail("Module execution failed: %s" % mod_return.get('msg'))

Best Practices

Avoid side effects before module execution: If your plugin modifies state before calling the module, ensure those changes are reversible in case the module fails.
Respect check mode: If your plugin supports check mode, ensure the module you're calling does too—or skip execution when self._play_context.check_mode is True.
Use Display.vvv() for debugging: Log the module arguments and return values at verbosity level 3+ to aid troubleshooting.

Return values

Action plugins return a dictionary that mirrors the structure of module results. This ensures consistency across tasks and allows Ansible to interpret outcomes correctly.

Common Keys

changed (bool): Indicates whether the task made changes to the system. Set this to True only when a state transition occurred.
failed (bool): Signals task failure. If True, Ansible halts execution unless ignore_errors is set.
msg (str): A human-readable message describing the outcome or error.
skipped (bool): Marks the task as intentionally skipped, often used with conditional logic.
invocation (dict): Automatically populated metadata about how the plugin was called. Use self._get_invocation(task_vars) to include it.

Custom Return Values

You can include additional keys specific to your plugin or module logic:

return {
    'changed': True,
    'pruned_files': pruned,
    'skipped_files': skipped,
    'msg': "Pruned %d files" % len(pruned),
    'invocation': self._get_invocation(task_vars),
}

For more common return values see Common Return values.

Conclusion

Action plugins offer deep control over task execution, making them ideal for implementing custom logic, enforcing policy, and handling edge cases like connection loss or remote cleanup. But with that control comes complexity; especially around templating, context flags, and runtime behavior.

This article outlined practical patterns for building reliable, maintainable plugins: from placement and layout to diagnostics, error handling, and module delegation. The goal is not just flexibility, but predictability -- plugins that behave consistently across environments, expose their decisions clearly, and fail gracefully when needed.

In automation, predictability builds trust. And trust is what lets teams move fast without breaking things.

Ansible Custom Modules

2025-09-15T00:00:00+02:00

Intro
What is a module
Where to store modules
Writing modules
Sample Shell module
JSON input
Additional parameters
check_mode
ansible_diff
Generating JSON output
Final thoughts

Intro

Ansible is at its best when it lets you declare what a system should look like, not micromanage how to get there. Its built-in modules cover most common UNIX configuration tasks, such as editing files, managing services, installing packages, etc.

However, some scenarios demand more than that. For example, when configuring storage subsystems, orchestrating complex workflows, or dealing with edge cases that require precise command execution, playbooks alone can get messy fast.

That’s where custom modules come in. They let you encapsulate logic, handle complexity cleanly, and keep your roles declarative and maintainable. Whether you're scripting disk array setup or wrapping a cloud API call, writing your own module gives you control, clarity, and reusability.

What is a module

At the heart of Ansible’s automation engine is the module. Modules are standalone, reusable script that performs a specific task. Modules are the building blocks behind nearly every Ansible action, whether you're installing packages, managing users, configuring services, or interacting wiht other resources.

Each module defines a clear interface: it accepts arguments, executes logic, and returns structured output (usually as JSON) back to Ansible. Modules are executed by Ansible on behalf of the user, and can interact with the target system, an API, or other resources depending on the task.

Where to store modules

When you write your own Ansible module, you have a few options for where to place it, depending on how you want to organize and reuse it:

Inside a Role

You can store the module in a role’s library/ directory:

roles/
  my_role/
    library/
      my_module.py

Ansible automatically loads modules from this path when the role is used in a playbook. This is ideal for role-specific logic that doesn't need to be shared across multiple projects.

Inside a Collection

For broader reuse and better packaging, custom modules can live inside a collection:

collections/
  ansible_collections/
    my_namespace/
      my_collection/
        plugins/
          modules/
            my_module.py

This structure allows you to version, distribute, and document your module cleanly. It also integrates with Ansible Galaxy and ansible-galaxy install.

Custom Path via `ansible.cfg` or Environment Variable

If you’re testing or prototyping, you can place your module in any directory and tell Ansible where to find it by setting:

[defaults]
library = ./custom_modules

Or by exporting the ANSIBLE_LIBRARY environment variable:

export ANSIBLE_LIBRARY=./custom_modules

When Ansible runs a task that uses your module, it searches these paths in order—starting with role-local library/, then collection plugins, then any configured custom paths. Once found, the module is copied to the target host (unless you’ve delegated execution to localhost) and executed with arguments passed via a temporary JSON file.

Writing modules

When writing custom Ansible modules, Python is the primary and officially supported language. It's tightly integrated with Ansible's internals and benefits from the AnsibleModule helper class, which simplifies argument parsing, error handling, and JSON output.

That said, other languages can technically be used, but with caveats:

Recommended: Python

Full support via ansible.module_utils.basic.AnsibleModule
Access to Ansible’s plugin APIs and utilities
Clean integration with roles, collections, and documentation
Easy to test and debug using ansible-test

Possible but Uncommon: Other Languages

You can write modules in any language that:

Can read JSON input from a temporary file.
Can write JSON output to stdout
Can exit with a proper return code

Using Shell script

Using Shell over the recommended Python for Ansible modules may be a better fit for:

Re-using existing Shell scripts
If you've built up a library of reliable shell scripts over time, reusing that logic inside Ansible modules saves effort and reduces risk. You don't need to reimplement or debug complex workflows in Python when they already work in shell.
Target Systems Don’t Have Python
Minimalist systems—like Alpine Linux, embedded devices, or hardened containers—often lack Python by default. Shell is universally available, making it the most portable option for modules that need to run directly on the managed node.
Closer to the System
Shell is ideal for tasks that interact directly with system utilities: nft, ip, vgcreate, mount, systemctl, and so on. These tools are designed to be used from the shell, and wrapping them in Python often adds unnecessary complexity.
Fast Prototyping
Shell scripts are quick to write, test, and iterate. If you’re building a module for a specific role or environment, shell lets you move fast without worrying about Python packaging, dependencies, or module APIs.
Minimal Dependencies
Shell modules don't require external libraries or Python environments. Tools like jq can help with, but that can be optional for simple use cases.

Shell isn't ideal for everything—error handling, argument validation, and testing are more limited than in Python, but for infrastructure tasks that are already shell-native, it's often the most efficient and maintainable choice.

Sample Shell module

#!/bin/sh
. "$1"

# something happens here

jq -Mn '$ARGS.named' \
    --argjson changed false \
    --arg msg "Hello world" \
    --argjson ansible_facts "$(jq -n '$ARGS.named' \
      --arg uptime "$(uptime)"
    )" \
    --arg input "$(cat "$1")"

This is a sample skeleton file, which does nothing but is a complete Ansible module.

. "$1"
Parses the input parameters. By default, Ansible assume legacy mode, so we can just source the input file as it will contain key=value pairs.
This means that simple flat parametes can be passed directly. Use WANT_JSON for more complex structures, but this adds a dependancy on jq for the JSON parsing.
jq ...
This generates a JSON response.
While we are using jq to generate the JSON response, this is not necessary. As long as you output valid JSON, you can do it simply with shell code, for example:
```
echo '{ "changed": false }'
```
The response should at least contain:
- changed : boolean, true or false depending on the outcome of the script.

In the example, we are also returning:

msg : Most modules seem to return this explaining what happened.
ansible_facts : Used for facts modules to return information on the host.

Other return values:

failed : boolean,
return true if the task failed.
skipped: boolean, return true if the task was skipped.

My convention is that failed is when there was an error, whereas skipped is more for when the task can not be executed.

For example, typically I use skipped when implementing check mode. Very often a play would have a task that installs package dependancies. Obviously if we are running in check mode, the package would not be installed. Rather than fail the play by returning failed: true as normally would do, if the module is running in check mode I would return skipped.

JSON input

For some scenarios, you may need to pass more complex paramters. You can easily achieve this by declaring your custom module to want JSON:

#!/bin/sh
# WANT_JSON

echo '{"changed": false}'

By having the string WANT_JSON around the start of the file, this tells Ansible that this module accepts JSON input.

Additional parameters

In addition to the parameters in the playbook, the following parameters are send to the module:

_ansible_check_mode : boolean, if true, check mode is being used (cli option: --check)
_ansible_diff : boolean, if true diff mode is being used (cli option: --diff)
_ansible_verbosity : int, cli options: -v, -vv or -vvv.
_ansible_keep_remote_files: boolean, environment ANSIBLE_KEEP_REMOTE_FILES=1.
_ansible_version: str, example: 2.18.8
_ansible_no_log : boolean
_ansible_debug : boolean
_ansible_version: str, example: 2.18.8
_ansible_module_name: str
_ansible_syslog_facility : str
_ansible_selinux_special_fs
_ansible_string_conversion_action : enum
_ansible_socket : null
_ansible_shell_executable : str
_ansible_tmpdir : str
_ansible_remote_tmp : str
_ansible_ignore_unknown_opts : boolean

check_mode

When modules are passed the parameter _ansible_check_mode=True, this means the playbook is running in dry-run mode. See Ansible check mode.

When in check mode, the module must simulate changes without actually applying them to the system. Modules that support check mode, must report what would change or simply skip execution.

This is useful for:

Validating playbooks before deployment
Auditing potential changes
Testing custom modules or logic safely

ansible_diff

When modules are passed the parameter _ansible_diff=True, it must show the before-and-after changes, especially those that modify files or configurations.

Typically, this is combined with check mode to preview changes without applying them.

To support this, you must include a diff key in the return JSON object:

diff: [{
    "before_header": "/etc/my_config.ini (original)",
    "before": "line1 = old content\n",
    "after_header": "/etc/my_config.ini (original)",
    "after": "line1 = new content\n",
    "prepared": ">>> reload apache",
}]

Not shown in the example, but you must return "changed": true, otherwise the diff sets will not be processed.

You can return at least before,after pair, or a prepared key. Returning all three is possible. before_header and after_header are optional.

Generating JSON output

While you can use echo statements to generate the JSON output, to simplify things you can use the jq command instead. Example:

jq -Mn '$ARGS.named' \
    --argjson changed false \
    --arg msg "Hello world" \
    --argjson ansible_facts "$(jq -n '$ARGS.named' \
      --arg uptime "$(uptime)"
    )"

In this example we are using:

-M or --monochrome-output : This prevents jq to use ANSI escape sequences to colorize its output.
-n or --null-input : Skip reading any input
'$ARGS.named' : outputs the arguments specified in the commnad line
--argjson key value : define the key as a valid json value. This can be a full json structure, but most commonly is used to define booleans or numbers.
--arg key value : define the key as the string value.

Final thoughts

While Python remains the default language for Ansible module development, shell scripting offers a pragmatic alternative, especially in environments where Python isn't readily available or where existing shell logic is already battle-tested. By understanding how Ansible passes arguments, handles check and diff modes, and expects structured output, you can write shell-based modules that integrate cleanly into your automation workflows without sacrificing clarity or control.

Whether you're managing minimalist systems, reusing hardened scripts, or simply prefer the transparency of shell, custom modules give you the flexibility to tailor Ansible to your infrastructure, not the other way around.

Using sphinx for doc generation

2025-07-11T00:00:00+02:00

Introduction
Basic functionality
Sample Usage
Generating documentation
Publish to gh-pages
Conclusion

I have started using sphinx for documention generation.

I do find sphinx quite prickly with the slightest mistake breaking things and error messages that are very uninformative or confusing.

Nevertheless, it is quite close to feature complete.

Introduction

Sphinx is a documentation generator written and used by the Python comunity. It is written in Python, but it can be also used to document projects written in other languages.

By default, it uses reSturcutredText but it there are extension to allow for Markdown compatibility.

It can generate output in multiple formats. Among others it supports:

HTML (in different configurations)
EPub
Man pages

Basic functionality

If you are using the default functionality you only really need the sphinx Python package.

In this scenario, you must write all your documentation in reStructuredText and use apidoc extension. Note, there is also an extension named autodoc. This is NOT the one you want, as it requires that you manually write reStructuredText files pointing to your source code. The apidoc extension, makes use of the autodoc extension and creates any needed reStructuredText files that may be needed. The limitation of apidoc is that it is hard-coded to always generated reStructuredText, so it can not be used for Markdown documentation.

Sample Usage

As mentioned earlier, sphinx is quite prickly, so this is the configuration I arrived at after a lot of trial and error.

sphinx has a sphinx-quickstart command which is meant to set things up for you. I prefer to do this manually.

First of all, I assume that you have a project directory with a structure like this:

your_package
│── your_package/                # Main package directory
│   ├── __init__.py              # Makes it a Python package
│   ├── module.py                # Your actual Python code
│── setup.py                      # Installation configuration
│── requirements.txt              # Optional: Dependencies list
│── README.md                     # Optional: Project info

To this I add a directory docs so the structure looks like:

your_package
│── docs/                        # Sphinx documentation directory
│── your_package/                # Main package directory
│   ├── __init__.py              # Makes it a Python package
│   ├── module.py                # Your actual Python code
│── setup.py                      # Installation configuration
│── requirements.txt              # Optional: Dependencies list
│── README.md                     # Optional: Project info

For document generation we require the following Python packages:

docutils : not really needed as it is a dependancy for sphinx
sphinx : main documentation generator
sphinx-autodoc2 : automatic documentation generation from source code
myst-parser : Markdown parser for sphinx
linkify-it-py : Dependancy for myst, adding convenient auto URL conversion.
sphinx-argparse : Used to document command line parsers

For convenience, I add this to my docs/requirements.txt file, since these are dependancies for doc generation and not for the package itself.

In docs, I create the files: Makefile, conf.py, index.md, make.bat.

docs/Makefile and docs/make.bat

The main items to pay attention in these two files are:

SOURCEDIR = .
This tells sphinx that current directory (docs) is where the documentation source files are.
BUILDDIR = _build
Sets-up the direction _build to be the output directory.

It is a good idea to add _build to the .gitignore file in docs.

docs/Makefile

docs/make.bat

docs/conf.py

This is where most of the sphinx relevant settings reside. This is what I am using:

This config file is loading version information from the module itself. Other meta-data can be set here.

The following extensions are enabled:

autodoc2 : generate documentation from doc strings
myst_parser : Markdown
sphinxarg.ext : Generate documentation from argument parser
sphinx.ext.doctest : Enable doctest unit tests.

autodoc2

autodoc2_packages : configures what source directories to find doc strings
autodoc2_render_plugin = 'myst' : We use markdown
autodoc2_sort_names = True : keeps names sorted. Otherwise the order is random.
autodoc2_hidden_objects = {'inherited','private'} : hides inherited classes and names that begin with a single underscore.

myst_parser

myst_enable_extensions : Enable extensions:
- fieldlist : Enables the use of :key value: in doc strings
- linkify : Converts raw URLs into links
- substitution : Lets you configure Jinja2 style substitutions. See: myst_substitutions.
- strikethrough : Enables ~~ markup.

sphinx.ext.doctest

Enables unit tests in documentation. Use:

doctest_global_setup and
doctest_global_cleanup to configure the execution environment

docs/index.md

This is the main page for the generated HTML documentation. Example contents:

In this example, we are using Jinja2 style substitutions. These are defined in the conf.py file in the myst_substituions dictionary. Then, they are refered in the document as {{ key }}.

The {toctree} structure is used to create a navigation tree.

The files listed there are the entry points for the documentation. These files are created as Markdown or possibly reStructuredText and reference here.

Important to note is the apidocs/index. This file is generated automatically by the autodoc2 extension.

In this example, we are also referencing a file cli.md. We are using this to document command line arguments.

docs/cli.md

This file is used by index.md and is used in the command line documentation.

Example content:

See argparse usage on more details on how argparse is used.

Docstrings

In your source code you can then use docstrings. Here is an example:

The example uses:

`:param type varname: to document parameters to the function
`:returns: to document the return value
It makes use of doctest to document examples and/or execute tests.

Generating documentation

In the docs directory you can:

make html : to generate HTML documentation. This will be placed in _build/html.
make man : To generate a man page. This will be placed in _build/man

Publish to gh-pages

If your project is hosted on [github][gh], you can publish the documentation directly to github-pages using a [github-action][action].

Enable this by navigating to your project settings. Click on Pages on the left sidebar. Change the Source to GitHub Actions.

Create the following workflow in .github/workflows/gh-pages.yml.

on: section: what triggers the workflow to run
- pushing to the default branch
- workflow_dispatch let's you manually run the workflow from the Web UI's actions tab.
Jobs:
- build: Use sphinx to generate documentation
  - Checkout : Uses fetch-depth: 0 to make sure we get tag data.
  - Setup Python : sets up python, enabling pip caching.
  - Install dependancies : uses the docs/requirements.txt file.
  - Generate sphinx documentation : Run sphinx with the HTML target
  - Afterwards, an artifact containing the web site is generated
- deploy: This job is only run from the default branch and actually deploys the web site to github-pages.

After the first succesful run, clock on the gear widget next to your About repository web part. Click Use your GitHub Pages website.

Conclusion

This article covers using sphinx as a document generator for Python projects and the eventual publishing of such documentation to github pages. This particular tutorial is geared towards using Markdown as the markup dialect instead of reStructured Text as I personally find Markdown easier to write.

javascript snippets

2025-05-12T00:00:00+02:00

Compute how an element is hidden
Iterate over LI elements
Adding/removing classes to/from elements
Get current date as YYYY-MM-DD
escape HTML characters
Copy to Clipboard

Compute how an element is hidden

Check if element is hidden

function isHidden(el) {
    var style = window.getComputedStyle(el);
    return (style.display === 'none')
}

Iterate over LI elements

<ul id="foo">
  <li>First</li>
  <li>Second</li>
  <li>Third</li>
</ul>

you can access the list items this way,

var ul = document.getElementById("foo");
var items = ul.getElementsByTagName("li");
for (var i = 0; i < items.length; ++i) {
  // do something with items[i], which is a <li> element
}

Adding/removing classes to/from elements

This snippet will add class foo to all the elements with spa class:

var items = document.querySelectorAll(".spa");
items.forEach(item => {
  item.classList.add("foo");
});

This snippet will remove class foo from all the elements with spa class:

var items = document.querySelectorAll(".spa");
items.forEach(item => {
  item.classList.remove("foo");
});

Get current date as YYYY-MM-DD

function getCurrentDate() {
    const today = new Date();
    const year = today.getFullYear();
    const month = String(today.getMonth() + 1).padStart(2, '0'); // Months are zero-based
    const day = String(today.getDate()).padStart(2, '0');

    return `${year}-${month}-${day}`;
}

escape HTML characters

function escapeHTML(str) {
    var div = document.createElement('div');
    div.appendChild(document.createTextNode(str));
    return div.innerHTML;
}

Of course, if you want to add strings to HTML you can just:

document.getElementById('element-id').textContent = str;

Which will automatically escape HTML.

Copy to Clipboard

function copyToClipboard(textToCopy) {
  var tempElement = document.createElement("textarea");
  tempElement.value = textToCopy;
  document.body.appendChild(tempElement);
  tempElement.select();
  document.execCommand("copy");
  document.body.removeChild(tempElement);
}

Creates a temporary textarea element, sets its value to the string you want to copy, and appends it to the document body.
The function then selects the text inside the temporary textarea, executes the copy command, and removes the textarea from the document body.

The reason we temporarily add a textarea element to the document is to leverage the browser’s built-in copy functionality, which typically works only on user-selected text or content within a focusable element like an input or textarea.

This workaround is necessary because document.execCommand('copy') relies on the text being selected within an editable field, and a hidden textarea is a simple way to fulfill that requirement. Without adding it to the document, the text wouldn't be part of the DOM, and the copy command wouldn’t work.

Additional OpenSSL tips

2025-05-12T00:00:00+02:00

Self-signed certificates
Display cert extensions
Viewing certificate information
Checking server certificate
How to create a certificate chain ?
Converting a .pem certificate to pkcs12
New Apple requirements since 2023
References

So the last few articles we have been exploring the world of certificate authorities (CA). This is an addendum to that covering simple one off tasks that are useful in the context certificates.

Self-signed certificates

Obviously, if you have a CA, you shouldn't need a self-signed certificate. However, being able to issue self-signed certificates can be useful specially in test scenarios.

This is the command I use for this:

    openssl req -newkey rsa:4096 \
            -x509 \
            -sha256 \
            -days 3650 \
            -nodes \
            -out "$fqdn.crt" \
            -keyout "$fqdn.key" \
            -subj "/CN=$fqdn" \
            -addext "subjectAltName=DNS:*.api.$fqdn,DNS:www.$fqdn,IP:10.0.0.1"

Replace $fqdn with the full domain name.

This creates a pair of files $fqdn.crt and $fqdn.key containing the certificate and key pair to add to your web server configuration.

The -subj "/CN=$fqdn" refers to the main name for the server. For example: example.com.

The -addext is if you want alternative names for the certificate. Usually this is useful for implementing wildcard certificates. It is optional.

Display cert extensions

$ openssl x509 -purpose -in root_ca.pem

Certificate purposes:
SSL client : Yes
SSL client CA : Yes
SSL server : Yes
SSL server CA : Yes
Netscape SSL server : Yes
Netscape SSL server CA : Yes
S/MIME signing : Yes
S/MIME signing CA : Yes
S/MIME encryption : Yes
S/MIME encryption CA : Yes
CRL signing : Yes
CRL signing CA : Yes
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : Yes

Viewing certificate information

This is done using the x509 command:

$ openssl x509 -in RapidSSLCA.pem -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 145105 (0x236d1)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=GeoTrust Inc., CN=GeoTrust Global CA
        Validity
            Not Before: Feb 19 22:45:05 2010 GMT
            Not After : Feb 18 22:45:05 2020 GMT
        Subject: C=US, O=GeoTrust, Inc., CN=RapidSSL CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:c7:71:f8:56:c7:1e:d9:cc:b5:ad:f6:b4:97:a3:
                    fb:a1:e6:0b:50:5f:50:aa:3a:da:0f:fc:3d:29:24:
                    43:c6:10:29:c1:fc:55:40:72:ee:bd:ea:df:9f:b6:
                    41:f4:48:4b:c8:6e:fe:4f:57:12:8b:5b:fa:92:dd:
                    5e:e8:ad:f3:f0:1b:b1:7b:4d:fb:cf:fd:d1:e5:f8:
                    e3:dc:e7:f5:73:7f:df:01:49:cf:8c:56:c1:bd:37:
                    e3:5b:be:b5:4f:8b:8b:f0:da:4f:c7:e3:dd:55:47:
                    69:df:f2:5b:7b:07:4f:3d:e5:ac:21:c1:c8:1d:7a:
                    e8:e7:f6:0f:a1:aa:f5:6f:de:a8:65:4f:10:89:9c:
                    03:f3:89:7a:a5:5e:01:72:33:ed:a9:e9:5a:1e:79:
                    f3:87:c8:df:c8:c5:fc:37:c8:9a:9a:d7:b8:76:cc:
                    b0:3e:e7:fd:e6:54:ea:df:5f:52:41:78:59:57:ad:
                    f1:12:d6:7f:bc:d5:9f:70:d3:05:6c:fa:a3:7d:67:
                    58:dd:26:62:1d:31:92:0c:79:79:1c:8e:cf:ca:7b:
                    c1:66:af:a8:74:48:fb:8e:82:c2:9e:2c:99:5c:7b:
                    2d:5d:9b:bc:5b:57:9e:7c:3a:7a:13:ad:f2:a3:18:
                    5b:2b:59:0f:cd:5c:3a:eb:68:33:c6:28:1d:82:d1:
                    50:8b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier: 
                6B:69:3D:6A:18:42:4A:DD:8F:02:65:39:FD:35:24:86:78:91:16:30
            X509v3 Authority Key Identifier: 
                keyid:C0:7A:98:68:8D:89:FB:AB:05:64:0C:11:7D:AA:7D:65:B8:CA:CC:4E

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 CRL Distribution Points: 
                URI:http://crl.geotrust.com/crls/gtglobal.crl

            Authority Information Access: 
                OCSP - URI:http://ocsp.geotrust.com

    Signature Algorithm: sha1WithRSAEncryption
        ab:bc:bc:0a:5d:18:94:e3:c1:b1:c3:a8:4c:55:d6:be:b4:98:
        f1:ee:3c:1c:cd:cf:f3:24:24:5c:96:03:27:58:fc:36:ae:a2:
        2f:8f:f1:fe:da:2b:02:c3:33:bd:c8:dd:48:22:2b:60:0f:a5:
        03:10:fd:77:f8:d0:ed:96:67:4f:fd:ea:47:20:70:54:dc:a9:
        0c:55:7e:e1:96:25:8a:d9:b5:da:57:4a:be:8d:8e:49:43:63:
        a5:6c:4e:27:87:25:eb:5b:6d:fe:a2:7f:38:28:e0:36:ab:ad:
        39:a5:a5:62:c4:b7:5c:58:2c:aa:5d:01:60:a6:62:67:a3:c0:
        c7:62:23:f4:e7:6c:46:ee:b5:d3:80:6a:22:13:d2:2d:3f:74:
        4f:ea:af:8c:5f:b4:38:9c:db:ae:ce:af:84:1e:a6:f6:34:51:
        59:79:d3:e3:75:dc:bc:d7:f3:73:df:92:ec:d2:20:59:6f:9c:
        fb:95:f8:92:76:18:0a:7c:0f:2c:a6:ca:de:8a:62:7b:d8:f3:
        ce:5f:68:bd:8f:3e:c1:74:bb:15:72:3a:16:83:a9:0b:e6:4d:
        99:9c:d8:57:ec:a8:01:51:c7:6f:57:34:5e:ab:4a:2c:42:f6:
        4f:1c:89:78:de:26:4e:f5:6f:93:4c:15:6b:27:56:4d:00:54:
        6c:7a:b7:b7

Checking server certificate

This is done using the s_client command:

$ openssl s_client -connect www.google.com:443

CONNECTED(00000003)
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDITCCAoqgAwIBAgIQL9+89q6RUm0PmqPfQDQ+mjANBgkqhkiG9w0BAQUFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wOTEyMTgwMDAwMDBaFw0x
MTEyMTgyMzU5NTlaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHFA1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKFApHb29nbGUgSW5jMRcw
FQYDVQQDFA53d3cuZ29vZ2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkC
gYEA6PmGD5D6htffvXImttdEAoN4c9kCKO+IRTn7EOh8rqk41XXGOOsKFQebg+jN
gtXj9xVoRaELGYW84u+E593y17iYwqG7tcFR39SDAqc9BkJb4SLD3muFXxzW2k6L
05vuuWciKh0R73mkszeK9P4Y/bz5RiNQl/Os/CRGK1w7t0UCAwEAAaOB5zCB5DAM
BgNVHRMBAf8EAjAAMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwudGhhd3Rl
LmNvbS9UaGF3dGVTR0NDQS5jcmwwKAYDVR0lBCEwHwYIKwYBBQUHAwEGCCsGAQUF
BwMCBglghkgBhvhCBAEwcgYIKwYBBQUHAQEEZjBkMCIGCCsGAQUFBzABhhZodHRw
Oi8vb2NzcC50aGF3dGUuY29tMD4GCCsGAQUFBzAChjJodHRwOi8vd3d3LnRoYXd0
ZS5jb20vcmVwb3NpdG9yeS9UaGF3dGVfU0dDX0NBLmNydDANBgkqhkiG9w0BAQUF
AAOBgQCfQ89bxFApsb/isJr/aiEdLRLDLE5a+RLizrmCUi3nHX4adpaQedEkUjh5
u2ONgJd8IyAPkU0Wueru9G2Jysa9zCRo1kNbzipYvzwY4OA8Ys+WAi0oR1A04Se6
z5nRUP8pJcA2NhUzUnC+MY+f6H/nEQyNv4SgQhqAibAxWEEHXw==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 1772 bytes and written 307 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: BDE08AD29E65CA4007711610FDD69165F9EB47065716D9AB469DAD9882E6E207
    Session-ID-ctx: 
    Master-Key: 6FE2F273ABAAACF8E99180AAB9D540708F6A392DE1285787121B8A438E68FB3D01C127B31CC39146741D3A8396E0FA79
    Key-Arg   : None
    Start Time: 1311928772
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

Things to note:

The server’s certificate is enclosed by -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----, this can be saved in a separate file for later reuse.
The Certificate name, the Subject is /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com.

The Issuer of the certificate, ie the one who signed the certificate, is /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA.
We were unable to check the validity of this certificate (unable to get local issuer certificate) because we do not have the Issuer certifcate.

By giving the -CAfile parameter we can give the CA certificate file of the CA who issued the server certificate. Here I guessed the CA certificate file from the Issuer name:

$ openssl s_client -connect www.google.com:443 -CAfile /etc/ssl/certs/Verisign_Class_3_Public_Primary_Certification_Authority.pem

CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
verify return:1
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify return:1
depth=0 /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
verify return:1

---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDITCCAoqgAwIBAgIQL9+89q6RUm0PmqPfQDQ+mjANBgkqhkiG9w0BAQUFADBM
[ ... ]
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 222E7CFF6DE8425B1B3635F706CDD65083F2758C87095D70E448E27C8272E377
    Session-ID-ctx: 
    Master-Key: B04EBA908A928D7EF7B9432771A0C95A78BA060FBB75383AA081DDF8F8AD31A2F95CD0EF63AC09827A3220E7856EFE28
    Key-Arg   : None
    Start Time: 1311929265
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

The server certificate is now considered as valid.

It is not always possible to guess the CA certificate filename, instead use the -CApath to give the directory where resides all the known CA certificates of your system:

$ openssl s_client -connect www.google.com:443 -CApath /etc/ssl/certs/

[ ... ]
   Verify return code: 0 (ok)
---

How to create a certificate chain ?

A certificate chain is just a file where multiple PEM files are concatened. PEM files must be ordered from the last certificate (for example server certificate or client certificate) to the top Root CA..

$ cat server_crt.pem operational_ca.pem intermetiade_ca.pem root_ca.pem > certificate_chain.pem

Converting a .pem certificate to pkcs12

PKCS12 is a format to store a certficate AND it’s private key. Because it contains the private key, it needs to by encrypted. Unlike .pem file, the pkcs12 format is binary. So you cannot include it in text configuration files; it need to be base64 if needed.

$ openssl pkcs12 -export \
          -in "blog.quicheaters.org.pem"
          -inkey "blog.quicheaters.org.key" \
          -out "blog.quicheaters.org.p12" \
          -password 'pass:XXXxxxXXX' \
          -certfile "CA/root-ca.pem"

New Apple requirements since 2023

For its newer iOS and macOS Operating Systems, Apple is enforcing new requirements for certificates. See apple suppor.

Here are ways to create Apple compatible certificates.

I’m using this small openssl configuration file:

$ cat openssl.cnf
[ req ]
distinguished_name = req

Create a self-signed CA

openssl req -config ./openssl.cnf \
            -new -x509 -days 3650 -newkey rsa:4096 \
            -keyout CA/-ca.key \
            -passout pass:my_secret_pass \
            -out CA/ca.pem \
            -subj "/C=FR/ST=Ile de France/L=Parise/O=SuperCompany/CN=SuperCompany CA" \
            -addext 'basicConstraints=critical,CA:true' \
            -addext 'subjectKeyIdentifier=hash' \
            -addext 'authorityKeyIdentifier=keyid:always,issuer:always' \
            -addext 'keyUsage=critical,cRLSign,digitalSignature,keyCertSign'

Create a Server Certificate Signing Request

openssl req -config ./openssl.cnf \
            -new -newkey rsa:4096 -nodes \
            -keyout private/vpn.key \
            -out certreqs/vpn.csr \
            -subj "/C=FR/ST=Ile de France/L=Paris/O=SuperCompany/CN=vpn.example.com" \
            -addext 'basicConstraints=critical,CA:FALSE' \
            -addext 'subjectKeyIdentifier=hash' \
            -addext 'keyUsage=critical,nonRepudiation,digitalSignature,keyEncipherment,keyAgreement' \
            -addext 'extendedKeyUsage=critical,serverAuth' \
            -addext 'subjectAltName=DNS:vpn.example.com,IP:1XX.XX.XX.23'

Sign the Certificate Request

openssl x509 \
        -req -in certreqs/vpn.csr \
        -CA CA/ca.pem -CAkey CA/ca.key -CAcreateserial \
        -passin pass:my_secret_pass \
        -days 825 \
        -out newcerts/vpn.pem \
        -extfile <(echo 'basicConstraints=critical,CA:FALSE';
                   echo 'subjectKeyIdentifier=hash';
                   echo 'authorityKeyIdentifier=keyid:always,issuer:always';
                   echo 'keyUsage=critical,nonRepudiation,digitalSignature,keyEncipherment,keyAgreement';
                   echo 'extendedKeyUsage=critical,serverAuth';
                   echo 'subjectAltName=DNS:vpn.example.com,IP:1XX.XX.XX.23';)

References

Deploying your CA's root certificate

2025-05-12T00:00:00+02:00

Linux
Windows 10
iOS 14
Authenticating clients
Authenticating clients with nginx
Authenticating clients with apache
Testing client certificates
References

In my previous article, we implemented a Certificate Authority (CA) using openssl commands.

To become a real CA, you need to get your root certificate on all the devices in the world. But we don’t need to become a real CA. We just need to be a CA for the devices you own. We need to add the root certificate to any laptops, desktops, tablets, and phones that access your HTTPS sites. This can be a bit of a pain, but the good news is that we only have to do it once. Our root certificate will be good until it expires.

For client side authentication, you only need to install the root certificates to the servers that will be doing the client validation.

Deploying the root certificate will vary for different Operating systems and sometimes different browsers.

Linux

There are so many Linux distributions, but Ubuntu is by far the most popular, therefore these instructions will cover Ubuntu.

If it isn’t already installed, install the ca-certificates package.
```
sudo apt-get install -y ca-certificates
```
Copy the myCA.pem or /home/root-ca/certs/ca.cert.pem file to the /usr/local/share/ca-certificates directory as a myCA.crt file.
```
sudo cp ~/certs/myCA.pem /usr/local/share/ca-certificates/myCA.crt
```
Update the certificate store.
```
sudo update-ca-certificates
```

You can test that the certificate has been installed by running the following command:


awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep Hellfish

Windows 10

Open the “Microsoft Management Console” by using the Windows + R keyboard combination, typing mmc and clicking Open
Go to File > Add/Remove Snap-in
Click Certificates and Add
Select Computer Account and click Next
Select Local Computer then click Finish
Click OK to go back to the MMC window
Double-click Certificates (local computer) to expand the view
Select Trusted Root Certification Authorities, right-click on Certificates in the middle column under “Object Type” and select All Tasks then Import
Click Next then Browse. Change the certificate extension dropdown next to the filename field to All Files (.) and locate the myCA.pem file, click Open, then Next
Select Place all certificates in the following store. "Trusted Root Certification Authorities store" is the default. Click Next then click Finish to complete the wizard.

If everything went according to plan, you should see your CA certificate listed under Trusted Root Certification Authorities > Certificates.

iOS 14

On iOS devices you can do so fairly easily by following these steps:

Email the root certificate to yourself, so you can access it on your iOS device. Make sure to use the default Mail app to access the email.
Tap on the attachment in the email on your iOS device. It will prompt you to review the profile in the Settings app.
Open the Settings app and click Profile Downloaded near the top.
Click Install in the top right, and then Install again on the Warning screen.
Once installed, hit Close and go back to the main Settings page.
Go to General > About.
Scroll to the bottom and click on Certificate Trust Settings.
Enable your root certificate under “ENABLE FULL TRUST FOR ROOT CERTIFICATES”.

Authenticating clients

If you are using certificates to validate clients, you only need to install your root certificate myCA.pem. Depending on your web browser, this might vary.

Authenticating clients with nginx

Authenticating clients with nginx, you need to enable TLS first.

server {
    listen              443 ssl;
    server_name         myserver.internal.net;
    ssl_certificate     server.crt;
    ssl_certificate_key server.key;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    # ...
}

Configure Nginx to require clients to authenticate with a certificate issued by your CA To tell Nginx to use mutual TLS and not just one-way TLS, we must instruct it to require client authentication to ensure clients present a certificate from our CA when they connect.

In your server's configuration block, specify the location of your CA root certificate to use for authenticating client certificates. You may choose to make client verification optional so your application can return a 403 message:

server {
    listen                 443 ssl;
    server_name            myserver.internal.net;
    # ...
    ssl_client_certificate /etc/nginx/client_certs/myca.crt;
    ssl_verify_client      optional;

    # ...

    location / {
      if ($ssl_client_verify != SUCCESS) {
        return 403;
      }
    # ...
}

Note that in this example ssl_verify_client is set to optional. This allows you to control what parts of the server need to be authenticated using certificates, and also lets you return 403 error http code.

Authenticating clients with apache

In your Apache configuration file add the following:

SSLVerifyClient
SSLVerifyDepth 10
SSLCACertificateFile /path/to/cert/selfsigned-ca.crt

Restart apache.

Testing client certificates

To test this using the command line, you can use curl, for exampe:

curl -v --cert client.crt.pem --key client.key.pem https://sample.domain.tld

You can also open the URL in your browser and you will be prompted to select a client certificate for authentication. Depending on your browser and system, you might have to import the client certificate into the system's (or browser's) certificate store first.

Create the client keys as described in a previous article.

For Windows clients, the key material can be combined into a single PFX. You will be prompted for the passphrase you set above:

openssl pkcs12 -export -out testuser.pfx -inkey testuser.key -in testuser.crt -certfile myca.crt

This includes the public portion of your CA’s key to allows Windows to trust your internally signed CA.

Importing the Client certificate onto a Windows machine

Double click the .PFX file, select “Current User”.

If you set a passphrase on the PFX above, enter it here. Otherwise, leave blank and hit next.

Next, add the site in question to "trusted sites" in Internet Explorer. This will allow the client certificate to be sent to the site for veritifcation. (Trusting it in Internet Explorer will trust it in chrome as well).

When you next visit the site, you should be prompted to select a client certificate. Select "OK" and you’re in!

Firefox

Firefox has its own certificate store. In my version, go to about:preferences#privacy, scroll to the Certificates section, click "View Certificates...", and on the first tab (Your Certificates), click Import. Select your .p12 file, enter the password, and it will be added.

Chrome based (Chromium, Brave) on Linux

On my debian, I have to import client certificate into storage:

pk12util -d sql:$HOME/.pki/nssdb -i certs/johnsmith.p12

Chrome based (Chromium, Brave) on Windows

Just double-click on certificate file in explorer, and go everything default.

References

Revoking Certificates

2025-05-12T00:00:00+02:00

Certificate revocation lists
Prepare the configuration file
Create the CRL
Revoke a certificate
Server-side use of the CRL
Client-side use of the CRL
References

In my previous article, we implemented a Certificate Authority (CA) using openssl commands. This is a continuation of that article to implement Certificate Revocation Lists (CRL).

This article assumes that you implemented the CA as described in my previous article.

Certificate revocation lists

A certificate revocation list (CRL) provides a list of certificates that have been revoked. A client application, such as a web browser, can use a CRL to check a server’s authenticity. A server application, such as Apache or OpenVPN, can use a CRL to deny access to clients that are no longer trusted.

Publish the CRL at a publicly accessible location (eg, http://example.com/intermediate.crl.pem). Third-parties can fetch the CRL from this location to check whether any certificates they rely on have been revoked.

Prepare the configuration file

When a certificate authority signs a certificate, it will normally encode the CRL location into the certificate. Add crlDistributionPoints to the appropriate sections. In our case, add it to the [ server_cert ] section.

[ server_cert ]
# ... snipped ...
crlDistributionPoints = URI:http://example.com/intermediate.crl.pem

Create the CRL

# cd /home/int-ca
# openssl ca -config openssl.cnf \
      -gencrl -out crl/intermediate.crl.pem

Note

The CRL OPTIONS section of the ca man page contains more information on how to create CRLs.

You can check the contents of the CRL with the crl tool.

# openssl crl -in crl/intermediate.crl.pem -noout -text

No certificates have been revoked yet, so the output will state No Revoked Certificates.

You should re-create the CRL at regular intervals. By default, the CRL expires after 30 days. This is controlled by the default_crl_days option in the [ CA_default ] section.

Revoke a certificate

Let’s walk through an example. Alice is running the Apache web server and has a private folder of heart-meltingly cute kitten pictures. Alice wants to grant her friend, Bob, access to this collection.

Bob creates a private key and certificate signing request (CSR).

$ cd /home/bob
$ openssl genrsa -out bob@example.com.key.pem 2048
$ openssl req -new -key bob@example.com.key.pem \
      -out bob@example.com.csr.pem

You are about to be asked to enter information that will be incorporated
into your certificate request.
-----
Country Name [XX]:US
State or Province Name []:California
Locality Name []:San Francisco
Organization Name []:Bob Ltd
Organizational Unit Name []:
Common Name []:bob@example.com
Email Address []:

Bob sends his CSR to Alice, who then signs it.

# cd /home/int-ca
# openssl ca -config openssl.cnf \
      -extensions usr_cert -notext -md sha256 \
      -in csr/bob@example.com.csr.pem \
      -out certs/bob@example.com.cert.pem

Sign the certificate? [y/n]: y
1 out of 1 certificate requests certified, commit? [y/n]: y

Alice verifies that the certificate is valid:

# openssl verify -CAfile intermediate/certs/ca-chain.cert.pem \
      intermediate/certs/bob@example.com.cert.pem

bob@example.com.cert.pem: OK

The index.txt file should contain a new entry.

V 160420124740Z 1001 unknown ... /CN=bob@example.com

Alice sends Bob the signed certificate. Bob installs the certificate in his web browser and is now able to access Alice’s kitten pictures. Hurray!

Sadly, it turns out that Bob is misbehaving. Bob has posted Alice’s kitten pictures to Hacker News, claiming that they’re his own and gaining huge popularity. Alice finds out and needs to revoke his access immediately.

# cd /hone/int-ca
# openssl ca -config openssl.cnf \
      -revoke certs/bob@example.com.cert.pem

Enter pass phrase for intermediate.key.pem: secretpassword
Revoking Certificate 1001.
Data Base Updated

The line in index.txt that corresponds to Bob’s certificate now begins with the character R. This means the certificate has been revoked.

R 160420124740Z 150411125310Z 1001 unknown ... /CN=bob@example.com

After revoking Bob’s certificate, Alice must re-create the CRL.

Server-side use of the CRL

For client certificates, it’s typically a server-side application (eg, Apache) that is doing the verification. This application needs to have local access to the CRL.

In Alice’s case, she can add the SSLCARevocationPath directive to her Apache configuration and copy the CRL to her web server. The next time that Bob connects to the web server, Apache will check his client certificate against the CRL and deny access.

Similarly, OpenVPN has a crl-verify directive so that it can block clients that have had their certificates revoked.

Client-side use of the CRL

For server certificates, it’s typically a client-side application (eg, a web browser) that performs the verification. This application must have remote access to the CRL.

If a certificate was signed with an extension that includes crlDistributionPoints, a client-side application can read this information and fetch the CRL from the specified location.

The CRL distribution points are visible in the certificate X509v3 details.

# openssl x509 -in cute-kitten-pictures.example.com.cert.pem -noout -text

    X509v3 CRL Distribution Points:

        Full Name:
          URI:http://example.com/intermediate.crl.pem

References

https://jamielinux.com/docs/openssl-certificate-authority/introduction.html

Your own Certificate Authority

2025-05-12T00:00:00+02:00

As mentioned in the previous article, there are some scenarios that it would be useful to run your own Certificate Authority (CA). For example:

When you require a lot of certificates.
Authenticate users or client devices.

It is possible to run your own mini Certificate Authority manually.

To request an SSL certificate from a CA like Verisign or GoDaddy, you send them a Certificate Signing Request (CSR), and they give you an SSL certificate in return that they have signed using their root certificate and private key. All browsers have a copy (or access to a copy from the operating system) of the root certificate from the various CAs, so the browser can verify that your certificate was signed by a trusted CA.

That’s why when you generate a self-signed certificate the browser doesn’t trust it. It hasn’t been signed by a CA. The way to get around this is to generate our own root certificate and private key. We then add the root certificate to all the devices we own just once, and then all the certificates we generate will be inherently trusted.

This tutorial uses OpenSSL. OpenSSL is a free and open-source cryptographic library that provides several command-line tools for handling digital certificates. Some of these tools can be used to act as a certificate authority.

Generating root cert

Acting as a certificate authority (CA) means dealing with cryptographic pairs of private keys and public certificates. The very first cryptographic pair we’ll create is the root pair. This consists of the root key (ca.key.pem) and root certificate (ca.cert.pem). This pair forms the identity of your CA.

Typically, the root CA does not sign server or client certificates directly. The root CA is only ever used to create one or more intermediate CAs, which are trusted by the root CA to sign certificates on their behalf. This is best practice. It allows the root key to be kept offline and unused as much as possible, as any compromise of the root key is disastrous.

Note

It’s best practice to create the root pair in a secure environment. Ideally, this should be on a fully encrypted, air gapped computer that is permanently isolated from the Internet. Remove the wireless card and fill the ethernet port with glue.

Preparation

Choose a directory (/home/root-ca) to store all keys and certificates.

# mkdir /home/root-ca

Create the directory structure. The index.txt and serial files act as a flat file database to keep track of signed certificates.

# cd /home/root-ca
# mkdir certs csr crl newcerts private
# chmod 700 private
# touch index.txt
# echo 1000 > serial

You must create a configuration file for OpenSSL to use in /home/root-ca/openssl.cnf

# OpenSSL root CA configuration file.
# Copy to `/home/root-ca/openssl.cnf`.

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = /home/root-ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.cert.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_strict

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
# organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
# stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
# 0.organizationName              = Organization Name
# organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = NL
# stateOrProvinceName_default   = 
localityName_default            = Den Haag
# 0.organizationName_default      = Alice Ltd
# organizationalUnitName_default  =
emailAddress_default            =

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

The [ ca ] section is mandatory. Here we tell OpenSSL to use the options from the [ CA_default ] section.

The [ CA_default ] section contains a range of defaults. Make sure you declare the directory you chose earlier (/home/ca).

We’ll apply policy_strict for all root CA signatures, as the root CA is only being used to create intermediate CAs.

We’ll apply policy_loose for all intermediate CA signatures, as the intermediate CA is signing server and client certificates that may come from a variety of third-parties.

Options from the [ req ] section are applied when creating certificates or certificate signing requests.

The [ req_distinguished_name ] section declares the information normally required in a certificate signing request. You can optionally specify some defaults.

The next few sections are extensions that can be applied when signing certificates. For example, passing the -extensions v3_ca command-line argument will apply the options set in [ v3_ca ].

We’ll apply the v3_ca extension when we create the root certificate.

We’ll apply the v3_ca_intermediate extension when we create the intermediate certificate. pathlen:0 ensures that there can be no further certificate authorities below the intermediate CA.

We’ll apply the usr_cert extension when signing client certificates, such as those used for remote user authentication.

We’ll apply the server_cert extension when signing server certificates, such as those used for web servers.

Creating the root key

Create the root key (ca.key.pem) and keep it absolutely secure. Anyone in possession of the root key can issue trusted certificates. Encrypt the root key with AES 256-bit encryption and a strong password.

Note

Use 4096 bits for all root and intermediate certificate authority keys. You’ll still be able to sign server and client certificates of a shorter length.

# cd /home/root-ca
# openssl genrsa -aes256 -passout pass:secretpassword -out private/ca.key.pem 4096
# chmod 400 private/ca.key.pem

Creating the root certificate

Use the root key (ca.key.pem) to create a root certificate (ca.cert.pem). Give the root certificate a long expiry date, such as twenty years. Once the root certificate expires, all certificates signed by the CA become invalid.

Warning

Whenever you use the req tool, you must specify a configuration file to use with the -config option, otherwise OpenSSL will default to /etc/pki/tls/openssl.cnf.

# cd /root/root-ca
# openssl req -config openssl.cnf \
      -passin pass:secretpassword \
      -key private/ca.key.pem \
      -new -x509 -days 7300 -sha256 -extensions v3_ca \
      -out certs/ca.cert.pem \
      -subj "/C=NL/ST=ZH/O=Alice Ltd/CN=Alice Ltd Root CA"
# chmod 444 certs/ca.cert.pem

To verify the root certificate do:

# openssl x509 -noout -text -in certs/ca.cert.pem

The output shows:

the Signature Algorithm used
the dates of certificate Validity
the Public-Key bit length
the Issuer, which is the entity that signed the certificate
the Subject, which refers to the certificate itself

The Issuer and Subject are identical as the certificate is self-signed. Note that all root certificates are self-signed.

        Issuer: C=NL, ST=ZH, O=Alice Ltd, CN=Alice Ltd Root CA
        Validity
            Not Before: May  8 22:29:26 2025 GMT
            Not After : May  3 22:29:26 2045 GMT
        Subject: C=NL, ST=ZH, O=Alice Ltd, CN=Alice Ltd Root CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

The output also shows the X509v3 extensions. We applied the v3_ca extension, so the options from [ v3_ca ] should be reflected in the output.

        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                64:2B:BA:1A:12:9F:D2:B6:62:45:1D:EC:07:17:DC:DB:CA:E8:E9:A5
            X509v3 Authority Key Identifier: 
                64:2B:BA:1A:12:9F:D2:B6:62:45:1D:EC:07:17:DC:DB:CA:E8:E9:A5
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign

Minimalistic Root certificate

Alternatively you can take the minimalistic approach:

Create certificate directory
```
mkdir ~/ca
cd ~/ca
```

Generate the private key:

openssl genrsa -des3 -passout pass:secretpassword -out myCA.key 4096

Generate root certificate:

openssl req -x509 -new -nodes -passin pass:secretpassword -key myCA.key \
     -sha256 -days 1825 -out myCA.pem \
     -subj "/C=NL/ST=ZH/O=Alice Ltd/CN=Alice Ltd Root CA"

Verify certificate:
```
openssl x509 -noout -text -in myCA.pem
```
You can skip the intermediat certificate steps.

Create intermediate cert

An intermediate certificate authority (CA) is an entity that can sign certificates on behalf of the root CA. The root CA signs the intermediate certificate, forming a chain of trust.

The purpose of using an intermediate CA is primarily for security. The root key can be kept offline and used as infrequently as possible. If the intermediate key is compromised, the root CA can revoke the intermediate certificate and create a new intermediate cryptographic pair.

Prepare the directory

The root CA files are kept in /home/root-ca. Choose a different directory (/home/int-ca) to store the intermediate CA files.

# mkdir /home/int-ca

Create the same directory structure used for the root CA files. It’s convenient to also create a csr directory to hold certificate signing requests.

# cd /home/int-ca
# mkdir certs crl csr newcerts private
# chmod 700 private
# touch index.txt
# echo 1000 > serial

Add a crlnumber file to the intermediate CA directory tree. crlnumber is used to keep track of certificate revocation lists.

# echo 1000 > crlnumber

Create a file in /home/int-ca/openssl.cnf:

# OpenSSL intermediate CA configuration file.
# Copy to `/home/int-ca/openssl.cnf`.

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = /home/int-ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/intermediate.key.pem
certificate       = $dir/certs/intermediate.cert.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/intermediate.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_loose

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
# organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
# stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
# 0.organizationName              = Organization Name
# organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = NL
# stateOrProvinceName_default   = 
localityName_default            = Den Haag
# 0.organizationName_default      = Alice Ltd
# organizationalUnitName_default  =
emailAddress_default            =

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

Five options have changed compared to the root CA configuration file:

[ CA_default ]
dir             = /home/int-ca
private_key     = $dir/private/intermediate.key.pem
certificate     = $dir/certs/intermediate.cert.pem
crl             = $dir/crl/intermediate.crl.pem
policy          = policy_loose

Create the intermediate key

Create the intermediate key (intermediate.key.pem). Encrypt the intermediate key with AES 256-bit encryption and a strong password.

This is done in the intermediate CA

# cd /home/int-ca
# openssl genrsa -aes256 \
      -passout pass:secretpassword \
      -out private/intermediate.key.pem 4096
# chmod 400 private/intermediate.key.pem

Create the intermediate certificate

Use the intermediate key to create a certificate signing request (CSR). The details should generally match the root CA. The Common Name, however, must be different.

Warning

Make sure you specify the intermediate CA configuration file (/home/int-ca/openssl.cnf).

# cd /home/int-ca
# openssl req -config openssl.cnf -new -sha256 \
      -passin pass:secretpassword \
      -key private/intermediate.key.pem \
      -out csr/intermediate.csr.pem \
      -subj "/C=NL/ST=ZH/O=Alice Ltd/CN=Alice Ltd Intermediate CA"

To create an intermediate certificate, use the root CA with the v3_intermediate_ca extension to sign the intermediate CSR. The intermediate certificate should be valid for a shorter period than the root certificate. Ten years would be reasonable.

Copy the intermeida certificates csr to the root CA. If you follow hard security practices you may need to manually copy the csr file to the air-gapped root CA system. In our example, we simply copy:

# cp -av /home/int-ca/csr/intermediate.csr.pem /home/root-ca/csr

Warning

This time, specify the root CA configuration file (/home/root-ca/openssl.cnf).

# cd /home/root-ca
# openssl ca -config openssl.cnf -extensions v3_intermediate_ca \
      -days 3650 -notext -md sha256 \
      -passin pass:secretpassword \
      -in csr/intermediate.csr.pem \
      -out certs/intermediate.cert.pem

Sign the certificate? [y/n]: y
1 out of 1 certificate requests certified, commit? [y/n]y

# chmod 444 certs/intermediate.cert.pem

The index.txt file is where the OpenSSL ca tool stores the certificate database. Do not delete or edit this file by hand. It should now contain a line that refers to the intermediate certificate.

V   350506223334Z       1000    unknown /C=NL/ST=ZH/O=Alice Ltd/CN=Alice Ltd Intermediate CA

Verify the intermediate certificate

As we did for the root certificate, check that the details of the intermediate certificate are correct.

# openssl x509 -noout -text \
      -in certs/intermediate.cert.pem

Verify the intermediate certificate against the root certificate. An OK indicates that the chain of trust is intact.

# openssl verify -CAfile certs/ca.cert.pem \
      certs/intermediate.cert.pem

intermediate.cert.pem: OK

Create the certificate chain file

When an application (eg, a web browser) tries to verify a certificate signed by the intermediate CA, it must also verify the intermediate certificate against the root certificate. To complete the chain of trust, create a CA certificate chain to present to the application.

To create the CA certificate chain, concatenate the intermediate and root certificates together. We will use this file later to verify certificates signed by the intermediate CA.

# cat certs/intermediate.cert.pem \
      certs/ca.cert.pem > certs/intermediate-full-chain.cert.pem
# chmod 444 certs/intermediate-full-chain.cert.pem

Note

Our certificate chain file must include the root certificate because no client application knows about it yet. A better option, particularly if you’re administrating an intranet, is to install your root certificate on every client that needs to connect. In that case, the chain file need only contain your intermediate certificate.

At this point you need to copy from the root CA to the intermediate CA the files:

certs/intermediate.cert.pem
certs/intermediate-full-chain.cert.pem

cp -av /home/root-ca/certs/intermediate*.pem /home/int-ca/certs

Minimalistic Approach

For the minimalistic approach, this entire section can be skipped.

Signing client/server certificates

We will be signing certificates using our intermediate CA. You can use these signed certificates in a variety of situations, such as to secure connections to a web server or to authenticate clients connecting to a service.

Note

The steps below are from your perspective as the certificate authority. A third-party, however, can instead create their own private key and certificate signing request (CSR) without revealing their private key to you. They give you their CSR, and you give back a signed certificate. In that scenario, skip the genrsa and req commands.

Create a key

Our root and intermediate pairs are 4096 bits. Server and client certificates normally expire after one year, so we can safely use 2048 bits instead.

Note

Although 4096 bits is slightly more secure than 2048 bits, it slows down TLS handshakes and significantly increases processor load during handshakes. For this reason, most websites use 2048-bit pairs.

If you’re creating a cryptographic pair for use with a web server (eg, Apache), you probably want to create a key without a password so that you don't need to enter the password every time you restart the web server:

# cd /home/int-ca
# openssl genrsa  \
      -out private/www.example.com.key.pem 2048
# chmod 400 private/www.example.com.key.pem

On the other hand, if you are creating a key for an interactive user, creating a key with password would be beneficial:

# cd /home/int-ca
# openssl genrsa  \
      -aes256 \
      -passout pass:secretpassword \
      -out private/user.alice.key.pem 2048
# chmod 400 private/user.alice.key.pem

Create a certificate

Use the private key to create a certificate signing request (CSR). The CSR details don’t need to match the intermediate CA. For server certificates, the Common Name must be a fully qualified domain name (eg, www.example.com), whereas for client certificates it can be any unique identifier (eg, an e-mail address). Note that the Common Name cannot be the same as either your root or intermediate certificate.

# cd /home/int-ca
# openssl req -config openssl.cnf \
      -key private/www.example.com.key.pem \
      -new -sha256 -out csr/www.example.com.csr.pem \
      -subj "/C=NL/CN=www.example.com"

For the user case you may use:

# cd /home/int-ca
# openssl req -config openssl.cnf \
      -passin pass:secretpassword \
      -key private/user.alice.key.pem \
      -new -sha256 -out csr/user.alice.csr.pem \
      -subj "/C=NL/CN=user.alice/emailAddress=alice@example.com"

To create a certificate, use the intermediate CA to sign the CSR. If the certificate is going to be used on a server, use the server_cert extension. If the certificate is going to be used for user authentication, use the usr_cert extension. Certificates are usually given a validity of one year, though a CA will typically give a few days extra for convenience.

# cd /home/int-ca
# openssl ca -config openssl.cnf \
      -passin pass:secretpassword \
      -extensions server_cert -days 375 -notext -md sha256 \
      -in csr/www.example.com.csr.pem \
      -out certs/www.example.com.cert.pem
# chmod 444 certs/www.example.com.cert.pem

The intermediate/index.txt file should contain a line referring to this new certificate.

V   350506223334Z       1000    unknown /C=NL/ST=ZH/O=Alice Ltd/CN=Alice Ltd Intermediate CA

For the user case you may use:

# cd /home/int-ca
# openssl ca -config openssl.cnf \
      -passin pass:secretpassword \
      -extensions usr_cert -days 375 -notext -md sha256 \
      -in csr/user.alice.csr.pem \
      -out certs/user.alice.cert.pem
# chmod 444 certs/user.alice.cert.pem

Verify the certificate

# openssl x509 -noout -text \
      -in certs/www.example.com.cert.pem

The Issuer is the intermediate CA. The Subject refers to the certificate itself.

        Issuer: C=NL, ST=ZH, O=Alice Ltd, CN=Alice Ltd Intermediate CA
        Validity
            Not Before: May  8 23:42:47 2025 GMT
            Not After : May 18 23:42:47 2026 GMT
        Subject: C=NL, CN=www.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

The output will also show the X509v3 extensions. When creating the certificate, you used either the server_cert or usr_cert extension. The options from the corresponding configuration section will be reflected in the output.

        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Server
            Netscape Comment: 
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier: 
                FA:B9:39:0C:09:67:53:20:D1:FC:2D:B9:00:05:91:98:4E:4D:B4:F0
            X509v3 Authority Key Identifier: 
                keyid:D9:13:85:BC:23:8A:24:BA:A0:D9:A2:21:C9:79:95:34:E3:85:E1:2A
                DirName:/C=NL/ST=ZH/O=Alice Ltd/CN=Alice Ltd Root CA
                serial:10:00
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication

Use the CA certificate chain file we created earlier (ca-chain.cert.pem) to verify that the new certificate has a valid chain of trust.

# openssl verify -CAfile certs/intermediate-full-chain.cert.pem \
      certs/www.example.com.cert.pem

www.example.com.cert.pem: OK

Deploy the certificate

You can now either deploy your new certificate to a server, or distribute the certificate to a client. When deploying to a server application (eg, Apache), you need to make the following files available:

intermediate-full-chain.cert.pem
www.example.com.key.pem
www.example.com.cert.pem

If you’re signing a CSR from a third-party, you don’t have access to their private key so you only need to give them back the chain file (ca-chain.cert.pem) and the certificate (www.example.com.cert.pem).

Minimal approach

Create private key:

openssl genrsa -out www.example.com.key 2048

Generate CSR:

openssl req -new \
         -key www.example.com.key \
         -out www.example.com.csr \
         -addext "subjectAltName = DNS:foo.co.uk, DNS:*.app.example.com" \
         -subj "/C=NL/CN=www.example.com"

Create the certificate: using our CSR, the CA private key and the CA certificate:

openssl x509 -req \
       -passin pass:secretpassword \
       -CA myCA.pem -CAkey myCA.key -CAcreateserial \
       -days 825 -sha256 \
       -copy_extensions copyall \
       -in www.example.com.csr \
       -out www.example.com.crt

We now have three files: www.example.com.key (the private key), www.example.com.csr (the certificate signing request, or csr file), and www.example.com.crt (the signed certificate). We can configure local web servers to use HTTPS with the private key and the signed certificate.

Conclusion

This articles covers two approaches for running Certificate Authorities only using openssl commands.

The full approach can be implementely quite securely and explicitly support server and client certificates.

The minimal approach is quite straight forward for a quick and dirty solution.

References

Certificate Authorities

2025-05-12T00:00:00+02:00

For home users there is not much use for running you own Certificate Authority (CA), and with availability of Letsencrypt and the plethora of ACME libraries setting TLS encryption is quite straight forward.

There are also some alternatives to Letsencrypt that offer free certificates:

These are commercial offerings that have a free tier.

I myself use letsencrypt with acme.sh because it is available for Alpine Linux, which is my preferred Operating System for my home servers.

Private Certificate Authority

There are some scenarios that it would be useful to run your own Certificate Authority (CA). For example:

When you require a lot of certificates.
Authenticate users or client devices.

There are some tools that take care of most of the details on how to run your own CA.

Still, this may be useful to know in case you ever need to. I tried the folowing myself:

caman
Written in shell script, and support intermediate certificates.
minica
Written in go language. Very simple and straightforward.
mkcert
Another one in go language. A simple zero-config tool to make locally trusted development certificates with any names you'd like.

NEW YEAR 2026

2025-05-10T00:00:00+02:00

Happy New Year!

More Python GUI programming

2025-05-12T00:00:00+02:00

Using threads with Python and Tkinter
Alternative implementation using Pipe's
- Comparison of Queue vs. Pipe + Select
- Which One is Better?
Using File handlers
Alternative approaches
Conclusion

In a previous article, I wrote about Python GUI programming.

I used to write simple User Interfaces with Tcl/Tk. Nowadays I find writing Python more often, so obviously I would start writing user interfaces in Python with tkinter, because of my Tcl/Tk background.

While Tcl/Tk has a complete event driven architecture, suitable for scripting, Python has a richer set of programming paradims to choose from.

One common pattern is GUI programming is to use threading to split the UI and the computations into separate threads. This has the advantage of making the computation thread only about doing calculations, while keeping the UI very responsive.

In Tcl/Tk because it did not support threading natively, you would either have to split the computation tasks into small units or use multiple processes.

Both approaches came with different set of disadvantages.

Splitting computation tasks into small units meant that you in essence had to program in a cooperative multitasking environment, and split a large task into smaller tasks so that the UI wouldn't block for long periods of time making it unresponsive. Most of the programming effort would be then spent creating the smaller units, and making sure that the application wouldn't block.

Alternatively, you could split your program into multiple processes using network sockets and fileevent handlers to keep the UI responsive. In here a lot of programming effort is spent coordinating and sharing the data between processes.

Using threads with Python and Tkinter

With python, the UI thread and separate computation thread would have been easy to implement becase python supports threads natively. Unfortunatley, because tkinter is based on Tcl/Tk, it does not really support multiple threads and has potential concurrency issues.

Impelementation using queue.Queue

There is essentially one main approach to implement this in a thread safe way. To use a python synchronization primitive such as queue.Queue, and poll it from an after function on a regular basis.

This is an exampe using [queue.Queue]:

import tkinter as tk
import threading
import time
import queue

class App:
    def __init__(self, root):
        self.root = root
        self.root.title("Counting UI")

        # Create UI elements
        self.listbox = tk.Listbox(root, width=40, height=20)
        self.listbox.pack()

        self.close_button = tk.Button(root, text="Close", command=self.close)
        self.close_button.pack()

        # Create a thread-safe queue
        self.queue = queue.Queue()

        # Start the counting thread
        self.running = True
        self.thread = threading.Thread(target=self.counter_thread, daemon=True)
        self.thread.start()

        # Start polling the queue using after()
        self.poll_queue()

    def counter_thread(self):
        """Thread that counts from 1 to 60, adding values to the queue each second."""
        for i in range(1, 61):
            if not self.running:
                break
            self.queue.put(str(i))
            time.sleep(1)

        # Signal completion
        self.queue.put("DONE")

    def poll_queue(self):
        """Polls the queue every 100ms and updates the listbox."""
        while not self.queue.empty():
            item = self.queue.get()
            if item == "DONE":
                self.close()
                return
            self.listbox.insert(tk.END, item)

        if self.running:
            self.root.after(100, self.poll_queue)  # Schedule next poll

    def close(self):
        """Closes the application."""
        self.running = False
        self.root.quit()

# Run the application
if __name__ == "__main__":
    root = tk.Tk()
    app = App(root)
    root.mainloop()

This Python script creates a simple GUI application using Tkinter and manages background processing using a thread and a queue. Here's what it does:

It opens a Tkinter window with a listbox and a "Close" button.
A separate thread runs in the background, counting from 1 to 60, adding each number to a queue once per second.
The main thread (UI thread) polls the queue every 100ms, retrieving numbers and displaying them in the listbox.
When counting reaches 60, the script signals completion with "DONE", triggering the application to close.

Key Components

Tkinter UI Setup:
- A listbox is used to display the counting numbers.
- A "Close" button is provided to manually stop execution.
Threading (counter_thread function):
- Runs independently from the UI thread to avoid freezing the interface.
- Adds numbers 1 to 60 into the queue every second.
- Uses "DONE" as a termination signal.
Queue for Communication (queue.Queue()):
- Ensures thread-safe interaction between the worker thread and the main UI thread.
- Acts as a buffer for numbers produced by the worker thread.
Polling Mechanism (poll_queue function):
- Checks the queue every 100ms.
- Retrieves numbers and updates the listbox.
- Recognizes "DONE" as the termination condition and shuts down the UI.
Application Lifecycle (close function):
- Stops the counting thread gracefully.
- Closes the application window.

Why Use Threads and Queues?

Threading prevents UI freezing since time delays (time.sleep) inside Tkinter's main loop would block interactions.
Queues ensure thread safety, avoiding direct UI modifications from background threads.
Polling allows non-blocking event handling, letting Tkinter remain responsive.

Alternative implementation using Pipe's

For comparison, we can achieve a similar result using select and UNIX Pipes. This in principle is very similar to what you would do using Tcl/Tk:


import tkinter as tk
import threading
import time
import os
import select

class App:
    def __init__(self, root):
        self.root = root
        self.root.title("Counting UI")

        # Create UI elements
        self.listbox = tk.Listbox(root, width=40, height=20)
        self.listbox.pack()

        self.close_button = tk.Button(root, text="Close", command=self.close)
        self.close_button.pack()

        # Create a pipe
        self.pipe_r, self.pipe_w = os.pipe()

        # Start the thread
        self.running = True
        self.thread = threading.Thread(target=self.counter_thread)
        self.thread.start()

        # Start polling the pipe using after()
        self.poll_pipe()

    def counter_thread(self):
        """Thread that counts from 1 to 60, writing to the pipe each second."""
        for i in range(1, 61):
            if not self.running:
                break
            os.write(self.pipe_w, f"{i}\n".encode())
            time.sleep(1)

        # Signal completion by writing "DONE" to the pipe
        os.write(self.pipe_w, "DONE\n".encode())

    def poll_pipe(self):
        """Polls the pipe every 100ms using select() and updates the listbox."""
        readable, _, _ = select.select([self.pipe_r], [], [], 0)
        if readable:
            try:
                data = os.read(self.pipe_r, 1024).decode()
                for line in data.splitlines():
                    if line == "DONE":
                        self.close()  # Call close method in UI thread
                        return
                    self.listbox.insert(tk.END, line)
            except OSError:
                pass  # Pipe closed

        if self.running:
            self.root.after(100, self.poll_pipe)  # Schedule next poll

    def close(self):
        """Closes the application."""
        self.running = False
        os.close(self.pipe_r)
        os.close(self.pipe_w)
        self.root.quit()

# Run the application
if __name__ == "__main__":
    root = tk.Tk()
    app = App(root)
    root.mainloop()

Both implementations serve the same purpose—communicating between a background thread and the Tkinter UI thread—but they differ in efficiency, complexity, and reliability. Here's a comparison:

Comparison of Queue vs. Pipe + Select

Feature	Queue (`queue.Queue`)	Pipe + Select (`os.pipe() + select.select()`)
Ease of Use	Simple API (`put/get`)	Requires handling raw byte streams
Polling Mechanism	Uses `queue.get()` (blocking or timeout-based)	Requires `select.select()` polling
Code Readability	Cleaner and more structured	More low-level, harder to maintain
Data Format	Stores Python objects (strings, numbers)	Requires encoding/decoding byte streams

Which One is Better?

Using queue.Queue is generally better in the context of Python multithreading and GUI applications:

More Pythonic and intuitive: Python's queue mechanisms integrate naturally with threading.
Automatic thread safety: No need for manual locking or polling overhead.
Better resource management: No need to manually open and close OS-level pipes.
Cross-platform reliability: Works consistently across Windows, macOS, and Linux.

However, Pipe + Select is useful in certain cases:

When dealing with multi-processing (instead of multi-threading), since queue.Queue is thread-safe but not process-safe (for processes, multiprocessing.Queue is preferred).
If interacting with external system processes, where pipes and low-level I/O are necessary.

For a Tkinter-based GUI with multi-threading, queue.Queue is the better choice - it's safer, simpler, and requires less manual handling compared to pipe-based synchronization.

Using File handlers

The alternative implementation using UNIX pipe's is interesting because it can be modified to use filehandlers. This is closer to a Tcl/Tk implementation which would probably use fileevent.


import tkinter as tk
import threading
import time
import os

class App:
    def __init__(self, root):
        self.root = root
        self.root.title("Counting UI")

        # Create UI elements
        self.listbox = tk.Listbox(root, width=40, height=20)
        self.listbox.pack()

        self.close_button = tk.Button(root, text="Close", command=self.close)
        self.close_button.pack()

        # Create a pipe
        self.pipe_r, self.pipe_w = os.pipe()

        # Start the thread
        self.running = True
        self.thread = threading.Thread(target=self.counter_thread)
        self.thread.start()

        # Register the pipe with Tkinter's event loop
        self.root.createfilehandler(self.pipe_r, tk.READABLE, self.read_pipe)

    def counter_thread(self):
        """Thread that counts from 1 to 60, writing to the pipe each second."""
        for i in range(1, 61):
            if not self.running:
                break
            os.write(self.pipe_w, f"{i}\n".encode())
            time.sleep(1)

        # Signal completion by writing "DONE" to the pipe
        os.write(self.pipe_w, "DONE\n".encode())

    def read_pipe(self, file_descriptor, event_mask):
        """Reads from the pipe and updates the listbox."""
        try:
            data = os.read(file_descriptor, 1024).decode()
            for line in data.splitlines():
                if line == "DONE":
                    self.close()  # Call close method in UI thread
                    return
                self.listbox.insert(tk.END, line)
        except OSError:
            pass  # Pipe closed

    def close(self):
        """Closes the application."""
        self.running = False
        os.close(self.pipe_r)
        os.close(self.pipe_w)
        self.root.quit()

# Run the application
if __name__ == "__main__":
    root = tk.Tk()
    app = App(root)
    root.mainloop()

Both approaches use pipes (os.pipe()) for communication between a background thread and the Tkinter UI thread. However, they differ in how they integrate with Tkinter's event loop. Here’s a breakdown:

Event Polling Mechanism

Feature	Pipe + Select (`select.select()`)	Pipe + `createfilehandler()`
Event Handling	Polls the pipe every 100ms using `select.select()`	Directly registers a pipe handler with Tkinter
Polling Overhead	Requires frequent polling even when there’s no data	More efficient—Tkinter triggers event on readable data
Performance	Slightly wasteful due to constant polling	More optimal as event processing is handled natively by Tkinter
Responsiveness	UI remains responsive but polling introduces some delay	Immediate handling of incoming data with better responsiveness

Code Complexity & Maintainability

Feature	Pipe + Select	Pipe + `createfilehandler()`
Complexity	Requires manual polling with `select.select()`	Simpler, as Tkinter automatically processes pipe events
Readability	More boilerplate code for polling	More concise and integrates cleanly with Tkinter
Ease of Debugging	Requires handling timeout and manual checks	Easier to debug since Tkinter invokes handler only when needed

Which One is Better?

Both approaches are Linux/UNIX dependant, so they wouldn't work on MS Windows. If that is not a requirement, is obvious that using File Handlers is the better approach as avoids polling overhead, giving better performance.

Alternative approaches

I found Asynchronous Tkinter Mainloop which is an asynchronous implementation of the mainloop for tkinter. This allows using async handler functions. It is intended to be as simple to use as possible. No fancy unusual syntax or constructions - just use an alternative function instead of root.mainloop() and wrap asynchronous handlers into a helper function.

This is interesting if you are used to using Python's asyncio. However keep in mind that this is very similar to using "after+poll" approach mentioned earlier.

Conclusion

In conclusion, while Tkinter's threading limitations stem from its Tcl/Tk foundation, various strategies exist to maintain a responsive UI alongside background computations. Using queue.Queue provides the most Pythonic and thread-safe approach, ensuring clear communication between worker threads and the UI. Pipes and file handlers, while viable, introduce added complexity and platform dependencies. Meanwhile, asynchronous frameworks like asyncio and event-driven alternatives further expand possibilities these suffer with the same polling overhead as other solutions.

Ultimately, the choice depends on your application's needs—whether you prioritize simplicity, efficiency, or compatibility. If responsiveness and ease of implementation are your goals, queue.Queue stands as the most robust option. However, developers familiar with lower-level event-driven architectures may find pipes and file handlers more suitable. Exploring these approaches helps refine a developer’s understanding of concurrency in GUI applications while balancing usability, performance, and maintainability.

Installing source using PIP

2025-05-05T00:00:00+02:00

Installing from github
Installing from ZIP file
- How It Works:
- Alternative: Local ZIP File Installation
Creating a Python package
- Explanation:
  - Key Components:
- Single file python modules

Installing from github

You can install a Python module directly from a GitHub repository using pip by specifying the repository's URL. Here’s how you do it:

pip install git+https://github.com/username/repository.git

Replace username with the owner of the repository and repository with the name of the repository. If the module is inside a subdirectory, or if the repository has different branches, you might need to specify the path like this:

pip install git+https://github.com/username/repository.git@branch_name

You can install a specific tag from a GitHub repository using pip by appending the tag name to the repository URL with an @ symbol. Here's how:

pip install git+https://github.com/username/repository.git@tag_name

When using pip to install a package directly from a GitHub repository, pip interacts with GitHub as follows:

Fetching the Repository – pip uses Git to clone the specified repository to your local machine. This is done using the git+https://github.com/... URL format.
Checking Out the Code – If you specify a branch, tag, or commit hash, pip checks out that specific version of the repository.
Looking for setup.py or pyproject.toml – Once cloned, pip searches for a setup.py file (for traditional Python packages) or pyproject.toml (for modern builds using tools like Poetry). It uses this file to determine dependencies and installation steps.
Installing Dependencies – If the package has dependencies listed, pip installs them before installing the main package itself.
Building & Installing – If the package needs compiling or building wheels, pip handles that and then installs the package into your environment.

Essentially, pip acts as both a package manager and a tool for fetching repositories from GitHub dynamically. It's a convenient way to install Python packages that aren't published on PyPI yet.

Installing from ZIP file

If you want to install a Python package from a .zip file hosted on a web server using pip, you can do it like this:

pip install https://example.com/path/to/package.zip

How It Works:

pip downloads the ZIP file from the specified URL.
It extracts the contents and looks for a setup.py or pyproject.toml file to determine how to install the package.
If dependencies are listed, pip installs them before installing the package itself.

Alternative: Local ZIP File Installation

If you’ve already downloaded the ZIP file, you can install it locally using:

pip install /path/to/package.zip

If the package is structured correctly, this method works seamlessly.

Creating a Python package

If you're creating a Python package, the minimal setup.py file should include:

from setuptools import setup

setup(
    name="your_package_name",
    version="0.1",
    packages=["your_package"],
)

Alternatively you could:

from setuptools import setup
from source import VERSION

setup(
    name="your_package_name",
    version=VERSION,
    packages=["your_package"],
)

Explanation:

name: Defines the package name.
version: Sets the version number. In the alternatively option, the version comes from the source package itself.
packages: Specifies the directory containing your Python modules.

For a more complete setup.py, you'd usually include:

author, description, long_description
install_requires for dependencies
entry_points if your package includes a command-line tool

The ZIP file should be structured like a typical Python package to ensure pip can install it properly. Here's an example of how the contents should be organized:

package.zip
│── your_package/                # Main package directory
│   ├── __init__.py              # Makes it a Python package
│   ├── module.py                # Your actual Python code
│── setup.py                      # Installation configuration
│── requirements.txt              # Optional: Dependencies list
│── README.md                     # Optional: Project info

I saw examples, where the actual python code is named core.py and in __init__.py they would have:

from .core import *

You may also want to import or define a VERSION id.

Key Components:

your_package/ – The actual package directory with your Python modules.
__init__.py – An empty file (or containing initialization code) that marks the directory as a package.
setup.py – Defines metadata, dependencies, and installation instructions for the package.
requirements.txt – If the package has dependencies, list them here.
README.md – Optional, but useful for explaining your package.

To create this ZIP file, navigate to the package directory and run:

zip -r package.zip your_package setup.py requirements.txt README.md

Once zipped, pip can install it like:

pip install https://example.com/package.zip

You can use the same structure for a Git repository.

Single file python modules

If your module is just a single Python file rather than a full package, you can still create a ZIP file for installation with pip. Here’s how your ZIP structure should look:

module.zip
│── module.py              # Your entire Python module
│── setup.py               # Installation configuration
│── requirements.txt       # Optional: Dependencies list
│── README.md              # Optional: Project info

`setup.py` Example for a Single File:

from setuptools import setup

setup(
    name="module_name",
    version="0.1",
    py_modules=["module"],  # Instead of 'packages', use 'py_modules'
)

Creating the ZIP File:

zip -r module.zip module.py setup.py requirements.txt README.md

Installing with `pip`:

pip install https://example.com/module.zip

This works similarly to installing a full package, but since there’s only one file, you use py_modules instead of packages in setup.py.

Copper anniversary

2024-10-22T00:00:00+02:00

Today, this blog reaches the 12.5 year mark (In the Netherlands, copper annyversary).

Actually 0ink.net has been running for much longer than that, but as a blog format, it is from 2013-05-15 when it was migrated to Wordpress.

You can find the firs post here.

Using the Github Container Registry

2025-05-05T00:00:00+02:00

Initialization
Basic env variables
Job declaration
Steps
Using the image

The GitHub Container Registry (GHCR) allows you to host and manage Docker container images in your personal or organisation account on GitHub. One of the benefits is that permissions can be defined for the Docker image, independent of any repository. Thus, your repository could be private and your Docker image public.

This feature is part of Github's Packages feature. To manage your images you need to go to your personal or organization's page and click on the Packages tab.

By default, packages are "Private". You can change this default by going to your account or org's settings page, click on the "Packages" option on the side bar. Under "Packages permissions" -> "Package creation", you can change the default visibility.

In general, you can have a Github repo containing the source for a container image, and through the use of Github Actions, you can automatically publish the container to the GHCR. From then on, you can pull the container image using docker engine.

To enable this you need to create a .github/workflows/docker-image.yml file with the contents:

Initialization

name: Docker Image CI

on:
  workflow_dispatch:
  push:
    branches: [ "prerel", "prerel-*" ]
    tags: [ "*" ]
  pull_request:
    branches: [ "main" ]

These control events on when to run the workflow. See Events that trigger workflows.

For rolling releases I would use:

on:
  workflow_dispatch:
  push:
    branches: [ "main", "prerel", "prerel-*", "bugfix*" ]
  schedule:
  # Run every 8th of the month
  - cron: "0 2 8 * *"

Basic env variables


env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

Declares some environmental variables. Here we are indicating that we will be using GHCR. Also, we are using the repository name (owner/repo) for the image name. Be careful that the IMAGE_NAME should be all lowercase.

Job declaration

jobs:

  build:

    runs-on: ubuntu-latest

    permissions:
      contents: read
      packages: write
      attestations: write
      id-token: write

This starts the actual jobs to run. The permissions block is important as these are needed to configure the automatic GITHUB_TOKEN. See Controlling permissions for GITHUB_TOKEN for details.

Steps

    steps:

Workflow requires the following steps:

Run-time additional env variables

    - name: Additional environment
      run: |
        (
          echo "IMAGE_NAME=$(echo "${{ env.IMAGE_NAME }}" | tr A-Z a-z)"
          echo "PKG_OWNER=$(echo "${{ env.IMAGE_NAME}}" | cut -d/ -f1)"
          echo "PKG_NAME=$(basename "${{ env.IMAGE_NAME}}")"
          echo "BUILD_DATE=$(date +'%Y%m%d')"
        ) >> $GITHUB_ENV

This makes sure that the env.IMAGE_NAME is all lowercase. This is a docker client limitation. It also splits the image name into env.PKG_OWNER and env.PKG_NAME. The env.PKG_NAME is needed later by the actions/delete-package-versions action.

Git source checkout

    - name: checkout repository
      uses: actions/checkout@v4

This is fairly standard. Check out our source code.

Additional configuration

    - name: Read additional environment from file
      uses: cosq-network/dotenv-loader@v1.0.2
      with:
        env-file: dotenv

Read additional environment variables from a file. In this example, dotenv.

This is an optional step. I am using this to define certain variables instead of hardcoding them into the Dockerfile.

Log in to GHCR

    - name: Log in to the Container registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

This steps logs you into the GHCR so you are able to push images. It uses docker/login-action. It supports to many container repositories such as docker hub. Not just GHCR.

Container metadata

    - name: Extract metadata (tags, labels) for Docker
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=schedule
          type=ref,event=branch
          type=ref,event=tag
          type=ref,event=pr
          type=raw,value=

Gets meta data using docker/metadata-action

Tags are generated according to:

Event	Ref	Docker Tags
`pull_request`	`refs/pull/2/merge`	`pr-2`
`push`	`refs/heads/master`	`master`
`push`	`refs/heads/releases/v1`	`releases-v1`
`push tag`	`refs/tags/v1.2.3`	`v1.2.3`, `latest`
`push tag`	`refs/tags/v2.0.8-beta.67`	`v2.0.8-beta.67`, `latest`
`workflow_dispatch`	`refs/heads/master`	`master`

In my example, I also generate a tag based on the date.

Alternatively, for rolling releae I would use:

        tags: |
          type=raw,value=latest,enable={{ is_default_branch }}
          type=raw,value=

Essentially, we are tagging by date, and the default branch also tags as latest.

Container build and push

    - name: Build and push Docker image
      id: push
      uses: docker/build-push-action@v3
      with:
        context: .
        push: true
        tags:  ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        build-args: |
          LIBC=${{ env.LIBC }}
          PYTHON_VERSION=${{ env.PYTHON_VERSION }}
          PYINSTALLER_VERSION=${{ env.PYINSTALLER_VERSION }}
          UPX_VERSION=${{ env.UPX_VERSION }}

In this step, we create the image and push it to the GHCR. It makes use of docker/build-push-action.

Tags make use of the output of docker/metadata-action, also make use of some pre-computed values from a previous step.
We also pass configuration values to the Dockerfile using build-args.

Artifact attestation

    - name: Generate artifact attestation
      uses: actions/attest-build-provenance@v2
      with:
        subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
        subject-digest: ${{ steps.push.outputs.digest }}
        push-to-registry: true

Generates signed build provenance attestations for workflow artifacts.

Artifact Attestations allow project maintainers to effortlessly create a tamper-proof, unforgeable paper trail linking their software to the process which created it.

For more details read Introducing Artifact attestations.

Cleaning Container registry

Storage isn't free and registries can often get bloated with unused images. Having a retention policy to prevent clutter makes sense in most cases.

In GCHR you can use the action actions/delete-package-versions to keep things tidy.

Example usage:

    - uses: actions/delete-package-versions@v5
      with:
        package-name: "${{ env.PKG_NAME }}"
        package-type: 'container'
        min-versions-to-keep: 9

This simply keeps only the last 9 images.

Using the image

Once the container image is build and published to the GCHR you can run it with a command:

docker run ghcr.io/pkgowner/imagename:latest

Using Excel files from Python

2025-05-05T00:00:00+02:00

Intro
XlsxWriter
openpyxl
xlwings
pywin32
Conclusions
Hints and Tips

Intro

The other day I had to create a report in Excel using Python for work.

Python has no shortage of libraries to create Excel files. There are portable solutions that would work on Windows and Linux, as well as solutions that only run on Windows and depend on Excel being available.

They have their pros and ther cons. This is a run down of the possible options.

XlsxWriter

The first option I tried was XlsxWriter. This solution is cross-platform, being compatible with Linux and Windows. It does not have many dependancies. It however is limited to writing only and does not support creating VBA macros in the created Excel files.

XlsxWriter is a Python module for writing files in the Excel 2007+ XLSX file format.

XlsxWriter can be used to write text, numbers, formulas and hyperlinks to multiple worksheets and it supports features such as formatting and many more, including:

100% compatible Excel XLSX files.
Full formatting.
Merged cells.
Defined names.
Charts.
Autofilters.
Data validation and drop down lists.
Conditional formatting.
Worksheet PNG/JPEG/GIF/BMP/WMF/EMF images.
Rich multi-format strings.
Cell comments.
Integration with Pandas and Polars.
Textboxes.
Support for adding Macros.
Memory optimization mode for writing large files.

It supports Python 3.4+ and PyPy3 and uses standard libraries only.

It is ideal for generating reports and has a very pythonic API.

openpyxl

This is the second option. The main advantage over XlsxWriter is that allows reading existing xlsx files as well as writing. Like XlsxWriter it is a portable library, able to run on Windows and Linux. It does not have any strange dependancies. The main disadvantage I found is that it is a bit temperamental, and the order of how you do things can impact the output of the generated file. The API specifically is a bit wonky with things being non-intuitive. Also, like XlsxWriter, openpyxl does not support reading or writing VBA macros.

openpyxl is a Python library to read/write Excel 2010 xlsx/xlsm/xltx/xltm files.

It was born from lack of existing library to read/write natively from Python the Office Open XML format.

It was initially based on the PHPExcel library.

If you have to read and write Excel files on Linux as well as Windows, this seems to be the way to go. However, be prepared for a steep learning curve and do a lot of trial and error to get things working right.

xlwings

The third option is xlwings. This is a freemium library that has multiple versions with different capabilities. These are:

xlwings open source
xlwings (Open Source) is a BSD-licensed Python library that makes it easy to call Python from Excel and vice versa:
- Scripting: Automate/interact with Excel from Python using a syntax close to VBA.
- Macros: Replace VBA macros with clean and powerful Python code.
- UDFs: Write User Defined Functions (UDFs) in Python (Windows only)
xlwings pro
xlwings pro adds these features:
- One-click Installer: Easily build your own Python installer including all dependencies—your end users don’t need to know anything about Python.
- Embedded code: Store your Python source code directly in Excel for easy deployment.
- xlwings Reports: A template-based reporting mechanism, allowing business users to change the layout of the report without having to touch the Python code.
- Markdown Formatting: Support for Markdown formatting of text in cells and shapes like e.g., text boxes.
- Permissioning of Code Execution: Control which users can run which Python modules via xlwings.
- Table.update(): An easy way to keep an Excel table in sync with a pandas DataFrame
xlwings lite
xlwings Lite brings the VBA experience into the modern age by offering a privacy-first, secure, and developer-friendly way to automate Excel and write custom functions with Python.
xlwings server
xlwings Server adds Python support to Microsoft Excel and Google Sheets without the need of a local Python installation. xlwings Server is self-hosted and runs on any platform that supports Python or Docker, including bare-metal servers, Linux-based VMs, Docker Compose, Kubernetes and serverless products like Azure functions or AWS Lambda.

For the simple use case of reading and writing Excel file data it is a bit of an overkill. It does support adding VBA code to Excel files, however there are limitations associated with the VBA object model. So while code can be added, classes and expred modules can not.

pywin32

You can always use the Python for Win32 with COM support. This gives you access to the Component Object Model (COM), which is the native automation functionality of Windows which Excel supports.

This is a quick example:

import win32com.client
import sys, io

# Open up Excel and make it visible (actually you don't need to make it visible)
excel = win32com.client.Dispatch('Excel.Application')
excel.Visible = True

# Redirect the stdout to a file
orig_stdout = sys.stdout
bk = io.open("Answers_Report.txt", mode="w", encoding="utf-8")
sys.stdout = bk

# Select a file and open it
file = "path_of_file"
wb_data = excel.Workbooks.Open(file)

# Get the answers to the Q1A and write them into the summary file
mission=wb_data.Worksheets("1ayb_MisiónyVisiónFutura").Range("C6")
vision =wb_data.Worksheets("1ayb_MisiónyVisiónFutura").Range("C7")
print("Question 1A")
print("Mission:",mission)
print("Vision:" ,vision)
print()

Conclusions

For strictly writing Excel files, XlsxWriter is more than enough. It is easier to use and have a nice Pythonic API.

For reading and writing Excel files, openpyxl is enough, despite its flaws.

If you need limited support for VBA code, xlwings is the best option.

Hints and Tips

OpenPyxl

openpyxl has a lot of quirks. Here is a number of things I discovered:

Data Validation

This is documented in openpyxl Data Validation and data validation module.

Data validators can be applied to ranges of cells but are not enforced or evaluated. Ranges do not have to be contiguous: eg. “A1 B2:B5” is contains A1 and the cells B2 to B5 but not A2 or B2.

Example:


# Create a data-validation object with list validation
dv = DataValidation(type="list", formula1='"Dog","Cat","Bat"', allow_blank=True)

The formula1 is a formula that can contain a list. The list is comma separated. You can use double quotes if you need to escape values with commas. Double quotes can be escaped by entering two double quotes. (i.e. in general, it folows Excel formula syntax).

The other strange thing is that if you want to show a drop down box with the list options you can add the keyword argument showDropDown. If you want to see the dropdown, set this to False. If you want to hide it, set it to True.

Creating groups

For creating groups you can use

  worksheet.column_dimensions.group(start, end, **kwargs)
  worksheet.row_dimensions.group(start, end, **kwargs)

Keyword args:

hidden : hide group
outline_level : for multilevel groupings

There a further restrictions with creating groups, in that:

you must create the cells in the groups before grouping
if you are using multi-level groups, you must create the outer-group first before creating the inter groups.

xlwings

Some tips when using xlwings:

Steps to Enable Programmatic Access to VBA

Open Excel:
- Launch Excel (you don't need to open a specific workbook).
Access Options:
- Click on File in the top menu.
- Select Options from the menu to open the Excel Options dialog. Options item should be at the bottom of the menu.
Trust Center:
- In the Excel Options dialog, click on Trust Center in the left pane.
- Click on Trust Center Settings... to open the Trust Center settings.
Macro Settings:
- In the Trust Center, click on Macro Settings.
- Check the option labeled "Trust access to the VBA project object model".
Save Settings:
- Click OK to save the Trust Center settings.
- Click OK again to exit the Excel Options dialog.

Additional Considerations

Security Implications: Enabling programmatic access to the VBA project object model can expose your system to potential security risks, especially if you run untrusted code. Ensure that the code you execute is from a trusted source.
IT Policies: If you are using Excel in a corporate environment, these settings might be managed by IT policies, and you may need to contact your IT department for assistance.

Using Podman on Alpine Linux

2025-04-01T00:00:00+02:00

Introduction
What is Podman?
Installation on Alpine Linux
- Rootful vs Rootless
- Configuring Podman
Compatibility with Docker
- Moving of --link'ed containers

Introduction

I've been using Docker for a few years, and recently decided to switch to Podman for a new server I'm building. I was pleasantly surprised by how quick and easy the transition was.

What is Podman?

Podman is an alternative to Docker Desktop.
Podman is an open source container management tool designed to provide a daemonless and rootless approach to running containers, setting it apart from traditional container platforms like Docker. It allows users to manage containers, pods, and images using a command-line interface similar to Docker's, making it accessible for those familiar with Docker commands. Podman is particularly focused on security, enabling containers to run without requiring root privileges, which helps minimize potential vulnerabilities. It integrates seamlessly with tools like Kubernetes, supporting native pod management. Developed as part of the Red Hat ecosystem, Podman is a versatile choice for users seeking a secure, lightweight, and flexible alternative for container management.

Podman bills itself as an open source alternative to Docker, despite Docker also being open source. Docker Inc., the company behind Docker, has a business model that includes offering commercial products and services around its open source technology.

Podman is an alternative to Docker Desktop. It's an open-source container management tool that offers a daemonless and rootless approach to running containers, differentiating it from traditional platforms like Docker. Podman allows users to manage containers, pods, and images using a command-line interface similar to Docker, making it user-friendly for those familiar with Docker commands. It emphasizes security by enabling containers to run without root privileges, reducing potential vulnerabilities. Podman integrates well with Kubernetes, supporting native pod management, and is developed as part of the Red Hat ecosystem. It's a versatile option for users looking for a secure, lightweight, and flexible alternative for container management.

While Podman promotes itself as an open-source alternative to Docker, it's worth noting that Docker is also open source. Docker Inc. offers commercial products and services around its open-source technology.

Installation on Alpine Linux

Since my preferred server's operating system is Alpine Linux, I installed Podman on a test Alpine Linux virtual machine.

First, enable the Alpine Linux community repository by modifying the /etc/apk/repositories file and uncommenting the following line:

http://dl-cdn.alpinelinux.org/alpine/vX.YY/community

Then, install Podman with the following command:

apk add podman

To enable Podman at startup, use:

rc-update add podman

This is all that's needed to run Podman in rootful mode.

Rootful vs Rootless

Podman supports two modes of operation: rootful and rootless. In rootful mode, the container runs as root on the Linux host (or VM on Mac/Windows), while in rootless mode, it runs under a standard Unix user account. Rootless mode offers stronger security, but some containers may not work under the increased restrictions. For example, containers that create new devices or perform restricted operations must run as root. This is different from the USER value in Containerfile/Dockerfile, which affects how processes inside the container perceive themselves. In rootless mode, processes that appear as root inside the container are actually running as a restricted user on the host system.

Since I'm migrating from Docker, I've decided to keep everything in rootful mode.

The Alpine Linux wiki provides steps for installing Podman in rootless mode. Note that the instructions for running as root are slightly outdated.

Configuring Podman

I usually run my servers in diskless and/or data disk modes. Because Podman expects some data to be persistent, I modify the configuration to store persistent data elsewhere:

Ensure Podman is not running:

service podman stop

Edit /etc/containers/storage.conf, find the [storage] section, and modify the graphroot key to point to your persistent storage location:

graphroot = "/media/data/containers/podman"

Make sure that the graphroot directory exists:

mkdir -p /media/data/containers/podman

If you stopped Podman earlier, you can start it now:

service podman start

Compatibility with Docker

For composer compatibility you can use Podman Compose or docker compose itself (with the previously mentioned socket API compatibility). I myself chose to use docker compose instead of Podman Compose becase Podman Compose depends on Python (and I wanted to keep my install leaner).

To enable this you need to install docker compose:

Podman includes a socket API that's drop-in compatible with Docker tools, allowing you to use the Docker CLI, docker-compose, and even Portainer.

For compatibility with compose tools, you can use Podman Compose or docker-compose itself, thanks to the socket API compatibility. I've chosen to use docker-compose instead of Podman Compose because the latter depends on Python, and I wanted to keep my installation leaner.

To enable this, install docker-compose:

apk add docker-cli-compose

For convenience, create a small script at /usr/local/bin/podman-compose:

#!/bin/sh
# Set DOCKER_HOST pass -H to override
export DOCKER_HOST=unix:///run/podman/podman.sock
exec docker-compose "$@"

Using the socket API, you can manage Podman with Docker commands via the -H option or the DOCKER_HOST environment variable. For more information, see the Docker manual. On the Podman host, you can use:

-H unix:///run/podman/podman.sock or
export DOCKER_HOST=unix:///run/podman/podman.sock

Interestingly, you can control a remote Podman instance using this method. For example, you can use the Docker CLI on your development PC to control the Podman instance on your server:

-H ssh://root@remote-server:22/run/podman/podman.sock
export DOCKER_HOST=ssh://root@remote-server/run/podman/podman.sock

Moving of --link'ed containers

Old containers using Container Links can not be moved directly to Podman without changes. The Container Links feature is now obsolete and Docker supports it only for compatibility.

In Podman you are better off creating a new network using:

podman create network NETWORK_NAME

Connect your containers to that network:

podman run -d \
    -p 8080:80 \
    --restart=always \
    --network=NETWORK_NAME \
    nginx:latest

Then you can use the container name using the DNS resolver. Note that the default Podman network has this internal DNS resolver disabled and you need to create a new one.

If you are using docker compose a network is created by default and all the containers in the stack will be connected to it.

AdGuard Home

2025-03-12T00:00:00+01:00

How does AdGuard Home compare to traditional ad blockers
Known limitations
Installation
Configuration
Reverse Proxy
Using it
Conclusion

AdGuard Home is a network-wide ads & trackers blocking DNS server.

After set-up, it will cover all your home devices, and you don’t need any client-side software for that.

With the rise of Internet-Of-Things and connected devices, it becomes more and more important to be able to control your whole network.

It operates as a DNS server that re-routes tracking domains to a “black hole”, thus preventing your devices from connecting to those servers.

As a network-wide ad-blocker, AdGuard Home competes with Pi-hole. Here is a comparison table between the two:

Feature	AdGuard Home	Pi-hole
Platform Support	Windows, macOS, Linux, Raspberry Pi, Docker	Linux, Raspberry Pi, Docker
DNS-over-HTTPS/TLS	Built-in support	Requires additional setup (using third-party tools)
Filtering Rules Syntax	AdGuard-specific syntax; supports more complex rules	Standard DNS filtering rules
Web Interface	Modern, user-friendly interface	Simple, functional interface
Block Page Customization	Limited customization	More customizable block page
Parental Control	Built-in support for parental control features	Lacks built-in parental control features
Regular Expressions	Supports regular expressions in filters	Supports regular expressions in filters
Automatic Updates	No built-in automatic updates	Built-in automatic updates for blocklists and core components
Community Support	Growing community	Larger, well-established community

As you can see, there is no clear winner. Most people would choose Pi-Hole as it has a larger, more well-established community. I opted for AdGuard Home because it has a simpler installation process due to very minimal dependancies.

I am using dnsmasq for my DNS and DHCP needs. It is quite flexible and configurable. There is some things that I am addressing with AdGuard Home that I was never able to address with dnsmasq itself.

Ad Blocking: While it is possible to download block lists and add them to dnsmasq, this has to be done manually and it was difficult to keep up to date.
Reporting and stats: dnsmasq is able to log DNS queries, you have to create repots and generate statistics on your own.
DNS interface: dnsmasq lacks this completely.

AdGuard Home does not cover all the functionality that dnsmasq provides such as:

tftp for PXE booting
No DHCP configuration for IPv6 (at least I couldn't find instructions for it)
CLI integration

At the end, I am using AdGuard Home and dnsmasq in combination.

How does AdGuard Home compare to traditional ad blockers

It depends.

DNS sinkholing is capable of blocking a big percentage of ads, but it lacks the flexibility and the power of traditional ad blockers. You can get a good impression about the difference between these methods by reading this article, which compares AdGuard for Android (a traditional ad blocker) to hosts-level ad blockers (which are almost identical to DNS-based blockers in their capabilities). This level of protection is enough for some users.

Additionally, using a DNS-based blocker can help to block ads, tracking and analytics requests on other types of devices, such as SmartTVs, smart speakers or other kinds of IoT devices (on which you can't install traditional ad blockers).

Known limitations

Here are some examples of what cannot be blocked by a DNS-level blocker:

YouTube, Twitch ads;
Facebook, Twitter, Instagram sponsored posts.

Essentially, any advertising that shares a domain with content cannot be blocked by a DNS-level blocker.

Is there a chance to handle this in the future? DNS will never be enough to do this. Our only option is to use a content blocking proxy like the standalone AdGuard applications.

Installation

AdGuard Home is hosted on github. You can download a pre-build binary from its releases page. Since AdGuard Home is developed in Go language, its binaries seem to be fairly portable.

Installing on Alpine Linux is fairly simple. I am using an Ansible playbook for this, but the process is as follows:

I am using dnsmasq with AdGuard Home. So the pre-requisite step is to change the dnsmasq so that it does not listen on port 53/UDP. This is done with the line:
```
port=5353
```
Download binary from the releases page.
Unpack the binary into your destination directory. I use /usr/local/lib/AdGuardHome.

AdGuard Home reads configuration and saves some data in its installation directory. I relocate these via symbolic links:

agh_home=/usr/local/lib/AdGuardHome
ln -s /etc/AdGuardHome.yaml ${agh_home}/AdGuardHome.yaml
ln -s /var/lib/AdGuardHome ${agh_home}/data

Create the start|stop scripts. I am using:

#!/bin/sh
# Startup script
appdir=/usr/local/lib/AdGuardHome
mkdir -p "$(readlink -f "$appdir/data")"
chmod 0700 "$(readlink -f "$appdir/data")"
if [ ! -f /etc/AdGuardHome.yaml ] ; then
  cp /etc/AdGuardHome.proto /etc/AdGuardHome.yaml
elif [ /etc/AdGuardHome.proto -nt /etc/AdGuardHome.yaml ] ; then
  cp /etc/AdGuardHome.proto /etc/AdGuardHome.yaml
fi
$appdir/AdGuardHome &

and

# Shutdown script
#!/bin/sh
killall AdGuardHome

The startup script is doing a few things:

Make sure that the data directory has the right permissions.
Initialize configuration if needed (More on that later).

The default configuration will listen on port 3000/TCP. Connect to it with a web browser to configure AdGuard Home.

Configuration

Actually, for the initial configuration I prefer to use prepared file. This is the AdGuardHome.proto file that is referenced in the start script.

This YAML file can contain the settings that you want to override from the defaults.

I am using this file:

http:
  address: 10.0.2.254:3000
users:
- name: admin
  password: $2a$....DELETED....
dns:
  bind_hosts:
    - 0.0.0.0
  port: 53
  upstream_dns:
    - '# https://dns10.quad9.net/dns-query'
    - 127.0.0.1:5353
  hostsfile_enabled: false
  use_private_ptr_resolvers: true
  local_ptr_upstreams:
    - 127.0.0.1:5353
log:
  enabled: true
  file: "/var/log/AdGuardHome.log"

The http section is fixing the IP address and port it will listen to.

The users section can be used to configure authentication. You can have multiple user names and passwords. The passwords are based on Bcryp2 hashing. You can use: https://www.devglan.com/online-tools/bcrypt-hash-generator to generate the hashes.

If you do not want to do user authentication, for example, if placing behind a reverse proxy that does the authentication already, you can use:

users: []

This disables the internal user name and password authentication.

In the dns section, I am doing a number of tweaks:

upstream dns and local_ptr_upstream are configured to point to 127.0.0.1:5353 which is the locally running dnsmasq that was re-configured earlier.
hostfile_enabled is set to false to disable /etc/hosts file usage. Personally I expect the host names defined by /etc/hosts only to be used by the local system, and shouldn't be exported to DNS.

Reverse Proxy

Reverse proxying can be used to provide for example single sign on (SSO) by handling the authentication on the reverse proxy itself. This is very simple to do. It can be done either at the virtual host level, or at a sub folder level. In nginx you can use the code:

location /adghome/ {
    proxy_pass http://10.0.2.254:3000/;
}

Or using a virtual host:

location / {
    proxy_pass http://10.0.2.254:3000/;
}

Using it

Once properly configured, clients will receive via DHCP the AdGuard Home and start using directly. You can browse to the user interface and you will see statistics and configuration.

Conclusion

In short, AdGuard Home is much like your run-of-the-mill browser ad blocker, but rather than being a plug-in on your favourite web browser, AdGuard Home is a fully-fledged server application which runs on a separate machine somewhere on your network (or perhaps even on a VPS you own). Its primary goal is to provide your network with a mechanism to actively block certain requests that websites you visit make – in this case, requests for adverts, malware, or various other malicious things.

Using SSLH on Alpine Linux

2025-01-21T00:00:00+01:00

Install sslh

apk add sslh

Configure sslh startup:

# /etc/conf.d/sslh
# configuration for /etc/init.d/sslh
#
# The sslh binary to run; one of:
#
# fork    Forks a new process for each incoming connection. It is well-tested
#         and very reliable, but incurs the overhead of many processes.
# select  Uses only one thread, which monitors all connections at once. It is
#         more recent and less tested, but has smaller overhead per connection.
#mode="fork"
mode="select"
# Path of the configuration file.
#cfgfile="/etc/sslh.conf"
# Additional options to pass to the sslh daemon. See sslh(1) man page.
#command_args=""
# Uncomment to run the sslh daemon under process supervisor.

We set mode to select as this is most efficient. For package versions before 2.1.2-r1 the default init script will fail to start when fork mode is selected.

Create sslh configuration. For more configuration examples see example.cfg.

# /etc/sslh.conf

timeout: 2;
# Timeout before forwarding the connection to the timeout protocol
# The default is 2s.  If using OpenVPN, this needs to be increased to
# 5s for OpenVPN to work reliably.

transparent: true;
# Make SSLH act as a transparent proxy.  See later for more
# configuration steps.

user: "nobody";
# Run process as the given user.  Note that transparent mode
# will not work if user is not root.

numeric: true;
# Do not attempt to resolve hostnames: logs will contain IP addresses.
# This is mostly useful if the system's DNS is slow and running the sslh-select variant,
# as DNS requests will hang all connections.

listen:
(
  { host: "0.0.0.0"; port: "443"; },
  { host: "::"; port: "443"; }
);
# List of listening address and ports.  This is probably enough, but sslh
# can also listen on UDP sockets, IPv6 addresses, and UNIX sockets.

protocols:
(
  { name: "ssh"; host: "localhost"; port: "22";
    fork: true;},
  { name: "tls"; host: "localhost"; port: "8443";
    sni_hostnames: [ "app1.example.com", "app2.example.com" ];
    log_level: 0;  tfo_ok: true },
  { name: "tls"; host: "localhost"; port: "9443";
    sni_hostnames: [ "app3.example.com", "app4.example.com" ];
    log_level: 0;  tfo_ok: true },
  { name: "tls"; host: "localhost"; port: "7443";
    log_level: 0; tfo_ok: true; },
  { name: "tls"; host: "localhost"; port: "443";
    log_level: 0; tfo_ok: true; },  
);
# Connections that match the given protocol will be forwarded to host:port.
# For TLS, you can match based on sni_hostnames from the TLS handshake.
# Options:
# log_level: 0 turn off logging
#            1 logs every incoming request.  Hor https, makes sense to turn off logging
# fork: For sslh-select, will fork a new process for the connection
# tfo_ok: Enable TCP fast open.

# on-timeout: "ssh";
# By default it will connect to the "ssh" specification on time-out.  If
# you want to change the time out target, create a "timeout" protocol
# and set on-timeout: to "timeout".  The default to connect to "ssh"
# on timeout is a sane default.

# Logging configuration
# Value: 1: stdout; 2: syslog; 3: stdout+syslog; 4: logfile; ...; 7: all
verbose-config: 0; #  print configuration at startup
verbose-config-error: 3;  # print configuration errors
verbose-connections: 3; # trace established incoming address to forward address
verbose-connections-error: 3; # connection errors
verbose-connections-try: 3; # connection attempts towards targets
verbose-fd: 0; # file descriptor activity, open/close/whatnot
verbose-packets: 0; # hexdump packets on which probing is done
verbose-probe-info: 3; # what's happening during the probe process
verbose-probe-error: 3; # failures and problems during probing
verbose-system-error: 3; # system call problem, i.e.  malloc, fork, failing
verbose-int-error: 3; # internal errors, the kind that should never happen```

For https traffic, it is advisable to enable tfo_ok as it enables TCP Fast Open feature, which reduces the connection time by reducing round-trips.

For trasnparent proxying, I have tested two scenarios.

One host running sslh and sshd and web server.
Two hosts, host A running sslh, host B running sshd and web server. Default route of host B points to host A.

These an additional scenario are described in the sslh documentation.

Single host transparent proxy

In this example, IP address 203.0.113.23 is being used. This is for a a host with a single external IP interface eth0.

Modify OpenSSh configuration:

# Restrict interfaces that sshd will bind to
ListenAddress 0.0.0.0:22            # Direct access
ListenAddress 203.0.113.23:2201     # Access via sslh

Modify nginx configuration:

listen 203.0.113.23:4433 ssl http2;

By using dedicated ports for sslh traffic, we make it simpler to identify the transparently proxied traffic.

Create a netfilter table ( /etc/nftables.d/sslh.nft ):


#!/usr/sbin/nft -f

# Use ip as we want to configure sslh only for IPV4
table ip sslh {
        chain output {
                type route  hook output  priority -150;
                oif eth0  tcp sport { 2201, 4433 }  counter  mark set 0x4155;
        }
}

Since we were using dedicated ports earlier, we can use these simple netfilter rules to identify the traffic that is being proxied. We then mark with 0x4155 so that it can be directed to the loopback interface.

In the /etc/network/interfaces configuration:

iface eth0 inet static
  address 203.0.113.23
  netmask 255.255.255.0
  up   ip rule add fwmark 0x4155 lookup 100
  down ip rule del fwmark 0x4155 lookup 100
  up   ip route add local 0.0.0.0/0 dev lo table 100
  down ip route del local 0.0.0.0/0 dev lo table 100

This grabs the traffic marked with 0x4155 by the netfilter rules and routes them to the loopback so sslh can pick it up.

See sslh installation on how to install on a debian system.

Two host transparent proxy

I find this configuration simpler/nicer as does not use nftables rules. You need to add the following routing rules somewhere. I am using a file in /etc/local.d directory for Alpine Linux. Other options are the startscript of sslh, the /etc/conf.d/sslh file, the configuration of the outgoing interface. The lines needed are:

ip rule add from $HOST_B_ADDRESS/32 table 100
ip route add local 0.0.0.0/0 dev lo table 100

If you are forwarding connections to more hosts, you need to add those rules here.

It is important to note that the $HOST_B_ADDRESS mentioned needs to be an IP address dedicated to sslh traffic. Otherwise normal traffic for host B would also be redirected to the host A loopback device.

Writting a Geany plugin

2025-01-15T00:00:00+01:00

Pre-requisites
Registering
Building
Binding keys
Callbacks
Convenience functions
Recording
Playback
Finishing touches

I wrote a simple C language plugin for Geany. It is similar to geany's standard keyrecord plugin which simply records and playbacks key sequences. However unlike keyrecord which uses key events, it uses the Scintilla macro recording functionality like geanymacros plugin. However, unlike geanymacros does not do macro editing, and binding macros to multiple keys.

My plugin can be found in this Github repo. Most of the boiler plate I got it from another plugin, Ctrl-Alt.

Pre-requisites

I use void-linux, so in void you first need to install the pre-requisites. This can be done with the command:

sudo xbps-install -S base-devel geany-devel

Registering

Firs you need to include <geanyplugin.h> which brings all the [Geany][geany] API as well as the necessary GTK headers.

#include <geanyplugin.h>

and export a function named geany_load_module(). This function is used to tell Geany of your plugin. This includes metadata such as name, description, version, etc. But also includes pointers to function such as:

init - plugin initialization
cleanup - finalization
configure - optional configuration function
help - optional help functionality
callbacks - optional array of PluginCallback functions

Building

To compile a plugin:

gcc -c plugin.c -fPIC `pkg-config --cflags geany`

Linking the plugin:

gcc plugin.o -o plugin.so -shared `pkg-config --libs geany`

If all went OK, put the library into one of the paths Geany looks for plugins, e.g. $prefix/lib/geany or $HOME/.config/geany/plugins.

Binding keys

Previous versions of the Geany API a plugin would register bindings itself. Currently, a plugin defines a binding as a _groupname and _keyname and the user can configure the actual key combination.

To define the key bindings you must add to the init function:

key_group = plugin_set_key_group(plugin, PLUGIN_KEY_NAME, KB_COUNT, NULL);
keybindings_set_item(key_group, KB_RECORD, macrec_record, 0, 0,
            "record", "Start/Stop recording macro", NULL);
keybindings_set_item(key_group, KB_PLAY, macrec_play, 0, 0,
            "playback", "Playback macro", NULL);

Where:

PLUGIN_KEY_NAME is a string for naming the key group.
KB_COUNT is the number of keys in the key group. This is defined using an C enum. KB_RECORD and KB_PLAY are in the same enum and are unique numbers identifying a key combination.
macrec_record and macrec_play are the callback functions executed when the given keybinding is pressed.

Callbacks

Callbacks are functions called whenever a Geany event is fired. These are defined in geany_load_module function as:

plugin->funcs->callbacks = macrec_callbacks;

macrec_callbacks is defined as:

static PluginCallback macrec_callbacks[] =
{
        { "editor-notify", (GCallback) &on_editor_notify, FALSE, NULL },
        { "document-close", (GCallback) &on_document_close, FALSE, NULL },
        { NULL, NULL, FALSE, NULL }
};

We are hooking to events, editor-notify which is used for recording macros and uses the Scintilla built-in recording facility.

document-close is used to abort recording of macros.

Convenience functions

For this plugin I am using a few convenience functions:

ui_set_statusbar - Display text on the status bar.
msgwin_status_add - Logs a formatted status message without setting the status bar.
utils_open_browser - Tries to open the given URI in a browser.

Recording

For macro recording we using Scintilla functionality. This is in contrast to the keyrecord plugin which uses low-level GTK functionality which for me is not vedry reliable.

Start recording:

scintilla_send_message(document_get_current()->editor->sci,SCI_STARTRECORD,0,0);

After that, we capture events using editor-notify callback. In that function we use a table to classify and identify the messages we want to handle:

/* structure to hold details of Macro for macro editor */
typedef struct {
  gint message;
  const gchar *description;
} MacroDetailEntry;

/* list of editor messages this plugin can handle & a description */
const MacroDetailEntry MacroDetails[]={
{SCI_CUT,N_("Cut to Clipboard")},
{SCI_COPY,N_("Copy to Clipboard")},
{SCI_PASTE,N_("Paste from Clipboard")},
{SCI_LINECUT,N_("Cut current line to Clipboard")},
{SCI_LINECOPY,N_("Copy current line to Clipboard")},

... content deleted ...

{0,NULL}
};

Once recording we stop Scintilla macro recording.

scintilla_send_message(document_get_current()->editor->sci,SCI_STOPRECORD,0,0);

Playback

Since the recording is done using Scintilla facilities we have to use Scintilla playback. This is realized with this function call:

scintilla_send_message(sci,me->message,me->wparam,(sptr_t)clipboardcontents);

To make sure that the undo history is consistent, we call:

scintilla_send_message(sci,SCI_BEGINUNDOACTION,0,0);

before playback, and call:

scintilla_send_message(sci,SCI_ENDUNDOACTION,0,0);

to finish the UNDO action.

Finishing touches

I also added a github action to the repository to compile the code (to ensure that things at least compile).

For that I am using a simple C/C++ workflow. What I added was the task to make sure that the correct dependancies are available:

- name: Install dependancies
  run: sudo apt-get install -y libgtk-3-dev autoconf automake autopoint gettext geany

Cinnamon Keyboard shortcuts

2025-01-21T00:00:00+01:00

To familiarize with the Cinnamon desktop, here is a list of its shortcuts:

Function	Category	Keys
Show the window selection screen	General	Ctrl+Alt+Down
Show the workspace selection screen	General	Ctrl+Alt+Up, Alt+F1, Super+Tab
Show desktop	General	Super+D
Show Desklets	General	Super+S
Cycle through open windows	General	Alt+Tab
Cycle backwards through open windows	General	Shift+Alt+Tab
Cycle through windows from all workspaces	General	Ctrl+Alt+Tab
Cycle backwards through windows from all workspaces	General	Ctrl+Shift+Alt+Tab
Run dialog	General	Alt+F2
Toggle Looking Glass	General:Troubleshooting	Ctrl+Alt+L (Default: Super+L)
Unmaximize window	Windows	Alt+F5
Close window	Windows	Alt+F4
Activate window menu	Windows	Alt+Space
Toggle maximization state	Windows	Alt+F10
Resize window	Windows:Positioning	Alt+F8
Move window	Windows:Positioning	Alt+F7
Push tile left	Windows:Tiling and Snapping	Super+Left
Push tile right	Windows:Tiling and Snapping	Super+Right
Push tile up	Windows:Tiling and Snapping	Super+Up
Push tile down	Windows:Tiling and Snapping	Super+Down
Move window to left workspace	Windows:Inter-workspace	Shift+Ctrl+Alt+Left
Move window to right workspace	Windows:Inter-workspace	Shift+Ctrl+Alt+Right
Move window to workspace above	Windows:Inter-workspace	Shift+Ctrl+Alt+Up
Move window to workspace below	Windows:Inter-workspace	Shift+Ctrl+Alt+Down
Move window to left monitor	Windows:Inter-monitor	Shift+Super+Left
Move window to right monitor	Windows:Inter-monitor	Shift+Super+Right
Move window to monitor above	Windows:Inter-monitor	Shift+Super+Up
Move window to monitor below	Windows:Inter-monitor	Shift+Super+Down
Switch to left workspace	Workspaces	Ctrl+Alt+Left
Switch to right workspace	Workspaces	Ctrl+Alt+Right
Log out	System	Ctrl+Alt+Delete
Shut down	System	Ctrl+Alt+End
Lock screen	System	Super+L, `Screensaver` (Default: Ctrl+Alt+L)
Suspend	System	`Sleep`
Hibernate	System	`Suspend`
Restart Cinnamon	System	Ctrl+Alt+Escape
Switch monitor configurations	System:Hardware	Super+P, `Display`
Rotate display	System:Hardware	`RotateWindows`
Orientation Lock	System:Hardware	Super+O
Increase screen brightness	System:Hardware	`MonitorBrignessUp`
Decrease screen brightness	System:Hardware	`MonitorBrignessDown`
Toggle keyboard backlight	System:Hardware	`KbdLightOnOff`
Increase keyboard backlight level	System:Hardware	`KbdLightUp`
Decrease keyboard backlight level	System:Hardware	`KbdLightDown`
Toggle touchpad state	System:Hardware	`TouchpadToggle`
Turn touchpad on	System:Hardware	`TouchpadOn`
Turn touchpad off	System:Hardware	`TouchpadOff`
Show power statistics	System:Hardware	`Battery`
Take a screenshot of an area	System:Screenshots and Recording	Shift+Print
Copy a screenshot of an area to clipboard	System:Screenshots and Recording	Ctrl+Shift+Print
Take a screenshot	System:Screenshots and Recording	Unassigned, Default: Print
Copy a screenshot to clipboard	System:Screenshots and Recording	Ctrl+Print
Take a screenshot of a window	System:Screenshots and Recording	Alt+Print
Copy a screenshot of a window to clipboard	System:Screenshots and Recording	Ctrl+Alt+Print
Toggle recording desktop	System:Screenshots and Recording	Shift+Ctrl+Alt+R
Launch terminal	Launchers	Ctrl+Alt+T
Launch calculator	Launchers	`Calculator`
Launch email client	Launchers	`Mail`
Launch web browser	Launchers	`WWW`
Home folder	Launchers	Super+E, `Explorer`
Search	Launchers	`Search`
Volume mute	Sound and Media	`AudioMute`
Volume down	Sound and Media	`AudioLowerVolume`
Volume up	Sound and Media	`AudioRaiseVolume`
Mic mute	Sound and Media	`AudioMicMute`
Launch media player	Sound and Media	`AudioMedia`
Play	Sound and Media	`AudioPlay`
Pause playback	Sound and Media	`AudioPause`
Stop playback	Sound and Media	`AudioStop`
Previous track	Sound and Media	`AudioPrev`
Next track	Sound and Media	`AudioNext`
Eject	Sound and Media	`Eject`
Rewind	Sound and Media	`AudioRewind`
Fast-forward	Sound and Media	`AudioFastForward`
Volume mute (Quiet)	Sound and Media:Quiet Keys	Alt+ `AudioMute`
Volume down (Quiet)	Sound and Media:Quiet Keys	Alt+ `AudioLowerVolume`
Volume up (Quiet)	Sound and Media:Quiet Keys	Alt+ `AudioRaiseVolume`
Zoom in	Universal Access	Alt+Super+=
Zoom out	Universal Access	Alt+Super+-
Turn screen reader on or off	Universal Access	Alt+Super+S

Spices, these relate to the panel applets:

Function	Category	Keys
Show Calendar	Calendar	Super+C
Global hotkey or cycling through thumbnail menus	Grouped window list	unassigned
Global hotkey to show the order of apps	Grouped window list	unnassigned, Default: Super+`
Keyboard shortcut to open and close the menu	Menu	Super
Show Network Mgr menu	Network Manager	Shift+Super+N
Show notifications	Notifications	Super+N
Clear notifications	Notifications	Shift+Super+C
Show Sound menu	Sound	Shift+Super+S

Unasigned functions

Function	Category
Move pointer to the next monitor	General:Pointer
Move pointer to the previous monitor	General:Pointer
Maximize window	Windows
Miminize window	Windows
Raise window	Windows
Lower window	Windows
Toggle fullscreen state	Windows
Toggle shaded state	Windows
Toggle always on top	Windows
Toggle showing window on all workspaces	Windows
Increase opacity	Windows
Decrease opacity	Windows
Toggle vertical maximization	Windows
Toggle horizontal maximization	Windows
Center window in screen	Windows:Positioning
Move window to upper-right	Windows:Positioning
Move window to upper-left	Windows:Positioning
Move window to lower-right	Windows:Positioning
Move window to lower-left	Windows:Positioning
Move window to right edge	Windows:Positioning
Move window to top edge	Windows:Positioning
Move window to bottom edge	Windows:Positioning
Move window to left edge	Windows:Positioning
Move window to new workspace	Windows:Inter-workspace
Move window to workspace 1-12	Windows:Inter-workspace
Switch to workspace 1-12	Workspaces:Direct Navigation
Launch help browser	Launchers
Repeat	Sound and Media
Shuffle	Sound and Media
Turn on-screen keyboard on or off	Universal Access
Increase text size	Universal Access
Decrease text size	Universal Access
High contrast on or off	Universal Access

My custom commands:

Function	Command	Keys
System Monitor	`gnome-system-monitor -r`	Ctrl+Alt+S
Switch Workspace	`wsswitch`	Super+`
Find Dialog	`catfish`	Super+F
Run Dialog	`rundlg`	Super+R
Screenshot UI	`gnome-screenshot -i`	`Print`
Show windows in All Workspaces	`rofi -show window`	Ctrl+Escape
patoggle	`patoggle output`	Super+KP_Insert

Lookup Ansible vars from the command line

2025-09-15T00:00:00+02:00

This is a quickie. If you want to look-up the value of an Ansible variable from the command line:


ansible localhost -m ansible.builtin.debug -a "var=my_variable"

Booting Refind on KVM

2025-07-15T00:00:00+02:00

This is a quick recipe to boot a refind image on KVM. This is just to test UEFI boot configuration in virtualized environments.

First step is to create a refind boot image. For that I use this script:

To use this script, you need to download the binary zip file refind image from here. Also, the following packages need to be installed:

util-linux : for sfdisk and fallocate
unzip
dosfstools : for mkfs.vfat.
mtools : for mcopy.

And create the image using the command:

sh mkimg.sh refind-bin-0.14.2.zip boot.img 12G

To use the newly created image in a KVM environment.

virt-install \
        --name vm1 \
        --memory 1024 \
        --vcpus 1 \
        --clock offset=utc \
        --disk boot.img,format=raw \
        --osinfo linux2022 \
        --graphics vnc \
        --autoconsole text \
        --virt-type kvm \
        --console pty \
        --rng /dev/urandom} \
        --import \
        --boot uefi

memory is in Megabytes.
osinfo values can be found using the command virt-install --osinfo list.
The main settings to pay attention here is --boot uefi, which tells the system to set up a UEFI VM.

Saving Cinnamon customizations

2025-01-03T00:00:00+01:00

When I switched Desktop Environments towards Cinnamon, I wanted to save configuration settings so I could restore them later if I moved to a different PC.

Pre-requisites:

We need the following packages

dconf

Optionally you may want to install:

dconf-editor
glib

The simplest way is to modify the settings using the built-in Cinnamon UI tools or cinnamon-settings. Once you have all the settings as you want run:

dconf dump / > customizations.ini

Or if you want to limit to cinnamon settings:

dconf dump /org/cinnamon/  > cinnamon.ini
dconf dump /org/gnome/ > gnome.ini

We need to dump cinnamon and gnome because cinnamon is gnome derived and some settings are stored there.

See the manpage for dconf for more details.

The result of these commands is a text file in INI format. Which can be edited with a text editor. I would normally edit the text file as it would contain some settings that I would prefer to leave as default.

When it is time to restore the settings you need to use the command:

dconf load / < customizations.ini

dconf load /org/cinnamon/  < cinnamon.ini
dconf load /org/gnome/ < gnome.ini

Modifying the INI file lets you access some settings not available directly from the GUI. To explore what is possible you can use the following commands:

gsettings list-schemas

or from the GUI:

dconf-editor

Linux booting Linux

2025-01-01T00:00:00+01:00

Introduction
Backing up
Installing Refind
Custom Linux
- voidlinux pre-requisites
- Setting up Buildroot

Introduction

This is an amusing weekend project. I have an old HP Stream 11 Pro G5 laptop that I wanted to repurpose. It originally came pre-installed with Windows 10, and I wanted to use it with Linux. Since my Linux distro of choice is voidlinux I downloaded the Live Image and wrote it to a USB drive.

I plugged the USB drive into the laptop and turned in on. I immeditely spammed the Escape key to get the boot menu. After selecting the USB drive as the boot device I booted into Linux. I was able to confirm that voidlinux was able to recognize most of the hardware in the laptop.

Normally I run my systems by booting from USB drive. However for this laptop that would pose a problem becase this laptop is very basic and only has two USB ports (which are USB2 to boot).

My standard configuration is boot from USB, and the data is in the internal drive. If later I want to change Operating Systems I only need to create a new USB drive and plug it in.

This approach with this HP Stream 11 Pro G5 is not good since as I mention earlier, there are only two USB2 ports. The laptop does have a Micro SD card slot, so instead of creating a USB stick I could simply write to a Micro SD card.

So I wrote the live voidlinux image to a Micro SD card and try to boot from it. Unfortunately, the UEFI BIOS did not recognize the Micro SD card reader, much less detect it as a bootable device. So what I did is use rEFInd as a boot manager to be able to boot either Windows or Linux. If Linux it would boot into a small Linux image. This small Linux image would then load a Linux kernel and InitRAM image from the Micro SD card and boot from it.

This makes it possible to boot SD card images just like I do with USB sticks.

Backing up

Since I really don't use MS-Windows much, I decided to remove the Windows system from laptop. However, since I am sure in the future I would want to send this laptop for re-purposing, I would first need to backup the internal MMC contents.

The internal drive is only 64GB, with compression, it can be backed-up to a 32GB USB stick. For this I am using clonezilla Live. Since this laptop has UEFI boot, preparing a suitable media is very easy.

Start with an empty USB stick.
Create two partitions:
- 1GB VFAT partition : for clonezilla software
- remaining space as an xfs or ext4 partition to store the backup images
Create the filesystems on the first and second partitions.
Download clonezilla Live. I am using the stable Debian based version. If you have more exotic hardware you may need to use the Ubuntu based version.
Unzip the contents of clonezilla Live to the first partition of USB drive.

With this you have suitable media to perform the backup. As before plug the USB stick to the laptop, spam the Escape key until you can get the boot menu and select to boot from the USB stick.

From the GRUB menu, select Clonezilla Live. Since I am old, I selected the VGA with Large Font option. Becaue this loads everything to RAM, this would take a bit longer than usual.
After configuring the region and keyboard type, select Start Clonezilla.
Choose device-image mode and select local_dev in the next menu.
Breeze through the next couple of screens as the target drive is already inserted.
For the /home/partimag repository select the second (large) partition on the USB stick.
The next skip asks for fsck. Just do no-fsck since it is a new partition.
Then select the root folder of the partition when prompted.
Select Beginner when asked for the Wizard mode to use.
Select savedisk to backup the whole MMC drive, and give a suitable name to the image set.
Select the drive to backup. In my laptop, it is shown as mmcblk1.
For compression option I chose z9p for max compression.
The next options are just for error checking. Pick what you think its best.

That starts the backup process. After some time the backup process will finnish. You can put away the USB drive in a safe place. You will need it to reset the laptop for re-use.

For this step there a several options. I chose clonezilla because was the simplest and most straightforward. Personally, I do not like the user experience of the Live USB, but it works.

Underneath it uses partimage which is no longer an active project. One could have used instead a standard live distro and use fsarchiver or partclone. With this we have a simple purpose build USB that contains the restore images in one single package.

Installing Refind

You can download refind from its download page:

https://www.rodsbooks.com/refind/getting.html

The detailed installation instructions can be found here:

https://www.rodsbooks.com/refind/installing.html#linux

For my case I used the voidlinux live USB I used before for testing.

Boot into voidlinux live USB.
Since we booted using the USB drive, the EFI partitions is usually not mounted. To mount:
```
mount -t vfat /dev/mmcblk1p1 /mnt
```
Download the binary zip file from getting refind

Copy refind to the EFI partition:

cp -av refind-bin-VERSION/refind /mnt/EFI/refind

Remove unneeded drivers and/or architectures:
```
rm -r drivers_ia32 drivers_aa64
```
Rename the refind.conf-sample to refind.conf and edit it to taste.
For example, make sure that showtools line contains bootorder which I use in a later step.

Add refind to the list of available boot loadders:

efibootmgr -c -l \\EFI\\refind\\refind_x64.efi -L rEFInd -d /dev/mmcblk1

Install additional components (See addons. Myself, I download the shell.efi file.
Re-boot system and spam the Escape key while booting to get to the BIOS menu. Select rEFInd. Use the refind utility to change the boot order to make sure that refind is before the Windows boot manager.

Up to now, we are able to boot Linux from either the internal MMC or from a USB drive. In order to boot from the MicroSD card slot we need to create a custom Linux image that can be used to select and load a kernel from the MicroSD card. For that we use buildroot.

The purpose of refind here is to easily support dual-booting Linux and Windows. It is unfortunate that the EFI run-time does not seem to support the MicroSD card reader. If that was the case we could stop now.

There are multiple alternatives to refind such as GNU Grub2. I find grub very Linux oriented and I find the configuration system overly complicated.

Custom Linux

Buildroot is a tool that simplifies and automates the process of building a complete Linux system for an embedded system, using cross-compilation.

In order to achieve this, Buildroot is able to generate a cross-compilation toolchain, a root filesystem, a Linux kernel image and a bootloader for your target. Buildroot can be used for any combination of these options, independently (you can for example use an existing cross-compilation toolchain, and build only your root filesystem with Buildroot).

For our use-case we need:

Linux kernel with kexec function call enabled and a number of drivers built-in
kexec utilities/userspace
Some basic user space utilties (mostly provided by busybox)
dialog utiltiy for user displays

With buildroot we can create a Linux kernel with the necessary drivers and a initrd image with enough utilities to read the MicroSD card reader, give the option to pick a kernel and boot into it.

voidlinux pre-requisites

As one of the steps in buildroot it will build its own fakeroot command. This does not work in voidlinux with musl runtime. You must either switch to voidlinux with glibc or follow my recipe in Mixing glibc with musl.

Preparing a voidlinux system with musl runtime:

sudo env XBPS_ARCH=x86_64 xbps-install \
        --repository=https://repo-default.voidlinux.org/current  \
        -r /glibc -S \
        base-voidstrap

You must also install the following package build dependancies:

base-devel ncurses-devel rsync wget cpio

Setting up Buildroot

buildroot is distributed via a git repository. From this repository you can select the version that you want to use.

Fetch buildroot:

git clone  https://git.buildroot.net/buildroot
cd buildroot
git co -b dev 2024.02.9

I have only tested version 2024.02.9. The buildroot project has a LTS schema where the YYYY.02 releases are considered for long term support/more stable.

I am using out-of-tree builds and also customizations are kept separte from buildroot using the BR2_EXTERNAL functionality.

This keeps things nice and tidy.

The buildroot customizatins are in separate github repository. To use it you need to fetch it with git.

Fech lnxboot repository:

git clone https://github.com/TortugaLabs/lnxboot.git

For the out-of-tree build, you just need to create an empty directory and and set-things up.

Create a folder for out-of-tree building:

mkdir build_dir
cd build_dir
glibc make O=$(pwd) BR2_EXTERNAL=$(readlink -f ../lnxboot) -C ../buildroot lnxboot_defconfig

glibc make

The first make sets up the out-of-tree build environment. We use defconfig target so that the savedefconfig function works correctly.
The second make applies the configuration settings for

You will find in images of build_dir two files:

bzImage
rootfs.cpio.gz

Copy this to the EFI partition and update refind.conf to boot it.

From the build directory you can customize things with this commands:

glibc make menuconfig - Configures buildroot
glibc make linux-menuconfig - Configures the Linux kernel
glibc make busybox-menuconfig - Configures busybox

Additional make targets:

glibc make img - Create a qemu bootable image (for testing)
glibc make run-qemu - Create a qemu bootable image and boot it (for testing)
glibc make lnxboot-update - Updates the lnxboot configuration files.

Nemo file explorer custom actions

2025-09-28T00:00:00+02:00

Adding a "Copy File Path" Option to Nemo in Linux Cinnamon
Variations
Other conditions
Conclusion

Adding a "Copy File Path" Option to Nemo in Linux Cinnamon

The Nemo file browser in Linux Cinnamon is highly customizable, allowing users to tweak its functionality to suit their workflow. One useful feature missing by default is a right-click menu option to copy a file's full path to the clipboard—an essential shortcut for developers, power users, and anyone working with file paths regularly.

Note that you could simply download the Copy Path to Cliboard from Cinnamon Spices, the official addons repository.

In this guide, we'll show you how to add this feature by creating a Nemo Action, both for individual users and system-wide use manually.

Step 1: Create a Nemo Action File

First, we need to create a .nemo_action file that tells Nemo how to handle the new menu option.

Open a terminal and navigate to the actions folder:

mkdir -p ~/.local/share/nemo/actions
cd ~/.local/share/nemo/actions

Create a new file:
```
touch copy_path.nemo_action
```

Step 2: Define the Action

Edit the copy_path.nemo_action file in your favorite text editor and add the following content:

[Nemo Action]
Name=Copy Full Filepath
Name[fr]=Copier chemin complet
Comment=Copies the full path of the selected file
Comment[fr]=Chemin complet %F 
Exec=sh -c "readlink -n -f '%F' | xclip -selection clipboard"
Icon-Name=edit-copy
Selection=notnone
Extensions=any
Separator=,
Dependencies=readlink;xclip;

Explanation of Key Parameters:

Name: Defines the name appearing in the right-click menu.
Exec: Runs a command using readlink (to get the full path) and xclip (to copy it to the clipboard).
Selection & Extensions: Ensures it only applies to files, not directories.

Step 3: Install Required Dependencies

Ensure xclip is installed, as it handles clipboard copying:

sudo apt install xclip

Step 4: Restart Nemo

To apply the changes, restart Nemo:

nemo --quit && nemo &

Now, when you right-click a file, you should see the "Copy Full Filepath" option. Clicking it copies the selected file's full path to the clipboard!

Step 5: Make It Available for All Users (Optional)

If you want all users on the system to have access to this feature, place the .nemo_action file in the system-wide Nemo actions directory:

sudo mv ~/.local/share/nemo/actions/copy_path.nemo_action /usr/share/nemo/actions/

Ensure proper permissions:

sudo chmod 644 /usr/share/nemo/actions/copy_path.nemo_action
sudo chown root:root /usr/share/nemo/actions/copy_path.nemo_action

Restart Nemo once again:

nemo --quit && nemo &

Now, all users on the system should have the "Copy Full Filepath" option available!

Variations

It is possible to restrict the custom action to specific conditions. For example if I want the action only to be available in a user's ~/custom directory you can add to the action defintion file:

Conditions=path =~ ~/custom/

You could also use Conditions=path matches $HOME/custom/* to specify the path.

Other conditions

The "Conditions" field in Nemo actions allows you to specify when an action should be available. Here are some common options:

mime: Restrict the action to specific file types (e.g., mime=text/plain for text files).
name: Match specific filenames (e.g., name=*.txt for all .txt files).
path: Limit the action to certain directories (e.g., path matches $HOME/special/*).
selection: Define how many files must be selected (e.g., selection=1 for single-file actions).
exec: Ensure a command exists before showing the action (e.g., exec=command-name).
extension: Restrict based on file extensions (e.g., extension=png,jpg for images).

You can combine multiple conditions to fine-tune when your action appears.

Conclusion

Adding custom Nemo Actions is a powerful way to extend the functionality of your Linux Cinnamon file manager. With just a few simple steps, you now have a quick and efficient way to copy file paths—saving time and effort, especially in development workflows.

Stretch Cluster notes

2024-12-12T00:00:00+01:00

Clustering
Stretched Clusters
Calculating latencies
Common storage stretched clusters
Conclusions

Clustering

At work, very often we need to implement disaster recovery solution. One of the methods to achieve this is to implement stretch clusters.

A computer cluster is a set of computers that work together so that they can be viewed as a single system. Computer clusters have each node set to perform the same task, controlled and scheduled by software.

The components of a cluster are usually connected to each other through fast local area networks, with each node (computer used as a server) running its own instance of an operating system. In most circumstances, all of the nodes use the same hardware and the same operating system.

Clusters are usually deployed to improve performance and availability over that of a single computer, while typically being much more cost-effective than single computers of comparable speed or availability.

Stretched Clusters

A very special case of clusters is when instead of using a Local Area Network, we connect them over a Wide Area Network.

For disaster recovery with low Recovery Point Objective (RPO) requirements, using stretched clusters to keep data synchronized accross data centers is a very common approach.

These typically you would have cluster nodes spread across two locations. With data being replicated synchronously between them. Latency over the Wide Area link becomes quite important as usually, synchronous protocols would require acknowledgement from cluster node members on the far away location before completing a transaction.

Another name for Stretched Clusters is geo-replicated clusters.

Calculating latencies

Latency is a term that is used to describe a time delay in a transmission medium such as a vacuum, air, or a fiber optic waveguide. In free space, light travels at 299,792,458 meters per second. This equates to 299.792 meters per microsecond (µs) or 3.34µs per kilometer. In fiber optics, the latency of the fiber is the time it takes for light to travel a specified distance through the glass core of the fiber. Light moving through the fiber optic core will travel slower than light through a vacuum because of the differences of the refractive index of light in free space and in the glass. See m2optics on how to calculate the latency based on the distance.

A rule of thumb for quickly calculating latency for every 100 Km add 1 ms round trip latency. This based on single mode fiber is using 5 microseconds per kilometer over a straigth line distance between locations. This provides a high level estimate of predicted latency. This estimate is enough to eliminate possible options that one may be considering when implementing a DR solution.

Common storage stretched clusters

These are common solutions for storage replication that implement stretched clustering:

Ceph - a free and open-source software-defined storage platform that provides object storage, block storage, and file storage built on a common distributed cluster foundation.
VMware vSAN - a software-defined storage that is embedded in VMware's ESXi hypervisor.
NetApp Metrocluster - a feature of NetApp's ONTAP software. It allows to synchronously replicate data between two sites using SyncMirror.

Ceph

Ceph provides distributed operation without a single point of failure and scalability to the exabyte level. Ceph does not rely on any other conventional filesystem and directly manages HDDs and SSDs with its own storage backend BlueStore and can expose a POSIX filesystem.

Ceph has a stretch mode that supports geo-replicated clusters.

A stretch cluster is a cluster that has servers in geographically separated data centers, distributed over a WAN. Stretch clusters have LAN-like high-speed and low-latency connections, but limited links. Stretch clusters have a higher likelihood of (possibly asymmetric) network splits, and a higher likelihood of temporary or complete loss of an entire data center (which can represent one-third to one-half of the total cluster).

Ceph is designed with the expectation that all parts of its network and cluster will be reliable and that failures will be distributed randomly across the CRUSH map. Even if a switch goes down and causes the loss of many OSDs, Ceph is designed so that the remaining OSDs and monitors will route around such a loss.

Sometimes this cannot be relied upon. If you have a "stretched-cluster" deployment in which much of your cluster is behind a single network component, you might need to use stretch mode to ensure data integrity.

Ceph has two standard configurations:

Configuration with two data centers (or, in clouds, two availability zones).
In this configuration, each site needs to hold a copy of the data. A third site is required as a tiebreaker monitor. This tiebreaker monitor picks a winner if the network connection fails and both data centers remain alive.
Configuration with three data centers (or, in clouds, three availability zones).
Because there is an odd number of sites, there is no tiebreaker monitor required.

RedHat in OpenShift implementations recomments a 10 ms RTT between sites, with 100 ms RTT for the tiebreaker monitor. This translates to 1,000 Km distance. The real limitation would be application itself, as 10 ms latency for disk writes is quite high.

Ceph also support asynchronous replication using rbd-mirror. This does not have a latency limitation, with the disadvantage that RPO can not be zero.

VMware VSAN

The vSphere and vSAN software runs on industry-standard x86 servers to form a hyper-converged infrastructure (or HCI). However, network operators need to have servers from HCL (Hardware Compatibility List) to put one into production.

Some vSAN features:

native data at rest encryption
local protection for stretched clusters
analytics
optimized solid-state drive performance

The latency limits for stretched vSAN clusters is documented in the vSAN network requirements documentation:

Site Communication	Bandwidth	Latency
Site to Site	Minimum of 10 Gbps	Less than 5 ms RTT
Site to Witness	2 Mbps per 1,000 vSAN components	Less than 500 ms RTT for 1 host per site Less than 200 ms RTT for up to 10 hosts per site Less than 100 ms RTT forr 11-15 hosts per site

The latency of 5 ms RTT translates to a maximum of 500 Km distance.

NetApp MetroCluster

NetApp MetroCluster configurations combine array-based clustering with synchronous replication to deliver continuous availability, immediately duplicating all of your mission-critical data on a transaction-by-transaction basis. MetroCluster configurations enhance the built-in high availability and nondisruptive operations of NetApp hardware and ONTAP storage software, providing an additional layer of protection for the entire storage and host environment.

Latency requirements are documented in MetroCluster consierations for ISLs.

The ISL speed must be least 10Gbps.
ISL can be no further than 300km and the max RTT cannot exceed 3ms – whichever is reached first.

Conclusions

In general a 3 to 5 ms RTT is a reasonable figure. So usually between 300 to 500 Km distances.

Recap of solutions:

Ceph - open source, software define storage solution
VMware vSAN - closed source, software defined storage solution
NetApp MetroCluster - Hardware based storage solution

Rewriting history in git

2025-05-22T00:00:00+02:00

Coordinate with your team
The secret is in the last commit and there is nothing else
The secret is in the last commit but there are other changes
The secret is beyond the last commit
Publishing your repo
Coordinate with your team

From gitguardian blog.

You know that adding secrets to your git repository (even a private one) is a bad idea, because doing so risks exposing confidential information to the world. But mistakes were made, and now you need to figure out how to excise confidential information from your repo. Because git keeps a history of everything, it's not often enough to simply remove the secret or file, commit, and push: we might need to do a bit of deep cleaning.

Thankfully, for simpler cases, git provides commands that make cleaning things up easy. And in more complicated cases, we can use git-filter-repo, a tool recommended by the core git developers for deep cleaning an entire repository.

First and foremost, if there is reason to think that the secret has escaped into the world, and you can revoke the secret, do so. How to revoke a secret is going to vary quite a lot depending on what the secret protects. If you don't know how to revoke it, you will need help from the owner of the resource protected by the secret.

Coordinate with your team

If you are working with a team, make sure you coordinate things with the team.

First of all, we need to determine who else is affected by the secret's presence, because we'll need to coordinate everyone's actions. If the secret only appears in the branch you're working on, you only need to coordinate with anyone else who is always working off of that branch. However, if you found the secret lurking further back in git history, perhaps in your master or main branch, you'll need to coordinate with everyone working in the repository.

Let the others affected know that a secret was found that needs to be excised from everyone's git history. When you edit the git history to remove a file, it can cause problems with your teammatesl local clones; moreover, they can end up re-inserting the secret back into the public repository when they push their work. So it is important that everyone affected is in sync for the excision to work. This means that everyone needs to stop what they are doing, close outstanding PRs, and push up work that’s in progress.

Now, let's make a fresh clone.

Delete your existing clone in its entirety.
Make a fresh clone with git clone [repository URL]
Change into the project directory with cd [project name]
Download the entire repository history: git pull --all --tags.

The last step will look a little bit familiar. git pull tells git to grab updates from the remote repository, and apply them in the current branch (when it makes sense to do so, that is, when the local branch is set to track a remote branch). But git is smart, it doesn’t pull everything down, only what’s needed. This is where the --all flag comes in. This flag tells git to grab every branch from the remote repository. And the --tags flag tells git to grab every tag as well. So this command tells git to download the entire repository history, a complete and total clone. We have to do this, because it is possible that the commit containing the secret exists in more than one branch or tag. We need to ensure we don't just scrub our secret from a portion of the repository history. As a result, this command can take a very long time, if the repository is very large.

The secret is in the last commit and there is nothing else

This is the simplest case. In this case, we can drop the last commit in its entirety. We do this with

git reset --hard HEAD~1

What does this command do? Let's break it apart a little bit. git reset is how we tell git that we want to undo recent changes. Normally, this command by itself tells git to unstage anything we've added with git add, but haven't committed yet. This version of resetting isn't sufficient for our purposes. But git reset is more flexible than that. We can tell git to take us back in time to a previous commit as well. We do that by telling git which commit to rewind to. We can use a commit's identifier (it's “SHA”), or we can use an indirect reference. HEAD is what git calls the most recent commit on the checked out branch. HEAD~1 means "the first commit prior to the most recent" (likewise HEAD~2 means "two commits prior to the most recent"). Finally, the --hard tells git to throw away any differences between the current state and the state we're resetting to. If you leave off the --hard your changes, including the secret, won't be discarded. With --hard, the differences will be deleted, gone forever (which is precisely what we want!).

The secret is in the last commit but there are other changes

In this case, we don't want to completely drop the last commit. We want to edit the last commit instead. Edit your code to remove the secrets, and then add your changes as usual. Then, instead of making a new commit, we'll tell git we want to amend the previous one:

git add [FILENAME]
git commit --amend

We all know git commit, but the --amend flag is our friend here. This tells git that we want to edit the previous commit, rather than creating a new one. We can continue to make changes to the last commit in this way, right up until we’re ready to either push our work, or start on a new commit.

The secret is beyond the last commit

It's complicated

If you know you committed a secret, but have since committed other changes, things get trickier quickly. In anything but the simplest cases, we are going to want a more powerful tool to help us do a deep clean of the repository We're going to use git-filter-repo, a tool recommended by the git maintainers that will help us to rewrite history in a more user-friendly way than the native git tooling.

A technical aside to those familiar with the concept of rebasing. (If you don't know what that means, feel free to skip this paragraph.) All of the cases covered below can of course be managed using native git tools, particularly by rebasing. Sometimes rebasing is relatively painless, of course, but in the kinds of scenarios we're presenting here, rebasing is going to be a tedious and deeply error-prone process. This is why I am favoring purpose-built tools like git-filter-repo over rebasing. It is far better to avoid opening the possibility for making mistakes. Experience shows, recovering from a botched rebase is extremely time consuming, and often nearly impossible. Better to use the right tool for the job.

First, install git-filter-repo.

Next, let's assess the situation to determine which technique is right for your situation. Sometimes secrets are files, and sometimes secrets are lines of code. For example, if you accidentally committed an SSH key or TLS certificate file, these are contained in specialized files that you'll need to excise. On the other hand, maybe you have a single line of code containing an API key that's part of a larger source file. In that case, you want to modify one or more lines of a file without deleting it.

Git-filter-repo can handle both cases, but requires different syntax for each case.

Backup your config

git-filter-repo will modify your repository's .git/config file to prevent you from accidentantly pushing your changes. So, make a backup of your git/config before proceeding.

Excise an entire file

To tell git-filter-repo to excise a file from the git history, we need only a single command:

git filter-repo --use-base-name --path [FILENAME] --invert-paths

The --use-base-name option tells git-filter-repo that we are specifying a filename, and not a full path to a file. You can leave off this option if you would rather specify the full path explicitly.

Normally, git-filter-repo works by ignoring the filenames specified (they are, as the name suggests, filtered out). But we want the inverse behavior, we want git-filter-repo to ignore everything except the specified file. So we must pass --invert-paths to tell it this. If you leave off the --invert-paths, you'll excise everything except the specified file, which is the exact opposite of what we want, and would likely be disastrous. Please don't do that.

Edit a file without removing it

If you only need to edit one or more lines in a file without deleting the file, git-filter-repo takes a sequence of search-and-replace commands (optionally using regular expressions).

First, identify all the lines containing secrets that need to be excised. You'll also need to work out a plan for how you will replace those lines. Perhaps just deleting them is enough. But perhaps they need to be modified to prevent a runtime crash.

Next, create a file containing the search-and-replace commands, called replacements.txt. Make sure it’s in a folder outside of your repo, for example, the parent folder.

The format of this file is one search-and-replace command per line, using the format:

ORIGINAL==>REPLACEMENT

For example, suppose that you’ve hard-coded an API token into your code, like this:

AUTH_TOKEN=’123abc’

Now suppose that you’ve decided that it’s better to load the API token from an environment variable, as such:

AUTH_TOKEN=ENV[‘AUTH_TOKEN’]

We can tell git-filter-repo to search for the hard-coded token, and replace with the environment variable by adding this line to replacements.txt:

‘123abc’==>ENV[‘AUTH_TOKEN’]

If you have multiple secrets you need to excise, you can have more than one rule like this in replacements.txt.

Finally, assuming you placed replacements.txt in the parent directory, we invoke git-filter-repo with our search-and-replace commands like this:

git filter-repo --replace-text ../replacements.txt

Sometimes you might get an error saying you’re not working from a clean clone. That's OK. Git-filter-repo is making irreversible changes to your local repository, and it wants to be certain that you have a backup before it does that. Of course, we do have a remote repository, and we're working from a local clone. And of course we are very interested in making irreversible edits to our commit history - we have a secret to purge! So there's no need for git-filter-repo to worry. We can reassure it that we are OK with making irreversible changes by adding the --force flag:

git filter-repo --replace-text ../replacements.txt --force

And now you have a clean git history! You’ll want to validate your work by compiling your software or running your test suite. Then once you’re satisfied that nothing is broken, move on to the next step to propagate the new history to your remote repository and the rest of the team.

Restoring your config

git-filter-repo might have modified your repository's .git/config file, removing the remote repository configuration. Restore your .git/config otherwise you won't be able to pubish to your remote repo.

Publishing your repo

If you only just added the secret, and haven't pushed any of your work yet, you're done. Just keep working like you had been, and no one will ever know.

Otherwise, we'll need to overwrite what's on your remote git repository (such as GitHub), as it still contains tainted history. We can't simply push, however: The remote repository will refuse to accept our push because we’ve re-written history. So we'll need to force push instead. Moreover, if our re-writes to history affect multiple branches or tags, we'll need to push them all up. We can accomplish all of this like so:

git push --all --force && git push --tags --force

Note: The --all argument does not automatically push any updated tags. Git does not allow the push arguments --all and --tags to be used in the same call. If you need to run both commands, you can do so with the single line, thanks to the &&.

Coordinate with your team

If you work as part of a team, now comes the hard part. Everyone you identified as affected at the beginning of this process still has the old history. They need to synchronize against the revised history you just force-pushed. This is where errors can happen, and more importantly, where frustration can occur.

Ideally everyone pushed their work up before you edited the history. In that case, everyone can simply make a clean clone of the repo and pick up where they left off.

But if someone failed to push their work up before you re-wrote history, they're going to find they have a number of conflicts that need to be resolved when they pull. Instead, they need to fetch the new history from the remote repository, and rebase their hard work on the re-written history. To do this:

git fetch
git rebase -i origin/[branchname]

If you aren't familiar with git fetch, this command tells git to download new data from the remote repository, but unlike git pull, it doesn't attempt to merge new commits into your current working branch. So the fetch here is requesting all the newly re-written history.

In general, this is a pain in the butt, so it is better to coordinate with the team to avoid this step.

Baremetal backup

2025-05-31T00:00:00+02:00

Preparation
- Disabling BitLocker
- Get Clonezilla
Create boot media
Running Clonezilla

I recently bought a used laptop. Before using it, I wanted to image the disk so that I can restore it to a pristine state. For that I am using Clonezilla. Clonezilla is a partition and disk imaging/cloning program similar to True Image or Norton Ghost. It helps you to do system deployment, bare metal backup and recovery. Three types of Clonezilla are available, Clonezilla live, Clonezilla lite server, and Clonezilla SE (server edition). Clonezilla live is suitable for single machine backup and restore and that is what I am using here. The other versions are used for mass system deployments.

Preparation

For this you need to prepare a USB thumb drive that will contain the Clonezilla software and also accept the baremetal backup. Clonezilla is smart enough to recognize filesystem structure and will only backup up used blocks so the size of the thumbdrive only needs to be as large as the used blocks. When I tested this with a November 2024 Windows 11 image, a 32 GB drive was enough.

In my case, my laptop came with BitLocker enabled. Clonezilla doesn't understand BitLocker drives, so you need to disable it first.

Disabling BitLocker

Boot the PC. In the Setup screens, press Shift+F10 to open a Command Prompt.

Enter the command:

manage-bde -status

To check the status of BitLocker. To disable BitLocker enter:

manage-bde -off C:

Where C: is the drive where BitLocker is enabled. This will start the BitLocker decryption and can take some time.

Get Clonezilla

Go to the Clonezilla download page, and select the stable branch, since I assume you are using a modern PC, you probably will be using a UEFI firmware. So select AMD64 and zip file type.

Supposely the Ubuntu based version have Kernel that includes non-free firmware which should have wider compatibility. In 2025, this is rare, so using the Debian based version is enough.

Create boot media

So with the USB drive you must first partition it. I would create a EFI partition to contain the Clonezilla software and a data partition to keep the backup image.

If using sfdisk you can use these commands:

sfdisk /dev/sde
> label: gpt
> ;500M;U;*
> ;;L;

sfdisk /dev/sde is the command with /dev/sde being the drive we want to partition.
label: gpt specifies that we want to use GPT partitioning since we are assuming an UEFI firmware. sfdisk would normally default to dos which generates an Master Boot Record (MBR) partition instead.
;500M;U;* : Define 1s partition as a 500M bootable EFI system partition.
;;L; : Define the remaining space as a Linux partition.

You don't have to use sfdisk, this is just an example.

Next you need to create a FAT32 filesystem for the UEFI boot partition.

mkfs.vfat \
    -F 32 \
    -n "CZILLA" \
    -v \
    /dev/sde1

mkfs.vfat : Create filesystem command
-F 32 : Specify that we are creating a FAT32 filesystem
-n "EFISYS" : Name of the volume. Can be anything.
-v : verbose flag
/dev/sde1 : Drive partition we are formatting

Create the filesystem for the data:

mkfs.ext4 \
      -b 4096 \
      -E stride=16,stripe-width=16 \
      -i 8192 \
      -m 0 \
      -L data \
      -O ^has_journal \
      /dev/sde2

mkfs.ext4 : Create filesystem command
-b 4096 : Use 4K blocks.
-E stride=16,stripe-width=16 : Options recommended for flash media
-i 8192 : i-node density, 8192 is a good number since for image backups we only have a few large files.
-m 0 : Reserved space set to 0. Recommended for removable media.
-L data : Volume label. Can be anything.
-O ^has_journal : Disable journaling, recommeded for flash media.

Copy the Clonezilla software:

mount -t vfat /dev/sde1 /mnt
unzip clonezilla-live-3.2.2-5-amd64.zip -d /mnt
umount /mnt

mount -t vfat /dev/sde1 /mnt : Mounts the EFI filesystem in the /mnt directory.
unzip clonezilla-live-3.2.2-5-amd64.zip -d /mnt extracs the Zip file to the USB disk
umount /mnt : Unmount the filesystem

At this point we are ready to image the baremetal system.

You can use this script to automate this step.

Running Clonezilla

Insert the USB drive into one of your PCs USB ports, and turn on your PC. You may need to get the boot menu. Usually this is done by pressing Escape or Delete keys while the system is booting. Select to boot from the USB drive.

You will be shown the GRUB Boot menu:

Since I am old, I would choose the large font option. Just choose any option that works for you. The system will first let you configure the keyboard and display. I just leave things as default.

Start Clonezilla

Select device-image on the next menu. This option is for backing up from a device to a image file in a mounted drive.

Select local_dev, since we want to save the image to the locally connected boot disk.

This screen gives you an opportunity to insert a removable drive. Usually it will recognized automatically shown here. Once you see the drive, press Control+C. Since we are using the same USB disk used for Clonezilla you should be able to press Control+C right away.

Select the partition to store images. In this example we are using sdb2 31.5G|ext4|data partition. Remember, we used the -L data option when formatting the data file system.

Since we are using newly initialized drives, we can safely use the no-fsck option.

You can select a different directory in the data partition to store images. Just leave as default.

If you have to read this article, you are NO expert, so select beginner mode.

Since we want to make a full backup, select the savedisk option.

You can change the image name here. I simply leave it as default.

Pick the right disk drive to backup...

Choose the compression level. Normally just select -z9p as that would generate the smallest file size.

Since we are backing up a Windows systems, the only possible option is sfsck. Only if you suspect that the filesystem is not in a good state you should use one of the remaining options.

Enable check the saved image. It is a good idea to always do this to make sure the the backup was stored correctly.

I normally would skip the encryption step as the backup does not contain any sensitive data and I store it off-line.

Keep this default (copy log files).

Leave this default, which is to ask what to do once things are done.

After some additional prompts that you may need to acknowledge you will see this screen:

Depending on the disk size, this may take awhile.

Since we selected check the saved image, you will see a checking image screen.

After the backup is completed, you will get this screen. Hit Enter to continue.

Just poweroff

tcpdump snippets

2024-12-12T00:00:00+01:00

List interfaces
GUI
- Capture file rotation
- stftime codes
Filter by MAC address
Monitor DHCP network traffic
Monitoring LLDP traffic
Showing vlan traffic
Filtering tagged/untagged VLAN (IEEE 802.1Q) traffic
Using ngrep
Piping tcpdump output
Printing packets in ASCII or Hex
Quieter

Some useful stuff related to using tcpdump

tcpdump is a data-network packet analyzer computer program that runs under a command line interface. It allows the user to display TCP/IP and other packets being transmitted or received over a network to which the computer is attached.

Tcpdump works on most Unix-like operating systems: Linux, Solaris, FreeBSD, DragonFly BSD, NetBSD, OpenBSD, OpenWrt, macOS, HP-UX 11i, and AIX. In those systems, tcpdump uses the libpcap library to capture packets. The port of tcpdump for Windows is called WinDump; it uses WinPcap, the Windows ersion of libpcap.

List interfaces

To list the interfaces available for sniffing you can use this option:

tcpdump -D

Example output:

1.enp1s0 [Up, Running, Connected]
2.any (Pseudo-device that captures on all interfaces) [Up, Running]
3.lo [Up, Running, Loopback]
4.eno1 [Up, Disconnected]
5.virbr0 [Up, Disconnected]
6.docker0 [Up, Disconnected]
7.nflog (Linux netfilter log (NFLOG) interface) [none]
8.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]

The -D or --list-interfaces option prints the list of the network interfaces available on the system and on which tcpdump can capture packets. For each network interface, a number and an interface name, possibly followed by a text description of the interface, are printed. The interface name or the number can be supplied to the -i flag to specify an interface on which to capture.

This can be useful on systems that don't have a command to list them (e.g., Windows systems, or UNIX systems lacking ifconfig -a); the number can be useful on Windows 2000 and later systems, where the interface name is a somewhat complex string.

GUI

tcpdump is a command line tool. If you prefer to have a full GUI interface you can use Wireshark.

Wireshark is an open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education. Originally named Ethereal, the project was renamed Wireshark in May 2006 due to trademark issues.

Wireshark is cross-platform, using the Qt widget toolkit in current releases to implement its user interface, and using pcap to capture packets; it runs on Linux, macOS, BSD, Solaris, some other Unix-like operating systems, and Microsoft Windows. There is also a terminal-based (non-GUI) version called TShark.

A feature of Wireshark is that it can also analyze capture traces from tcpdump. See Capturing with “tcpdump” for viewing with Wireshark

The use case is for example you have a router, switch or a headless system. You can use tcpdump to capture traffic and analyze it off-line by reading it into Wireshark.

To capture for off-line analysis use the command:

tcpdump -i <interface> -s 65535 -w <file>

Older versions of tcpdump when writing to file would truncate packets to 68 or 96 bytes. Current versions default to 262144 bytes. Use the -s options to set the max packet size to capture, or use the -S option to get the entire packet. Since this impacts the total size of the file, you can use it to tweak the disk usage. You can pair this with option -c to specify the number of packets to capture.

You can use tcpdump -r option to analyze captured traffic files.

# tcpdump -r dns.pcap
reading from file dns.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144
Warning: interface names might be incorrect
dropped privs to tcpdump
20:33:45.240421 wlp0s20f3 Out IP kkulkarni.attlocal.net.37376 > dsldevice.attlocal.net.domain: 8860+ PTR? 89.1.168.192.in-addr.arpa. (43)
20:33:45.250107 wlp0s20f3 In  IP dsldevice.attlocal.net.domain > kkulkarni.attlocal.net.37376: 8860* 1/0/0 PTR kkulkarni.attlocal.net. (79)
20:33:45.253418 wlp0s20f3 Out IP kkulkarni.attlocal.net.54366 > dsldevice.attlocal.net.domain: 23092+ PTR? 1.112.168.192.in-addr.arpa. (44)
20:33:45.260212 wlp0s20f3 In  IP dsldevice.attlocal.net.domain > kkulkarni.attlocal.net.54366: 23092 NXDomain* 0/0/0 (44)

Capture file rotation

You can make tcpdump automatically rotate files. Example:

tcpdump -i en0 -G 1800 -C 100 -w /var/tmp/trace-%H-%M.pcap

Options:

-C filesize
Before writing a raw packet to a savefile, check whether the file is currently larger than filesize and, if so, close the current savefile and open a new one. Savefiles after the first savefile will have the name specified with the -w flag, with a number after it, starting at 1 and continuing upward. The units of filesize are millions of bytes (1,000,000 bytes, not 1,048,576 bytes).
-G rotateseconds
If specified, rotates the dump file specified with the -w option every rotateseconds seconds. Savefiles will have the name specified by -w which should include a time format as defined by strftime (3).
If no time format is specified, each new file will overwrite the previous. Whenever a generated filename is not unique, tcpdump will overwrite the preexisting data; providing a time specification that is coarser than the capture period is therefore not advised.
If used in conjunction with the -C option, filenames will take the form of file<count>.
-W
Used in conjunction with the -C option, this will limit the number of files created to the specified number, and begin overwriting files from the beginning, thus creating a 'rotating' buffer. In addition, it will name the files with enough leading 0s to support the maximum number of files, allowing them to sort correctly.
Used in conjunction with the -G option, this will limit the number of rotated dump files that get created, exiting with status 0 when reaching the limit.
If used in conjunction with both -C and -G, the -W option will currently be ignored, and will only affect the file name.

stftime codes

%m=month
%d=day of month
%H=hour of day
%M=minute of day
%S=second of day
%s=millisecond of day

Filter by MAC address

tcpdump supports the ether qualifier to specify ethernet addresses in the standard colon-separated format. For example, to capture any broadcast traffic,

$ tcpdump ether dst ff:ff:ff:ff:ff:ff

To capture any traffic sent to or from a given MAC address,

$ tcpdump ether host e8:2a:ea:44:55:66

(Here the first three octets identify the MAC in question as belonging to an Intel NIC, e8:2a:ea being an OUI assigned to Intel.)

Reference: https://www.pico.net/kb/how-does-one-filter-mac-addresses-using-tcpdump/

Monitor DHCP network traffic

The tcpdump command can be used to monitor DHCP related network traffic. This is very useful in cases where DHCP issues may have to be investigated. Basically, the tcpdump command can be used to do some packet sniffing on the network.

The method to capture DHCP traffic is to define a filter so that tcpdump dumps only DHCP related traffic. In DHCP, UDP port number 67 is used by a DHCP server, and UDP port number 68 is used by DHCP clients. Thus, you want to capture traffic with port number 67 or 68 as follows, assuming that eth0 is the network interface that will be used to monitor:

# tcpdump -i eth0 port 67 or port 68 -e -n -vv

By using the -vv option (for very verbose) you may see a lengthy output from the tcpdump command displaying a lot of information. In it you can see which DHCP server is responding, and what IP address is assigned. For example:

Client-ID Option 61, length 7: ether ec:9b:f3:6b:97:4b
Requested-IP Option 50, length 4: 192.168.0.3
Server-ID Option 54, length 4: 192.168.0.1
MSZ Option 57, length 2: 1500
Vendor-Class Option 60, length 16: "android-dhcp-7.0"
Hostname Option 12, length 16: "SAMSUNG-SM-G890A"

In the example above, you can see that a Samsung SM-G890A phone, running Android, gets IP address 192.168.0.3 assigned from DHCP server 192.168.0.1. You can also see the MAC (or "hardware") address of the phone: ec:9b:f3:6b:97:4a.

One you're finished sniffing the network for DHCP related traffic, you can simply CTRL-C out of the tcpdump command.

Reference: https://unixhealthcheck.com/blog?id=433

For IPv6 the command is slightly different:

# tcpdump -i <interface name> -n -vv '(udp port 546 or 547) or icmp6'

In IPv6, DHCP is using different ports. Also, some of the configuration items are sent via ICMP6.

Monitoring LLDP traffic

tcpdump -i eth0 -v -e ether proto 0x88cc

-i nic : Select the interface
-v or -vv : more protocol decoding
-e : Show Ethernet Frame fields
ether proto 0x88cc : select LLDP frames (Protocol ID 0x88cc)

Showing vlan traffic

You can verify the incoming traffic to see if they have VLAN tags by using tcpdump with the -e and vlan option.

This will show the details of the VLAN header:

tcpdump -i bond0 -nn -e  vlan

Reference: https://access.redhat.com/solutions/2630851

Filtering tagged/untagged VLAN (IEEE 802.1Q) traffic

If you want use a tcpdump filter that matches both tagged and untagged traffic you have to watch out for the fact that using VLAN changes the way the traffic is encapsulated. The issue is that tagging traffic inserts four more bytes (namely the VLAN ID) to the ethernet (or more precisely IEEE 802.1Q) header. Without specifically asking for VLAN traffic in the BPF filter, every traffic is parsed as untagged traffic. Thus, the specified filter delivers only untagged UDP packets (i.e., their frames) and drops all tagged traffic.

Now watch out: similar things happen if you specify the mysterious vlan keyword in the tcpdump filter. After specifiying the vlan keyword, the subsequent filters are matched against traffic shifted by 4 bytes to the right. Note that this is also true if you specify not vlan as filter.

If we want to match both tagged and untagged UDP traffic, we have to specify the following filter:

Filter UDP traffic, both VLAN tagged and untagged:

tcpdump -nn -d "udp or (vlan and udp)"

Or, the generic solution:

Generic filter expression that matches VLAN tagged and untagged traffic:

tcpdump -nn -d "<filter> or (vlan and <filter>)"

If you want to filter only untagged traffic, specify the following:

Generic filter to match only untagged traffic:

tcpdump -nn -d <filter> and not vlan

Long story short: When using tcpdump (or libpcap), be careful where to put the vlan keyword in your expression. In general, it's a very bad idea to specify the keyword twice, unless you pack VLAN traffic into VLAN traffic. Maybe these examples are more explanative than the quote below taken from the tcpdump manpage:

Note that the first vlan keyword encountered in expression changes the decoding offsets for the remainder of expression on the assumption that the packet is a VLAN packet.

Recall this (admittedly sometimes strange) behavior is not a bug...

Reference: https://www.christian-rossow.de/articles/tcpdump_filter_mixed_tagged_and_untagged_VLAN_traffic.php

Using ngrep

tcpdump uses berkeley packet filter expressions to select the packets to analyze. If you prefer to use regular expressions you can use ngrep.

ngrep (network grep) is a network packet analyzer. It has a command-line interface, and relies upon the pcap library and the GNU regex library.

ngrep supports Berkeley Packet Filter (BPF) logic to select network sources or destinations or protocols, and also allows matching patterns or regular expressions in the data payload of packets using GNU grep syntax, showing packet data in a human-friendly way.

Examples:

Capture network traffic incoming/outgoing to/from eth0 interface and show parameters following HTTP (TCP/80) GET or POST methods:
```
ngrep -l -q -d eth0 -i "^GET |^POST " tcp and port 80
```
Capture network traffic incoming/outgoing to/from eth0 interface and show the HTTP (TCP/80) User-Agent string
```
ngrep -l -q -d eth0 -i "User-Agent: " tcp and port 80
```
Capture network traffic incoming/outgoing to/from eth0 interface and show the DNS (UDP/53) querys and responses
```
ngrep -l -q -d eth0 -i "" udp and port 53
```

Piping tcpdump output

When piping tcpdump output to another command you should enable line buffered mode with -l or -C.

Without the option to force line (-l) buffered (or packet buffered -C) mode you will not always get the expected response when piping. By using this option the output is sent immediately to the piped command.

tcpdump -nl | awk '/10.14.34.132/'
tcpdump -i eth0 -s0 -l port 80 | awk '/Server:/'

Ref: https://community.exabeam.com/s/article/tcpdump-cheat-sheet

Printing packets in ASCII or Hex

Print each packet (minus its link level header) in ASCII with -A.

Handy for capturing web pages. Also note that -x can be used for ASCII and hex characters.

tcpdump -A -vv -i eno1
tcpdump -x -vv -i eno1

Same as -X, but also shows the ethernet header.

tcpdump -XX

Quieter

Using -q supresses some protocol information, -t supresses timestamps.

tcpdump -q -i eth0
tcpdump -t -i eth0
tcpdump -A -n -q -i eth0 'port 80'
tcpdump -A -n -q -t -i eth0 'port 80'

Ref: https://github.com/tcpdump-examples/how-to-use-tcpdump

About JS2Py

2025-05-06T00:00:00+02:00

Js2Py translates JavaScript to Python code. Js2Py is able to translate and execute virtually any JavaScript code.

Js2Py is written in pure python and does not have any dependencies. Basically an implementation of JavaScript core in pure python.

While intriguing, I do not see the point to do this. Going from Python to JavaScript seems to be more useful.

The only use case I would have for this is to parse those Proxy Auto Configuration scripts or PAC files.

Speaking of Python to JavaScript translators, Google search of py2js has multiple hits. The one that seems more supported and included in the PyPi repository is am230/py2js. Then again, you are better off using Brython.

How SanDisk's ranges compare

2025-05-22T00:00:00+02:00

This is not a SanDisk advertisement!

Using hooks in git

2024-11-22T00:00:00+01:00

Add additional files to commit
List files that will be committed
Making hooks available

Here are some tips for using hooks in git.

Like many other Version Control Systems, Git has a way to fire off custom scripts when certain important actions occur. There are two groups of these hooks: client-side and server-side. Client-side hooks are triggered by operations such as committing and merging, while server-side hooks run on network operations such as receiving pushed commits. You can use these hooks for all sorts of reasons. Typically these are used to customize Git for:

automate tasks
enforce policies through workflows
ensuring code quality
running test automation

Add additional files to commit

Take some files and create items that will be added to commit.

Create script to generate meta data

Create pre-commit hook

#!/bin/sh
python generate_commit_artifacts.py
git add path/to/generated/artifacts

Make hook executable

The use case for this is to include in the source code repository files that require specialized tools to generate.

List files that will be committed

In a pre-commit hook, you can use git diff to find the files that are staged for commit. The git diff --cached command shows the changes between the index (staging area) and your last commit, effectively listing the files that will be included in the next commit.

#!/bin/sh
# Pre-commit hook script to find staged files

# Get the list of staged files
staged_files=$(git diff --cached --name-only --diff-filter=ACMR)

# Print the staged files
echo "Staged files:"
echo "$staged_files"

# Run your script for each staged file
for file in $staged_files; do
    # Add your metadata generation script here
    echo "Processing $file"
    # Example: python generate_metadata.py $file
done

In this script:

git diff --cached --name-only --diff-filter=ACMR lists the names of the files that are staged to be committed. The --diff-filter=ACMR option filters the results to include only Added (A), Copied (C), Modified (M), and Renamed (R) files.
staged_files stores the list of these files.
The script then loops through each staged file and performs any desired actions, such as running a metadata generation script.

By using this approach, you can ensure that your pre-commit hook processes only the files that are actually staged for the next commit. This helps in generating and committing relevant metadata efficiently.

Making hooks available

To ensure that your hooks are available when someone forks your repository, you can include the hook script within your repository itself and provide instructions on how to set it up. Here's a recommended approach:

Create a Directory for Hooks: Inside your repository, create a directory named something like githooks where you can store your hook scripts.
Store your Hook script: Place your hook scripts in this directory, make sure they are executable.

Updated Hooks: In the root directory of your repository create or update a setup script (e.g., setup-hooks.sh) that will install your hooks in the appropriate .git/hooks directory.

#!/bin/sh
#
# setup-hooks.sh
#
cd "$(dirname "$0")" || exit 1
ls -1 githooks | while read hook
do
  [ -x "githooks/$hook" ] || continue
  ln -sf ../../githooks/"$hook" .git/hooks/$hook
done

Include Setup Instructions: Add a section in your README.md or create a separate CONTRIBUTING.md file to include instructions for setting up the Git hooks:
```
## Setting Up Git Hooks

To set up the Git hooks for this repository, run the following command:

  sh setup-hooks.sh
```

By following this approach, anyone who forks your repository can easily set up the repository hooks by running the setup script. This keeps the process simple and ensures that everyone working with the repository has the necessary hooks in place.

Switching to Giscus

2025-05-08T00:00:00+02:00

So since last year I have been using utterances to add comments to my BLOG. After reading this post from Max Brenner, I decided to switch to giscus.

Giscus is a comments system powered by GitHub Discussions. It lets visitors leave comments and reactions on your website via GitHub!

I agree with the comments from Mr. Brenner:

Post reactions
While utterances allows to add reactions to comments, giscus also adds reactions to the post itself.
Lack of maintenance
Apparently, utterances hasn't been updated in three years and has a long issue list which is not really being worked upon.

Brython

2025-05-08T00:00:00+02:00

Brython (Browser Python) is an implementation of Python 3 running in the browser, with an interface to the DOM elements and events.

In a way it is similar to pyjs, however, looks like development has stalled.

Brython is designed to replace Javascript as the scripting language for the Web. As such, it is a Python 3 implementation (you can take it for a test drive through a web console), adapted to the HTML5 environment, that is to say with an interface to the DOM objects and events.

The demos seem to be impressive enough.

Brython development is hosted in github.

Brython supports the syntax of Python 3, and the modules of the CPython distribution written in Python, except for the features that are not relevant in the browser context (writing on disk for instance).

It includes libraries to interact with DOM elements and events, and with existing Javascript libraries such as jQuery, D3, Highcharts, Raphael etc. It supports the latest specs of HTML5/CSS3, and can use CSS Frameworks like Bootstrap3, LESS, SASS etc.

A tutorial can be found in Brython: Python in Your Browser.

While interesting, I think it is better to learn TypeScript and JavaScript directlly as these are more marketable as Front End devlopment languages.

Using LLDP on Linux

2024-11-23T00:00:00+01:00

Introduction
Enabling LLDP on Linux
- Using open-lldp
- Using lldpd
Hints and Tips

Introduction

Link Layer Discovery Protocol (LLDP) is a layer 2 neighbor discovery protocol that allows devices to advertise device information to their directly connected peers/neighbors. It is best practice to enable LLDP globally to standardize network topology across all devices if you have a multi-vendor network.

Commonly used layer 2 discovery protocols are often vendor-proprietary, for instance, Cisco’s CDP, Foundry’s FDP, Extreme’s EDP and Nortel’s SONMP. This makes layer 2 discovery difficult in a heterogeneous environment. To counter this, IETF has introduced a standard vendor-neutral configuration exchange protocol -- the LLDP.

Using LLDP, device information such as chassis identification, port ID, port description, system name and description, device capability (as router, switch, hub…), IP/MAC address, etc., are transmitted to the neighboring devices. This information is also stored in local Management Information Databases (MIBs), and can be queried with the Simple Network Management Protocol (SNMP). The LLDP-enabled devices have an LLDP agent installed in them, which sends out advertisements from all physical interfaces either periodically or as changes occur.

Enabling LLDP on Linux

On Linux there are two solutions that can be used to provide LLDP support on your systems:

I personally like lldpd as you only need to install it and it will start working. Another benefit of lldpd is that it has multiple protocol support. lldpd supports CDP, FDP, EDP and SONMP. Unfortunately, lldpd is only available in Alpine Linux's community repositories. The main repositories only contain open-lldp. See this article for explanation of the Alpine Linux repositories.

Because I usually do not like to enable community repos, I had to figure out how to use open-lldp. I found an article which explains how to use open-lldp here.

Using open-lldp

Install and enable the agent:

apk add open-lldp
rc-update add lldpad
service lldpad start

Once running, you need to configure the agent. Currently I am using a start-up script, but the agent supports a configuration file in /etc/lldpad.conf. Note the config file is updated automatically by the lldpad agent when you change its configuration.

Configure things:

lldptool -L -i eth0 adminStatus
lldptool -i eth0 -T -V portDesc enableTx=yes
lldptool -i eth0 -T -V sysName  enableTx=yes
lldptool -i eth0 -T -V sysDesc enableTx=yes
lldptool -i eth0 -T -V sysCap enableTx=yes
lldptool -i eth0 -T -V mngAddr ipv4=192.168.42.66 enableTx=yes

To query neighbors:

# lldptool -t -n -i eth0

Chassis ID TLV
        MAC: 9c:8e:99:bd:b1:20
Port ID TLV
        Local: 12
Time to Live TLV
        120
Port Description TLV
        Port #12
System Name TLV
        switch1
System Description TLV
         HP ProCurve 1810G - 24 GE, P.2.12, eCos-2.0, CFE-2.1
System Capabilities TLV
        System capabilities:  Bridge
        Enabled capabilities: Bridge
Management Address TLV
        IPv4: 192.168.40.11
        Ifindex: 25
End of LLDPDU TLV

View from switch Web GUI:

To get configured TLV values:

# lldptool -l -i ne0 adminStatus -c
adminStatus=rxtx

# lldptool -i ne0 -t -V portDesc -c
enableTx=yes

# lldptool -i ne0 -t -V mngAddr -c
ipv4=192.168.2.6
ipv6=
enableTx=yes

Using lldpd

Enable the community repo in /etc/apk/repositories

Install and enable the daemon:

apk add lldpd
rc-update add lldpd
service lldpd start

This is enough by default. If you want to enable additional protocols you must modify the file /etc/conf.d/lldpd and add the relevant options for additional protocol support:

-c : Enable the support of CDP protocol. (Cisco)
-e : Enable the support of EDP protocol. (Extreme)
-f : Enable the support of FDP protocol. (Foundry)
-s : Enable the support of SONMP protocol. (Nortel)

Query neighbors:

# lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    eth0, via: LLDP, RID: 1, Time: 0 day, 09:50:19
  Chassis:     
    ChassisID:    mac d4:01:c3:2f:cc:2e
    SysName:      swap4
    SysDescr:     MikroTik RouterOS 6.49.15 (stable) RBD52G-5HacD2HnD
    MgmtIP:       192.168.2.204
    MgmtIface:    4
    MgmtIP:       192.168.2.3
    MgmtIface:    4
    Capability:   Bridge, on
    Capability:   Router, on
    Capability:   Wlan, on
  Port:        
    PortID:       ifname bridge/ether2
    TTL:          120
-------------------------------------------------------------------------------

Hints and Tips

When I first tried this I did a mistake of creating a loop on my network, so STP disabled the port I was running LLDP. Weirdly enough, I would occasionally see neighbors on that port. So to troubleshoot, I was using tcpdump. These are some useful options:

-i nic : Select the interface
-v or -vv : more protocol decoding

-e : Show Ethernet Frame fields
ether proto 0x88cc : select LLDP frames (Protocol ID 0x88cc)

Also when configuring open-lldp, I would want to configure on the physical devices. I use this loop in bash:

for nic in $(ls -1 /sys/class/net/)
do
  [ ! -d /sys/class/net/"$nic"/device ] && continue
  if [ -f /sys/class/net/"$nic"/device/modalias ] ; then
    # Special rule to filter out xen backend interfaces
    grep -q xen-backend /sys/class/net/"$nic"/device/modalias && continue
  fi

  # CONFIG COMMANDS GO HERE
done

Normally, LLDP frames will not propagate through Linux bridges. This is the intended bridge behaviour. If you are for example using virtualization on your system and would like the VMs to see LLDP frames, you can force LLDP frames to be forwarded. See this article.

Enable LLDP forwarding:

echo 0x4000 > /sys/class/net/<bridge_name>/bridge/group_fwd_mask

This is a runtime change to make this more permanent, I do it from the Alpine Linux /etc/network/interfaces file:

# /etc/network/interfaces

auto br1
iface br1 inet dhcp
  bridge-ports eth0 eth1
  up echo 0x4000 > /sys/class/net/$IFACE/bridge/group_fwd_mask

On the other hand, on the MikroTik RouterOS switch I am using LLDP frames are passed through by default. So on the neighbors list I would see not just the switch but all the other devices on the other side of the switch. The switch has routing, bridging and switch capabilities. In RouterOS, bridging is done by software, while switching is done by a hardware chip on the device. Since obviously you would want to use switching when possible, you can add a switch rule to filter out LLDP frames:

/interface/ethernet/switch/rule> add switch=switch1 ports=ether3,ether4 mac-protocol=lldp copy-to-cpu=no redirect-to-cpu=yes mirror=no

This needs to be done in the CLI as the WebFIG UI won't let you select the LLDP protocol. See this. The redirect-to-cpu make sure the CPU gets incoming LLDP for book keeping but restricts forwarding.

Using 7z for benchmarks

2024-11-22T00:00:00+01:00

Syntax
LZMA benchmark details
LZMA benchmark in multithreading mode
7-Zip benchmark
Examples
References

7zip comes with a b (Benchmark) command. 7zip usually is packaged as p7z or similar...

This command measures speed of the CPU. Its execution also can be used to check RAM for errors.

Syntax

b [number_of_iterations] [-mmt{N}] [-md{N}] [-mm={Method}]

Options:

-md{N} : change the upper dictionary size to increase memory usage.
-mmt{N} : n change the number of threads. Default will use all available cores.
-mm=* : run complex 7-Zip benchmark.

The LZMA benchmark is default benchmark for benchmark command.

There are two tests for LZMA benchmark:

Compressing with LZMA method
Decompressing with LZMA method

$ 7z b

7-Zip (z) 24.08 (x64) : Copyright (c) 1999-2024 Igor Pavlov : 2024-08-11
 64-bit locale=en_US.UTF-8 Threads:16 OPEN_MAX:1024

Compiler:  ver:13.2.0 GCC 13.2.0 : SSE2
Linux : 6.6.56_2 : #1 SMP PREEMPT_DYNAMIC Tue Oct 15 02:54:10 UTC 2024 : x86_64
PageSize:4KB THP:madvise hwcap:178BFBFF hwcap2:2
AMD Ryzen 7 5800U with Radeon Graphics
(A50F00) 

1T CPU Freq (MHz):  4361  4408  4409  4421  4422  4421  4417
8T CPU Freq (MHz): 796% 4246   797% 4175  
16T CPU Freq (MHz): 1419% 3623   1569% 3987  

RAM size:   15412 MB,  # CPU hardware threads:  16
RAM usage:   3559 MB,  # Benchmark threads:     16

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:      41524  1462   2763  40395  |     633594  1547   3491  54026
23:      36519  1426   2609  37209  |     596376  1531   3370  51591
24:      34094  1404   2612  36658  |     595727  1556   3360  52270
25:      32470  1379   2689  37074  |     578205  1547   3326  51443
----------------------------------  | ------------------------------
Avr:     36152  1418   2668  37834  |     600975  1545   3387  52333
Tot:            1481   3028  45083

The LZMA benchmark shows a rating in MIPS (million instructions per second). The rating value is calculated from the measured speed, and it is normalized with results of Intel Core 2 CPU with multi-threading option switched off. So if you have modern CPU from Intel or AMD, rating values in single-thread mode must be close to real CPU frequency.

The Dict column shows dictionary size. For example, 21 means 2^21 = 2 MB.

The Usage column shows the percentage of time the processor is working. It's normalized for a one-thread load. For example, 180% CPU Usage for 2 threads can mean that average CPU usage is about 90% for each thread.

The R / U column shows the rating normalized for 100% of CPU usage. That column shows the performance of one average CPU thread.

Avr shows averages for different dictionary sizes.

Tot shows averages of the compression and decompression ratings.

The test data that is used for compression in that test is produced with special algorithm, that creates data stream that has some properties of real data, like text or execution code. Note that the speed of LZMA for real data can be slightly different.

LZMA benchmark details

Compression speed strongly depends from memory (RAM) latency, Data Cache size/speed and TLB. Out-of-Order execution feature of CPU is also important for that test.

Decompression speed strongly depends on CPU integer operations. The most important things for that test are: branch misprediction penalty (the length of pipeline) and the latencies of 32-bit instructions ("multiply", "shift", "add" and other). The decompression test has very high number of unpredictable branches. Note that some CPU architectures (for example, 32-bit ARM) support instructions that can be conditionally executed. So such CPUs can work without branches (and without pipeline flushing) in many cases in LZMA decompression code. And such CPUs can have some speed advantages over other architectures that don't support complex conditionally execution. Out-of-Order execution capability is not so important for LZMA Decompression.

The test code doesn't use FPU and SSE. Most of the code is 32-bit integer code. Only some minor part in compression code uses also 64-bit integers. RAM and Cache bandwidth are not so important for these tests. The latencies are much more important.

The CPU's IPC (Instructions per cycle) rate is not very high for these tests. The estimated value of test's IPC is 1 (one instruction per cycle) for modern CPU. The compression test has big number of random accesses to RAM and Data Cache. So big part of execution time the CPU waits the data from Data Cache or from RAM. The decompression test has big number of pipeline flushes after mispredicted branches. Such low IPC means that there are some unloaded CPU resources. But the CPU with Hyper-Threading feature can load these CPU resources using two threads. So Hyper-Threading provides pretty big improvement in these tests.

LZMA benchmark in multithreading mode

When you specify (N * 2) threads for test, the program creates N copies of LZMA encoder, and each LZMA encoder instance compresses separated block of test data. Each LZMA encoder instance creates 3 unsymmetrical execution threads: two big threads and one small thread. The total CPU load for these 3 threads can vary from 140% to 200%. To provide better CPU load during compression, you can test the mode, where the number of benchmark threads is larger than the number of hardware threads.

Each LZMA encoder instance in multithreading mode divides the task of compression into 3 different tasks, where each task is executed in separated thread. Each of these tasks is simpler than original task, and it uses less memory. So each thread uses the data cache and TLB more effectively in multithreading mode. And LZMA encoder is slightly more effective in multithreading mode in value of "the Speed" divided to "CPU usage".

Note that there is some data traffic between 3 threads of LZMA encoder. So data exchange bandwidth via memory between CPU threads is also can be important, especially in multi-core system with big number of cores or CPUs.

All LZMA decoder threads are symmetrical and independent. So the decompression test uses all hardware threads, if the number of hardware threads is used.

7-Zip benchmark

With -mm=* switch you can run a complex benchmark for 7-Zip code. It tests hash calculation methods, compression and encryption codecs of 7-Zip. Note that the tests of LZMA have big weight in "total" results. And the results are normalized with AMD K8 cpu in that complex benchmark.

$ 7z b -mm=*

7-Zip (z) 24.08 (x64) : Copyright (c) 1999-2024 Igor Pavlov : 2024-08-11
 64-bit locale=en_US.UTF-8 Threads:16 OPEN_MAX:1024

 m=*
Compiler:  ver:13.2.0 GCC 13.2.0 : SSE2
Linux : 6.6.56_2 : #1 SMP PREEMPT_DYNAMIC Tue Oct 15 02:54:10 UTC 2024 : x86_64
PageSize:4KB THP:madvise hwcap:178BFBFF hwcap2:2
AMD Ryzen 7 5800U with Radeon Graphics
(A50F00) 

1T CPU Freq (MHz):  4404  4313  4421  4426  4405  4422  4415
8T CPU Freq (MHz): 799% 4259   797% 4175  
16T CPU Freq (MHz): 1434% 3653   1573% 4002  

RAM size:   15412 MB,  # CPU hardware threads:  16
RAM usage:   3647 MB,  # Benchmark threads:     16

Method           Speed Usage    R/U Rating   E/U Effec
                 KiB/s     %   MIPS   MIPS     %     %

CPU                     1564   4061  63537
CPU                     1574   3944  62086
CPU                     1560   3921  61174   103  1600

LZMA:x1          81225  1491   2009  29943    53   783
                642809  1554   3308  51409    87  1345
LZMA:x3          39193  1442   1670  24080    44   630
                581692  1499   3255  48803    85  1276
LZMA:x5:mt1      29052  1489   2437  36294    64   949
                582429  1523   3223  49106    84  1284
LZMA:x5:mt2      30660  1507   2541  38304    66  1002
                580898  1531   3199  48977    84  1281
Deflate:x1      559275  1548   4589  71015   120  1857
               2071309  1557   4132  64349   108  1683
Deflate:x5      174091  1484   4516  67029   118  1753
               2045737  1532   4144  63501   108  1661
Deflate:x7       64640  1542   4646  71619   122  1873
               2071119  1548   4152  64262   109  1681
Deflate64:x5    157003  1505   4507  67846   118  1774
               2056814  1528   4209  64326   110  1682
BZip2:x1         82001  1549   3198  49542    84  1296
                517893  1510   3718  56136    97  1468
BZip2:x5         36407  1441   2108  30384    55   795
                197521  1504   2577  38765    67  1014
BZip2:x5:mt2     32927  1515   1814  27480    47   719
                191315  1510   2486  37547    65   982
BZip2:x7         17540  1518   2994  45442    78  1189
                195467  1506   2545  38329    67  1002
PPMD:x1          58411  1552   3894  60412   102  1580
                 46787  1551   3552  55097    93  1441
PPMD:x5          31269  1520   3486  52994    91  1386
                 26949  1511   3342  50501    87  1321
Swap4        411409038  1525   1726  26330    45   689
             407703538  1531   1704  26093    45   682
Delta:4       19134561  1550   3794  58781    99  1537
              11982527  1545   3177  49080    83  1284
BCJ           21499405  1554   2833  44031    74  1152
              21507528  1563   2819  44047    74  1152
ARM64         30825468  1560   2024  31565    53   826
              30076127  1552   1984  30798    52   806
RISCV         20395348  1547   1350  20885    35   546
              16120110  1557   1060  16507    28   432
AES256CBC:1    1482399  1562   2332  36431    61   953
               1370916  1550   2174  33692    57   881
AES256CBC:2   11935671  1526   6407  97777   168  2557
              42505953  1544   2818  43526    74  1138
AES256CBC:3   11619172  1509   6307  95184   165  2490
              74469288  1512   2521  38128    66   997
CRC32:12      24817472  1541   1649  25413    43   665
CRC32:32    
CRC32:64    
CRC64         20803049  1537   1386  21302    36   557
XXH64         84786144  1565   1387  21705    36   568
SHA256:1       1864713  1565   2431  38040    64   995
SHA256:2      20485066  1547   2755  42609    72  1114
SHA1:1         3175442  1567   1897  29722    50   777
SHA1:2        20937576  1546   2644  40870    69  1069
BLAKE2sp:1     3329474  1562   3491  54550    91  1427
BLAKE2sp:2     8505983  1558   2236  34841    58   911
BLAKE2sp:3    16257632  1538   2165  33296    57   871

CPU                     1563   3391  53000
------------------------------------------------------
Tot:                    1518   2916  44317    76  1159

The CPU rows show CPU frequency. It's measured for sequence of simple CPU instructions. Note: It can be inaccurate, if hyper-threading is used.

The Effec column shows Efficiency - the Rating normalized to CPU frequency.

The E / U column shows the Efficiency normalized for 100% of CPU usage.

Examples

run benchmarking
```
7z b
```
run benchmarking with one thread and 64 MB dictionary.
```
7z b -mmt1 -md26
```
run benchmarking for 30 iterations. It can be used to check RAM for errors.
```
7z b 30
```
run complex 7-Zip benchmark.
```
7z b -mm=*
```
run complex 7-Zip benchmark for different number of threads : (1, max/2, max), where max is number of available hardware threads. So it can test 3 main modes: single-thread, multi-thread without hyper-threading, multi-thread with hyper-threading.
```
7z b -mm=* -mmt=*
```

References

Replacing failed drives in a md array

2025-04-01T00:00:00+02:00

In a Linux softraid, you may have to replace failed drives. As usual, backups are still needed, but hardware failures can be covered by RAID levels.

Check the array status with cat /proc/mdstat:

xm3:~# cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 sdb[0] sdc[2]
      67042304 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Check the serial number of your drives to make sure you are replacing the right one: lsblk -do +VENDOR,MODEL,SERIAL:

NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS VENDOR MODEL                 SERIAL
loop0   7:0    0 108.1M  1 loop /.modloop                
sda   253:0    0   232G  0 disk             ATA    WDC_WD2500AVVS-73M8B0 WD-WCAV94350152
sdb   253:16   0   232G  0 disk             ATA    WDC_WD2500AVVS-73M8B0 WD-WCAV94350152
sdc   253:32   0   232G  0 disk             ATA    WDC_WD2500AVVS-73M8B0 WD-WCAV94350152

Remove the faulty drive
- Hot plugging, you can do this while the system is running, if your hardware supports it.
  - Remove the drive from the system.
  - Run:
```
mdadm --manage /dev/md0 --remove failed
```
    Note that failed worked for me. Other examples mentioned detached.
- Warm plugging, some hardware requires you to enter some commands:
  - mark drive as faulty
```
mdadm --manage /dev/md0 --fail /dev/sdc`
```
  - remove drive from array
```
mdadm --manage /dev/md0 --remove /dev/sdc
```
  - remove the drive from the kernel
```
echo 1 > /sys/block/sdc/device/delete
```
- This is how mdstat looks like after:
```
xm3:~# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 vdb[0]
      67042304 blocks super 1.2 [2/1] [U_]

 unused devices: <none>
```
  When removing the physical drive, if possible, unplug the power cable first, data cable next.
Insert the new drive. When possible, plug the data cable first, power cable next. Wait 10-15 seconds

Run

for file in /sys/class/scsi_host/*/scan; do
  echo "- - -" > $file;
done;

This will re-scan the SATA buses. This is not always needed.

Add the new drive to the array:
```
mdadm --add /dev/md0 /dev/sdc
```

Check that the configuration updated correctly:

xm3:~# mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Tue Nov  5 14:44:26 2024
        Raid Level : raid1
        Array Size : 67042304 (63.94 GiB 68.65 GB)
     Used Dev Size : 67042304 (63.94 GiB 68.65 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Wed Nov 13 14:03:42 2024
             State : clean, degraded, recovering 
    Active Devices : 1
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

    Rebuild Status : 0% complete

              Name : xm3.virtual:0  (local to host xm3.virtual)
              UUID : 95d17a6b:bee76241:c72a05d1:3cfd7d62
            Events : 28

    Number   Major   Minor   RaidDevice State
       0     253       16        0      active sync   /dev/sdb
       2     253       32        1      spare rebuilding   /dev/sdc

You can monitor the re-build process with:

# watch cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 vdc[2] vdb[0]
      67042304 blocks super 1.2 [2/1] [U_]
      [=========>...........]  recovery = 45.6% (30607360/67042304) finish=3.0min speed=200060K/sec

unused devices: <none>

You should check the output of dmesg:

[  703.836264] md/raid1:md0: Disk failure on sdc, disabling device.
[  703.836264] md/raid1:md0: Operation continuing on 1 devices.
[ 1789.815521] md: recovery of RAID array md0

Simlarly, you should use:

smartctl -a /dev/sdc

To check the health status of the drive.

In here I am using raw drives (without partitioning). If you are using partitions make sure to run fdisk when appropriate.

If you have smartmontools installed and running, we need to reset the daemon so it doesn't keep warning about the drive we removed.

3D Prints

2025-03-16T00:00:00+01:00

The other day, the 3D model I created back in 2016 finally broke.

So, as such I created an updated model:

For this I am using the same service from my 3D Printing updates article. So far they have been quite reliable.

This model took only two attempts to get to the working version. The first attempt was soooo stupid. I created the model thinking in "centimeters", but uploaded the model using "milimeters" as the unit. Obviously, the results were tiny.

The new model, did not use any support material. I achieve this by splitting the model into two sections which later are glued together. The top section and the bottom section.

This also allows the two sections to be printed with different fill densities, which can reduce the cost of the printing.

After I did the print, I tweaked the model, so that the bottom is 1mm smaller than the top. Also the top uses less material. I did not try printing the tweaked model, but I think it would have a nicer effect than the original.

The models can be found here:

Printed version in cm (scale by 10x when printing)
- Top part
- Bottom part
Tweaked model (scale in mim now)
- Top part
- Bottom part

Filesystem discard

2025-03-16T00:00:00+01:00

Now with the prevalence of SSD's for storage, it is important to make sure that the DISCARD operation is used. This is specially true as this can increase the lifetime of your flash storage by reducing the need to re-map blocks by simply marking them as freed.

In Linux, because the flexibility of the block device infrastructure it is very easy to layer different block device drivers to add functionality. The risk of doing that is that Discard capabilities can be left out accidentally.

To examine if Discard is enabled accross the different layers, you can use the command:

lsblk --discard

This will output something like:

NAME            DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
loop0                  0        4K       4G         0
sda                    0        4K       2G         0
└─crypt-pool           0        4K       2G         0
  └─pool-imglib        0        4K       2G         0
sdb                    0        0B       0B         0
├─sdb1                 0        0B       0B         0
└─sdb2                 0        0B       0B         0

Make sure that the values under column DISC-GRAN and DISC-MAX are non-zero.

Once you have identified that discard is properly configured you can start discarding using these commands:

Discard on demand, by simply running:
```
fstrim --all
```
Add the discard option to the mount command. This is supported by several filesystems and will call discard as files are deleted.
Run fstrim --all on a cron job either weekly or monthly.

In addition, if you are going to delete a logical volume, you could do:

blkdiscard DEVICE

on the logical volume before running lvremove to discard the storage associated to the logical volume.

As I mentioned earlier, it is important to make sure that all layers have discard enabled.

Low level disc drivers. These should detect that the underlying drive is capable of using discard automatically.
mdadm : supported and enabled by default. (1)
dmcrypt - supported, defaults to off for security, requires explicity enabling. (2)
lvm2 : suppored and enabled by default. (3)
Filesystems:
- ext3/ext4: supports discard mount option and fstrim
- btrfs: supports discard mount option and fstrim
- f2fs: supports discard mount option and fstrim
- xfs: supports discard mount option and fstrim
- jfs: supports discard mount option and fstrim
- ntfs: supports discard mount option with the ntfs3 driver and fstrim
- vfat: supports discard mount option and fstrim
Xen VMs using block devices will automatically detect discard and enable the interface for virtual drives.

DRBD Preload

2024-11-22T00:00:00+01:00

This is a continuation to VM disk replication.

One of the steps needed for VM disk replication is the need to pre-load the replicated volume.

For the project I wrote, we had two mechanisms:

Copy from an attached virtual disk image
Download from an image server

Copying from an attached virtual disk is straightforward use of the dd command.

Downloading from an image server was slightly triciker as it needed to:

Resume downloads (because the large image sizes)
Images should be stored as compressed qcow2 files.
Use gzip on the fly so as to compress large empty disk volume areas.

This was accomplished through a script that would convert qcow2 files into raw:


#!/usr/bin/env python3
#
# Stream qemu image as a raw image
#
from argparse import ArgumentParser
import nbd  # Requires python3-libnbd
import sys
import subprocess

def mycli():
  '''Create an ArgumentParser

  :returns ArgumentParser:
  '''
  cli = ArgumentParser(
    description = 'Stream a qemu image in raw format'
  )
  cli.add_argument('-c','--compress',dest='gzip',help='Enable gzip compression', action='store_true')
  cli.add_argument('--blocksize',help='Read block size',type=int, default=32*1024*1024)
  cli.add_argument('file',help='Disk image')
  cli.add_argument('offset',help='Byte offset to skip',nargs='?',type=int,default=0)
  cli.add_argument('count',help='Byte count to send',nargs='?',type=int)
  return cli

if __name__ == '__main__':
  cli = mycli()
  args = cli.parse_args()

  n = nbd.NBD()
  cmd = ['qemu-nbd', '--read-only', '--persistent', args.file]
  n.connect_systemd_socket_activation(cmd)

  if args.gzip:
    gzip = subprocess.Popen(['gzip'], stdin=subprocess.PIPE)
    fp = gzip.stdin
  else:
    fp = sys.stdout.buffer

  offset = args.offset
  size = n.get_size()
  if not args.count is None and offset + args.count < size:
    size = offset + args.count

  while offset < size:
    c = min(args.blocksize, size-offset)
    b = n.pread(c, offset)
    fp.write(b)
    offset += c

  n.shutdown()

And some bash script to retrieve the data:

comma() {
  echo "$1" | sed ':a;s/\B[0-9]\{3\}\>/,&/;ta'                                                              
}

preload_in_guest() {
   [ -z "$preload_url" ] && return
   ( echo "$preload_url" | grep '^https*://') || return 0

   # We do this stupid process because we would time out for large
   # downloads
   cksz=$(expr 32 '*' 1024 '*' 1024) # 32M chunks

   imgsz=$(wget -nv -O- "${preload_url}?chonky=size")
   if ! ( echo "$imgsz" | grep -q '^[0-9][0-9]*$' ) ; then
     echo "Unable to determine image size"
     exit
   fi

   # OK, break it into chunks
   count=$(expr $imgsz / $cksz)
   if [ $(expr $imgsz - $(expr $count \* $cksz)) -gt 0 ] ; then
     count=$(expr $count + 1)
   fi
   echo "WILL DOWNLOAD IMAGE $(comma $imgsz)b IN $(comma $count) CHUNKS"

   offset=0
   seek=0
   while [ $seek -lt $count ]
   do
     echo "Reading chunk: $(comma $seek) of $(comma $count) ( $(expr ${seek}00 / $count)% )"
     if ! (wget -nv -O- "${preload_url}?chonky=$offset,$cksz" \
        | gunzip \
        | dd of=/dev/drbd0 bs=$cksz seek=$seek) ; then
       echo "Error retrieving chunk: $seek"
       exit 1
     fi
     seek=$(expr $seek + 1)
     offset=$(expr $offset + $cksz)
   done
}

On the server size, a PHP script is used that gets chonky from the query parameters and uses the value formatted as offset,size to call the python script.

VM managed disk replication

2024-11-15T00:00:00+01:00

Introduction
High level overview
Deployment
Client setup
- KVM NBD client
- KVM iSCSI client
Performance Impact

Introduction

This recipe is for adding disk replication to VMs using KVM hypervisor. The idea is to keep VM operations independant of storage operations in non-vertically integrated support teams. In this scenario, VM Operators only need to manage VMs, and storage is delegated to a storage team that manages virtualized storage delivery services, one of these being replicated storage service.

High level overview

This solution is based on Ubuntu 22.04. Note that normal disk images can not be used as repliacted disks unless they have a DRBD device signature already embedded on them.

There are two KVM hosts, for providing a High Availability cluster. There are two VMs, each running on the different KVM hosts. These mirror VMs are managing the replication.

There are two additional VMs for running the applicaiton. One one of these VMs should be active.

Deployment

Create mirror VMs with the same configuration:
- 2 vCPUs
- 4096MB RAM
- Netwoks:
  - 1 vNIC for replication (does not need to be routed)
  - 1 vNIC for iSCSI/NBD protocol (only needs to be routed to clients, i.e. KVM host or client VMs)
  - 1 vNIC for management and accessing Ubuntu software repositories.
  - 1 OS disk, 1 data disk
Primary (mirror-a):
- Download required packages from Ubuntu repos
- Configure DRBD and start first sync (this happens in the background)
- Download or copy disk pre-load image
- Set-up NBD server or iSCSI target
Secondary (mirror-b):
- Download required packages from Ubuntu repos
- Configure DRBD (as secondary)
- Set-up but NOT start NBD server or iSCSI target

Required packages

drbd-utils : Needed for DRBD
tgt : Needed for iSCSI
nbd-server : Used for the NBD server

DRBD configuration

Recommended: Modify /etc/hosts to include IP address of peer VMs.

Install DRBD drivers:

apt -y install linux-modules-extra-$(uname -r)
modprobe drbd

path: /etc/drbd.conf

     global { usage-count no; }
     common { disk { resync-rate 100M; } }
     resource r0 {
        protocol C;
        startup {
                wfc-timeout  15;
                degr-wfc-timeout 60;
        }
        net {
                cram-hmac-alg sha1;
                shared-secret "secret";
        }
        on a {
                device /dev/drbd0;
                disk /dev/sdb;
                address ${vm1_ipaddr}:7788;
                meta-disk internal;
        }
        on b {
                device /dev/drbd0;
                disk /dev/sdb;
                address ${vm2_ipaddr}:7788;
                meta-disk internal;
        }
     }

Initialize DBRD device:

#!/bin/sh
#
# Do not initialize it twice!
(blkid /dev/sdb | grep 'TYPE="drbd"') && exit 0
. /root/mirdata.txt # defines $drbd_role as primary or secondary
drbdadm create-md r0
systemctl enable drbd.service
systemctl start drbd.service
#
case "$drbd_role" in
primary)
  drbdadm -- --overwrite-data-of-peer primary all
  ;;
esac

At this point /dev/drbd0 can be pre-loaded with the contents of the replicated drive.

Configure NBD server

path: /etc/nbd-server/config

[generic]
[r0]
exportname = /dev/drbd0
# Default to port 10809

Start primary:

systemctl enable nbd-server
systemctl start nbd-server

Start secondary:

systemctl disable ndb-server
systemctl stop nbd-server

Configure iSCSI target

path: /etc/tgt/conf.d/target1.conf

# if you set some devices, add <target>-</target> and set the same way with follows
# naming rule : [ iqn.(year)-(month).(reverse of domain name):(any name you like) ]
<target iqn.2023-10.world.srv:dlp.target01>
  # provided devicce as a iSCSI target
  backing-store /dev/drbd0
  # iSCSI Initiator's IQN you allow to connect
  # initiator-name iqn.2023-10.world.srv:node01.initiator01
</target>

Primary server

systemctl enable tgt
systemctl restart tgt

Secondary server

systemctl disable tgt
systemctl stop tgt

Client setup

The mirrored storage is exported either as NBD or iSCSI target.

VM clients can mount the storage using the facilities available for that VM's Operating System.

For KVM volumes (to use as OS for example) these are configured in the VM XML depending on the protocol to use.

KVM NBD client

Firewall configuration:

sudo ufw allow out 10809/tcp

VM XML configuration snippet:

<disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source protocol='nbd' name='r0' >
        <host name='ip-address-of-server' port='10809'/>
      </source>
      <target dev=‘vdc' bus=‘virtio'/>
</disk>

Port 10809 is the default, so that setting can be omitted.

KVM iSCSI client

For simplicity we are not using iSCSI authentication and simply uses the iqn identifiers and potentially IP addresses to control access.

Firewall Configuration:

sudo ufw allow out 3260/tcp

VM XML Configuration snippets:

<disk type='network' device='disk'>
    <driver name='qemu' type='raw'/>
    <source protocol='iscsi' name='iqn.2023-10.world.srv:dlp.target01/1'>
      <host name='192.168.39.184' port='3260'/>
      <initiator>
        <iqn name='iqn.2023-10.world.srv:node01.initiator01'/>
      </initiator>
    </source>
    <target dev='sdc' bus='sata'/>
</disk>

Performance Impact

Protocol	Emulation	512 Reads (MB/s)	512 Writes (MB/s)	1MB Reads (MB/s)	1MB Writes (MB/s)
iSCSI	sata	174.0	11.5	316.0	131.0
iSCSI	virtio	201.0	15.1	317.0	133.0
nbd	sata	164.0	12.7	308.0	130.0
nbd	virtio	210.0	20.0	347.0	135.0
Direct	sata	162.0	12.7	317	339.0

iSCSI with SATA emulation provides the most compatibility
NBD with virtio emulation provides the highest performance.
iSCSI with SATA has the highest per IOP overhead.
nbd has lower protocol overhead than iSCSI
virtio has better virtualization performance than sata emulation
For large data transfers overhead ceases to be relevant.

Documenting Architecture Decisions

2024-12-13T00:00:00+01:00

Context
Approach
Format
Conclusion

This is a modified version of Documenting Archittecture Decisions by Michael Nygard.

Context

Architecture for agile projects has to be described and defined differently. Not all decisions will be made at once, nor will all of them be done when the project begins.

Agile methods are not opposed to documentation, only to valueless documentation. Documents that assist the team itself can have value, but only if they are kept up to date. Large documents are never kept up to date. Small, modular documents have at least a chance at being updated.

Nobody ever reads large documents, either. Most developers have been on at least one project where the specification document was larger (in bytes) than the total source code size. Those documents are too large to open, read, or update. Bite sized pieces are easier for for all stakeholders to consume.

One of the hardest things to track during the life of a project is the motivation behind certain decisions. A new person coming on to a project may be perplexed, baffled, delighted, or infuriated by some past decision. Without understanding the rationale or consequences, this person has only two choices:

Blindly accept the decision.
This response may be OK, if the decision is still valid. It may not be good, however, if the context has changed and the decision should really be revisited. If the project accumulates too many decisions accepted without understanding, then the development team becomes afraid to change anything and the project collapses under its own weight.
Blindly change it.
Again, this may be OK if the decision needs to be reversed. On the other hand, changing the decision without understanding its motivation or consequences could mean damaging the project's overall value without realizing it. (E.g., the decision supported a non-functional requirement that hasn't been tested yet.)

It's better to avoid either blind acceptance or blind reversal.

Approach

We will keep a collection of records for "architecturally significant" decisions: those that affect the structure, non-functional characteristics, dependencies, interfaces, or construction techniques.

An architecture decision record is a short text file in a format similar to an Alexandrian pattern. (Though the decisions themselves are not necessarily patterns, they share the characteristic balancing of forces.) Each record describes a set of forces and a single decision in response to those forces. Note that the decision is the central piece here, so specific forces may appear in multiple ADRs.

We will keep ADRs in the project repository under doc/arch/YYYY-MM-DD-title.md. Where YYYY-MM-DD is the date when the issue was first brought up (not necessarily resolved).

We should Markdown because it is a lightweight text formatting language.

If a decision is reversed, we will keep the old one around, but mark it as superseded. (It's still relevant to know that it was the decision, but is no longer the decision.)

Format

Header:
```
---
title: short noun phrases
status: wip/proposed/accepted/superseded
---
```
- Title: These documents have names that are short noun phrases. For example, "ADR 1: Deployment on Ruby on Rails 3.0.10" or "ADR 9: LDAP for Multitenant Integration".
- Status: A decision may be "proposed" if the project stakeholders haven't agreed with it yet, or "accepted" once it is agreed. If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement.
# Context
This section describes the forces at play, including technological, political, social, and project local. These forces are probably in tension, and should be called out as such. The language in this section is value-neutral. It is simply describing facts.
# Decision
This section describes our response to these forces. It is stated in full sentences, with active voice. "We will ..."
# Alternatives
Other options being considered.
# Justification
Explanation on why decision was made.
# Consequences
This section describes the resulting context, after applying the decision. All consequences should be listed here, not just the "positive" ones. A particular decision may have positive, negative, and neutral consequences, but all of them affect the team and project in the future.

The whole document should be one or two pages long. We will write each ADR as if it is a conversation with a future developer. This requires good writing style, with full sentences organized into paragraphs. Bullets are acceptable only for visual style, not as an excuse for writing sentence fragments. (Bullets kill people, even PowerPoint bullets.)

Conclusion

One ADR describes one significant decision for a specific project. It should be something that has an effect on how the rest of the project will run.

The consequences of one ADR are very likely to become the context for subsequent ADRs. This is also similar to Alexander's idea of a pattern language: the large-scale responses create spaces for the smaller scale to fit into.

Developers and project stakeholders can see the ADRs, even as the team composition changes over time.

The motivation behind previous decisions is visible for everyone, present and future. Nobody is left scratching their heads to understand, "What were they thinking?" and the time to change old decisions will be clear from changes in the project's context.

Alpine Boot

2025-01-23T00:00:00+01:00

This is a quick note. In Alpine linux you can run start-up scripts by enabling the local service:

rc-update add local default

When this is enabled, it will run all executable scripts that end with .start on start-up and scripts that end with .stop on shutdown.

By default, these scripts would run silently and you will see:

* Starting local ... [ ok ]

during boot-up if things went well. If there was an error you would see:

* Starting local ... [ !! ]

To see more details on what happened create the file /etc/conf.d/local with the following:

# /etc/conf.d/local
rc_verbose=yes

Y2K38 filesystems

2024-11-22T00:00:00+01:00

Background
What to do
xfs
ext4

This is a small note...

This happens to me some times. I mount a filesystem and I get the warning:

xfs filesystem being mounted at /mntimg supports timestamps until 2038-01-19 (0x7fffffff)

This has to do with the year 2038 problem.

Background

The year 2038 problem (also known as Y2038, Y2K38, Y2K38 superbug or the Epochalypse) is a time computing problem that leaves some computer systems unable to represent times after 03:14:07 UTC on 19 January 2038.

The problem exists in systems which measure Unix time -- the number of seconds elapsed since the Unix epoch (00:00:00 UTC on 1 January 1970) -- and store it in a signed 32-bit integer. The data type is only capable of representing integers between -(2³¹) and 2³¹ − 1, meaning the latest time that can be properly encoded is 2³¹ − 1 seconds after epoch (03:14:07 UTC on 19 January 2038). Attempting to increment to the following second (03:14:08) will cause the integer to overflow, setting its value to -(2³¹) which systems will interpret as 2³¹ seconds before epoch (20:45:52 UTC on 13 December 1901). The problem is similar in nature to the year 2000 problem, the difference being the Year 2000 problem had to do with base 10 numbers, whereas the Year 2038 problem involves base 2 numbers.

See wikipedia article

What to do

Of course, you can just ignore it, it is merely a warning and not an error. If you want to keep your logs clean of these, you can update your filesystems or just create new ones. Ironically, this only affects UNIX style filesystems such as xfs and ext4. It does not affect other filesystems like vfat used by MSDOS.

For xfs and ext4, the Linux Kernel introduced patches back in version 5.10 circa 2021 to address these.

xfs

For xfs, when you createa new file system it should default now to using 64 bit timestamps. Otherwise you can force it with this command:

mkfs.xfs -m bigtime=1 device

If you have existing file systems you can convert them with this process:

Run xfs_repair -n to make sure that there are no errors
Run:
```
xfs_amdin -O bigtime=1 device
```

ext4

Check if the filesystem needs to be upgraded:

$ tune2fs -l device | grep "Inode size:" Inode size: 128

Where an inode size of 128 is insufficient beyond 2038 and an inode size of 256 is what you want.

Current version of mkfs will create a filesystem with the appropriate inode size. Otherwise, you can use the command with the right inode size:

mkfs.ext4 -I 256 device

Unfortunately, there is no safe way to change an existing filesystem. The current version of e2fsprogs does accept a -I 256 option, however, the functionality is disabled when flex_bg (which is on by default) is enabled. But even making sure that flex_bg is off, resizing the inodes fails.

The best way to do the conversion is then to:

backup filesystem
re-create filesystem
restore backup

NEW YEAR 2025

2024-10-22T00:00:00+02:00

This year this Blog would have been up and running for 12 years!

Happy new year sticker created by Icon home - Flaticon

Single Page Application

2024-12-13T00:00:00+01:00

As the last post of the year, I am posting my attempt at a simple "Single Page Application".

A single-page application (SPA) is a web application or website that interacts with the user by dynamically rewriting the current web page, instead of the default method of loading entire new pages. The goal is faster transitions that make the website feel more like a native app.

In a SPA, a page refresh never occurs; instead, all necessary HTML, JavaScript, and CSS code is either retrieved by the browser with a single page load, or the appropriate resources are dynamically loaded and added to the page as necessary, usually in response to user actions.

From wikipedia article.

For my simple single-page application, I wanted a web page that would present a form that will let the user enter some input parameters, then click a button and a page showing a QR code base on the input parameters will be displayed. If the user were to scan the QR code, then the page will be open showing some payload text and an optional URL.

The SPA makes use of these libraries:

base32-js: Used to encode the URL to escape special characters.
qrcodejs: QR-code generation library

To make things easy for me, I chose to limit the javascript to only those libraries and using only:

HTML
JavaScript
CSS

There is a single HTML that gets loaded at the beginning and contains all the interactive screens:

Loading screen
Initial screen, usually gets replaced immediatly so you should hardly see it.
home screen
Default screen, showing a form to get input parameters.
qrdisplay screen
Screen showing the QR code
error screen
Screen showing that decoding failed for some reason
decoded screen
Screen showing the decoded QR code

On load all screens except for Loading are disabled (CSS style display: none).

After loading, script.js will check if the URL contains a # character indicating there is a fragment identifier. If found, it will decode and show the contents of the fragment identifier.

If there was no fragment identifier, the home screen will be shown.

Note that all the interaction happens in the web browser. Even entering the URL with the fragment identifier, will only send the web page request to the server. The fragment identifier remains local to the browser and is never send to the network.

The script source can be found in github.

You can check-it out in action in this demo.

So, that is for this year. See you next in year 2025!.

Fitbit reset

2024-12-21T00:00:00+01:00

How to reset your Fibit Charge 5, 6 or Fitbit Luxe?

You can reset your Fitbit Charge 5, Charge 6, or Fitbit Luxe in 2 ways:

Soft reset: restart your Fitbit. You keep your data.
Hard reset: revert to factory setting. Your data is removed from your Fitbit.

Method 1:

Swipe down from the start screen (with the clock face on it).
Go to Settings.
Tap Restart device.
Tap Restart again.

Doesn't the restart solve anything? Follow the following steps.
Note: You will lose all your data:

Connect your Fitbit to your charger.
Press the button on the flat side of the charger 3 times. This flat piece is fixed to the USB-A connector. Wait 1 second after each press.
Wait 10 seconds until the Fitbit logo appears.

Python development tips 2024

2025-05-03T00:00:00+02:00

Local install packages
Better debugging
Built-in exceptions
Adding site specific customizations
Constants
Using Environment Variables for Configuration

Local install packages

Install using a venv and pip

python3 -m venv --system-site-packages tmpenv
. tmpenv/bin/activate
pip install icecream

Copy the resulting python lib/python3.12 to /usr/local/lib/python3 (omitting pip)

Normally, copying virtual environments would not work as paths are hard-coded in many files. However, here we are only copying the files in lib/python3.1 to /usr/local/lib/python3. Most python distributions would automatically pick up files from there. However, you can force it using PYTHONPATH environment variable.

Better debugging

So instead of using print for outputing debuging code, you can use icecream:

icecream

Interestingly, there are versions for other languages:

It looks very interesting as it is a better way to get debugging output. I am thinking of using it like this:

try:
    from icecream import ic
    ic.configureOutput(includeContext=True) # Optional... shows source and line numbers
except ImportError:  # Graceful fallback if IceCream isn't installed.
    ic = lambda *a: None if not a else (a[0] if len(a) == 1 else a)  # noqa

Requires: python3-executing python3-asttokens python3-colorama

Built-in exceptions

Python prefers the use of exceptions instead of return values to indicate errors. Since you wouldn't want to create a new exception class for every error you may need to raise, it is good to have a list of Python's built-in exception handy.

Python Built-in exceptions

List of exceptions:

Generic exceptions
- Exception
- ArithmeticError
- BufferError
Specific exceptions
- AssertionError - related to assert calls
- AttributeError - related to object attributes
- EOFError
- GeneratorError - raised when a generator or a corroutine closes
- ImportError
- ModuleNotFoundError
- IndexError - for numeric subscripts (i.e. list or tupples)
- KeyError - for dict
- KeyboardInterrupt - Control+C
- NotImplementedError
- OSError
- OverflowError
- RecursionError
- ReferenceError
- RuntimeError - this is the "others" exception.
- StopIteration - raised by next()
- StopAsyncIteration
- SyntaxError
- IndentationError
- TabError
- SystemError - An Python interpreter internal error. These should be reported to the python maintainers.
- SystemExit - raised by sys.exit().
- TypeError
- UnboundLocalError
- UnicodeError
- UnicodeEncodeError
- UnicodeDecodeError
- UnicodeTranslateError
- ValueError
- ZeroDivisionError
- EnvironmentError, IOError, WindowsError - Only applicable to Microsoft Windows
OS exceptions: These are subclasses of OSError.
- BlockingIOError
- ChildProcessError
- ConnectionError
- BrokenPipeError
- ConnectionAbortedError
- ConnectionRefusedError
- ConnectionResetError
- FileExistsError
- FileNotFoundError
- InterruptedError
- IsADirectoryError
- NotADirectoryError
- PermissionError
- ProcessLookupError
- TimeoutError

There are more exceptions but these are not commonly used.

For catching exception, it is useful to refer to the Exception class tree.

Raising exceptions uses the following syntax:

if x < 0:
  raise ValueError(f'Sorry, {x} is not valid')

And you can then catch it with:

try:
  execute_code()
except ValueError as e:
  print(e)
except:
  print("generic")

Adding site specific customizations

When python starts, it will load a module called sitecustomize where site specific customizations can be done. My idea is to have a script create this:

#!/bin/sh
type python || exit 1
site_dir=$(python -m site | grep site-packages | tr -d "'," | grep -v USER_SITE | xargs)
if [ -z "$site_dir" ] ; then
  echo "Unable to determine site dir" 1>&2
  exit 2
fi
if [ ! -d "$site_dir ] ; then
  echo "$site_dir: not found" 1>&2
  exit 3
fi
if [ ! -w "$site_dir" ] ; then
  echo "$site_dir: no write access" 1>&2
  exit 4
fi

dd of="$site_dir/sitecustomize.py" <<_EOF_
  import site
  site.addsitedir("/usr/local/lib/python3")
_EOF_

Constants

While Python doesn't have a constant of constants, the convention in Python is to write the name in capital letters with underscores separating words.

By using capital letters only, you're communicating that the current name is intended to be treated as a constant—or more precisely, as a variable that never changes. So, other Python developers will know that and hopefully won't perform any assignment operation on the variable at hand.

NOTE: Python doesn't support constants or non-reassignable names. Using uppercase letters is just a convention, and it doesn't prevent developers from assigning new values to your constant. So, any programmer working on your code needs to be careful and never write code that changes the values of constants. Remember this rule because you also need to follow it.

While this may seem pointless (or redundant) this has the following benefits:

Advantage	Description
Improved readability	A descriptive name representing a given value throughout a program is always more readable and explicit than the bare-bones value itself. For example, it's easier to read and understand a constant named `MAX_SPEED` than the concrete speed value itself.
Clear communication of intent	Most people will assume that `3.14` may refer to the `Pi` constant. However, using the `Pi`, `pi`, or `PI` name will communicate your intent more clearly than using the value directly. This practice will allow other developers to understand your code quickly and accurately.
Better maintainability	Constants enable you to use the same name to identify the same value throughout your code. If you need to update the constant's value, then you don't have to change every instance of the value. You just have to change the value in a single place: the constant definition. This improves your code's maintainability.
Lower risk of errors	A constant representing a given value throughout a program is less error-prone than several explicit instances of the value. Say that you use different precision levels for `Pi` depending on your target calculations. You've explicitly used the values with the required precision for every calculation. If you need to change the precision in a set of calculations, then replacing the values can be error-prone because you can end up changing the wrong values. It's safer to create different constants for different precision levels and change the code in a single place.
Reduced debugging needs	Constants will remain unchanged during the program's lifetime. Because they'll always have the same value, they shouldn't cause errors and bugs. This feature may not be necessary in small projects, but it may be crucial in large projects with multiple developers. Developers won't have to invest time debugging the current value of any constant.
Thread-safe data storage	Constants can only be accessed, not written. This feature makes them thread-safe objects, which means that several threads can simultaneously use a constant without the risk of corrupting or losing the underlying data.
Detect typos	A very common mistake specially for string constants is to mispell the value itself. By making the value a constant, if the constant is mispelled, an error will be raised

From realpython.

Using Environment Variables for Configuration

Learn how experienced developers use environment variables in Python, including managing default values and typecasting.

From Doppler blog.

As a developer, you’ve likely used environment variables in the command line or shell scripts, but have you used them as a way of configuring your Python applications?

This guide will show you all the code necessary for getting, setting, and loading environment variables in Python, including how to use them for supplying application config and secrets.

Why use environment variables for configuring Python applications?

Before digging into how to use environment variables in Python, it's important to understand why they're arguably the best way to configure applications. The main benefits are:

Deploy your application in any environment without code changes
Ensures secrets such as API keys are not leaked into source code

Environment variables have the additional benefit of abstracting from your application how config and secrets are supplied.

Finally, environment variables enable your application to run anywhere, whether it's for local development on macOS, a container in a Kubernetes Pod, or platforms such as Heroku or Vercel.

Here are some examples of using environment variables to configure a Python script or application:

Set FLASK_ENV environment variable to "development" to enable debug mode for a Flask application
Provide the STRIPE_API_KEY environment variable for an Ecommerce site
Supply the DISCORD_TOKEN environment variable to a Discord bot app so it can join a server
Set environment specific database variables such as DB_USER and DB_PASSWORD so database credentials are not hard-coded

How are environment variables in Python populated?

When a Python process is created, the available environment variables populate the os.environ object which acts like a Python dictionary. This means that:

Any environment variable modifications made after the Python process was created will not be reflected in the Python process.
Any environment variable changes made in Python do not affect environment variables in the parent process.

Now that you know how environment variables in Python are populated, let's look at how to access them.

How to get a Python environment variable

Environment variables in Python are accessed using the os.environ object.

The os.environ object seems like a dictionary but is different as values may only be strings, plus it's not serializable to JSON.

You've got a few options when it comes to referencing the os.environ object:

# 1. Standard way

import os

# os.environ['VAR_NAME']

# 2. Import just the environ object

from os import environ

# environ['VAR_NAME']

# 3. Rename the `environ` to env object for more concise code

from os import environ as env

# env['VAR_NAME']

Accessing a specific environment variable in Python can be done in one of three ways, depending upon what should happen if an environment variable does not exist.

Let's explore with some examples.

Option 1: Required with no default value

If your app should crash when an environment variable is not set, then access it directly:

print(os.environ['HOME']

# >> '/home/dev'

print(os.environ['DOES_NOT_EXIST']

# >> Will raise a KeyError exception

For example, an application should fail to start if a required environment variable is not set, and a default value can't be provided, e.g. a database password.

If instead of the default KeyError exception being raised (which doesn't communicate why your app failed to start), you could capture the exception and print out a helpful message:

import os
import sys

# Ensure all required environment variables are set
try:  
  os.environ['API_KEY']
except KeyError: 
  print('[error]: `API_KEY` environment variable required')
  sys.exit(1)

Option 2: Required with default value

You can have a default value returned if an environment variable doesn't exist by using the os.environ.get method and supplying the default value as the second parameter:


# If HOSTNAME doesn't exist, presume local development and return localhost

print(os.environ.get('HOSTNAME', 'localhost')

If the variable doesn't exist and you use os.environ.get without a default value, None is returned

assert os.environ.get('NO_VAR_EXISTS') == None

Option 3: Conditional logic if value exists

You may need to check if an environment variable exists, but don't necessarily care about its value. For example, your application can be put in a "Debug mode" if the DEBUG environment variable is set.

You can check for just the existence of an environment variable:

if 'DEBUG' in os.environ:
  print('[info]: app is running in debug mode')

Or check to see it matches a specific value:

if os.environ.get('DEBUG') == 'True':
  print('[info]: app is running in debug mode')

How to set a Python environment variable

Setting an environment variable in Python is the same as setting a key on a dictionary:

os.environ['TESTING'] = 'true'

What makes os.environ different to a standard dictionary, is that only string values are allowed:

os.environ['TESTING'] = True
# >> TypeError: str expected, not bool

In most cases, your application will only need to get environment variables, but there are use cases for setting them as well.

For example, constructing a DB_URL environment variable on application start-up using DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, and DB_NAME environment variables:

os.environ['DB_URL'] = 'psql://{user}:{password}@{host}:{port}/{name}'.format(
  user=os.environ['DB_USER'],
  password=os.environ['DB_PASSWORD'],
  host=os.environ['DB_HOST'],
  port=os.environ['DB_PORT'],
  name=os.environ['DB_NAME']
)

Another example is setting a variable to a default value based on the value of another variable:

# Set DEBUG and TESTING to 'True' if ENV is 'development'
if os.environ.get('ENV') == 'development':
  os.environ.setdefault('DEBUG', 'True') # Only set to True if DEBUG not set
  os.environ.setdefault('TESTING', 'True') # Only set to True if TESTING not set

How to delete a Python environment variable

If you need to delete a Python environment variable, use the os.environ.pop function:

To extend our DB_URL example above, you may want to delete the other DB_ prefixed fields to ensure the only way the app can connect to the database is via DB_URL:

Another example is deleting an environment variable once it is no longer needed:

auth_api(os.environ['API_KEY']) # Use API_KEY
os.environ.pop('API_KEY') # Delete API_KEY as it's no longer needed

Why default values for environment variables should be avoided

You might be surprised to learn it's best to avoid providing default values as much as possible. Why?

Default values can make debugging a misconfigured application more difficult, as the final config values will likely be a combination of hard-coded default values and environment variables.

Relying purely on environment variables (or as much as possible) means you have a single source of truth for how your application was configured, making troubleshooting easier.

Using a .env file for Python environment variables

As an application grows in size and complexity, so does the number of environment variables.

Many projects experience growing pains when using environment variables for app config and secrets because there is no clear and consistent strategy for how to manage them, particularly when deploying to multiple environments.

A simple (but not easily scalable) solution is to use a .env file to contain all of the variables for a specific environment.

Then you would use a Python library such as python-dotenv to parse the .env file and populate the os.environ object.

Using python-dotenv, you can save the below to a file named .env (note how it's the same syntax for setting a variable in the shell):

API_KEY="357A70FF-BFAA-4C6A-8289-9831DDFB2D3D"
HOSTNAME="0.0.0.0"
PORT="8080"

Then save the following to dotenv-test.py:

# Rename `os.environ` to `env` for nicer code
from os import environ as env

from dotenv import load_dotenv
load_dotenv()

print('API_KEY:  {}'.format(env['API_KEY']))
print('HOSTNAME: {}'.format(env['HOSTNAME']))
print('PORT:     {}'.format(env['PORT']))

Then run dotenv-test.py to test the environment variables are being populated:

python3 dotenv-test.py
# >> API_KEY:  357A70FF-BFAA-4C6A-8289-9831DDFB2D3D
# >> HOSTNAME: 0.0.0.0
# >> PORT:     8080

While .env files are simple and easy to work with at the beginning, they also cause a new set of problems such as:

How to keep .env files in-sync for every developer in their local environment?
If there is an outage due to misconfiguration, accessing the container or VM directly in order to view the contents of the .env may be required for troubleshooting.
How do you generate a .env file for a CI/CD job such as GitHub Actions without committing the .env file to the repository?
If a mix of environment variables and a .env file is used, the only way to determine the final configuration values could be by introspecting the application.
Onboarding a developer by sharing an unencrypted .env file with potentially sensitive data in a chat application such as Slack could pose security issues.

Z-Wave Associations with Home Assistant

2024-11-22T00:00:00+01:00

Introduction
Configuring associations
Setting associations
Removing associations
Last words

This is an update to Z-Wave associations with Vera UI.

Introduction

Z-Wave associations lets you control slave devices from a master device without the Z-Wave controller or Hub.

For this to really work the master device sending commands must support this functionality. This varies from device to device, so you must look up the documentation of the master device and find the supported associations groups. In a nutshell, you look-up what each association group is for, and then you add to that group the slave devices that will receive the Z-wave commands from the master.

See this article for more information on associations.

In Home Assistant, Z-Wave support is provided by add-ons. There are two add-ons available:

Z-Wave JS : from the official Add On store
Z-Wave JS UI : from the Community Add Ons

Both are based on the Z-Wave JS project. However the Z-Wave JS UI Add On from the Community store, exposes the Z-Wave JS control panel, where as the official Z-Wave JS Add on does NOT.

Home assistant by default does not expose the functionality and UI interface to modify Z-Wave group associations. That is why Z-Wave JS UI Add On is required.

Configuring associations

Open the Z-Wave JS UI user interface by using the sidebar link (if enabled in the addon settings page) or from the settings addon page, using the link OPEN WEB UI.

Select control panel from the tab menu.

Find the master device from the list of devices, and open its properties. Click on the GROUPS tab.

Setting associations

To create a new association click on the ADD option.

Complete the form:

Node Endpoint : Simply select from the list. Usually Root Endpoint
Group : Select the group. Usually the group refers to the event that will be used. For example, it could be a button group.
Target Node : Select from the list the slave device that we want to control.
Target Endpoint : Leave blank unless the slave device has multiple controls (for example a double switch).

Removing associations

Simply click on the red trashcan icon next to the association you want to delete.

Last words

Note that some associations are always available by default and can not be changed. These are used to report the device state to the Z-Wave Controller/Hub.

Ansible Snippets

2024-11-22T00:00:00+01:00

Bootstraping
Execution order
Create a file without external template
Creating inventory script
Writing ansible modules in sh
- Check mode
- Diff mode
Writing Ansible Action Plugins
- Transferring data from an ActionPlugin
Using Ansible playbooks through SSH bastion hosts/jump servers
Report no changes when running a script
- Adding check_mode to a script
Role File structure
Disable gathering facts
vars and defaults in roles
- Best practices

Bootstraping

Create a default config file:
ansible-config init --disabled > ansible.cfg
Createa new ansible role directory:
ansible-galaxy init --init-path roles _role-name_
This creates the directory role-name in the roles directory.

Execution order

pre_tasks
roles (in the order they are listed)
tasks
handlers (only if notified by a task and before post_tasks)
post_tasks

Create a file without external template

Normally you can use the template module to send a Jinja2 templated file to the managed host. This requires that the Jinja2 template be stored in its own file. Some times I find it more convenient to place the template directly into the YAML file.

This can be achieved using the copy module and the content parameter. Example:

- name: Create a customized file
  copy:
    content: |
      Hello {{ who }},
      This file was created by Ansible.
    dest: /path/to/your/file.txt
  vars:
    who: "World"

Creating inventory script

When specifying inventories in ansible, an executable file can be specified. This will be executed to generate the inventory. It takes the following command line arguments:

script --list
Returns a JSON formatted inventory. Example:
```
{
    "group001": {
        "hosts": ["host001", "host002"],
        "vars": {
            "var1": true
        },
        "children": ["group002"]
    },
    "group002": {
        "hosts": ["host003","host004"],
        "vars": {
            "var2": 500
        },
        "children":[]
    }
    "_meta": {
        "hostvars": {
            "host001": {
                "var001" : "value"
            },
            "host002": {
                "var002": "value"
            }
        }
    }
}
```
The output should be a JSON object containing all the groups to be managed as dictionary items. Each group's value should be either an object containing a list of each host, any child groiups, and potential group variables, or simply a list of hosts. Empty elements of a group can be omitted.
An optional _meta block can be added to contain host specific variables. If this is omitted, the --host command line argument is used to query host variables.
script --host hostname
Where hostname is a host from the --list. The script should return either an empty JSON object, or a JSON dictionary containing meta varaibles specific to the host. Example:
```
{
    "VAR001": "VALUE",
    "VAR002": "VALUE"
}
```

Other arguments are allowed but ansible will not use them.

See Inventory scripts for more details.

You can use the ansible-inventory command to see what ansible will process for its inventory.

Writing ansible modules in sh

See Ansible Module architecture for more details.

This documents how to create Old-style Ansible module. These are less efficient but are asier to implement in shell script.

Ansible playbooks are meant to be declarative in nature. So for handling more complex tasks the recommendation is to write modules, which then can be used from a playbook. To create a module in shell script, you just need to create a file in your module's path (ANSIBLE_LIBRARY). Input parameters are given as a file in "$1". This file is formatted as a sourceable file so using:

. "$1"

The return code of the script is used to determine if the module was succesful or an error happened.

The output of the script must be in JSON format. And it should contain the following keys:

changed : boolean indicating if changes were made
msg : optional informational message, particularly useful in an error condition.
ansible_facts : dictionary containing facts that will be added to the playbook run. This is optional.

There are additional arguments sent to the script. These are internal ansible arguments. Worth mentioning are:

_ansible_no_log (boolean) : do not log output. Used to keep sensitive strings from logging.
_ansible_debug (boolean) : debugging
_ansible_diff (boolean) : Running in diff mode.
_ansible_check_mode (boolean) : Running in check mode.

Check mode

If _ansible_check_mode is set to True, the user is running the playbook with the --check flag. Modules should only show that changes would be made, without making any actual changes.

More details here

Diff mode

If _ansible_diff is set to True, the user is running the playbook with the --diff flag. Modules that support this should add a key named diff with either:

two keys, before and after. This contains the contents before and after the change.
single prepared key. This contains a list with a textual descriptions of the changes to be made.

See article

Writing Ansible Action Plugins

Modules described earlier, run on the managed node. While they can return data to the control node, sometimes is necessary to do some preparatory work on the control node before calling a Module proper. This is done using Action Plugins

Essentially, action plugins let you integrate local processing and local data with module functionality.

To create an action plugin, create a new class with the Base(ActionBase) class as the parent:

from ansible.plugins.action import ActionBase

class ActionModule(ActionBase):
    pass

From there, execute the module using the _execute_module method to call the original module. After successful execution of the module, you can modify the module return data.

module_return = self._execute_module(module_name='<NAME_OF_MODULE>',
                                     module_args=module_args,
                                     task_vars=task_vars, tmp=tmp)

A simple example template:

#!/usr/bin/python
# Make coding more python3-ish, this is required for contributions to Ansible
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

from ansible.plugins.action import ActionBase
from datetime import datetime

class ActionModule(ActionBase):
    def run(self, tmp=None, task_vars=None):
        result = super(ActionModule, self).run(tmp, task_vars)
        module_args = self._task.args.copy()
        module_return = self._execute_module(module_name='setup',
                                             module_args=module_args,
                                             task_vars=task_vars, tmp=tmp)
        result.update(module_return)
        return result

Transferring data from an ActionPlugin

So you need to copy a large file from the control node to the managed node in an Action Plugin. To do this you need to declar in your class:

TRANSFERS_FILES = True

Next in your Action implementation, you can use this command to create temporary file:

tmp_src = self._connection._shell.join_path(self._connection._shell.tmpdir, 'archive.zip')

Then you can copy bytes:

self._transfer_data(tmp_src, bytes_object)

or copy a file:

self._transfer_file(local_file, tmp_src)

A complete example:

#!/usr/bin/python

# Make coding more python3-ish, this is required for contributions to Ansible
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

from ansible.plugins.action import ActionBase
from ansible.errors import AnsibleActionFail, AnsibleError

class ActionModule(ActionBase):
  TRANSFERS_FILES = True

  def run(self, tmp=None, task_vars=None):
    result = super(ActionModule, self).run(tmp, task_vars)

    # Get the arguments passed to the action plugin
    src = self._task.args.get('src', '')
    if len(src) == 0:
      raise AnsibleActionFail('Missing or empty src parameter',result=result)

    # Copy archive to remote/managed node:
    try:
      tmp_src = self._connection._shell.join_path(self._connection._shell.tmpdir, 'archive.zip')
      self._transfer_file(src, tmp_src)
    except AnsibleError as e:
      raise AnsibleActionFail(to_text(e))

    return result

Using Ansible playbooks through SSH bastion hosts/jump servers

There are two approaches:

Inventory vars

The first way to do it with Ansible is to describe how to connect through the proxy server in Ansible's inventory. This is helpful for a project that might be run from various workstations or servers without the same SSH configuration (the configuration is stored alongside the playbook, in the inventory).

Example Inventory:

[proxy]
bastion.example.com

[nodes]
private-server-1.example.com
private-server-2.example.com
private-server-3.example.com

[nodes:vars]
ansible_ssh_common_args='-o ProxyCommand="ssh -p 2222 -W %h:%p -q username@bastion.example.com"'

This sets up an SSH proxy through bastion.example.com on port 2222 (if using the default port, 22, you can drop the port argument). The -W argument tells SSH it can forward stdin and stdout through the host and port, effectively allowing Ansible to manage the node behind the bastion/jump server.

The important config line is ansible_ssh_common_args, which adds the relevant options to the ansible ssh command. A few notes on the options:

Recent SSH versions could just use:

ansible_ssh_common_args='-J username@bastion.example.com:2222"'

SSH config

The alternative, which would apply the proxy configuration to all SSH connections on a given workstation, is to add the following configuration inside your ~/.ssh/config file:


Host private-server-*.example.com
   ProxyJump user@bastion:2222

Ansible will automatically use whatever SSH options are defined in the user or global SSH config, so it should pick these settings up even if you don't modify your inventory.

This method is most helpful if you know your playbook will always be run from a server or workstation where the SSH config is present. Also, this applies to normal ssh invokations so if you also use the ssh and related utilities directly, then, htey will use the same configuration.

TCP Tunneling

These options assume that the bastion host has TCP Tunneling/Forwarding enabled. If your bastion host has this feature disabled, you can replace the ProxyJump with proxy command:

  ProxyCommand ssh user@bastion nc %h %p

This replaces the -W option with the nc (netcat) command.

Source: https://www.jeffgeerling.com/blog/2022/using-ansible-playbook-ssh-bastion-jump-host

Report no changes when running a script

Running a script always assume that it makes changes.

Using scripts is not recommenteded because it is just as easy to convert the script into a proper Ansible module. Doing so makes it also possible to:

Return ansible facts
Report change status
Support check and diff modes.

Regardless, this is an example:

tasks:

- name: Exec sh command
  shell:
    cmd: "echo ''; exit 254;"
  register: result
  failed_when: result.rc != 0 and result.rc != 254
  changed_when: result.rc != 254

I have customized command module and the script action plugin to simplify these three lines of code into a single line. So the previous example becomes:

tasks:

- name: Exec sh command
  shell:
    cmd: "echo ''; exit 254;"
  no_change_rc: 254

Adding check_mode to a script

In addition, it is actually possible to support check_mode in a script. You need to:

Pass the appropriate settings to your script indicating to it that it is in check_mode.
This can be done by adding a check for the ansible_check_mode variable in a Jinja2 template.
Set-up the task so that it will also execute in check_mode by adding the tag: check_mode: false to it.

Example:

- name: "Apply config to "
  command: |
    xop cfg --no-change-rc=127 {% if ansible_check_mode %}--dry-run{% endif %} 
  register: res
  failed_when: res.rc != 0 and res.rc != 127
  changed_when: res.rc != 127
  check_mode: false

In this example, the xop cfg command gets executed regardless if check_mode is on or off.
The script then is passed the --dry-run option if ansible_check_mode is on. So the script will not make any actual changes to the system.

Scripts:

Role File structure

roles
- role name
  - tasks
    - main.yml : tasks file can include smaller files if warranted
  - handlers
    - main.yml : handlers file
  - templates : files to use in the template resource
    - ntp.conf.j2 : templates end in .j2
  - files
    - bar.txt : files for use with the copy resource
    - foo.sh : script files for use with the script resource
  - vars
    - main.yml : variables associated with this role
  - defaults
    - main.yml : default lower priority variables for this role
  - meta
    - main.yml : role dependencies
  - library : roles can also include custom modules
  - module_utils : roles can also include custom module_utils
  - lookup_plugins : or other types of plugins, like lookup in this case

See: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html

Disable gathering facts

This needed for managing hosts that do not have a python interpreter by default. Specially for scenarios where it is needed to bootstrap the python interpreter.

There are different ways to do this.

Disabling on a per playbook basis.
Example:

# sample playbook
- hosts: sites
  user: root
  tags:
  - configuration
  gather_facts: no
  tasks:
  - ....
  # ... tasks to install python ...
  - ...
  # Optionally, call setup explicitly
  - setup
  # ... more tasks ...

Using an environment variable
ANSIBLE_GATHERING
This can be set to explicit. For example:
ANSIBLE_GATHERING=0 ansible-playbook playbook.yaml -i inventory.yaml
From ansible.cfg:
- plays will gather facts by default, which contain information about the remote system.
- Options:
  - smart - gather by default, but don't regather if already gathered
  - implicit - gather by default, turn off with gather_facts: False
  - explicit - do not gather by default, must say gather_facts: True
- example:
```
gathering = explicit
```

vars and defaults in roles

In an ansible role, there are two places where you can define variables, defaults and vars directories. It always was confusing to me when would you use one over the other one.

defaults
- Contains default variables for the role.
- Variables in this directory have the lowest precedence, meaning they can be easily overridden by other variable sources (e.g., playbooks, command line).
- Use this for variables that can be set by users or other roles but need to be initialized with a default value.
- Example:
```
# defaults/main.yml
my_default_variable: "default_value"
```
vars
- Contains variables that are generally more static and not meant to be easily overridden.
- Variables in this directory have higher precedence than those in defaults but lower than those defined in inventory files or extra vars.
- Use this for essential role-specific variables that should rarely change.
- Example:
```
# vars/main.yml
my_static_variable: "static_value"
```

Best practices

Use defaults/ for configurable options: Place variables here if you expect them to be overridden by the user or other roles.
Use vars/ for crucial settings: Use this directory for variables that are critical to the role and not intended to be changed.
Document Variables: Always document the purpose and acceptable values for your variables, especially in defaults/ since users might change them.
Keep it Simple: Avoid overly complex variable hierarchies. Keep your role's variables understandable and maintainable.

Modifying VM configuration with libvirtd

2024-11-22T00:00:00+01:00

The other day, I had to update a VM configuration managed via libvirt from the command line. There are different ways to do this. The easiest probably is to use the virt-manager application and use the GUI to modify things.

... virt-manager screenshot ...

Alternatively you can use virsh with dumpxml+ undefine + define or edit to modify the XML definition directly. While straight forward this is more diffiecult.

A good in-between is to use the virt-xml command, which lets you add/remove devices directly. See man page.

Examples

Adding a network interface:

virt-xml vmname --add-device --network bridge=br-ens13f3,model=virtio

Removing sda definition:

virt-xml vmname --remove-device --disk target=sda

Adding sda definition as an iscsi:

virt-xml vmname --add-device --disk target=sda,driver.type=raw,source.protocol=iscsi,source.name=iqn.2023-10.world.srv:dlp.target01/1,source.host.name=192.168.39.184,source.host.port=3260,target.bus=sata

Note that this can not specify iSCSI initiator IQN

Adding a serial console:

virt-xml vmname --add-device --serial pty,target.port=0

This functionality can be access programmatically from python directly (not via a command) using:

import os
import sys
sys.path.insert(0, '/usr/share/virt-manager')
from virtinst import virtxml

Unfortunately there is no API documentation for this so this needs to be read from the source.

Comments enabled

2024-11-22T00:00:00+01:00

Today I enabled comments on this blog.

You need to have an github account in order to make comment. This is because the main audience for this blog is developers and this should cut down on the spam.

Comments are stored GitHub Issues using the Utterances comments widget.

I chose Utterances because:

it is lightweight
makes use of the issues API, which at the moment should be more stable than the discussions API.
Has its own bot, so there is no need to embed javascript with an app ID/secret.

Tweaking HDMI dummies

2024-11-22T00:00:00+01:00

Prepare EDID blob
Modify Kernel command line
Update initramfs
Reboot
Post boot EDID

This is a follow-up to my post about HDMI hotplug.

Back then I wrote it to handle a KVM switch and how it would deal with switching monitors and how Linux handled when the monitor was not plugged in.

Since then I discovered HDMI dummy plugs. These are small adapters that connect to an HDMI and fool the computer into thinking there is a monitor still connected.

Using it is really a matter of preference.

On my desk, I have a KVM connecting a Linux desktop and a Windows laptop. When the KVM connects to the Windows laptop I would configure the system to extend the desktop to two monitors. When I switch from the Windows laptop to my Linux desktop with the KVM switch, the Windows Operating system detects that the monitor is disconnected and moves all the opened windows from the external monitor (which is now disconnected) to the Laptop display. While this is the common sense thing to do, I don't find the experience pleasent.

By using the HDMI dummy plug adaptor, Windows thinks that the monitor still is connected while the KVM switches to the other computer. So when I switch back, all the windows stay put.

So the issues that I was having back as described in HDMI hotplug would not happen as the monitor looks like is always connected.

Furthermore, looking at the state of software under Linux, the Operating System already handles monitor hotplugs gracefully. Similarly, Desktop Environments work without much issues with monitors being plugged or unplugged on the fly. Also in HDMI hotplug I was using the XDM (Display Manager) which was initially written back 1988, and is one of the most rudimentary display managers.

Of course things in life are never simple. So this is what happened.

When I first started using HDMI dummies I had a 1080p FHD monitor and 1080p dummies. Recently I upgraded my main monitor to a QHD monitor. So the 1080p dummy was not able to reach the QHD (1440p resoluation). Of course, there is no 1440p dummy, so I bought a 4K HDMI dummy. Which made things a little bit challenging because the cheapo 4K HDMI dummy would insist to the computer that the preferred resolution was 4K, but my monitor would not be able to display that.

Normally monitors use EDID data to tell the computer what are the preferred resolutions. A good HDMI dummy would then cache the EDID information from the monitor and only present a compatible EDID to the computer. For some reason this did not work.

Luckly Linux allows to force EDID to be overriden. To do that this is what i had to do.

Prepare EDID blob

First plug in the monitor directy without the HDMI dummy. Determine the connection in use:

for p in /sys/class/drm/*/status; do con=${p%/status}; echo -n "${con#*/card?-}: "; cat $p; done

Read the EDID blob from /sys/class/drm/card0{connection}/edid.

cat < /sys/class/drm/card0-HDMI-A-1/edid` > monitor.bin

Use the command parse-edid to read the blob and make sure it contains valid data.

Save it to /usr/lib/firmware/edid. I would save it to a file with the monitor model name and bin extension.

Modify Kernel command line

Add the following option to the Linux kernel command line:

drm.edid_firmware=HDMI-A-1:edid/PHL325B1L.bin

Change HDMI-A-1 with the right connector as seen using a previous command. Replace PHL325B1L.bin with the name of the EDID blob you created earlier.

Update initramfs

Under voidlinux, the dracut utility is used to generate the initramfs.

Create a file in /etc/dracut.conf.d/ with the configuration:

# /etc/dracut.conf.d/50-fw.conf 
install_items+=" /usr/lib/firmware/edid/PHL325B1L.bin "

And generate the new initramfs:

dracut --force

Reboot

At this point all is ready. Shutdown the computer, plug the monitor using the HDMI dummy.

Post boot EDID

Alternatively, you could use this script to force EDID after boot:

#!/bin/sh
#
# Script to force EDID settings
#
set -euf -o pipefail

#
# Customize these settings
#
EDID_BIN=/usr/lib/edid/PHL325B1L.bin
output=HDMI-A-1
rows=45
cols=160

# Use parse-edid to verify
# save /sys/class/drm/card*/edid
debugfs=/sys/kernel/debug

if [ ! -d $debugfs/dri ] ; then
  umount=true
  mount -t debugfs debugfs $debugfs
  trap "umount $debugfs" EXIT
fi

dd if=$EDID_BIN of=$debugfs/dri/0/$output/edid_override
echo 1 | dd of=$debugfs/dri/0/$output/trigger_hotplug

for n in $(seq 1 6)
do
(
  tput sc
  printf '\033[1;'$rows'r'
  tput rc
  stty rows $rows cols $cols
) < /dev/tty$n >/dev/tty$n 2>&1
done

References:

Docker in Docker

2024-11-22T00:00:00+01:00

Introduction
Method 1: Mounting /var/run/docker.sock
- docker.sock permission error
Method 2: Docker in Docker Using DinD
Method 3: Docker in Docker Using Sysbox Runtime
Method 4: Don't run docker in docker
Conclusions

Introduction

In simple terms, Docker Inside Docker involves running Docker within a Docker container.

These are the major use cases that I know of for doing this:

Running an application to manage docker from a container (Example of this is portainer).
Building docker containers from within a container
- CI/CD pipelines that build and push docker images to a container registry.
- Modern CI/CD systems that support Docker-based runners
Container isolation for: multi-tenancy, enhanced security, resource management.
- Sandboxed environments
- For experimental purposes on your local development workstation.
- Resource isolation between different groups or development teams.

This use cases bring the following benefits (Quote):

Isolated Development and Testing:
Running Docker inside Docker allows developers to create isolated environments specifically tailored for their applications. This ensures that dependencies, configurations, and runtime environments remain consistent across different development stages, making it easier to reproduce and debug issues.
Enhanced Security and Isolation:
Running Docker inside Docker allows developers to create isolated environments specifically tailored for their applications. This ensures that dependencies, configurations, and runtime environments remain consistent across different development stages, making it easier to reproduce and debug issues.
Simplified CI/CD Pipelines:
Docker Inside Docker is widely used in Continuous Integration and Continuous Deployment (CI/CD) workflows. It enables the creation of self-contained, disposable environments for building, testing, and deploying applications, allowing for faster and more reliable automation pipelines.
Multi-tenancy and Resource Management:
Nested containerization can be incredibly useful in scenarios where multiple teams or users require isolated environments on shared infrastructure. By launching Docker inside Docker, you can provide each team or user with a separate Docker engine, ensuring resource isolation and preventing interference between different applications.

There are a number of approaches for doing this depending on the use case one may be more appropriate than others:

Mounting /var/run/docker.sock
Run Docker in Docker using DnD
Run Docker in Docker using Sysbox run-time
Don't run Docker in Docker

See here.

Method 1: Mounting `/var/run/docker.sock`

/var/run/docker.sock is the default Unix socket. Sockets are meant for communication between processes on the same host.

Docker daemon by default listens to docker.sock. If you are on the same host where the Docker daemon is running, you can use the /var/run/docker.sock to manage containers. meaning you can mount the Docker socket from the host into the container

For example, if you run the following command, it will return the version of the docker engine.

curl --unix-socket /var/run/docker.sock http://localhost/version

To run docker inside docker, all you have to do is run docker with the default Unix socket docker.sock as a volume.

For example,

docker run -v /var/run/docker.sock:/var/run/docker.sock \
           -ti docker

Just a word of caution: If your container gets access to docker.sock, it means it has more privileges over your docker daemon. So when used in real projects, understand the security risks, and use it.

Now, from within the container, you should be able to execute docker commands for building and pushing images to the registry.

Here, the actual docker operations happen on the VM host running your base docker container rather than from within the container. Meaning, even though you are executing the docker commands from within the container, you are instructing the docker client to connect to the VM host docker-engine through docker.sock

This method is the way to run applications like portainer which are used to manage docker environments.

Because of the security implications I wouldn't recommend this for building docker images or for doing container isolation.

docker.sock permission error

While using docker.sock you may get permission denied error. In that case, you need to change the docker.sock permission to the following.

sudo chmod 666 /var/run/docker.sock

Also, you might have to add the --privileged flag to give privileged access.

The docker sock permission gets reset server restarts. To avoid this you need to add the permission to system startup scripts.

For example, you can add the command to /etc/rc.local so that it runs automatically every time your server starts up.

Also, Keep in mind that 666 permissions open a security hole.

Method 2: Docker in Docker Using DinD

This method actually creates a child container inside a Docker container. Use this method only if you really want to have the containers and images inside the container. Otherwise, I would suggest you use the first approach.

For this, you just need to use the official docker image with dind tag. The dind image is baked with the required utilities for Docker to run inside a docker container.

Note: This requires your container to be run in privileged mode.

Step 1: Create a container named dind-test with docker:dind image '''bash docker run --privileged -d --name dind-test docker:dind
Step 2: Log in to the container using exec.
```
docker exec -it dind-test /bin/sh
```
Step 3: Once you are inside the container, execute docker commands as needed.

This approach is usually geared towards creating Docker images within docker. The main disadvantage is that you are still runnning your container in priviledge mode which can be considered a security risk.

Method 3: Docker in Docker Using Sysbox Runtime

This method avoids running containers in priviledge mode by using the sysbox runtime.

Sysbox is an open-source and free container runtime (a specialized "runc"), that enhances containers in two key ways:

Improves container isolation:
- Linux user-namespace on all containers (i.e., root user in the container has zero privileges on the host).
- Virtualizes portions of procfs & sysfs inside the container.
- Hides host info inside the container.
- Locks the container's initial mounts, and more.
Enables containers to run same workloads as VMs:
- With Sysbox, containers can run system-level software such as systemd, Docker, Kubernetes, K3s, buildx, legacy apps, and more seamlessly & securely.
- This software can run inside Sysbox containers without modification and without using special versions of the software (e.g., rootless variants).
- No privileged containers, no complex images, no tricky entrypoints, no special volume mounts, etc.

If you create a container using sysbox runtime, it can create virtual environments inside a container that is capable of running systemd, docker, kubernetes without having privileged access to the underlying host system.

Step 1: Install the sysbox runtime environment.
Step 2: Once you have the sysbox runtime available, all you have to do is start the docker container with a sysbox runtime flag as shown below. Here we are using the official docker dind image.
```
docker run --runtime=sysbox-runc --name sysbox-dind -d docker:dind
```
Step 3: Now take an exec session to the sysbox-dind container.
```
docker exec -it sysbox-dind /bin/sh
```
Step 4: Once you are inside the container, execute docker commands as needed.

This method is intented for sandboxing, running Container as VMs, isolating, etc. Let's you run system software that requires special priviledges into isolated and secure environment.

Sysbox solves problems such as:

Enhancing the isolation of containerized microservices (root in the container maps to an unprivileged user on the host).
Enabling a highly capable root user inside the container without compromising host security.
Securing CI/CD pipelines by enabling Docker-in-Docker (DinD) or Kubernetes-in-Docker (KinD) without insecure privileged containers or host Docker socket mounts.
Enabling the use of containers as "VM-like" environments for development, local testing, learning, etc., with strong isolation and the ability to run systemd, Docker, IDEs, and more inside the container.
Running legacy apps inside containers (instead of less efficient VMs).
Replacing VMs with an easier, faster, more efficient, and more portable container-based alternative, one that can be deployed across cloud environments easily.
Partitioning bare-metal hosts into multiple isolated compute environments with 2X the density of VMs (i.e., deploy twice as many VM-like containers as VMs on the same hardware at the same performance).
Partitioning cloud instances (e.g., EC2, GCP, etc.) into multiple isolated compute environments without resorting to expensive nested virtualization.

Method 4: Don't run docker in docker

If you only need to docker images (For example inside a CI/CD pipeline with Docker or K8s executors), you can simply use a tool that is focused on building docker images without using docker itself. For example:

buildah
img

These two are available in void linux

Conclusions

Running docker in docker used to be a requirement to build docker images from within a docker container (as part of a CI/CD pipeline based on docker container executors).

Today in 2024, that is no longer needed, one you can just use a solution as indicated in Method 4.

However, docker in docker solutions are still needed to run things like portainer using Method 1 to run docker management solutions from within a container. I remember seeing other applications such as Infobox Blox One DDI that took a similar approach.

Lastly, there is still a need to run Containers as VMs or to create Docker like environments for testing and isolation. This use case exists, but I find it a bit niche. In my view, you are better off runninge these as a full VM which gives you better flexibility and more discrete isolation. It is less efficient, but in today's world, the overhead due to virtualization is minimal compared to the complexity these special solutions bring.

Telekom Cloud CI/CD demo

2024-11-22T00:00:00+01:00

Pre-requisites
Preparation
Base infrastructure
- Notes
Sample Workload
- Accessing Nginx
- Notes
Code Repository
Setting up Jenkins
Creating pipeline
- Build steps
Test run the workload

At work, I got asked to do a demo for a simple CI/CD pipeline on the Open Telekom Cloud. Since it was short notice, I kept it simple and used the web console to set things up. In the future, a Terraform script would have been more re-usable, but I did not have enough time.

The created CI/CD infrastructure is made of the following:

Code Repository: Hosted on GitHub
Jenkins CI : Hosted on Telekom Cloud VM
- GitHub Integration (plugin)
SWR for Containers
Cloud Container Engine
- K8s Master
- Compute Nodes
- Shared Load Balancer

Architecture:

When implemented the CI/CD pipeline looks as follows:

Trigger: code commit
Prepare for build
Build Java components via Maven
Build Docker Image
Push image to SWR
Deploy Image to running Pod

Additional tasks can be added such as Static code analysis and linting, Unit tests, Integration tests, Static application security testing, vulnerability scanning, etc.

Pre-requisites

SSH key-pairs to login.
- Go: Service List → Computing → Elastic Cloud Server
- Click: Cloud Server Console ... Network and Security ... Key Pair

Preparation

To keep things clean, the first thing I like to do is:

Create a tag for reporting.
- Go: Service List → Management & Deployment → Tag Management Service
- Click: TMS ... Predefined Tags
- Click: Create Tag
- NOTE: The pre-defined tag list doesn't seem to be honored in all possible scenarios.
Create a project for isolating resources.
- Go: Service List → Management & Deployment → Identity and Access Management
- Click: IAM ... Projects
- Click: Create Project
Create an access group scopped to the project.
- Go: Service List → Management & Deployment → Identity and Access Management
- Click: IAM ... User Groups
- Click: Create User Group
- Select the new group and click Authorize.
- Select Policy/Roles:
  - Tenant Administrator : I suspect this is the only needed one.
  - The following are used by the Full admin role, while I have not tested them are not needed:
    - Agent Operator
    - Security Administrator
- Click Next and select the Scope. Change to Region-specific projects and select the project as created in the previous step.
- Click Ok to confirm.
Create an account scoped for the given project.
- Go: Service List → Management & Deployment → Identity and Access Management
- Click: IAM ... Users
- Save the Login Link for later:
- Click Create User
- Fill-in:
  - Username
  - Description
  - Access Type:
    - Programmatic access
    - Management console access
  - Credential Type:
    - Password
    - Set now
    - untick Require password reset at first login
- Click Next add the user group created in the previous step.

Base infrastructure

Create VPC
- Go: Service List → Network → Virtual Private Cloud
- Create VPC. Don't forget to add tags.
- Configure the first subnet and add a tag for it.
- Click Network Console → Access Control → Security Groups
- Click Create Network Security Group, and use the template for General-purpose web server.
- Modify the new network security group. For Jenkins we need to add ports 8080 and 50000.
Create Jenkins CI host
- Go: Service List → Computing → Elastic Cloud Server
- Click Create ECS
  - Select the project from Region.
  - Select flavor. I suggest a General-purpose flavor as these are the most cost-effective.
  - For the image, I used: Public Image : Ubuntu : 22.04_latest.
  - System disk: Used the default High I/O setting, but changed the size to around 100GB for this example.
  - Pick the network and network security group created in the previous step.
  - EIP set to Auto assign.
  - Select the Key Pair to use. And click Advanced Options to apply tags.
- For this demo we are using a single VM for Jenkins CI host. We use EIP so that the VM can received Web Hooks from GitHub. Since Jenkins is Java based I would recommend at least 4GB of memory.
- Make sure there is sufficient storage. Set the system disk to be at least 16 GB or more.
- It is recommended to create a suitable DNS record. Otherwise you would need to use IP addresses.
- After the VM is created you can use the SSH keys to log in. The default user name is ubuntu.
- Install Java dependancies:
  - openjdk-17-jdk-headless openjdk-17-jre-headless maven
  - In this example we are using Java v17 because some dependancies on the selected applications.
- Install kubectl
- Install and configure Jenkins.
- Also install these additional packages:
  - docker.io plus cgroupfs-mount debootstrap rinse pigz elfutils
  - Add the jenkins user to the docker group:
    - adduser user group
    - The Jenkins service may need to be restarted.
- Retrieve initial password from: /var/lib/jenkins/secrets/initialAdminPassword
Create cloud container engine cluster
- Go: Service List → Computing → Cloud Container Engine
- Click Create Cluster
  - Type: CCE Standard Cluster
  - Enter Cluster Name
  - Cluster Scale: 50 nodes
  - Master Nodes: Single
    Make sure these two settings are set to 50 and Single so that you get a free of cost set of master nodes.
  - Select the VPC and Subnet created in a previous step.
  - Network Model: VPC Network
    Read Network model comparison for explanation, but the TLDR is:
    - VPC Network: Has low overhead but only scales up to small-medium networks.
    - Tunnel Network: Has higher overhead (thus lower performance) but can scale to large networks.
  - Select Advanced Settings to configure relevant tags.
  - Add-Ons configuration can be left on defaults.
- Add a node to the cluster by clicking on the Create Node link.
  - Specifications: Change to a suitable side for the cluster nodes.
  - OS: I am using Ubuntu 22.04 as that is what I am personally familiar with.
  - Login Mode: Select a key from the available key pairs. You can use the private key to log in to the cluster node (use ubuntu for the user name due to using the Ubuntu 22.04 image).
  - Network Settings, change EIP to Auto create but make sure that the Line is set to Dynamic BGP and Bandwidth to 1.
  - Select Advanced Settings to configure relevant tags.
  - In the Advanced Settings you can add additional commands to run pre or post K8s installation.
  - Conveniently you can create more than one node here by adjusting the quantity at the bottom of the page.
Create Software Repository for Containers
- Go: Service List → Application → Software Repository for Container
- Click Create Organization
- Create a Long-Term valid login command.

At this point we have all the infrastructure needed.

Notes

This was all done from the web console. Ideally all this should have been generated via Terraform scripts.

We are creating a VM to run Jenkins. This could be replaced with a Jenkins Pod on the K8s cluster with external executors as K8s pods. Initially I did not do it like that because of my dependancies on docker for the application build process. However you can use tools like the ones listed here or there and still perform the builds from a running K8s pod. To create dynamic executor from K8s pods you need the Kubernetes plugin.

For simplicity I am using Compute nodes with EIPs. Probably it should be more secure to remove the EIPs and use a NAT gateway if the compute nodes need to download packages from external repositories.

Sample Workload

To test that the infrastructure is working, we can deploy a sample workload:

Go: Service List → Computing → Cloud Container Engine
Click the name of the target cluster to access the cluster console.
In the navigation pane, choose Workloads. Then, click Create Workload.
Configure the following parameters and retain the default value for other parameters:
- Basic Info
  - Workload Type: Select Deployment.
  - Workload Name: Set it to nginx.
  - Namespace: Select default.
  - Pods: Set the quantity of pods to 1.
- Container Settings
  Enter nginx:latest in the Image Name text box.
- Service Settings
  Click the plus sign (+) to create a Service for accessing the workload from an external network. This example shows how to create a LoadBalancer. Configure the following parameters in the window that slides out from the right:
  - Service Name: Enter nginx. The name of the Service is exposed to external networks.
  - Service Type: Select LoadBalancer.
  - Service Affinity: Retain the default value.
  - Load Balancer: If a load balancer is available, select an existing load balancer. If not, select Auto create to create one.
  - Ports:
    - Protocol: Select TCP.
    - Service Port: Set this parameter to 80, which is mapped to the container port.
    - Container Port: port on which the application listens. For containers created using the nginx image, set this parameter to 80. For other applications, set this parameter to the port of the application.
Click Create Workload.
Wait until the workload is created.
The created Deployment will be displayed on the Deployments tab.

Accessing Nginx

Obtain the external access address of Nginx.
Click the Nginx workload name to enter its details page. On the page displayed, click the Access Mode tab, view the IP address of Nginx. The public IP address is the external access address.
Enter the external access address in the address box of a browser. The following shows the welcome page if you successfully access the workload.

Notes

For this demo we are letting CCE create the load balancer. And the load balancer rules are then managed directly by it. Probably creating the load balancer should be done via Terraform but I have not tried this. In theory you could use kubectl to create the service as described here.

Code Repository

For this demonstration we are using a github hosted code repository. Since our customer made use of the Spring Framework I chose to use the Spring Petclinic example application. Note that this example is interesting as there is a github org containing different variations of the petclinic, such as different frontends, microservices architecture, clouds, etc.

Create your code repository in github. You can do this by either doing a Fork of the Spring Petclinic application or using the Import functionality. The difference between the two is that Forking will keep the link between the original repository and your new repository, whereas importing will keep the two repositories independant. For the demo I chose to import.

After creating the code repository link it to Jenkins CI:

Navigate to your code repository.
Click on the Settings tab.
On the navigation pane, go to Webhooks and click on Add webhook.
- Payload URL : jenkins url /github-webhook/ e.g.
  http://demo240619-ci.otc1.cloudkit7.xyz:8080/github-webhook/
- Content type : application/json

This will trigger the webhook whenever changes are comitted.

Setting up Jenkins

Browse to the Jenkins CI web page. It defaults to port 8080. You will need the password from:

/var/lib/jenkins/secrets/initialAdminPassword

I configure the following plugins:

Dashboard View - nice dashboard view.
Credentials Binding - Needed to use credentials
Git
GitHub
Locale - Enabled this because otherwise things show as being partially translated.

I did not test these, but seem useful:

Config File Provider - Use it to store kubectl config?
Embeddable Build Status - add badges to your repos?
Warnings - Extract warnings or issues from different tools
Git Parameter - choose branches, tags or revisions

Let the set-up wizard run.

Create the first admin user. Modify the Jenkins URL if necessary. You may need to change the Jenkins URL if running behind a reverse proxy.

Creating pipeline

Go to the Jenkins CI main page and Click New Item:

Enter an item name
Select Frestyle project

Configure:

GitHub project
- Enter Project url
Source Code Management
- Choose Git
- Enter Repository URL. Add credentials if needed.
Build Triggers
- GitHub hook trigger for GITScm plling
Build Environment → Enable Use secret text(s) or file(s)
- Add Secret File: Use this to include the kubeconfig file.
- Add Username and password (separated): Use this to include docker push credentials and bind to SWR_USER and SWR_PASSWD.

Build steps

For the build steps I am just using shell scripts.

Common vars

cat > vars.sh <<-_EOF_
swr=swr.eu-de.otc.t-systems.com
org=demo240619
app=spring_petclinic
ver=3.3.0
k8sid="spring-petclinic1"
_EOF_

Maven build

./mvnw clean package -Dmaven.test.skip=true

Image build

(set +x ; docker login -u "${SWR_USER}"  -p "${SWR_PASSWD}" $swr )
docker buildx build -t $app:$ver-$BUILD_NUMBER .
docker tag "$app:$ver-$BUILD_NUMBER" "$swr/$org/$app:$ver-$BUILD_NUMBER"
docker push "$swr/$org/$app:$ver-$BUILD_NUMBER"

Update running pods

k8scmd="kubectl --kubeconfig=$KUBECFG"
$k8scmd config use-context internal 
if [ -z "$($k8scmd get deployment | awk -vID="$k8sid" '$1 == ID')" ] ; then
  echo "Deployment not found"
  exit 0
fi
$k8scmd set image deployment/$k8sid container-1="$swr/$org/$app:$ver-$BUILD_NUMBER"

Note that this step will be skipped since there won't be any running pods. They are created later.

Up to here, we have a Jenkins pipeline that

compiles a jar file
creates a docker image
pushes the image to the SWR for containers.

Test run the workload

Follow the steps as outlined Sample Workload. Select the image created by the Jenkins Pipeline.

Updates to the source code will trigger a webhook that will start the Jenkins pipeline. The pipeline will update the running pod to the newly build application.

Linux desktop default apps

2024-11-22T00:00:00+01:00

User configuration
System-wide configuration
Additional files
References

This is one of the most annoying bits about configuring things in Linux.

When you install more than one Linux application that handles the same file type sometimes the default application that will be run will not match your expectations. To check what application will be run you can use the UI or you can use check it from the command line using the xdg-mime command. Examples:

xdg-mime query default x-scheme-handler/mailto
xdg-mime query default image/jpeg

User configuration

You can use your Linux Desktop environment's User interface to set you preferred application to handle that file type.

In the case of LXQT which is the current Desktop Environment that I am using this can be found in:

Preferences → LXQt Settings → File Associations

Or run: lxqt-config-file-associations

If you want to modify files this configuration can found here:

~/.config/mimeapps.list
~/.config/lxqt-mimeapps.list Only in LXQt, in case you have application defaults specific to LXQt.

These files are in standard .ini format, example:

[Default Applications]
x-scheme-handler/http=Firefox.desktop
x-scheme-handler/https=Firefox.desktop
text/html=Firefox.desktop
application/pdf=qpdfview.desktop
inode/directory=pcmanfm-qt.desktop
image/jpeg=lximage-qt.desktop
image/jpg=lximage-qt.desktop
image/png=lximage-qt.desktop
image/gif=lximage-qt.desktop
image/webp=lximage-qt.desktop

In this file you can specify:

The default application to run when double clicking an icon
The list of applications that will be shown in the Open with... menu
The list of applications that will be REMOVED from the Open with... menu

For more info on the file format see specs.

Additonally you can use the xdg-mime command to modify the defaults. Examples:

xdg-mime default thunderbird.desktop x-scheme-handler/mailto
xdg-mime default firefox-esr.desktop text/html

System-wide configuration

For configuring the default applications for all users you can use:

/etc/xdg/mimeapps.list

The format follows the same spec as the user file and is documented in this [specification][spec].

The configuration file location can be overriden with the XDG_CONFIG_DIRS environment variable.

Additional files

Keep in mind if you are using Flatpak's, these will also introduce its own extensions that you may need to pay attention to.

Due ot the way of how the Linux desktop have evolved the following locations will also be used. Make sure these do not exist as they may cause unexpected configurations:

/etc/lxqt-mimeapps.list
/etc/mimeapps.list
/etc/xdg/lxqt-mimeapps.list
/usr/share/lxqt-mimeapps.list
/usr/share/mimeapps.list
/usr/local/share/applications/lxqt-mimeapps.list
/usr/local/share/applications/mimeapps.list
~/.local/share/applications/lxqt-mimeapps.list
~/.local/share/applications/mimeapps.list
~/.local/share/applications/defaults.list

References

Linux Serial Consoles

2026-04-07T00:00:00+02:00

Intro

In my previous article I went about adding a serial port to a NAS. The reason why this interesting to me is because it lets me remote manage my home servers at a fairly low level.

This works because I only use Linux for my home servers and only am interested in managing the Linux boot process which the serial port lets me do.

There are more powerful (and at the same more costly) alternatives for servers, such as:

Management boards like DRAC, ILO, RSA. These are propietary to specific manufacturers, but they give the most control of managed servers and can be quite costly.
IP based KVM. This also gives you full manageability while being an open solution. Systems like this can also quite expensive. The cheapest I found is Pi-KVM, however this solution is still more expensive compared to simply adding a serial port.

The big advantage of these more expensive solution is that these give you access to the BIOS menu and are compatible with graphical Operating Systems such as Windows.

Linux on the other hand can work fine with only a serial console. For Bootloader I use grub or syslinux which happily support serial consoles.

Note, that for this to work you either need a on-board serial port or a PCI/PCIe/ISA serial port. A USB to Serial converter will not work as they are not enabled early in the boot process.

In addition, for Virtual Machines, it is useful to configure them with serial console as these can be used early in the boot process of the operating system.

Specially for debugging early boot problems, simply capturing the output of the serial port as opposed to carefully watching the screen (be that physical or virtual) can be quite useful.

There used to be a line of products called PC Weasel. This was a line of graphics card that instead of output to a monitor they would output to a serial port. In addition to the graphics emulation, it would also provide keyboard emulation. It would be a manufacturer independent remote console. Unfortunately, they are no longer around.

Hardware

Serial ports can be found on many motherboards. They could either be standard DB-9 or DB-25 (very rare nowadays) or Cisco style console ports. For system that are not equiped with serial ports, adding a PCIe serial port is quite inexpensive (usually around 20EUR). However, these systems would need to have space for a PCIe card. I have seen serial port cards in M.2 format. I have not tested these.

On PCs, usually ports use a male DB9 connector. This is an old convention related to DTE/DCE terminology. Back in the day, PCs were called DTE's (Data Terminal Equipment) and the standard for DTE is to use a Male DB9 (or DB25) connector for DTEs. The PC would connect to a modem. Modems were catagorized as DCE (Data COmmunication Equipment) and the standard for DCE is to use a Female DB9 (or DB25) connector.

Since we are connecting two PCs, we would have 2 male DB9 connectors on each end. Now, do NOT use a simple gender changer to connect the two. Unlike modern equipment that can automatically negotiate transmit and receive pins, serial ports require what is called a null-modem cable/connector. So a gender changer is not enough.

Null modem

Null modem is a communication method to directly connect two DTEs (computer, terminal, printer, etc.) using an RS-232 serial cable.

Wiring:

Signal	One Side (DB9)	Signal Direction	Other Side (DB9)	Signal
Carrier Detect (CD)	1	←	4	DTR
Received Data (RxD)	2	←	3	TxD
Transmitted Data (TxD)	3	→	2	RxD
Data Terminal Ready (DTR)	4	→	1	CD
Data Terminal Ready (DTR)	4	→	6	DSR
Ground (GND)	5	-	5	GND
Data Set Ready (DSR)	6	←	4	DTR
Request to Send (RTS)	7	→	8	CTS
Clear to Send (CTS)	8	←	7	RTS

Cisco style ports

Some motherboards and small form factor (SFF) systems come with a serial port in the form of a cisco compatible console port. These are nice in the sense that they can use a rollover cable which act as a null-modemm cable, so you can connect these ports back to back (with the right cable), in addition to being smaller than even a DB9 connector.

While sometimes the right adaptors are not readily availabke, you can buy DB9 to RJ45 adaptors that need to be wired (Example).

This is not difficult to do and once wired everything snaps into place.

The basic wiring is as follows:

Signal	Color	RJ45 Pin
RTS	Blue	1
DTR	Orange	2
TxD	Black	3
GND	Red	4
GND	Green	5
RxD	Yellow	6
DSR	Brown	7
CTS	White	8

For a female adapter, the wiring becomes:

DB9 Pin	Signal	Color	RJ45 Pin
1	CD	N/A	N/A
2	TXD	Black	3
3	RXD	Yellow	6
4	DSR	Brown	7
5	GND	Red	4
5	GND	Green	5
6	DTR	Orange	2
7	CTS	White	8
8	RTS	Blue	1

Alternative Pin-out for cross-connection.

DB9 Pin	Signal	Color	RJ45 Pin
1	CD	N/A	N/A
2	RXD	Yellow	6
3	TXD	Black	3
4	DTR	Orange	2
5	GND	Red	4
5	GND	Green	5
6	DSR	Brown	7
7	RTS	Blue	1
8	CTS	White	8

For a male adapter, the wiring meaning is reversed however the RJ45 to DB9 pin assignments are the same:

DB9 Pin	Signal	Color	RJ45 Pin
1	CD	N/A	N/A
2	RXD	Black	3
3	TXD	Yellow	6
4	DTR	Brown	7
5	GND	Red	4
5	GND	Green	5
6	DSR	Orange	2
7	RTS	White	8
8	CTS	Blue	1

Raspberry Pi serial ports

On a Rasbperry Pi, there is a serial port available in the GPIO header. The configuration of GPIO header is as follows:

(See Raspberry-Pi GPIO)

For the built-in serial port is available on GPIO14 (TXD, pin8) and GPIO15 (RXD, pin10). In addition you would need to connect a ground signal. I would usually use the GND from Pin6 as is adjacent to the TXD,RXD lines. This is usually enough, however, if you are using a TTL to Serial level convert, you would need to connect a 5V power supply. In that case I would use Pin4 (again, because it is adjacent).

Note, the serial port in the Raspberry Pi is using TTL levels (3.3V). Do NOT connect directly to a standard serial port interface.

USB cables

While earlier I stated that using USB to Serial cables are not good for remote management, that is not the case for the systems being used as console. The reason is that in this scenario, we are using the serial port after the system is up and running and not during the early boot process. For that reason, for the console systems, USB cables are OK.

For me, I am using a Raspberry Pi 3B as a console server and/or a Windows laptop as as direct system console. For either use case, using the USB serial cable is OK.

In the case of Linux, after you plug in the serial cable, they will be shown to the user as /dev/ttyUSBX. Where X is a number starting from 0 (zero).

In the case of Windows, the USB serial port are usually show as device COMN. Where N is a number starting from 1 (one). However very often, USB port start being numbered from 4 onwards.

For my console server use-case, I got a USB to 4 serial ports cable. I preferred to use this to using a USB-hub and having multiple USB to serial conversion because it makes it simpler when enumerating the serial ports.

Configuration

Boot configuration

Depending on the bootloader and your hardware you need to do certain configuration changes for the serial port to be available during boot.

grub config

If booting using grub you need the following configuration settings in your grub.cfg:

serial --unit=0 --speed=115200
terimanl_input console serial
terminal output console serial

The first line configures the serial port to use. 0 (zero) is the first serial port. 1 for the second serial port. Change this accordingly.

Similarly, --speed specifies the baud rate to use. In the example we are using 115200. Select the appropriate speed. Common values are: 9600, 38400, etc.

The next two lines configure that input and output should go simultaneously to the display console and the serial port.

syslinux

To enable the serial port add the following line:

SERIAL 0 115200

As in the previous example, we are using the first serial port (0) and setting the speed to 115200.

Raspberry pi configuration

For Raspberry Pi before the Pi3 the serial port is enabled by default. From the Raspberry pi an onwards, you need to add the config line to /boot/config.txt:

enable_Uart=1

In my case I am also using an UEFI boot loader for loading grub as the boot menu. If that is the case, the UEFI bootloader will recognize the serial port automatically. However the Linux kernel needs to be passed the following command line to enable the serial port:

8250.nr_uarts=1

This is done automatically by the Raspberry PI firmware, but if you are using grub to boot Linux, make sure you add this to the kernel command line.

Linux configuration

The serial console can be configured using the kernel command line. See serial-console article.

By default under Linux, the current display and keyboard combination is the console. You can redefine the console by adding the console command line option in the Linux kernel command line. (See kernel options for all the options available for the Linux kernel command line).

For the console, the following options are possible:

console= : Output console device and options.

console can be specified multiple times. The following options after the = equal sign are recognized:

tty<n> Use the virtual console device <n>.
If <n> is zero 0, it will use the current foreground virtual console.
ttyS<n>[,options]
ttyUSB0[,options]
Use the specified serial port. The options are of the form "bbbbpnf", where "bbbb" is the baud rate, "p" is parity ("n", "o", or "e"), "n" is number of bits, and "f" is flow control ("r" for RTS or omit it). Default is "9600n8".
uart[8250],io,<addr>[,options]
uart[8250],mmio,<addr>[,options]
uart[8250],mmio16,<addr>[,options]
uart[8250],mmio32,<addr>[,options]
uart[8250],0x<addr>[,options]
Start an early, polled-mode console on the 8250/16550 UART at the specified I/O port or MMIO address, switching to the matching ttyS device later. MMIO inter-register address stride is either 8-bit (mmio), 16-bit (mmio16), or 32-bit (mmio32). If none of [io|mmio|mmio16|mmio32], <addr> is assumed to be equivalent to 'mmio'. 'options' are specified in the same format described for ttyS above; if unspecified, the h/w is not re-initialized.
hvc<n>
Use the hypervisor console device <n>. This is for both Xen and PowerPC hypervisors.

If console is specified multiple times output will appear on all of them. The last device will be used when you open /dev/console.

For example, normally I would use:

console=tty0 console=ttyS0,115200

This would generate output on the current foreground virtual console and the first serial port. The /dev/consoel device would be attached to the serial port.

If no console device is specified, the first device found capable of acting as a system console will be used. At this time, the system first looks for a VGA card and then for a serial port. So if you don’t have a VGA card in your system the first serial port will automatically become the console.

You will need to create a new device to use /dev/console. The official /dev/console is character device 5,1.

Configuring getty

To enable logins on the serial port, make sure a getty runs it, so you can login once the system is done booting. This is done by adding a line like this to /etc/inittab (exact syntax depends on your getty):

ttyS0::respawn:/sbin/getty -L 0 ttyS0 vt100

Booting Xen

When booting Xen, you need to pass the following parameters to the xen command line:

com1=115200,8n1 console=com1

This configures the first serial port and makes it the Xen console. This effectively hides that serial port from Linux. Similarly the Linux kernel command line needs to have:

console=tty0 console=hvc0

Finally, to enable logins add to /etc/inittab:

hvc0::respawn:/sbin/getty -L 0 hvc0 vt100

When using xen on the serial port you can press CTRL-a three times to switch between dom0 and xen input. In xen mode you have access to the low-level xen hypervisor. Press h for help on available options.

The CTRL-a escape key can be configured from the xen command line using option:

conswitch=<switch-char[x]

The default is:

conswitch=a

The optional x at the end means that xen will not switch to dom0 but remain in xen input mode.

During run-time, you can change the escape key using:

xl set-parameters conswitch=k

Where k is the character you want to use as escape.

Change serial parameters

During run-time you can modify the serial port setting use the stty command:

stty -a

Will display all available settings. You also check/set settings of a different serial port:

stty -a < /dev/ttyS1

Resizing the console

When running interactive commands you may need to tell server the current size of your screen. This can be done with the command:

stty rows 25 cols 80

We set the screen size to 80 columns, and 25 lines in this example. To automate this you can use this rsz script.

Accessing the Serial Console

To access the serial console you need a computer with a serial port and connecting it to the serial port of the managed system. As mentioned above, you would normally require a null modem to perform the connection.

Windows

In Windows you can use putty. Either plug in a USB to serial converter or use an on-board serial port.

First determine the name of the port that the system allocated to the serial port using device manager. The quickets way is to click on the Start button and type in device manager. It may warn you that you are running as standard user. That is fine as we only need to look up stuff, and not make any changes.

Expand Ports (COM & LPT), the serial adapter should be listed:

Alternatively you can use the mode command from the Windows command line.

Open PuTTY, and click Serial from the Category: Connection.

Edit the settings, eg: COM1, 115200, 8, 1, None. Flow Control: None.

Select the Category: Session, click the Serial radio button.

If you want, you may save the session.

Click Open to start the serail session.

Linux

The screen command can be used in Linux. Install screen if not already installed. Most distributions should have a package for it.

To find the serial port device you can:

ls /dev/ttyS* for on-board serial ports.
ls /dev/ttyUSB* for USB serial ports.

You can now use the screen command to to establish a simple serial connection.

Type screen <port_name> <baud_rate> to create a connection:

screen /dev/ttyUSB0 115200

The terminal will go blank with just a cursor. You are now connected to that port!

To disconnect, type control-a followed by shift-k. The screen will then ask if you are sure you want to disconnect.

There are other options you can control from screen, however it is recommended that you only use this method if you are comfortable with the command line. Type man screen for a full list of options and commands.

To disconnect, type control-a then shift-k.

That's all there is to it.

tinyserial

I myself use an uncomplicated command called tinyserial. This is a barebones command that sets up the serial port and connects you to it.

My local copy can be found here.

Serial ports in VMs

Configuring serial ports when creating Linux VMs lets you monitor the VM boot process from the start.

Configuring
- xen: enabled by default
- libvirt: Either add the following xml:
```
  <serial type="pty">
    <target port="0"/>
  </serial>
```
  or via virt-manager UI.
Connecting:
- xen: xl console vmname
- libvirt: virsh console vmname

Using GNU screen as Serial Console Server

You can use GNU screen mentioned earlier, to set-up a Serial Console server. For this Serial Console Server we have this functionality:

Let you connect to a serial port via the network
Log traffic over the serial port to a file

For this I am using a [Raspberry Pi]]rpi] with a multi-serial port USB adaptor. During system start-up I start GNU screen as a detached session and connect it to the serial port. Each serial port has a named session.

Using ssh I log-on to the system and use screen -r *SESSION* to re-attach to the session. The GNU screen command also logs all serial port traffic to a file.

To do this, I use the following options when starting screen:

-d -m : start the session in a detached/background state
-L : turn on automatic output logging
-Logfile file : and log to the given file.
-S name : give the session a name (useful for re-attaching)

I usually start the session with this additional arguments:

/dev/ttyUSB0 115200,crtscts

This set the given serial port to 115200 speed, and turns on RTS/CTS flow-control.

I also use the SCREENRC environment variable to specify an initialization file with further configuration.

Adding a serial port to a QNAP TS-251D

2024-11-22T00:00:00+01:00

I am using a QNAP TS-251D NAS. Because I would like to switch from QTS to Alpine Linux I though it would be useful to enable the serial port.

The TS-251D has a built-in serial port that is already enabled and only needs to be connected. For that you need a number of parts:

JST 4-pin connector. I selected Adafruit accessoires JST PH 2mm 4-pins - 3950
TTL to Serial port level converter. I am using JZK 4 PCs 3V-5V RS232 to TTL serial port module.

In addition, I am using a 3D printed bracket to hold the serial port.

Examinig the QTS environment you can see that there is indeed a /dev/ttyS0 device (a serial port) and it is enabled in the Kernel command line with:

console=ttyS0,115200n8

Simmilarly you can find in the /etc/inittab a line:

ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100

So there is a serial port available and being used for debugging.

To make it visible to the outside, you only need to connect to JST1

Pins from left to right.

For the bracket to hold the serial port I printed this 3D Model.

Place the Serial module on the 3D printed bracket.

And mount it on the case:

3D Printing updates

2024-11-22T00:00:00+01:00

Background

So I am back at playing with 3D printing.

What happened is that I am trying to add a serial port to my QNAP TS-251D NAS which uses a special "flat" low-profile bracket. Since there a none of those available in the open market, I decided to 3D print a bracket.

I was able to find a 3D model for a QNAP TS-563 PCI Bracket for AMD R7 250 Video Card PCIe, which I modified to allow for a DB9 serial port opening.

Also found a local 3D Priting service which lets me upload a model, select different config settings. They would then (for fee) print it and ship it to you.

So far the service is fairly quick and seems reliable so far.

The challenge has been the first few tries did not provide enough structural integrity for it to work. Specifically, it would bend when pushing in the serial port connector so it wouldn't set properly.

Currently I am trying modifying the shape so it bends less easily with a stronger material. Also relocated the port location so it is closer to the edge so it benefits of the support provided by the mounting screw.

The 3D Printing service I am using let's you choose the printing material among the following:

PLA (Polylactic Acid)

What do you use PLA for?

PLA is ideal for projects with limited structural requirements, such as decorative objects or small gadgets.

What are the properties of PLA?

Biodegradable and compostable.
Made from renewable resources such as corn starch or sugar cane.
Has a relatively low melting temperature.
Possesses a glossy surface, which makes it attractive for visual applications.
For which projects do I use PLA?

Beginner projects, decorative objects, prototypes that do not require high heat or impact resistance.

PETG (Polyethylene Terephthalate Glycol)

What do you use PETG for?

PETG is suitable for projects that require durability, such as functional parts or waterproof objects.

What are the properties of PETG?

Strong and reliable for long-term use.
Resistant to many chemicals.
Suitable for humid environments.

For which projects do I use PETG?

Functional parts, parts that are used outdoors, or objects that require some flexibility and strength.

TPU (Thermoplastic Polyurethane)

What do you use TPU for?

TPU is suitable for flexible and durable projects, such as phone cases, flexible hinges or shock-absorbing parts.

What are the properties of TPU?

Ideal for flexible objects.
Excellent wear resistance.
Provides good protection against shocks.
Resists oil, grease, and more.

For which projects do I use TPU?

Phone cases, hinges, shoe soles, and other products that require flexibility and durability.

ABS (Acrylonitrile Butadiene Styrene)

What do you use ABS for?

ABS is used for projects that require strength and heat resistance, such as mechanical parts or housings.

What are the properties of ABS?

ABS is strong and tough, suitable for solid objects.
Can withstand higher temperatures than many other plastics.
Can be sanded, drilled and painted for various finishes.

For which projects do I use ABS?

Functional prototypes, tools, and parts for the automotive industry

ASA (Acrylonitrile Styrene Acrylate)

What do you use ASA for?

ASA is ideal for outdoor applications that require UV resistance and weather resistance, such as automotive parts or outdoor decorations.

What are the properties of ASA?

ASA does not fade under exposure to sunlight.
Resistant to water and many weather influences.
Provides a strong and durable finish.

For which projects do I use ASA?

Outdoor projects, parts for cars or garden furniture, and any object exposed to the elements.

PA-CF (Polyamide Carbon Fiber)

What do you use PA-CF for?

PA-CF is used for highly load-bearing applications that require stiffness and wear resistance, such as industrial tools or parts with high mechanical loads.

What are the properties of PA-CF?

PA-CF provides exceptional stiffness for precision parts.
Highly resistant to wear, ideal for moving parts.
Retains properties at higher temperatures.
Good resistance to many chemicals and solvents.

For which projects do I use PA-CF?

Aerospace, automotive, and mechanical engineering, where parts must be able to withstand high temperatures and wear.

For the bracket I am using PA-CF as it seems to be the most rigid material I have tried so far.

UEFI boot on Raspberry PI

2024-11-22T00:00:00+01:00

Introduction
Prepare SD card
Copy UEFI
UEFI boot
Adding grub
Installing an operating system

Introduction

This recipe is to run UEFI boot on the raspberry pi. This is possible because there is an opensource UEFI firmware that has been ported to the Raspberry Pi.

I have tested the fimrware from pftf/RPi3. Which says:

The use of this firmware can greatly simplify the installation of generic Linux distributions such as Debian or Ubuntu as well as Windows 10 (in regular GUI mode, not IoT mode), straight from their ISO images.

This is an installable build of the official EDK2 Raspberry Pi 3 UEFI firmware.

This project states:

This is meant as a generally useful 64-bit ATF + UEFI implementation for Raspberry Pi variants based on the BCM2837 SoC, which should be good enough for most kind of UEFI development, as well as for running consummer Operating Systems in such as Linux, Windows or the BSDs.

Personally, I am using this to load grub and from there boot AlpineLinx. This allows me to have multiple versions of AlpineLinux in one SD card, and use the grub menu to select the version to boot.

This makes it possible to perform and rollback AlpineLinux upgrades without having to reflash the SD card.

Prepare SD card

First step is to create an SD card t o be used (or uSB drive), with a MBR partition table. Create a partition of of type 0x0c (FAT32 LBA) or 0x0e (FAT16 LBA). Then format this partition to FAT32. NOTE: Actually I found an article stating that the partition type should be 0x0c and the filesystem should be FAT16. I have tested this configuration, which limits the boot partition to 2GB.

Note: Do not try to use GPT for the partition scheme or 0xef (EFI System Partition) for the type, as these are unsupported by the CPU-embedded bootloader

Copy UEFI

Retrieve the UEFI Firmware from here and extract its files on the partition created in the previous step.

This firmware supports Pi revisions based on the BCM2837 SoC:

Raspberry Pi 2B v1.2 (older versions are not compatible)
Raspberry Pi 3A+
Raspberry Pi 3B
Raspberry Pi 3B+
Raspberry Pi CM3

I found additional versions however, these seem to have some hardware compatibility issues:

UEFI boot

Insert the SD card/plug the USB drive and power up your Raspberry Pi. You should see a multicoloured screen (which indicates that the CPU-embedded bootloader is reading the data from the SD/USB partition) and then the Raspberry Pi black and white logo once the UEFI firmware is ready.

At this stage, you can press Esc to enter the firmware setup, F1 to launch the UEFI Shell, or, provided you also have copied an UEFI bootloader in efi/boot/bootaa64.efi, you can let the UEFI system run that (which it should do by default if no action is taken).

Adding grub

At this point you should have an UEFI boot capable Raspberry PI. Now you need to get grub bootloader for the AArch64 architecture. I got mine from the debian netinstall iso.

For convenience the needed files can be found here.

As it is UEFI, you can simply copy the files to the boot partition created earlier.

You should create a boot menu and place it in the boot partition as boot/grub/grub.cfg.

This is an example grub.cfg:

set timeout=10
set default=alpine-rpi-3.19.1,rpi

menuentry "alpine-rpi-3.19.1,rpi (2023-12-27)" --id alpine-rpi-3.19.1,rpi {
  echo "Booting alpine-rpi-3.19.1,rpi ..."
  linux /boot/vmlinuz-rpi modloop=/boot/modloop-rpi modules=loop,squashfs,sd-mod,usb-storage quiet console=tty1 
  initrd /boot/initramfs-rpi
}

Installing an operating system

At this point you can boot your Linux based operating system as you would with any other grub based installation.

Ansible Best Practices

2024-11-22T00:00:00+01:00

This is a conversion from a presentation/pdf by Tim Appnel.

I attached a copy here too.

Roles and Modules

Complexity Kills Productivity

That's not just a marketing slogan. We really mean it and believe that. We strive to reduce complexity in how we've designed Ansible tools and encourage you to do the same. Strive for simplification in what you automate.

Optimize for Readbility

If done properly, it can be the documentation of your workflow automation.

Think Declaratively

Ansible is a desired state engine by design. If you're trying to "write code" in your plays and roles, you're setting yourself up for failure. Our YAML-based playbooks were never meant to be for programming.

Roles + Modules

Use the right tool for the job:

Roles	Modules
Self-contained portable units of Ansible automation	Small programs that perform actions on remote hosts or on their behalf
Expressed in YAML and bundled with associated assets	Expressed as code - i.e. Python, PowerShell
Decoupled from assumptions made by plays	Called by an Ansible tas
-	Modules do all of the heavy lifting in Ansible

Roles

Roles are Ansible Content

The same best practices for your plays still apply:

Use native YAML syntax
Version control your Ansible content
Use command modules sparingly
Always seek out a module first
Name your plays, blocks and tasks
Use human meaningful names with variables, hosts, etc.
Clean up your debugging messages

Role Design

Keep the purpose and function of a role self-contained and focused to do one thing well

Think about the full life-cycle of a service, microservice or container — not a whole stack or environment
Keep provisioning separate from configuration and app deployment
Roles are not classes or object or libraries - those are programming constructs
Keep roles loosely-coupled - limit hard dependencies on other roles or external variables

EXHIBIT A

# blackbox_role_playbook.yml
---
- hosts: all
  roles:
  - umbrella_corp_stack

EXHIBIT B

# componentized_roles_playbook.yml
---
- hosts: localhost
  roles:
  - azure_provisioner
- hosts: all
  roles:
  - system_security
- hosts: webservers
  roles:
  - python_common
  - python_django
  - nginx_uwsgi
  - racoon_app
- hosts: databases
  roles:
  - pgsql-replication

Maximize your role design for portability and reuse

Use ansible-galaxy to install your roles
Use a roles files (i.e. requirements.yml) to manifest your project roles
When using a shared role always declare a specific version such as a tag or commit

# requirements.yml
---
- src: nginxinc.nginx
  version: 0.8.0
- src: samdoran.pgsql-replication
  version: b5013e6
- src: geerlingguy.firewall
  version: 2.4.

Role Usability

Roles should run with as few, if any, parameter variables as possible

Practice convention over configuration
Provide sane defaults
Use variable parameters to modify default behaviour
Easier to develop, test and use quickly & securely
A role should always be more than a single task file

EXHIBIT A

# defaults_no_playbook.yml
---
- hosts: webservers
  roles:
- role: apache_simple
  apache_http_port: 80
  apache_doc_root: /var/www/html
  apache_user: apache
  apache_group: apache
- role: apache_simple
  apache_http_port: 8080
  apache_doc_root: /www/example.com
  apache_user: apache
  apache_group: apache

EXHIBIT B

# defaults_yes_playbook.yml
---
- hosts: webservers
  roles:
- role: apache_simple
- role: apache_simple
  apache_http_port: 8080
  apache_doc_root: /www/example.com

# default/main.yml
---
apache_http_port: 80
apache_doc_root: /var/www/html
apache_user: apache
apache_group: apache

Use variables in your roles appropriately

defaults/ are easy to override and most commonly used to modify behavior

i.e. port number or default user

# default/main.yml
---
apache_http_port: 80
apache_doc_root: /var/www/html
apache_user: apache
apache_group: apache

vars/ are used by the role and not likely to be changed
- i.e. a list of packages, lookup table of machine images by region
```
# vars/main.yml
---
apache_packages:
redhat:
```
- httpd
- mod_wsgi debian:
- apache2
- libapache2-mod-wsgi

Automate the testing of your roles

Use molecule, a testing framework designed to aid in the development and testing of Ansible Roles.

Initially developed by the community, led by John Dewey of Cisco, and adopted by Red Hat as an official Ansible project.

https://github.com/ansible/molecule

Use ansible-lint, a command-line static analysis tool that checks playbooks and roles for identifying behaviour that could be improved.

Initially developed by Will Thames and recently adopted by Red Hat as an official Ansible project.

https://github.com/ansible/ansible-lint

HINT: ansible-lint can be run as part of your Molecule test runs.

Role Readability

Still using command modules a lot?

- name: check cert
  shell: certify --list --name={{ cert_name }} --cert_store={{ cert_store }} | grep "{{ cert_name }}"
  register: check_output
- name: create cert
  command: certify --create --user=chris --name={{ cert_name }} --cert_store={{ cert_store }}
  when: check_output.stdout.find(cert_name) != -1
  register: create_output
- name: sign cert
  command: certify --sign --name={{ cert_name }} --cert_store={{ cert_store }}
  when: create_output.stdout.find("created") != -1

Develop your own module

- name: create and sign cert
  certify:
    state: present
    sign: yes
    user: chris
    name: "{{ cert_name }}"
    cert_store: "{{ cert_store }}

Modules

Module Design

Good modules are user-centric

Modules are how Ansible balances simple and powerful
They implement common automation tasks for a user
They make easy tasks easier and complicated tasks possible
They abstract users from having to know the details to get things done
They are not one-to-one mapping of an API or command line tool interface
- This is why you should not auto-generate your modules
They are not monolithic does everything modules that are hard to understand and complicated to use correctly

Module Implementation

Making the powerful simple starts with the implementation

No side-effects with multiple runs
Err on the side of safety
- i.e. use a temporary file and atomic_move when writing to a file
Fail fast -- immediately detect and report failure conditions
Support check mode
Minimal use of dependencies

Module Interface

Modules should provide a predictable user interface

Think desired state, think declaratively
Avoid action or command parameters
Keep parameters focused and narrowly defined - refrain from parameters that take complex data structures
Parameter names should be in lowercase and use underscores:
- update_cache # YES
- UpdateCache # NO
- updateCache # NO
Normalize common parameter names with other modules such as:
- name
- state
- dest
- src
- path
- username
- password

Modules in the wild

For your consideration...

kubernetes
- monolithic and requires expert knowledge of k8s
ansible-kubernetes-modules
- fine grained API mapping that is autogenerated
k8s
- better implementation but complex parameters abound and expert knowledge still required
k8s_scale
- more focused on a specific task — more of this please

Module Responses

Provide informative and consistent responses

Be consistent in what you return
Make response data reusable by a play or role
Return only relevant output — no logs files please
Accurately report changed status
Support diff mode if applicable - and return the diff conditionally

Module Exception Handling

Handle errors gracefully and predictably

Apply defensive programming
Fail fast — validate upfront and use the built-in argument spec function
Fail predictably and informatively when errors happen
Avoid catch all exceptions

Module Implementation

Don't reinvent the weheel

Make use of module_utils/ -- they’re your friends
- basic.py
- api.py
- facts/
- urls.py
- six.py
- noteworthy others: ec2.py, docker.py, database.py, mysql.py, powershell.ps1 — and many many more!

Module Documentation

Documentation is a requirement

Examples should include the most common and real world usage
Examples should be in native YAML syntax
Return responses must be included and described
Document your dependencies in the requirements section

Module Testing

Test before you commit and push your code

Utilize the testing tools in ansible/hacking/
- test-module
- ansible-test sanity
Create roles and playbooks and to test and verify all your documented examples
- molecule
Test locally - not with the CI/CD system

More Modules in the wild

For your consideration...

sysctl
- a master class in writing a “best practice” module
ping
- the hello world of Ansible
cron
- module implementing an interface to a command line tool
get_url
- module implementing an interface to a python library

Module and Action Plugins

Sometimes modules need something more

Local controller execution of a module entirely possible or required...
Special setup requirements before the module is dispatched to the host...
Need to supplement a module with the services of another core module such as copy and a role won’t cut it...

An Action Plugin executes on the controller and perform logic before dispatching a module

More resources

Ansible Developers Guide

http://docs.ansible.com/ansible/devel/dev_guide/index.html

hoses: enhanced socks5 tools

2024-03-05T00:00:00+01:00

Introduction
Features
Where to find
Examples
References

Introduction

The hose is a python package that implements:

socks5 proxy with SSL support
netcat style cli to use the proxy
simple stunnel implementation

For a project I had, I needed for remote systems to connect back to a central management system over the Internet. This provides a simple application level proxy that implements this connectivity.

Rather than re-inventing the wheel, it makes use of the python SSL module to provide security and certificate based authentication.

The certificate authentication uses standard X509 certifcates as supported by OpenSSL.

Application proxy uses the SOCKS, which is properly defined as RFC1928.

As functionality goes, it focus mostly in connecting network sockets and in netcat functionality piping to running commands.

Access control can be defined in static files, but also it can be delegated to external scripts or a REST API server.

Features

The following is implemented:

SOCKS v5 protocol, with CONNECT and BIND commands. It does not implement UDP Associate commands. The Authentication handshake is performed, but no Authentication is implemented. hoses expects the connection to be wrapped in a SSL/TLS stream and clients are authenticated using certificates at that level.
As mentioned earlier, SSL/TLS is used to secure the application proxy and it is performed before SOCKS protocol starts running.
The SOCKS protocol is further extended to allow connecting/binding to UNIX sockets.
Proxy activity is logged in an optional audit.log.
Access control is largely untested, but implemented as either:
- script/command that runs in the background and accepts JSON requests with client data and returns allow or deny.
- Or a combination of:
  - static access list
  - script that runs with environment variables containing request data and returns an access list
  - REST API endpoint that access JSON content and retunrs an access list.

In order to use the application proxy, netcat type functionality is implemented with the following features:

Connect to remote hosts either directly or via hoses application proxy.
Listen to incoming connections and spawn an external program.
SSL/TLS wrapped connections with optional client certificate validation.
Connections to TCP ports and UNIX sockets.

To facilitate securing legacy servers, stunnel functionality is available:

Port forwarding with optional SSL/TLS
SSL/TLS wiht optional client certificate validation

Where to find

The project is hosted on github.

Examples

Proxy

The main usage is as a socks proxy.

Basic socks5 compatible proxy:

hoses -S * -P 1080 proxy

-S * will listen on all IPv4 and IPv6 addresses.
-P 1080 listen on port 1080

Enable TLS and listen only on loopback address:

hoses -S 127.0.0.1 -P 3340 --cert server.crt --key server.key

Enable TLS and enable client certificate verification:

--cert server.crt : file path to server certificate
--key server.key : file path to corresponding key.

hoses -S 127.0.0.1 -P 3340 --cert server.crt --key server.key --ca ca.crt

--ca ca.crt Certificate file signing client certificates (or the client certificate itself for self-signed certificates.

netcat server

It can be used as netcat --listen replacement.

Listen on localhost, port 4040 and run a command:

hose listen --exec localhost 4040 'sh -c "python3 eliza.py"

Listen on unix socket on the socks server, and forward connections to port 22 on remotehost. And persist bindings.

hose -S socks-server -P 1080 listen -p unix:/tmp/sshsock 0 remotehost 22

Like previous but with TLS:

hose -S socks-server -P 1080 --cert client.crt --key client.key --ca ca.crt listen -p unix:/tmp/sshsock 0 remotehost 22

netcat client

It can be used as a netcat client to connect to network ports.

Connect to remote:

hose connect remotehost 4583

Connect to a remote TLS server with certificate based client authentication

hose --cert client.crt --key client.key --ca ca.crt connect remotehost 4583

Connect to a remote host through a SSL socks proxy with client authentication

hose -S socks-server -P 1080 --cert client.crt --key client.key --ca ca.crt connect remotehost 4583

stunnel

This is used as a replacement for stunnel.

Accept TLS connections and forward them to a different port. This is used to provide TLS encryption to protocol servers that do not support this out of the box.

hose --cert server.crt --key server.key listen --unwrap -p localhost 8011 remotehost 11

Accept unencrypted connection and forward them to a TLS server. This is used to give TLS encryption to protocol clients that do not support it out of the box.

hose --cert server.crt --key server.key listen --wrap -p localhost 11 remotehost 8011

SSH tunnel

You can use it as an ssh proxy command to make ssh go through a Socks proxy:

ssh -o "ProxyCommand hoses -S sockserver -P 1080 connect %h %p" remotehost

References

OpenSSH Certificate Authority

2024-03-05T00:00:00+01:00

Intro
Creating CA
Configuring Hosts
Singing user public keys
Using certificates
Further reading

Intro

OpenSSH supports a Certificate Authority (CA) function.

This allows you to distribute one or more trusted CA keys. Then if you want to give access to users, you simply sign their public keys with the CA private key.

Users then can login to a server without having to have their public keys in the authorized_keys file as long as their public key was signed by a trusted CA.

Furthermore, when signing keys, you can apply further restrictions such as:

user allowed to login as
validity periods
SSH restrictions similar to the options in the authorized_keys file. e.g. forced command, no-pty`, restrict tcp forwarding, etc.
IP addresses that the key can originate from
and more...

For larger organizations with lots of users and servers, makes managing authorizition keys a much simpler process.

Creating CA

To begin using OpenSSH certificates you first must generate an ssh key that will be kept secret and used as the certificate authority in your environment. This can be done with a command like:

ssh-keygen -f my_ssh_cert_authority

That command outputs two files:

my_ssh_cert_authority: The encrypted private key for your new authority
my_ssh_cert_authority.pub: The public key for your new authority.

Be sure you choose a passphrase when prompted so that the secret is stored encrypted. Other options to ssh-keygen are permitted including both key type and key parameters. For example, you might choose to use ECDSA keys instead of RSA.

Grab the fingerprint of your new CA:

$ ssh-keygen -l -f my_ssh_cert_authority
2048 2b:a1:16:84:79:0a:2e:38:84:6f:32:96:ab:d4:af:5d my_ssh_cert_authority.pub (RSA)

Configuring Hosts

Now that you have a certificate authority you'll need to tell the hosts in your environment to trust this authority. This is done very similar to user SSH keys by setting up the authorized_keys on your hosts (the expectation is that you're setting this up at launch time via cloudinit or perhaps baking the change into an OS image or other form of snapshot).

You have a choice of putting this authorized_keys file into $HOME/.ssh/authorized_keys or the change can be made system wide. For system wide configuration see sshd_config(5) and the TrustedUserCAKeys option.

If you are modifying the user's authorized_keys file simply add a new line to authorized_keys of the form:

cert-authority <paste the single line from my_ssh_cert_authority.pub>

A valid line might look like this for an RSA key:

cert-authority ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAYQC6Shl5kUuTGqkSc8D2vP2kls2GoB/eGlgIb0BnM/zsIsbw5cWsPournZN2IwnwMhCFLT/56CzT9ZzVfn26hxn86KMpg76NcfP5Gnd66dsXHhiMXnBeS9r6KPQeqzVInwE=

If configuring system-wide, modify your /etc/ssh/sshd_config to include:

TrustedUserCAKeys /etc/ssh/ca_keys

Copy the CA public keys:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAYQC6Shl5kUuTGqkSc8D2vP2kls2GoB/eGlgIb0BnM/zsIsbw5cWsPournZN2IwnwMhCFLT/56CzT9ZzVfn26hxn86KMpg76NcfP5Gnd66dsXHhiMXnBeS9r6KPQeqzVInwE=

You can have multiple CA public keys in this file. This can be used to keep CA keys off-line and rotate them on a regular basis. Storing offline CA keys is also useful to provide emergency access in the event of a compromised CA key.

Singing user public keys

At this point your host has been configured to accept a certificate signed by your authority's private key. Let's generate a certificate for ourselves that permits us to login as the user ubuntu and that is valid for the next hour (This assumes that our personal public SSH key is stored at ~/.ssh/id_rsa.pub) :

ssh-keygen -V +1h -s my_ssh_cert_authority -I bvanzant -n ubuntu ~/.ssh/id_rsa.pub

The output of that command is the file ~/.ssh/id_rsa-cert.pub. If you open it it's just a base64 encoded blob. However, we can ask ssh-keygen to show us the contents:

$ ssh-keygen -L -f ~/.ssh/id_rsa-cert.pub
/tmp/test_main_ssh-cert.pub:
    Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT f6:e3:42:5e:72:85:ce:26:e8:45:1f:79:2d:dc:0d:52
    Signing CA: RSA 4c:c6:1e:31:ed:7b:7c:33:ff:7d:51:9e:59:da:68:f5
    Key ID: "bvz-test"
    Serial: 0
    Valid: from 2015-04-13T06:48:00 to 2015-04-13T07:49:13
    Principals:
            ubuntu
    Critical Options: (none)
    Extensions:
            permit-X11-forwarding
            permit-agent-forwarding
            permit-port-forwarding
            permit-pty
            permit-user-rc

Using certificates

Let's use the certificate now::

# Add the key into our ssh-agent (this will find and add the certificate as well)
ssh-add ~/.ssh/id_rsa
# And SSH to a host
ssh ubuntu@<the host where you modified authorized_keys>

If the steps above were followed carefully you're now SSHed to the remote host. Fancy?

At this point if you look in /var/log/auth.log (Ubuntu) (/var/log/secure on Red Hat based systems) you'll see that the user ubuntu logged in to this machine. This isn't very useful data. If you change the sshd_config on your servers to include LogLevel VERBOSE you'll see that the certificate key id is also logged when a user logs in via certificate. This allows you to map that user bvanzant logged into the host using username ubuntu. This will make your auditors happy.

Why Ansible?

2024-03-05T00:00:00+01:00

As part of an effort of standardising my home lab I decided to migrate my ad-hoc configuration scripts into a more standard tool set. So I looked at:

At the end I opted for ansible, as it was closer in operation to my configuration scripts. The main points for me where:

Central control node, with push semantics
No agent needed, all communications via ssh. But it has python dependancies.
Written in python
Configuration mostly in YAML files.
Very wide acceptance in the IT community.

Later I found about cdist, while it is closer to my ad-hoc scripts, it is a bit niche and not well known. However, like ansible has a central control node push model, all communications via ssh and has no python dependancies, requiring only a compatible shell.

Configuration of cdist is written in essentially shell scripts. According to this wikipedia article:

Ansible makes a distinction between roles, written in a declarative YAML-based language, and modules, written in Python. Cdist only has "types" which serve the purposes of both modules and roles and are mostly written in Bourne Shell. Cdist's approach might be preferable because Shell is familiar to many system administrators who have never used a configuration management system before, but Ansible's declarative language is arguably more readable and appropriate.

The remaining options all require agents, which is not ideal as I really wanted a low footprint on the managed nodes.

cfengine has a nice theoretical framework but also, not as popular as ansible or the other entries in this list. While written in C it uses a DSL for its configurations.

puppet is written in a mix of of Ruby, Clojure and C++ and uses a DSL for configuration. chef is written in a mix of Ruby and ERLAN and uses DSL for Configuration. Personally, I see the use of Ruby as a turn-off. So far several softwares I tried to use written in Ruby did not lead to good experiences for me.

saltstack is written in python.

Removing Python as a dependancy

I am mostly using Alpine Linux for servers. As such, adding python adds to a base image of 17.1MB, an additional 50MB. So it is a bit unreasonable. I found a project ansible-openwrt which removes the python dependancies but it has a lot of OpenWRT.

I manage remove the OpenWRT dependancies, and the result can be found HERE.

It makes use of the Ansible's var_plugin functionality to hook into the ActionBase._configure_module method.

Also adds a number of modules written in shell script that replace the built-in python modules. Specially important are:

setup: which is called automatically to gather facts at start.
ping: because, it is the first thing to test things.
stat: which is used by several modules specially copy.
command: which is used by command and shell modules.

Interestingly enough, there are other modules such as script that do not have a python dependency.

OpenTofu

2024-11-22T00:00:00+01:00

Introduction
Origins
Why use OpenTofu
Using OpenTofu

Introduction

OpenTofu is open-source fork of Terraform from HashiCorp.

OpenTofu is an "Infrastructure-as-code" software tool. Users can use it to define and provide data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language (HCL), or optionally JSON.

I first started to explore OpenTofu because I wanted to migrate some hosted VMs from the OpenTelekomCloud. Back then, I was using a custom script to deploy infrastructure to the cloud from description files written in YAML.

I found the approach of deploying cloud infrastructure from description files much more convenient to doing the same from the Web UI.

While the Web UI is very useful to explore the capabilities of a cloud service, for a more "production-like" infrastructure, I would consider that approach very poor. Deploying several VMs, would require multiple clicks, depending on good memory skills to make sure that deployment were consistent.

When I was contemplating migrating this VMs from OTC to Oracle Cloud I felt doing it via the Web UI very unapealing. And while there is an API available, I did not find the inclination to write a custom script.

Recently, speaking to some co-workers, I was reminded of Terraform. And decided to try that this time. Of course, at the time, HashiCorp had recently switched licenses, so switched to OpenTofu instead.

My expectation is that migrating between OpenTofu and Terraform and back should be a simple experience, and any learned skills in OpenTofu should translate directly into Terraform. In fact, except for the installation of OpenTofu, I have been following tutorials for Terraform instead. So far everything has been essentially the same.

Origins

In August 2023 Terraform (along with several of HashiCorp's products) switched from an Open Source license (MPL2.0) to non Open-Source Business Source License. This change prompted a group of users to fork the last available version of Terraform, v1.5.5, as OpenTofu.

Why use OpenTofu

OpenTofu is an "Infrastructure-as-code" software, this allows you to build, change, and manage your infrastructure in a safe, consistent, trackable, and repeatable way by defining resource configurations that you can version (in a version control system like GitHub), reuse, and share.

This yields to more reliable infrastructure, and at the same time allow you to quickly make/implement changes to it in response of changing requirements.

Using OpenTofu

Installation

The official installation instructions can be found in their Documentation Webpage.

Whenever possible using your native package manager (e.g. apt on Debian Linux, dnf on Fedora, etc) would be the preferred option. In my case, I am using Void Linux, so that option was not available to me.

Luckly, OpenTofu is implemented as a single binary executable (It is developed in go language), so simply downloading this from their release page and adding it somewhere in your executable path to be sufficient.

There even is a Windows version which can be installed in a similar way.

For your convenience they provide with an "installation" script, but being it a single binary, I felt that this was irrelevant.

Configuring for OCI

Sign up for Oracle Cloud
You can skip these steps if using an Administrator user. However, it is highly recommended to use a "non-Administrator" account when available.
- Create Compartment:
  - Go to Identity, Compartments.
  - Click Create Compartment. Complete the form and create the compartment.
  - Wait a couple of minutes for the compartment to be create.
  - Note and record the compartment name and ID (which will be needed later)
- Create Group:
  - Go to Identity, Domains.
  - Select the desired domain. Either create a new domain or use the default.
  - On the sidebar, click on Groups.
  - Click on Create group and complete the form. You can leave users empty and create the API user later.
- Create a Policy:
  - Go to Identity, Policies.
  - Click Create policy. Enter Name and Description.
  - Note: The form can select a Compartment. I don't know how this works as I only tested on the root compartment.
  - Select Show manual editor and enter this as the policy:
```
allow group <group-name> to read all-resources in tenancy
allow group <group-name> to manage all-resources in compartment <compartment>
```
- Create API user:
  - Go to Identity, Domains.
  - Select the desired domain. Either create a new domain or use the default.
  - On the sidebar, click on Users.
  - Click on Create user and complete the form. Add the user to the relevant group.
    This way the scope of access for a given user/group is limited to a compartment.
Create RSA keys, this can be done either via command line or using the Web Console.
- From the command line:
  - openssl genrsa -out <your-home-directory>/.oci/<your-rsa-key-name>.pem 2048
  - chmod 600 <your-home-directory>/.oci/<your-rsa-key-name>.pem
  - openssl rsa -pubout -in <your-home-directory>/.oci/<your-rsa-key-name>.pem -out $HOME/.oci/<your-rsa-key-name>_public.pem
  - Assign key to user account:
    - Go to Identity, Domains.
    - Select the desired domain. Either create a new domain or use the default.
    - On the sidebar, click on Users.
    - Select the API user.
    - On the left sidebar click on API Keys.
    - Click Add API Key. Select Paste Public Keys and paste the value of the contents of the public key file. Click Add
- From the Web Console:
  - Go to Identity, Domains.
  - Select the desired domain. Either create a new domain or use the default.
  - On the sidebar, click on Users.
  - Select the API user.
  - On the left sidebar click on API Keys. Select Generate API Key pair.
  - Download Private key. You may download the public key, but that is strictly optional.
  - Click Add
- After adding the key, save the sample config information. You will need it later.

Setting up API Key-Based authentication

Create a working directory to be used in our OpenTofu project.

Create a file named provider.tf with the following contents:
```
provider "oci" {
 tenancy_ocid = "<tenancy-ocid>"
 user_ocid = "<user-ocid"
 fingerprint = "<fingerprint>"
 region = var.region_name
 private_key_path = "<private-key-path>"
}
```
The tenancy_ocid, user_ocid and fingerprint would have come from the config settings in the last step of "Configuring OCI". The region name is used in more locations so we will record it as variable so we don't specify it here.
The private-key-path is the path to the private RSA key from an earlier step. Use Linux/UNIX style "slash" (/) directory separators here even under MS-Windows.

Create a vars.tf with contents:

#
# Define variables
#
variable "compartment_ocid" {
  type        = string
  description = "ID of the compartment in OCI to use"
  default     = "<compartment-ocid>"
}
variable "region_name" {
  type        = string
  description = "Region to deploy resources"
  default     = "<region>"
}

The compartment-ocid comes from the compartment that was created during "Configuring OCI".
The region comes from the sample config from "Configuring OCI".

Create a file availability-domains.tf with contents:

data "oci_identity_availability_domains" "tutorial_ads" {
 compartment_id = var.compartment_ocid
}

Declare networking

Create a file vcn-module.tf with contents:
```
module "vcn" {
  source  = "oracle-terraform-modules/vcn/oci"
  version = "3.1.0"
  #
  # Required Inputs
  compartment_id = var.compartment_ocid
  region = var.region_name
  internet_gateway_route_rules = null
  local_peering_gateways = null
  nat_gateway_route_rules = null
  #
  # Optional Inputs
  vcn_name = "<vcn-name>"
  vcn_dns_label = "<vcn-dns-label>"
  vcn_cidrs = ["10.0.0.0/16"]
  #

  create_internet_gateway = true
  create_nat_gateway = false
  create_service_gateway = false
}
```
NOTE: In this example, we are fixing the module version. Don't know if this is needed or not.
Select a suitable name for virtual cloud's vcn-name.
Similarly, select a suitable vcn-dns-label. However, in Oracle Free tiers there are no DNS zones, so this setting is useless.
vcn_cidrs in this example is set to 10.0.0.0/16. Feel free to modify as needed.
We are only setting create_internet_gateway to true. This allows VMs in the public subnets to be reachable from the Internet. This is the only gateway allowed in the Oracle Free tier.
create_nat_gateway is set to false as this is not supported by the Oracle Free tier. This gateway allows VMs in the private subnets to communicate to the Internet. There is no in-bound access from the Internet to the private subnets.
create_service_gateway is set to false as this is not supported by the Oracle Free tier. This gateway allows VMs to communicate to the Oracle service end-points without going over the Internet.

create a file pub-seclst.tf with contents:

 resource "oci_core_security_list" "public-security-list" {

   # Required
   compartment_id = var.compartment_ocid
   vcn_id = module.vcn.vcn_id

   # Optional
   display_name = "security-list-for-public-subnet"

   ingress_security_rules { 
       stateless = false
       source = "0.0.0.0/0"
       source_type = "CIDR_BLOCK"
       # Get protocol numbers from https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml TCP is 6
       protocol = "6"
       tcp_options { 
           min = 22
           max = 22
       }
     }
   ingress_security_rules { 
       stateless = false
       source = "0.0.0.0/0"
       source_type = "CIDR_BLOCK"
       # Get protocol numbers from https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml ICMP is 1  
       protocol = "1"

       # For ICMP type and code see: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
       icmp_options {
         type = 3
         code = 4
       } 
     }   

   ingress_security_rules { 
       stateless = false
       source = "10.0.0.0/16"
       source_type = "CIDR_BLOCK"
       # Get protocol numbers from https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml ICMP is 1  
       protocol = "1"

       # For ICMP type and code see: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
       icmp_options {
         type = 3
       } 
     }

   egress_security_rules {
       stateless = false
       destination = "0.0.0.0/0"
       destination_type = "CIDR_BLOCK"
       protocol = "all" 
   }

 }

This defines the security list protecting the public subnet.

Create pub-subnet.tf with contents:

 resource "oci_core_subnet" "vcn-public-subnet"{

   # Required
   compartment_id = var.compartment_ocid
   vcn_id = module.vcn.vcn_id
   cidr_block = "10.0.255.0/24"

   # Optional
   route_table_id = module.vcn.ig_route_id
   security_list_ids = [oci_core_security_list.public-security-list.id]
   display_name = "public-subnet"
 }

Change cidr_block, display_name as needed.

Declare compute instance

Create login keys:
```
ssh-keygen -t rsa -N "" -b 2048 -C <your-ssh-key-name> -f <your-ssh-key-name>
```
The command generates some random text art used to generate the keys. When complete, you have two files:
- The private key file: your-ssh-key-name
- The public key file: your-ssh-key-name.pub
Create cloud-init.yaml with contents:
```
#cloud-config
runcmd:
- echo 'Hello world' >> /etc/motd
```
This is a quick example to demonstrate that it works. See cloud-init for more examples.
Create compute.tf with contents:
```
 resource "oci_core_instance" "<vm-name>" {
     # Required
     availability_domain = data.oci_identity_availability_domains.tutorial_ads.availability_domains[0].name
     compartment_id = var.compartment_ocid
     shape = "VM.Standard.A1.Flex"
     shape_config {
         memory_in_gbs = 2
         ocpus = 1
     }
     source_details {
         source_id = "ocid1.image.oc1.eu-amsterdam-1.aaaaaaaa7o2ilw6qsabd7qgnfjxncygvy442pzxkzcmsogkxeqhtwsgwlnwq"
         source_type = "image"
     }
     #~ shape = "VM.Standard.E2.1.Micro"
     #~ source_details {
         #~ source_id = "ocid1.image.oc1.eu-amsterdam-1.aaaaaaaa57ipifc7jj3m7nskxw66czipcrf4hpehsbx473uauvxot2im67dq"
         #~ source_type = "image"
     #~ }

     # Optional
     display_name = "<vm-name>"
     create_vnic_details {
         assign_public_ip = true
         subnet_id = oci_core_subnet.vcn-public-subnet.id
     }
     metadata = {
         ssh_authorized_keys = file("<you-sh-key-name>,pub")
         user_data = "${base64encode(file("cloud-init.yaml"))}"

     } 
     preserve_boot_volume = false
 }
```
- availability_domain is configured from availability-domains.tf which retrieves the list of availability domains and selects the first one.
- In this example, we are setting vm-name as resource-id and also as display_name. This is not necessary, but it makes things simpler this way. The resource-id is used internally by OpenTofu to refer created resources, while the display_name is shown in the Oracle web console.
- source_id can be looked up from the documentation. Simply find the image you want, and locate the region to use, and get the ID from there.
- shape and shape_config are used to configure the VM, in the Oracle Free tier, you can use: VM.Standard.A1.Flex or VM.Standard.E2.1.Micro.
  The Flex shape requires further configuration with shape_config which requires memory_in_gbs to configure memory and ocpus to configure CPU count.
- metadata.ssh_authorized_keys : Configure authorized ssh keys. The default user for the Oracle Ubuntu images is ubuntu.
- metadata.user_data : User data for initialization. This must be a base64 encoded script. NOTE that because it is base64 encoded under Microsoft Windows, scripts may not be recognized properly as including a file (as in the example) will contain Windows style line terminations which will make the #cloud-config test fail. The following cloud-init formats are supported:
  - #cloud-config Cloud Config Data
  - #! User-Data Script (e.g. #!/bin/bash)
  - #include Include file
  - #cloud-boothook Cloud boothook

Run Scripts

Initialize
```
tf init
```
This initializes a working directory. Specifically it would download from the Terraform repository any providers and/or modules.
Create an execution plan
```
tf plan
```
use this to preview what will OpenTofu will eventually execute.
Run your terraform scripts:
```
tf apply
```
This will execute changes to the infrastructure.

Destroying Infrastructure

Run:

 tf destroy

This will destroy any created resources.

The given example creates a ARM server. At the time of this writing, the Oracle Free tier does not have any Arm servers available. Running these scripts will generate error:

Error: 500-InternalError, Out of host capacity.

Unfortunately, there is not much that can be done here. You try running things later hopping that some capacity may be freed-up.

Inetd like service with systemd

2024-11-22T00:00:00+01:00

This is an example of a socket-activated per-connection service (which is usually referred to as inetd-like service).

A thorough explanation can be found at 0pointer.de.

Define a socket unit

The key point here is to specify Accept=yes, which will make the socket accept connections (behaving like inetd) and pass only the resulting connection socket to the service handler.

Create /etc/systemd/system/baz.socket:

[Unit]
Description=Baz Socket

[Socket]
ListenStream=127.0.0.1:9999
Accept=yes

[Install]
WantedBy=sockets.target

Define a service template unit

We now create a service template from which the actual service will be instantiated on-demand. The lifetime of this service is usually very short (thus it may not appear at systemctl).

Create /etc/systemd/system/baz@.service:

[Unit]
Description=Baz Service
Requires=baz.socket

[Service]
Type=simple
ExecStart=/usr/bin/python /opt/baz-service/serve.py %i
StandardInput=socket
StandardError=journal
TimeoutStopSec=5
#RuntimeMaxSec=10

[Install]
WantedBy=multi-user.target

An example service handler would be (located at /opt/baz-service/serve.py as specified at the service unit file):

#!/usr/bin/python
import sys
import logging
logging.basicConfig(level=logging.INFO)

instance = sys.argv[1]

# The connected socket is duplicated to stdin/stdout
data = sys.stdin.readline().strip()
logging.info('baz-service: at instance %s, got request: %s', instance, data)
sys.stdout.write(data.upper() + '\r\n')

Test

Start (or enable it to start at boot time) the socket (verify it via systemctl status baz.socket), and make a request to the service:

echo Hello| nc localhost 9999

You 'll notice that the socket's status has changed (i.e. Accepted: 1). Also, any log message generated by the on-demand service should be present at the journal (e.g. journalctl --all| grep -e baz).

https://gist.github.com/drmalex07/28de61c95b8ba7e5017c

Locking down SFTP

2024-11-22T00:00:00+01:00

This is a small recipe to increase the security around a SFTP interface.

In the /etc/ssh/sshd_config file include the following settings:

Subsystem sftp internal-sftp

This configures the sftp subsystem to use the internal sftp implementation. This is because inside the chroot, we usually will not have the normal sftp-server executable.

For each user that will be doing sftp do:

Match User sftp-only-user-name
  ChrootDirectory /only/path
  ForceCommand internal-sftp
  X11Forwarding no
  AllowTcpForwarding no
  PermitTTY no

Alternative you could do Match Group and have multiple sftp-only users in the specified group.

The options are:

ChrootDirectory /only/path : Note that this directory must have mode 0755 and be owned by root. If this is not the case, logins will fail with error:
bad ownership or modes for chroot directory \
ForceCommand internal-sftp : Only allow sftp. No other command will be allowed.
X11Forwarding, AllowTcpForwarding, PermitTTY as no : These make sure that the remote user doesn't try to open holes at the SSH protocol levels.

References:

Sftp icons created by Freepik - Flaticon

Python GUI

2024-11-22T00:00:00+01:00

After looking a multiple options of GUI programming under python I eventually settled for tkinter. The main reason was that tkinter is very ubiquitous and initially though the learning curve wuld have shorter as I was very used to GUI programming using TCL/TK. Turned out that what I known TCL/TK did not translate very well to tkinter in python.

Also I found out some BASIC features that I was used to in TCL/TK were not available in python. For example:

Implementing optional scrollbars
Scrollable frames

At some time I considered using kivy but at the end, I did not. Since the main advantage for it is that you can create mobile apps. But since my primary phone is an iPhone, I don't think I would be able to create iPhone apps due to Apple's walled garden restrictions.

So try things out, I wrote a couiple of scripts:

patoggle
This one is not that interesting as it only shows things on the screen but does not have any inputs. But I though was a good starting project.
xprtmgr
This is a more complete application. It was a good learning experience.

I assume that as I get more experience, things should be easier.

cisco bridging

2024-11-22T00:00:00+01:00

This article is here as a reminder.

So, for testing, I needed to configure a Cisco CSR1000V virtual router as a bridge. So I used a version 16 Cisco IOS XE image. To make my life easier I used the "wizard" that runs the first time to automatically configure bridgning. Ironically, this created an invalid configuration.

Over the years, cisco has transitioned through multiple ways to configure bridging, searching the Internet, it was not clear to me how to configure bridging. Eventually I manage to configure using bridge domains. The configuration is as follows:

Configure spanning tree features

These are cisco global settings. For my test I was using the following:

spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree portfast bpduguard default
spanning-tree extend system-id

Some of these setting are ON by default, so in some cases you don't need to.

Configure bridge domains

bridge-domain 1
bridge-domain 200
bridge-domain 201

Actually, these are not needed as they are automatically created when configuring the bridge-domain interfaces. However, these would show on the running-config.

Configure bridge members

For network interface, you need:

interface GigabitEthernet1
 no ip address
 service instance 1 ethernet
  encapsulation dot1q 1
  bridge-domain 1
 !
 service instance 200 ethernet
  encapsulation dot1q 200
  bridge-domain 200
 !
 service instance 201 ethernet
  encapsulation dot1q 201
  bridge-domain 201
 !

The interface line for the given port that is part of the switch.
no ip address : We are doing Layer-2, so no IP is needed.
For each VLAN that we are bridging we need:
- service instance ID ethernet
- encapsupation dot1q VLAN_ID
- bridge-domain ID
Note that I made the VLAN_ID the same as the instance ID and the bridge-domain ID. This is not necessary but makes things less confusing.
encapsulation is used for VLAN tagging. It is possible to use encapsulation untagged. However, Spanning Tree protocol doesn't run on the untagged VLAN.

Tunneling NFS over SSH

2024-11-22T00:00:00+01:00

This recipe is for tunneling NFS traffic over SSH. This adds encryption and Public Key authentication to otherwise insecure NFS traffic.

For this recipe to work, requires NFSv4. Earlier versions were not tested, but I expect not all the functionality to work.

server configuration

Install packages:

nfs-kernel-server
ncat or netcat-openbsd

Configure /etc/exports. Add a line:

/export/path 127.0.0.1(insecure,... other options...)

/export/path: File system to export
127.0.0.1: Loopback address, we only allow local connections.
insecure: Normally, the NFS server only allows connections from ports less than 1024. This option removes that restriction. We need this because the ssh traffic is running as a normal user.

Additional NFS export options:

rw : Allow read/write access
sync : sync I/O (recommended to prevent data loss)
no_subtree_check : When exporting full filesystem, this remove the subtree checks. This has to do with the fact that NFS uses inodes. This check is needed to make sure that the inode is within the exported filesystem sub-tree. However if you are exporting the entire filesystem, there should never be the case that an inode falls outside the subtree.
no_root_squash : allow root access
mountpoint=/mount/path : Only export if a filesystem is mounted on /mount/path

Obtain a public/private key. Either use ssh-keygen or copy it from elsewhere. Install authorized_keys on the account to be used for SSH/TCP forwarding:

command="nc -N  localhost 2049",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa ...

This account does not need to be root. The additional settings make sure that this key can only be used for forwarding traffic.

client configuration

This is for Ubuntu. Other distros may need different packages and/or approaches.

Install packages:

nfs-common
ncat (netcat-openbsd is not enough, ncat needs to support -e or -c)

Make sure the NFS server is in the SSH forwarder's known_hosts file. Use this command to make sure this happens:

ssh -n -i $ssh_key -T \
        -o StrictHostKeyChecking=accept-new \
        -o BatchMode=yes \
        -o ConnectTimeout=10 \
        $nfs_server

In this approach we are using ncat to implement SSH forwarder. Run this command from the /etc/rc.local file:

lport=4096
ssh_key=/path/to/ssh/private/key
ssh_opts="-o BatchMode=yes -o ConnectTimeout=10 -a -C"
nfs_srv=nfs-server

( ncat -l $lport -k --allow localhost -c "exec ssh -i $ssh_key $ssh_opts $nfs_srv" ) &

Options:

-a : disable agent forwarding
-C : request compression
-T : disable pty

An alternative to this is to use inetd.conf or use systemd.

At this point we are ready to mount NFS filesystems:

mount -t nfs \
  -o nfsvers=4,nolock,nosuid,nodev,port=4096,sec=sys,tcp,soft,intr,fg \
  localhost:/export/nfs/path \
  /mnt

Options:

nfsvers=4 : make sure we are running NFSv4
nosuid : disable SUID executables
nodev : disable device files
sec=sys : traditional UNIX security modes
tcp : use TCP protocol
soft,intr : how we handle CTRL+C and other errors

Caveats

showmount command does not show NFSv4 information.
I have not tried to use this with autofs. autofs configure for NFS shares is in /etc/autofs/auto.net, but it uses showmount command, so it probably would not work out of the box.

Notes

Some implementations of the nc command supports a -p source_port option. This would remove the need to use the insecure option from the nfs export options. However this requires netcat to run as root.

References:

Lock icons created by Freepik - Flaticon

Optimizing shell scripts

2024-03-05T00:00:00+01:00

Introduction
Input Data
Desired Output
Approach
Original Script
Optimized Script
Conclusion

Introduction

I consider myself a fairly competent shell scripter. I typically prefer to program towards readabilty, but at the end tend to write towards terse code.

Usually, readability is important because other people (including myself in the future) will need to read the code and figure out what is going on.

I think because I used to write perl for a long time, I tend also to write fairly terse code. This is not something to be proud of.

The other day, I was writing a small shell script to generate a menu of links in markdown from a simple specially written text file.

Input Data

See input:

Desired Output

See output:

Approach

I initially wrote it on my PC with a comparatively faster CPU. The response time on my PC was about one second. When I transferred the script to my Web server (with an small CPU), the reponse time was 15 seconds to generate the menu. This was surprising to me, as I did not expect the performance difference between my PC and my Web server to be so massive.

I figured that this need to be optimized. The simplest way to do this is to add caching. Which I did very quickly. In reality, I needed to improve the code.

The first thing you need to do when optimizing code is to measure the effects of changes on the code. In order to do that, I created a simple test harness:

The options to the harness are:

-1 or --no-count : run the code once. Essentially to check that the output is correct.
--count=int : Will run the code the specified number of times
1 or 2 : Run implementation 1 or implementation 2.

Run this with the time built-in to measure running times.

So running:

sh th.cgi -1 1

sh th.cgi -1 2

Can be used to make sure that implementation 1 and 2 are correct.

Then, you can run:

time sh th.cgi --count=100 1

time sh th.cgi --count=100 2

This will show the how fast/slow the code peforms. For testing, I would also test with busybox sh, as that is the shell that I have in my web server. On my PC, sh is bash

So the general approach is to measure the original performance, then make some changes and measure if the performance improves (or not).

Original Script

The original un-optimized code is:

Running this on my PC executes in almost 10 minutes.

Optimized Script

The optimized version is:

This version executes in 16 seconds.

So overall is a pretty good performance enhancement. So the approach I took is first to isolate what part of the code is the one that executes the slowest. This script essentially runs in a loop and runs on two halves, the first half is scanning lines, the second half is outputing markdown.

Commenting out the different parts of the code, I was able to determine that most of the time was spent in the scanning half.

The next step was to review the code and re-factor it to faster versions:

Moving invariant code out of loops

Move invariable code outside the loop. Assigning elem2o and elem2c was originally done in the loop. This was moved outside the loop. This did not yield much improvement, but in general this is always a good optimizaiton.

Replacing if/then/else with case

I was doing:

if (echo "$ARGS" | grep -q '>') ; then
  ...
else
  ...
fi

This actually forks and execs two additional commands and a subshell. This was replaced with:

case "$ARGS" in
*\>*)
  ...
  ;;
*)
  ...
  ;;
esac

Since this runs entirely inside the shell, this removes spawning commands and forking subshells.

Using IFS for parsing

I replaced:

local i=1
while [ -n "$(echo "$ARGS" | cut -d'>' -f$i-)" ] ; do
  set - "$@" "$(echo "$ARGS" | cut -d'>' -f$i | xargs)"
  i=$(expr $i + 1)
done

Which is a terrible way to parse a line. This was replaced with:

Adding -e 's/[ ]*>[ ]*/>/g' to the sed command at the top in front of the loop, and then...

IFS=">" ; set - $ARGS; IFS="$oIFS"

The performance difference here is massive.

Conclusion

So there you have it. Turns out that my skills at scripting are poor. On the other hand it could be that I am so used writing shell scripts that I am not using the best tools for the job. In this particular example, I shouldn't have started with shell script but use something more suitable for text manipulation, such as perl or awk. However, most scripting languages come with useful string processing capabilities and would have done the job well.

Happy New Year 2024

2024-11-22T00:00:00+01:00

Best wishes for 2024!

IPv6 on 2023

2024-11-22T00:00:00+01:00

This is a sequel to my article IPv6 blues.

Layout
Enabling forwarding
Configure networking
Prefix delegation
Router advertisement
Conclusion

At the time it looked that only /64 prefix address was being allocated. However, when I recently checked my ADSL modem router configuration I noticed that actually the ADSL modem gets assigned /48 prefix. Thids makes the configuration much easier.

What is nice, is that for "reasonable" configurations, IPv6 usually does the right thing and configures most things by itself. For IPv6 routing, you only need to enable the right functionality and most of the addressing is determined automatically.

This article expands on this Alpine Linux Wiki article.

Layout

This is a very simple configuration.

The current Alpine Linux kernel (v3.17.3) has IPv6 enabled by default, so nothing special needs to be done for that.

Enabling forwarding

By deault, IP (v4 or v6) forward is disabled on a Linux kernel. To enable, you need to modify syctl.conf. Create a file:

/etc/sysctl.d/router.conf

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# http://vk5tu.livejournal.com/37206.html
# What's this special value "2"? Originally the value was "1", but this 
# disabled autoconfiguration on all interfaces. That is, you couldn't appear 
# to be a router on some interfaces and appear to be a host on other 
# interfaces. But that's exactly the mental model of a ADSL router. 

# Controls IP packet forwarding
net.ipv6.conf.all.forwarding = 2
net.ipv6.conf.default.forwarding = 2

# Accept Router Advertisments
net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.default.accept_ra = 2

# We are a router so disable temporary addresses
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.use_tempaddr = 0

Configure networking

Configure IPv6 address for eth1. We don't need to configure eth0 as that will be done by dhcpcd through the ISP's router advertisements:

# Conect to ISP
auto eth0
iface eth0 inet static
  address 192.168.2.250
  netmask 255.255.255.0
  broadcast 192.168.2.255

# Connected to local LAN
auto eth1
iface eth1 inet static
  address 192.168.3.1
  netmask 255.255.255.0
  broadcast 192.168.3.255

iface eth1 inet6 static
  address fde4:8dba:82e1:fff4::1
  netmask 64
  autoconf 0
  accept_ra 0
  privext 0

Prefix delegation

The next step will be to configure DHCPv6 Prefix Delegation with your ISP. Install dhcpcd.

apk add dhcpcd

Configure it:

/etc/dhcpcd.conf


# Enable extra debugging
#debug
#logfile /var/log/dhcpcd.log

# Allow users of this group to interact with dhcpcd via the control
# socket.
#controlgroup wheel

# Inform the DHCP server of our hostname for DDNS.
hostname gateway

# Use the hardware address of the interface for the Client ID.
#clientid
# or
# Use the same DUID + IAID as set in DHCPv6 for DHCPv4 ClientID as
# per RFC4361. Some non-RFC compliant DHCP servers do not reply with
# this set. In this case, comment out duid and enable clientid above.
duid

# Persist interface configuration when dhcpcd exits.
persistent

# Rapid commit support.
# Safe to enable by default because it requires the equivalent option
# set on the server to actually work.
option rapid_commit

# A list of options to request from the DHCP server.
option domain_name_servers, domain_name, domain_search, host_name
option classless_static_routes

# Most distributions have NTP support.
option ntp_servers

# Respect the network MTU.
# Some interface drivers reset when changing the MTU so disabled by
# default.
#option interface_mtu

# A ServerID is required by RFC2131.
require dhcp_server_identifier

# Generate Stable Private IPv6 Addresses instead of hardware based
# ones
slaac private

# A hook script is provided to lookup the hostname if not set by the
# DHCP server, but it should not be run by default.
nohook lookup-hostname

# IPv6 Only
ipv6only

# Disable solicitations on all interfaces
noipv6rs

# Wait for IP before forking to background
waitip 6

# Don't touch DNS
nohook resolv.conf

# Use the interface connected to WAN
interface eth0
    ipv6rs # enable routing solicitation get the default IPv6 route
    iaid 1
    ia_pd 1/::/56 eth1/2/64

Add dhcpcd to the default run level:

rc-update add dhcpcd default

Router advertisement

Now we need to configure radvd to give router advertisements to our internal network for addressing and routing.

apk add radvd

Once radvd is installed, you may configure it:

/etc/radvd.conf


interface eth0 {

  # We are sending advertisements (route)
  AdvSendAdvert on;

  # When set, host use the administered (stateful) protocol
  # for address autoconfiguration. The use of this flag is
  # described in RFC 4862
  AdvManagedFlag on;

  # When set, host use the administered (stateful) protocol
  # for address autoconfiguration. For other (non-address)
  # information.
  # The use of this flag is described in RFC 4862
  AdvOtherConfigFlag on;

  # Suggested Maximum Transmission setting for using the
  # Hurricane Electric Tunnel Broker.
  # AdvLinkMTU 1480;

  # We have native Dual Stack IPv6 so we can use the regular MTU
  # http://blogs.cisco.com/enterprise/ipv6-mtu-gotchas-and-other-icmp-issues
  AdvLinkMTU 1500;

  prefix ::/64 {
    AdvOnLink on;
    AdvAutonomous on; ## SLAAC based on EUI
    AdvRouterAddr on;
  };
};

interface eth1 {

  AdvSendAdvert on;
  AdvManagedFlag on;
  AdvOtherConfigFlag on;
  AdvLinkMTU 1500;

  # Helps the route not get lost when on WiFi with packet loss
  MaxRtrAdvInterval 30;
  AdvDefaultLifetime 9000;

  prefix fde4:8dba:82e1:fff3::/64 {
    AdvOnLink on;
    AdvAutonomous on; ## SLAAC based on EUI
  };
};

Add radvd to the default run level:

rc-update add radvd default

Conclusion

At this point you should have a working IPv6 set-up. Things that you may want to add:

Firewall rules
Additional static routes and subnets
DHCP daemon configuration to have more control on IP address assignments
OpenVPN

libnss-db HOWTO

2023-09-05T00:00:00+02:00

This mini-howto illustrate how to use libnss-db on a Ubuntu Linux system.

Other installations should work to after adjusting package names and directory paths.

I myself use as a "serverless" lightweight user directory. Essentially, I mount the db directory and the home directory from an NFS server.

Package installation

Install the following packages:

apt install -y libnss-db make

This creates a directory /var/lib/misc in Ubuntu. Other distributions this may be in /var/db.

Preparing nss-db data directory

I like to keep a separate set of users in the db files. For that, create a directory /var/lib/misc/etc. This will contain the additional users and groups.

for f in passwd group shadow
do
  cp -av /etc/$f /var/lib/misc/etc/$f
  >  /var/lib/misc/etc/$f
done

This creates, passwd, group, and shadow files. NOTE: gshadow is unsupported. This is only relevant if you are using passwords to control user group changes.

Because in this scenario, we are using flat files (i.e. /etc/passwd vs. /var/lib/misc/passwd.db) and db files, we want to not have uid/gid overlaps.

Copy /etc/login.defs to /var/lib/misc/etc/login.defs and change UID_MIN,UID_MAX,GID_MIN,GID_MAX to a different space (so as not to overlap with the flatfiles spaces).

When users are created with the useradd command, you can pass the --prefix /var/lib/misc argument, so then it would create users in the /var/lib/misc/etc directory (ignoring /etc) and would get defaults from /var/lib/misc/etc/login.defs (instead of /etc/login.defs).

For /home directories to be created properly I create the symlink:

ln -s /home /var/lib/misc/home

I also like to add sudoers configuration to the /var/lib/misc directory (so it can be shared via NFS)

cp -av /etc/sudoers.d $dbdir

Moving nss-db data

If you are storing nss-db in a different location, you can use a mount --bind to make it available in /var/lib/misc. This can be configured on /etc/fstab as follows:

# /etc/fstab
/mount/point/dir /var/lib/misc none defaults,bind 0 0

Configuring nss-db

In /var/lib/misc there is a Makefile that is used to create the relevant db files. To control how this you can configure things in /etc/default/libnss-db:

# /etc/default/libnss-db
# settings for libnss-db

# Directory where the databases are kept
VAR_DB = /var/lib/misc
# Location of files
ETC = $(VAR_DB)/etc

# Databases to generate
DBS = passwd group shadow

# Programs used
AWK = awk
MAKEDB = makedb --quiet

You must also add the db setting to the /etc/nsswitch.conf lines for passwd, group and shadow.

# Configure nsswitch.conf
sed -i~ \
        -e 's/^\(passwd:[ \t]*\).*$/\1files systemd db/' \
        -e 's/^\(group:[ \t]*\).*$/\1files systemd db/' \
        -e 's/^\(shadow:[ \t]*\).*$/\1files db/' \
        /etc/nsswitch.conf

This is not strictly part of nss-db, but I like to add to /etc/sudoers the line:

@includedir /var/lib/misc/sudoers.d

From then on, adding, modifying and removing users/groups should be done on the files in /var/lib/misc/etc. Afterwards, use make in /var/lib/misc to recrate db files.

For convenience I created a script in /usr/local/bin named nssdb:

#!/bin/sh
#
# NSSDB command
#
nssdb_dir=/var/lib/misc
nssdb_opts="--prefix $nssdb_dir"
extra_group_add_opts="--key GID_MIN=13000 --key GID_MAX=13999"

if [ $# -eq 0 ] ; then
  cat <<-_EOF_
        Usage: $0 useradd|userdel|usermod|groupadd|groupdel|groupmod [options]
        _EOF_
  exit 1
fi

case "$1" in
  useradd|userdel|usermod|groupdel|groupmod)
    op="$1" ; shift
    ;;
  groupadd)
    op="$1" ; shift
    nssdb_opts="$nssdb_opts $extra_group_add_opts"
    ;;
  *) echo "$1: Unknown sub-command" ; exit 1
esac

"$op" $nssdb_opts "$@" && ( cd $nssdb_dir && make )

What it does is that you can run:

nssdb useradd
nssdb usermod
nssdb userdel
nssdb groupadd
nssdb groupmod
nssdb groupdel

This will run the specified commands but with --prefix option so these will modify files in /var/lib/misc and the Makefile called accordingly.

Note the following commands can not be supported:

chfn: does not support --prefix or --root options.
chsh, passwd, newusers : only support --root which requires root priviledges.

Alpine Linux Custom Interface names

2023-09-29T00:00:00+02:00

This article is a copy of this article and shows how to rename/change name of a network interface.

Alpine Linux uses busybox mdev to manage devices in /dev. mdev reads /etc/mdev.conf and according to mdev documentation one can define a command to be executed per device definition. The command which is going to be used to change network interface name is nameif.

`/etc/mdev.conf` configuration

-SUBSYSTEM=net;DEVPATH=.*/net/.*;.*     root:root 600 @/sbin/nameif -s

Here we tell mdev to call nameif for devices found in /sys/class/net/.

# ls -d -C -1 /sys/class/net/eth*
/sys/class/net/eth1
/sys/class/net/eth2
/sys/class/net/eth3
/sys/class/net/eth4
/sys/class/net/eth5

`nameif` configuration

nameif itself reads /etc/mactab' by default. Example line for a network interface with following hwaddr

# cat /sys/class/net/eth0/address
90:e2:ba:04:28:c0

would be

# grep 90:e2:ba:04:28:c0 /etc/mactab 
dmz0 90:e2:ba:04:28:c0

finalization

To use renamed network interface without reboot, just call nameif while the network interface is down.

# nameif -s

And finally reboot...

Alpine boot menu

2023-06-03T00:00:00+02:00

This article is an update to my Alpine Boot Switcher article.

Contents:

Preparing boot device
Booting the system
Adding a new kernel
After a succesful reboot
Alternative workflow
Removing installed kernels
Downloads

The weakness of that approach was that you needed to be on a running system to select the active kernel. So, if you switched to a broken kernel then you wouldn't be able to revert back without having to get the media device out and modifying the file system to switch to a different kernel.

The approach described here makes use of syslinux or grub menu system to allow you to select a new kernel. It requires modifying the init script in the initramfs image so that you can specifically select the right APK boot repository otherwise it will use the first one found in the boot file system.

This solution is suitable for physical and virtualized systems using BIOS and UEFI boot methods.

The high level approach is:

prepare a bootable USB or virtual disk image.
during system operation, use the inst_iso script to install a new Alpine release.
You can use the mkmenu script to select the new kernel or ...
Re-boot the system, syslinux (on BIOS systems) or grub (on UEFI) systems will show a menu to let you select the boot kernel or boot a default kernel after 10 seconds (unless configured for a different time-out).
If the boot was succesful, you can use mkmenu to make the current kernel the default kernel.
If the boot fails, then you can hard boot the system and use the boot menu to select a known working kernel.

Preparing boot device

Use the mkuub script to create a bootable USB drive or a bootable disk image.

The USB drive can be used on a physical system and supports UEFI and BIOS boot methods (BIOS boot is untested).

The bootable disk image is meant to be used on a virtualized environment. I have tested it on virsh on kvm and uses the BIOS boot method. It has UEFI boot support files but I have not tested this.

Usage:

    ./mkuusb.sh [options] isofile [usbdev]

Options:

--serial : Enable serial console
--ovl=ovlfile : overlay file to use
--boot-label=label : boot partition label
Defaults to a random label unless ovl is specifed. In that case, it will take the label from the filesystem mounted as /media/boot from /etc/fstab.
--boot-size=size : boot partition size
If not specified it will default to the entire drive or up to half the drive (if data partition is enabled) up to 8GB.
--data : create a data partition
--data-label=label : label for data partition_disc
Defaults to a random label unless ovl is specifed then will take the label from the filesystem mounted as /media/data from /etc/fstab.
--data-size=size : data partition size
size defaults to the remaining of the disk
isofile: ISO file to use as the base alpine install
usbhdd : /dev/path to the thumb drive that will be installed.
It will try to use a suitable default by looking an unused drive from the currently connected drives on the system. Otherwise a target image file can be specified using:
- img:path/to/image/file[,size]

if --serial was used, the boot menu will be configure to use the first serial port (Usually COM1) with a speed of 115200. So you can use a serial console or the normal display to select the desired kernel.

When configured using serial, kernel boot message would be displayed on the serial console.

Booting the system

After creating the boot device, you can connected to a physical server or in the case of a bootable image add it to a virtual machine configuration.

Booting the system will show a boot menu letting you select a kernel. If nothing is selected after the time-out period, a configured default will be used.

Adding a new kernel

During normal operation, you may want to upgrade to a new kernel. To do this you can use the script:

$bootmedia/scripts/inst_iso.sh file-or-url

You can pass a iso file name or the URL of an ISO over the network. If using an URL, the script will download the image first.

This will prepare the environment to include the new kernel and add a menu entry to boot the new kernel. The default kernel remains as the currently running kernel.

At this point, the system can be rebooted. You can then use the boot menu to select the latest kernel manually.

If the new kernel fails to boot succesfully, then you may need to hard boot the system.

Since the default kernel was not changed, it should boot to the last working kernel and proceed normally.

After a succesful reboot

If the new kernel was succesfully booted, you can use the command:

$bootmedia/mkmenu.sh

to make the currently running kernel the default.

Alternative workflow

Alternatively, after the new kernel is installed, you can use the command:

$bootmedia/mkmenu.sh --latest

to make the new kernel the default one. You can then re-boot the system and if everything goes fine, you are done.

If the system fails to boot then you probably will need to do a hard boot of the system and use the boot menu to select a working Alpine Linux kernel.

After a succesful boot, then you can use the command:

$bootmedia/mkmenu.sh

To make the current running kernel the default.

Removing installed kernels

Alpine Linux boot images are installed in their own folders. If you want to remove a kernel, just remove the folder that you want to un-install and run mkemnu.sh again to update the boot menu.

Downloads

The code can be found in my repository:

github

Alpine Linux boot images:

Alpine Linux ISO downloads

Adding static routes in alpine linux

2023-06-03T00:00:00+02:00

There are several ways to do this documented in the alpine linux wiki.

My preferred way is to configure it in /etc/network/interfaces.

For example:

auto eth0
iface eth0 inet static
       address 192.168.0.1
       netmask 255.255.255.0
       up ip route add 10.14.0.0/16 via 192.168.0.2
       up ip route add 192.168.100.0/23 via 192.168.0.3

Note that you can actually add these lines in a dhcp stanza.

The benefit of doing this is that those routes are added when that interface is brought up.

Also, they are kicked off by the networking init.d file.

New NacoWiki

2023-10-01T00:00:00+02:00

Release new version (3.2.1) of NacoWiki.

The following changes are included:

Added document properties
Added opts.yaml
API improvements
Bug fixes and UI improvements
Additional Plugins:
- Versions
- AutoTag
- Albatros : Blog site generator (similar to Pelican).

Most of the changes are to support the new plugins.

Versions

Now, NacoWiki keeps track of previous versions of articles. This can be disabled on a per directory basis by creating an opts.yaml and adding the disable-props: true entry.

AutoTag

When an article is save it will create tags automatically based on a tagcloud.md file containing a list of tagging words. This is enabled only if there is a tagcloud.md file in the sub-directory or any parent sub-directory.

Albatros

This is a Blog site generator inspired by Pelican. It was mainly written to migrate this web site from Pelican which uses a Python based markdown implementation to the same markdown implementation used in NacoWiki. The reason being that I use NacoWiki to edit this Blog, so it makes sense to use the same code to preview and generate the static web site.

It uses a slightly modified version of the template I was using for this website, so the change should be transparent to most peope.

Migrated to NacoWiki Albatros

2023-09-29T00:00:00+02:00

This edition marks the migration of my Blog from Pelican to NacoWiki Albatros.

This is a mostly transparent change. I wrote Albatros specifically to migrate this web site from Pelican to a php based markdown implementation. As such, it uses a slightly modified version of the Pelican theme.

The main changes from the previous site are:

Re-arranged file structure
Built-in search functionality. This no longer depends on duck-duck-go.
Uses the same renderer as NacoWiki.

File system encryption in Alpine Linux

2024-11-07T00:00:00+01:00

This is similar to my previous article Encrypting Filesystem in Void Linux but for Alpine Linux

The point of this recipe is to create a encrypted file sytem so that when the disc is disposed, it does not need to be securely erased. This is particularly important for SSD devices since because of block remapping (for wear levelling) data can't be overwritten consistently.

The idea is that the boot/root filesystem containing the encryption keys are stored in a different device as the encrypted file system.

Create block devices

apk add lvm2 cryptsetup

dd if=/dev/urandom of=/etc/crypto_keyfile.bin bs=1024 count=4
chmod 000 /etc/crypto_keyfile.bin
cryptsetup luksFormat /dev/xda2 /etc/crypto_keyfile.bin
cryptsetup luksOpen --key-file=/etc/crypto_keyfile.bin /dev/xda2 crypt-pool

Look up the UUID

blkid /dev/xda2

Edit dmcrypt service configuration in /etc/conf.d/dmcrypt:

target=crypt-pool
source="UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
key=/etc/crypto_keyfile.bin

And enable it with this command:

rc-update add dmcrypt boot

At this point it would be good to backup:

/etc/conf.d/dmcrypt
/etc/crypto_keyfile.bin

Reboot and make sure that the block device gets created on start-up.

Add to LVM:

vgcreate pool /dev/mapper/crypt-pool
lvcreate --name home0 -L 20G pool

Create your file-system and add it to /etc/fstab.

Alpine Linux uses the same encryption infrastructure as gentoo, so the gentoo instructions apply here.

CSS Tips

2023-05-18T00:00:00+02:00

Style editor
Sending query string with PHP
Position sticky
Flexbox and Grid

Personally, I dislike CSS. I find it difficult to use because of the lacking documentation and the fact that different Browsers implement things subtly different. Unfortunately, it is something that one has to use today in order to create funtioning web sites, and things are getting better all the time.

Some things that I have learned over time:

Style editor

A feature of Firefox, let's me edit and try CSS styles on the fly. I find it very conventient.

To access it:

Open the Firefox menu
More tools
Developer tools
Click on the Style Editor tab.

In addition, Firefox comes with a "Responsive Design" mode that let's you simulate the form factor of different devices. Granted, you can do this by just resizing the window but this is very convenient.

Sending query string with PHP

Normally, web browsers would cache JS and CSS files for performance. For normal usage, this is the best approach. For developing, this is a bit inconveneint as you may be debugging problems that may have been fixed, but the web browser is still using buggy versions of resource files.

To get around this, you can code your URLs to include a query string so that you can change the query string every time you make changes to resources.

In PHP, what I do, is add to the resource URL, a query string containing the modification time of the file. That way, when I update a resource file, it the query string changes automatically, so the cache expires and renders the latest version.

Position sticky

Very often you would like to have a header part of the page to be always displayed. I create this effect with the following CSS:

css-selector {
  position: sticky;
  top: 0;
  z-index: 100;

The important setting is the posistion: sticky which makes that element stick to the window. The top: 0 positions the element to the top of the page. Finally, the z-index: 100 makes it so that it will be rendered above other elements rather than being obscured by them.

Flexbox and Grid

These are layout managers that can be used for arranging the page. For navigation bars, I prefer to use flexbox. Whereas for table like layouts, Grid is usually a good option.

You should consider using grid layout when:

You have a complex design to work with and want maintainable web pages
You want to add gaps over the block elements

You should consider using flexbox when:

You have a small design to work with a few rows and columns
You need to align the element
You don’t know how your content will look on the page, and you want everything to fit in.

Local Startup

2024-02-02T00:00:00+01:00

This is a method to control start-up of applications in a Linux Desktop session that are run by a local default configuration, but can also be overriden by the user.

This is unlike the /etc/xdg/autostart which is mostly under the control of the distro packager.

Aslo unlike the /etc/X11/profile.d directory, this runs inside the Desktop Session. /etc/X11/profile.d gets started before the Desktop session is available.

There is a configuration /etc/xdg/local-startup.cfg, which contains the local configuration. It is a text file:

# comments start with #
#
delay 3000 # Number of milliseconds to wait before starting applications
run-cmd /etc/xdg/local-startup.run # Run script
application1.desktop  enable # Enable the application
application2.dekstop  disable # disable this application

Applications that can be auto started are defined either in /usr/share/applications or in $HOME/.local/share/applications as .desktop files.

Adding a enabled application global config, will start the application by default.

Adding a disabled application to the global config, will not start the application by default, but it makes it possible for the user to override this setting in their $HOME/.config/local-startup.cfg file.

Files:

local-startup.tk : main implementation script
local.cfg : example configuration file
local-startup-autostart.desktop : desktop file to run when session starts
local-startup-prefs.desktop : desktop file for preferences editor
local-startup.run : example run file.

local-startup.run

This is an arbitrary script that can be run on boot-up after the desktop environment is loaded. It is meant to run some scripts that can be used to tweak the User Experiance. Currently it does:

Closes the pidgin window (that always open regardless)
Monitors windows that have the size of a maximized window but are not maximized. These windows still have window re-size grabbers which I personally find annoying.

Pulse Audio control in python

2024-02-02T00:00:00+01:00

I have been using a shell script to toggle pulse audio sinks for some time. It worked well enough for switching output among several profiles on a single audio card. I recently upgraded my set-up to new hardware. This hardware for some reason, reported the analog stereo output and the digital HDMI output as different sound cards. So my patoggle script did not work well enough anymore.

Since parsing the output of the pacmd in shell script was becoming a pain, I decided to re-write the toggling script in python. The new script is here.

This script depends of two packages:

pulsectl : used to control pulse audio
tkinter : used to show info on screen

It can be used to control volume and switch audio output.

Global hotkeys

2024-02-02T00:00:00+01:00

To make it easier to switch desktop environment I am using a Desktop Environment independant hot keys configuration using xbindkeys. This lets me use the same keybindings on different Window managers and Desktop Environments.

This code can be found in github.

Included are the follwoing:

hk_helper : a bash or a tcl/tk implementations. The latest version is based on tcl/tk.
xbindkeysrc : hotkey configuration file
xbindkeys.desktop : /etc/xdg/startup file.
obsolete profile.sh : /etc/X11/profiles.d file (not used).

The script starts from /etc/xdg/startup, so as to make sure to let the desktop environment work and grab as many keys as possible.

Afterwards, if there is a $HOME/.xbindkeysrc it will start it first. This is to allow home user keys to work.

Finally it will use the global xbindkeysrc file.

Defined hotkeys can be seen in the xbindkeysrc config file here:

XPrinterMgr

2024-02-02T00:00:00+01:00

This is a small utility to manage a local printer(s) in a home office setting.

It will show the state of the printer:

accepting or rejecting jobs
enable or disabled printing

And if sudo is configured correctly would allow users to:

enable or disable printing
accept or reject new print jobs
print a test page
Show device info. It would try to start hp-toolbox if relevant.

It also let's you manage the local printer job queue. Allowing you to:

show what jobs are in the queue
cancel a job
cancel all job

It is not intended to replace the CUPS WebUI. It is just a simple gui to handle the most common cases for a home office allowing end-users some simple control of printer management functions.

It makes use of:

The code is in github.

rsync filter rules

2023-04-25T00:00:00+02:00

Rules
Simple include/exclude rules
Simple include/exclude example
Filter rules when deleting
Filter rules in depth
Pattern matching rules
Filter rule modifiers
Merge-file filter rules
List-clearing filter rule
Anchoring include/exclude patterns
Per-directory rules and delete

The basic options for filtering files in rsync are:

--exclude=PATTERN
This option pecifies an exclude rule.
--exclude-from=FILE This option is related to the --exclude option, but it specifies a FILE that contains exclude patterns (one per line). Blank lines in the file are ignored, as are whole- line comments that start with ';' or '#' (filename rules that contain those characters are unaffected).
If a line consists of just "!", then the current filter rules are cleared before adding any further rules.
If FILE is '-', the list will be read from standard input.
--include=PATTERN
This option specifies an include rule.
--include-from=FILE This option is related to the --include option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file are ignored, as are whole- line comments that start with ';' or '#' (filename rules that contain those characters are unaffected).
If a line consists of just "!", then the current filter rules are cleared before adding any further rules.
If FILE is '-', the list will be read from standard input.
--cvs-exclude, -C
This is a useful shorthand for excluding a broad range of files that you often don't want to transfer between systems. It uses a similar algorithm to CVS to determine if a file should be ignored.
The exclude list is initialized to exclude the following items (these initial items are marked as perishable:
```
RCS SCCS CVS CVS.adm RCSLOG cvslog.*  tags TAGS
.make.state .nse_depinfo *~ #* .#* ,* _$* *$ *.old
*.bak *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj
*.so *.exe *.Z *.elc *.ln core .svn/ .git/ .hg/ .bzr/
```
then, files listed in a $HOME/.cvsignore are added to the list and any files listed in the CVSIGNORE environment variable (all cvsignore names are delimited by whitespace).
Finally, any file is ignored if it is in the same directory as a .cvsignore file and matches one of the patterns listed therein. Unlike rsync's filter/exclude files, these patterns are split on whitespace.
If you're combining -C with your own --filter rules, you should note that these CVS excludes are appended at the end of your own rules, regardless of where the -C was placed on the command-line. This makes them a lower priority than any rules you specified explicitly. If you want to control where these CVS excludes get inserted into your filter rules, you should omit the -C as a command- line option and use a combination of --filter=:C and --filter=-C (either on your command-line or by putting the ":C" and "-C" rules into a filter file with your other rules). The first option turns on the per-directory scanning for the .cvsignore file. The second option does a one-time import of the CVS excludes mentioned above.

Rsync supports old-style include/exclude rules and new-style filter rules. The older rules are specified using --include and --exclude as well as the --include-from and --exclude-from. These are limited in behavior but they don't require a "-" or "+" prefix. An old-style exclude rule is turned into a "- name" filter rule (with no modifiers) and an old-style include rule is turned into a "+ name" filter rule (with no modifiers).

For more control on included/excluded files you should use these options:

--filter=RULE, -f
This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. This is most useful in combination with a recursive transfer.
You may use as many --filter options on the command line as you like to build up the list of files to exclude. If the filter contains whitespace, be sure to quote it so that the shell gives the rule to rsync as a single argument. The text below also mentions that you can use an underscore to replace the space that separates a rule from its arg.
-F
The -F option is a shorthand for adding two --filter rules to your command. The first time it is used is a shorthand for this rule:
```
--filter='dir-merge /.rsync-filter'
```
This tells rsync to look for per-directory .rsync-filter files that have been sprinkled through the hierarchy and use their rules to filter the files in the transfer. If -F is repeated, it is a shorthand for this rule:
```
--filter='exclude .rsync-filter'
```
This filters out the .rsync-filter files themselves from the transfer.

Rules

The filter rules allow for custom control of several aspects of how files are handled:

Control which files the sending side puts into the file list that describes the transfer hierarchy
Control which files the receiving side protects from deletion when the file is not in the sender's file list
Control which extended attribute names are skipped when copying xattrs

The rules are either directly specified via option arguments or they can be read in from one or more files. The filter-rule files can even be a part of the hierarchy of files being copied, affecting different parts of the tree in different ways.

Simple include/exclude rules

We will first cover the basics of how include & exclude rules affect what files are transferred, ignoring any deletion side- effects. Filter rules mainly affect the contents of directories that rsync is "recursing" into, but they can also affect a top-level item in the transfer that was specified as a argument.

The default for any unmatched file/dir is for it to be included in the transfer, which puts the file/dir into the sender's file list. The use of an exclude rule causes one or more matching files/dirs to be left out of the sender's file list. An include rule can be used to limit the effect of an exclude rule that is matching too many files.

The order of the rules is important because the first rule that matches is the one that takes effect. Thus, if an early rule excludes a file, no include rule that comes after it can have any effect. This means that you must place any include overrides somewhere prior to the exclude that it is intended to limit.

When a directory is excluded, all its contents and sub-contents are also excluded. The sender doesn't scan through any of it at all, which can save a lot of time when skipping large unneeded sub-trees.

It is also important to understand that the include/exclude rules are applied to every file and directory that the sender is recursing into. Thus, if you want a particular deep file to be included, you have to make sure that none of the directories that must be traversed on the way down to that file are excluded or else the file will never be discovered to be included. As an example, if the directory "a/path" was given as a transfer argument and you want to ensure that the file "a/path/down/deep/wanted.txt" is a part of the transfer, then the sender must not exclude the directories "a/path", "a/path/down", or "a/path/down/deep" as it makes it way scanning through the file tree.

When you are working on the rules, it can be helpful to ask rsync to tell you what is being excluded/included and why. Specifying --debug=FILTER or (when pulling files) -M--debug=FILTER turns on level 1 of the FILTER debug information that will output a message any time that a file or directory is included or excluded and which rule it matched. Beginning in 3.2.4 it will also warn if a filter rule has trailing whitespace, since an exclude of "foo " (with a trailing space) will not exclude a file named "foo".

Exclude and include rules can specify wildcard PATTERN MATCHING RULES (similar to shell wildcards) that allow you to match things like a file suffix or a portion of a filename.

A rule can be limited to only affecting a directory by putting a trailing slash onto the filename.

Simple include/exclude example

With the following file tree created on the sending side:

mkdir x/
touch x/file.txt
mkdir x/y/
touch x/y/file.txt
touch x/y/zzz.txt
mkdir x/z/
touch x/z/file.txt

Then the following rsync command will transfer the file "x/y/file.txt" and the directories needed to hold it, resulting in the path "/tmp/x/y/file.txt" existing on the remote host:

rsync -ai -f'+ x/' -f'+ x/y/' -f'+ x/y/file.txt' -f'- *' x host:/tmp/

Aside: this copy could also have been accomplished using the -R option (though the two commands behave differently if deletions are enabled):

rsync -aiR x/y/file.txt host:/tmp/

The following command does not need an include of the "x" directory because it is not a part of the transfer (note the traililng slash). Running this command would copy just "/tmp/x/file.txt" because the "y" and "z" dirs get excluded:

rsync -ai -f'+ file.txt' -f'- *' x/ host:/tmp/x/

This command would omit the zzz.txt file while copying "x" and everything else it contains:

rsync -ai -f'- zzz.txt' x host:/tmp/

Filter rules when deleting

By default the include & exclude filter rules affect both the sender (as it creates its file list) and the receiver (as it creates its file lists for calculating deletions). If no delete option is in effect, the receiver skips creating the delete- related file lists. This two-sided default can be manually overridden so that you are only specifying sender rules or receiver rules, as described in the FILTER RULES IN DEPTH section.

When deleting, an exclude protects a file from being removed on the receiving side while an include overrides that protection (putting the file at risk of deletion). The default is for a file to be at risk -- its safety depends on it matching a corresponding file from the sender.

An example of the two-sided exclude effect can be illustrated by the copying of a C development directory between two systems. When doing a touch-up copy, you might want to skip copying the built executable and the .o files (sender hide) so that the receiving side can build their own and not lose any object files that are already correct (receiver protect). For instance:

rsync -ai --del -f'- *.o' -f'- cmd' src host:/dest/

Note that using -f'-p *.o' is even better than -f'- *.o' if there is a chance that the directory structure may have changed. The "p" modifier is discussed in FILTER RULE MODIFIERS.

One final note, if your shell doesn't mind unexpanded wildcards, you could simplify the typing of the filter options by using an underscore in place of the space and leaving off the quotes. For instance, -f -_*.o -f -_cmd (and similar) could be used instead of the filter options above.

Filter rules in depth

Rsync builds an ordered list of filter rules as specified on the command-line and/or read-in from files. New style filter rules have the following syntax:

RULE [PATTERN_OR_FILENAME]
RULE,MODIFIERS [PATTERN_OR_FILENAME]

You have your choice of using either short or long RULE names, as described below. If you use a short-named rule, the ',' separating the RULE from the MODIFIERS is optional. The PATTERN or FILENAME that follows (when present) must come after either a single space or an underscore (_). Any additional spaces and/or underscores are considered to be a part of the pattern name. Here are the available rule prefixes:

exclude, '-'
specifies an exclude pattern that (by default) is both a hide and a protect.
include, '+'
specifies an include pattern that (by default) is both a show and a risk.
merge, '.'
specifies a merge-file on the client side to read for more rules.
dir-merge, ':'
specifies a per-directory merge-file. Using this kind of filter rule requires that you trust the sending side's filter checking, so it has the side-effect mentioned under the --trust-sender option.
hide, 'H'
specifies a pattern for hiding files from the transfer. Equivalent to a sender-only exclude, so -f'H foo' could also be specified as -f'-s foo'.
show, 'S'
files that match the pattern are not hidden. Equivalent to a sender-only include, so -f'S foo' could also be specified as -f'+s foo'.
protect, 'P'
specifies a pattern for protecting files from deletion. Equivalent to a receiver-only exclude, so -f'P foo' could also be specified as -f'-r foo'.
risk, 'R'
files that match the pattern are not protected. Equivalent to a receiver-only include, so -f'R foo' could also be specified as -f'+r foo'.
clear, '!' clears the current include/exclude list (takes no arg)

When rules are being read from a file (using merge or dir-merge), empty lines are ignored, as are whole-line comments that start with a '#' (filename rules that contain a hash character are unaffected).

Note also that the --filter, --include, and --exclude options take one rule/pattern each. To add multiple ones, you can repeat the options on the command-line, use the merge-file syntax of the --filter option, or the --include-from / --exclude-from options.

Pattern matching rules

Most of the rules mentioned above take an argument that specifies what the rule should match. If rsync is recursing through a directory hierarchy, keep in mind that each pattern is matched against the name of every directory in the descent path as rsync finds the filenames to send.

The matching rules for the pattern argument take several forms:

If a pattern contains a / (not counting a trailing slash) or a "**" (which can match a slash), then the pattern is matched against the full pathname, including any leading directories within the transfer. If the pattern doesn't contain a (non-trailing) / or a "**", then it is matched only against the final component of the filename or pathname. For example, foo means that the final path component must be "foo" while foo/bar would match the last 2 elements of the path (as long as both elements are within the transfer).
A pattern that ends with a / only matches a directory, not a regular file, symlink, or device.
A pattern that starts with a / is anchored to the start of the transfer path instead of the end. For example, /foo/ or /foo/bar/ match only leading elements in the path. If the rule is read from a per-directory filter file, the transfer path being matched will begin at the level of the filter file instead of the top of the transfer. See the section on ANCHORING INCLUDE/EXCLUDE PATTERNS for a full discussion of how to specify a pattern that matches at the root of the transfer.

Rsync chooses between doing a simple string match and wildcard matching by checking if the pattern contains one of these three wildcard characters: '*', '?', and '[' :

a '?' matches any single character except a slash (/).
a '*' matches zero or more non-slash characters.
a '**' matches zero or more characters, including slashes.
a '[' introduces a character class, such as [a-z] or [[:alpha:]], that must match one character.
a trailing *** in the pattern is a shorthand that allows you to match a directory and all its contents using a single rule. For example, specifying "dir_name/***" will match both the "dir_name" directory (as if "dir_name/" had been specified) and everything in the directory (as if "dir_name/**" had been specified).
a backslash can be used to escape a wildcard character, but it is only interpreted as an escape character if at least one wildcard character is present in the match pattern. For instance, the pattern "foo\bar" matches that single backslash literally, while the pattern "foo\bar*" would need to be changed to "foo\\bar*" to avoid the "\b" becoming just "b".

Here are some examples of exclude/include matching:

Option -f'- *.o' would exclude all filenames ending with .o
Option -f'- /foo' would exclude a file (or directory) named foo in the transfer-root directory
Option -f'- foo/' would exclude any directory named foo
Option -f'- foo/*/bar' would exclude any file/dir named bar which is at two levels below a directory named foo (if foo is in the transfer)
Option -f'- /foo/**/bar' would exclude any file/dir named bar that was two or more levels below a top-level directory named foo (note that /foo/bar is not excluded by this)
Options -f'+ */' -f'+ *.c' -f'- *' would include all directories and .c source files but nothing else
Options -f'+ foo/' -f'+ foo/bar.c' -f'- *' would include only the foo directory and foo/bar.c (the foo directory must be explicitly included or it would be excluded by the "- *")

Filter rule modifiers

The following modifiers are accepted after an include (+) or exclude (-) rule:

A / specifies that the include/exclude rule should be matched against the absolute pathname of the current item. For example, -f'-/ /etc/passwd' would exclude the passwd file any time the transfer was sending files from the "/etc" directory, and "-/ subdir/foo" would always exclude "foo" when it is in a dir named "subdir", even if "foo" is at the root of the current transfer.
A ! specifies that the include/exclude should take effect if the pattern fails to match. For instance, -f'-! */' would exclude all non-directories.
A C is used to indicate that all the global CVS-exclude rules should be inserted as excludes in place of the "-C". No arg should follow.
An s is used to indicate that the rule applies to the sending side. When a rule affects the sending side, it affects what files are put into the sender's file list. The default is for a rule to affect both sides unless --delete-excluded was specified, in which case default rules become sender-side only. See also the hide (H) and show (S) rules, which are an alternate way to specify sending-side includes/excludes.
An r is used to indicate that the rule applies to the receiving side. When a rule affects the receiving side, it prevents files from being deleted. See the s modifier for more info. See also the protect (P) and risk (R) rules, which are an alternate way to specify receiver-side includes/excludes.
A p indicates that a rule is perishable, meaning that it is ignored in directories that are being deleted. For instance, the --cvs-exclude (-C) option's default rules that exclude things like "CVS" and "*.o" are marked as perishable, and will not prevent a directory that was removed on the source from being deleted on the destination.
An x indicates that a rule affects xattr names in xattr copy/delete operations (and is thus ignored when matching file/dir names). If no xattr-matching rules are specified, a default xattr filtering rule is used.

Merge-file filter rules

You can merge whole files into your filter rules by specifying either a merge (.) or a dir-merge (:) filter rule (as introduced in the FILTER RULES section above).

There are two kinds of merged files -- single-instance ('.') and per-directory ('😂. A single-instance merge file is read one time, and its rules are incorporated into the filter list in the place of the "." rule. For per-directory merge files, rsync will scan every directory that it traverses for the named file, merging its contents when the file exists into the current list of inherited rules. These per-directory rule files must be created on the sending side because it is the sending side that is being scanned for the available files to transfer. These rule files may also need to be transferred to the receiving side if you want them to affect what files don't get deleted (see PER-DIRECTORY RULES AND DELETE below).

Some examples:

merge /etc/rsync/default.rules
. /etc/rsync/default.rules
dir-merge .per-dir-filter
dir-merge,n- .non-inherited-per-dir-excludes
:n- .non-inherited-per-dir-excludes

The following modifiers are accepted after a merge or dir-merge rule:

A - specifies that the file should consist of only exclude patterns, with no other rule-parsing except for in-file comments.
A + specifies that the file should consist of only include patterns, with no other rule-parsing except for in-file comments.
A C is a way to specify that the file should be read in a CVS-compatible manner. This turns on 'n', 'w', and '-', but also allows the list-clearing token (!) to be specified. If no filename is provided, ".cvsignore" is assumed.
A e will exclude the merge-file name from the transfer; e.g. "dir-merge,e .rules" is like "dir-merge .rules" and "- .rules".
An n specifies that the rules are not inherited by subdirectories.
A w specifies that the rules are word-split on whitespace instead of the normal line-splitting. This also turns off comments. Note: the space that separates the prefix from the rule is treated specially, so "- foo + bar" is parsed as two rules (assuming that prefix-parsing wasn't also disabled).
You may also specify any of the modifiers for the "+" or "-" rules (above) in order to have the rules that are read in from the file default to having that modifier set (except for the ! modifier, which would not be useful). For instance, "merge,-/ .excl" would treat the contents of .excl as absolute-path excludes, while "dir-merge,s .filt" and ":sC" would each make all their per-directory rules apply only on the sending side. If the merge rule specifies sides to affect (via the s or r modifier or both), then the rules in the file must not specify sides (via a modifier or a rule prefix such as hide).

Per-directory rules are inherited in all subdirectories of the directory where the merge-file was found unless the 'n' modifier was used. Each subdirectory's rules are prefixed to the inherited per-directory rules from its parents, which gives the newest rules a higher priority than the inherited rules. The entire set of dir-merge rules are grouped together in the spot where the merge-file was specified, so it is possible to override dir-merge rules via a rule that got specified earlier in the list of global rules. When the list-clearing rule ("!") is read from a per-directory file, it only clears the inherited rules for the current merge file.

Another way to prevent a single rule from a dir-merge file from being inherited is to anchor it with a leading slash. Anchored rules in a per-directory merge-file are relative to the merge- file's directory, so a pattern "/foo" would only match the file "foo" in the directory where the dir-merge filter file was found.

Here's an example filter file which you'd specify via --filter=". file":

merge /home/user/.global-filter
- *.gz
dir-merge .rules
+ *.[ch]
- *.o
- foo*

This will merge the contents of the /home/user/.global-filter file at the start of the list and also turns the ".rules" filename into a per-directory filter file. All rules read in prior to the start of the directory scan follow the global anchoring rules (i.e. a leading slash matches at the root of the transfer).

If a per-directory merge-file is specified with a path that is a parent directory of the first transfer directory, rsync will scan all the parent dirs from that starting point to the transfer directory for the indicated per-directory file. For instance, here is a common filter (see -F):

--filter=': /.rsync-filter'

That rule tells rsync to scan for the file .rsync-filter in all directories from the root down through the parent directory of the transfer prior to the start of the normal directory scan of the file in the directories that are sent as a part of the transfer. (Note: for an rsync daemon, the root is always the same as the module's "path".)

Some examples of this pre-scanning for per-directory files:

rsync -avF /src/path/ /dest/dir
rsync -av --filter=': ../../.rsync-filter' /src/path/ /dest/dir
rsync -av --filter=': .rsync-filter' /src/path/ /dest/dir

The first two commands above will look for ".rsync-filter" in "/" and "/src" before the normal scan begins looking for the file in "/src/path" and its subdirectories. The last command avoids the parent-dir scan and only looks for the ".rsync-filter" files in each directory that is a part of the transfer.

If you want to include the contents of a ".cvsignore" in your patterns, you should use the rule ":C", which creates a dir-merge of the .cvsignore file, but parsed in a CVS-compatible manner. You can use this to affect where the --cvs-exclude (-C) option's inclusion of the per-directory .cvsignore file gets placed into your rules by putting the ":C" wherever you like in your filter rules. Without this, rsync would add the dir-merge rule for the .cvsignore file at the end of all your other rules (giving it a lower priority than your command-line rules). For example:

cat <<EOT | rsync -avC --filter='. -' a/ b
+ foo.o
:C
- *.old
EOT
rsync -avC --include=foo.o -f :C --exclude='*.old' a/ b

Both of the above rsync commands are identical. Each one will merge all the per-directory .cvsignore rules in the middle of the list rather than at the end. This allows their dir-specific rules to supersede the rules that follow the :C instead of being subservient to all your rules. To affect the other CVS exclude rules (i.e. the default list of exclusions, the contents of $HOME/.cvsignore, and the value of $CVSIGNORE) you should omit the -C command-line option and instead insert a "-C" rule into your filter rules; e.g. "--filter=-C".

List-clearing filter rule

You can clear the current include/exclude list by using the "!" filter rule (as introduced in the FILTER RULES section above). The "current" list is either the global list of rules (if the rule is encountered while parsing the filter options) or a set of per-directory rules (which are inherited in their own sub-list, so a subdirectory can use this to clear out the parent's rules).

Anchoring include/exclude patterns

As mentioned earlier, global include/exclude patterns are anchored at the "root of the transfer" (as opposed to per- directory patterns, which are anchored at the merge-file's directory). If you think of the transfer as a subtree of names that are being sent from sender to receiver, the transfer-root is where the tree starts to be duplicated in the destination directory. This root governs where patterns that start with a / match.

Because the matching is relative to the transfer-root, changing the trailing slash on a source path or changing your use of the --relative option affects the path you need to use in your matching (in addition to changing how much of the file tree is duplicated on the destination host). The following examples demonstrate this.

Let's say that we want to match two source files, one with an absolute path of "/home/me/foo/bar", and one with a path of "/home/you/bar/baz". Here is how the various command choices differ for a 2-source transfer:

Example cmd: rsync -a /home/me /home/you /dest
+/- pattern: /me/foo/bar
+/- pattern: /you/bar/baz
Target file: /dest/me/foo/bar
Target file: /dest/you/bar/baz

Example cmd: rsync -a /home/me/ /home/you/ /dest
+/- pattern: /foo/bar               (note missing "me")
+/- pattern: /bar/baz               (note missing "you")
Target file: /dest/foo/bar
Target file: /dest/bar/baz

Example cmd: rsync -a --relative /home/me/ /home/you /dest
+/- pattern: /home/me/foo/bar       (note full path)
+/- pattern: /home/you/bar/baz      (ditto)
Target file: /dest/home/me/foo/bar
Target file: /dest/home/you/bar/baz

Example cmd: cd /home; rsync -a --relative me/foo you/ /dest
+/- pattern: /me/foo/bar      (starts at specified path)
+/- pattern: /you/bar/baz     (ditto)
Target file: /dest/me/foo/bar
Target file: /dest/you/bar/baz

The easiest way to see what name you should filter is to just look at the output when using --verbose and put a / in front of the name (use the --dry-run option if you're not yet ready to copy any files).

Per-directory rules and delete

Without a delete option, per-directory rules are only relevant on the sending side, so you can feel free to exclude the merge files themselves without affecting the transfer. To make this easy, the 'e' modifier adds this exclude for you, as seen in these two equivalent commands:

rsync -av --filter=': .excl' --exclude=.excl host:src/dir /dest
rsync -av --filter=':e .excl' host:src/dir /dest

However, if you want to do a delete on the receiving side AND you want some files to be excluded from being deleted, you'll need to be sure that the receiving side knows what files to exclude. The easiest way is to include the per-directory merge files in the transfer and use --delete-after, because this ensures that the receiving side gets all the same exclude rules as the sending side before it tries to delete anything:

rsync -avF --delete-after host:src/dir /dest

However, if the merge files are not a part of the transfer, you'll need to either specify some global exclude rules (i.e. specified on the command line), or you'll need to maintain your own per-directory merge files on the receiving side. An example of the first is this (assume that the remote .rules files exclude themselves):

rsync -av --filter=': .rules' --filter='. /my/extra.rules' --delete host:src/dir /dest

In the above example the extra.rules file can affect both sides of the transfer, but (on the sending side) the rules are subservient to the rules merged from the .rules files because they were specified after the per-directory merge rule.

In one final example, the remote side is excluding the .rsync-filter files from the transfer, but we want to use our own .rsync-filter files to control what gets deleted on the receiving side. To do this we must specifically exclude the per-directory merge files (so that they don't get deleted) and then put rules into the local files to control what else should not get deleted. Like one of these commands:

rsync -av --filter=':e /.rsync-filter' --delete host:src/dir /dest
rsync -avFF --delete host:src/dir /dest

Munin-tweaks

2023-04-25T00:00:00+02:00

Small recipes to tweak munin configurations.

Overriding critical and warning levels

In the node configuration enter:

plugin.field_name.critical value
plugin.field_name.warning value

The plugin name can be found by clicking in the graph with the value you want to override. The last component of the url (without the .html) is the plugin name.

The field name is in this view under the column Internal name.

Example:

[xenhosts;cn4.localnet]
  address cn4.localnet
  use_node_name no

  vgs.pool0_.warning 99.00
  vgs.pool0_.critical 99.50

Reference documentation

config field directives

Desktop Environments 2023

2024-02-02T00:00:00+01:00

Around April 2023, I decided to look for a new Linux Desktop Environment for my personal void linux. So I tried these distros:

lxqt: This is the one I eventually chose to switch to. I liked it that it was very small and light, and very modular, almost like a kit that you assemble yourself. Because I have thinking for some time that I would like to make my own Desktop environment from a Window Manager, lxqt is very close to what I wanted to do with that idea.
- PROS:
  - Light and modular
  - Classic User Experience (great since I am very old computer user).
  - Uses XScreenSaver
  - Uses QT widget set.
- CONS
  - not as visually appealing as the other desktops here.
MATE Desktop: This is the desktop that I was using before. It is light on resources and has all the features you would expect. Nothing against it, but I thought it was time to move on. Also my XScreenSaver stopped working in the latest void update.
- PROS
  - Full feature yet light on resources
  - Classic User Experience (great since I am very old computer user).
  - Was possible to use XScreenSaver.
standard gnome: This is supposed to be the Premier Linux [Desktop Environment]. Personally, I find it hard to use, as it completely deviates from User Experienced of Microsoft Windows 95. WHile it may be innovative, for me, an old computer user, it just gets in the way of getting things done. So for me it is a total turn-off. Also, under void I was only able to use the Wayland session.
- PROS
  - Very modern and popular desktop
- CONS
  - Too modern for my taste
  - Could only run on wayland.
gnome flashback : This returns the gnome desktop back to a classic user experience. I personally find this UX much better than the standard gnome user experience. Visually, was very smooth and appealing. I found it quite pleasant. I was also able to run it as part of the X session.
- PROS
  - Buttery smooth look and feel
KDE plasma : This is the KDE project Desktop Environment. I found it OK, but not particularly special. I just tried it on void and managed it to get it to work and do stuff, but didn't feel worth it to stick around too much.
XFCE4: This is another classic UX Desktop Environment. Just like MATE is a full featured desktop that tends to be light on resources. I find it stable and a good performer. The main reason why I did not opt for xfce is that this is the desktop I usually use for Virtual Machines that I spin up on the cloud. So it is useful to have a visually different UX so as not to get confused between working locally and working on a cloud system.
- PROS:
  - Classic User Experience (great since I am very old computer user).
budgie: Did not work for me on void linux. It would start, but the mouse and keyboard would be unresponsive.
cinnamon: Did not work for me on void linux. It would start, but the mouse and keyboard would be unresponsive.
pantheon: This is a desktop that I would like to try, but it is not available on Void Linux.

Pet Peeves

Wayland

The Wayland project started back in 2008. It started as a modern Window system implementation, slated to replace Xorg. I am writing this is 2023, and after 15 years, it is only partially available. As far as I can tell, the only desktop environments that support wayland are gnome and plasma.

I think this is a pity, but that is how things are in Open Source without a real corporate sponsor.

GTK+ libraries

I find this is a bit of a mess. I think it is ude with how the gnome project is evolving it software components. I use void linux and at the time of this writing I can install from the void repositories three different versions of GTK+, v2, v3 and v4. Also, I tried compiling a package called Xnee and that actually seemed to require v1.

Anyways, because different programs require different GTK+ release levels, this means that my working void desktop has all three versions installed.

I personally find this quite messy. Again, this is a fact of life with fully Open Source Software.

Personal thoughts on GUI programming

2024-02-02T00:00:00+01:00

I have been programming for about 40 years. Going through a lot of different languages and programming paradims.

Lately most of my programming is done in:

bash/shell script
Python
PHP, with some bits in JavaScript

So lately, after hitting some bugs in a recent update of the MATE desktop environment I decided to re-do my desktop set-up switching to a new desktop environment. After testing several desktop environments I decided for LxQt because of its minimalistic feel. While doing this, I figured, I needed some GUI scripts to spice things up.

In the past, most of these simple GUI scripts, I have done them in TCL/TK. So I am very familiar writting GUI applications using TCL.

Since, TCL is not a popular language (in a survey at the beginning of 2023 it wasn't even mentioned. On the other hand, python seemed quite popular.

I decided, that it probably would be a good idea to write these GUI scripts in python. Since, in python you can also get a tkinter module that seems to be quite ubiquitous. i.e. "batteries included" distributions have it, and most Linux distributions package it. Also, since is supposed to be a straight-port from TCL/TK I though that the learning curve would be pretty smooth.

So, I tried it using for a couple of scripts, a volume control and a [macro recording][xm] utilities.

I would have to admit that I find writting python3 tkinter GUIs very awkward. Because it is a direct translation of TCL/TK I keep thinking in TCL terms, but things do not work the same in python.

Because, it was easier for me, the other GUI utilities, local-startup and hk_helper were written in TCL

So, my conclusion is that while I can write GUI scripts in python, the learning curve is still steep, and I need to get a lot more experience before I can consider myself familiar with it.

Also, the decision to switch to tcl for the scripts that were eventually written in tcl, was driven not just for the familiarity, but because the use case required spawning new processes, which in tcl is far easier than in python.

Sometimes I think using a more pythonic GUI library instead of tkinter would be better. But, like I mentioned earlier, tkinter gets the job done, and can be found with python very often.

NacoWiki

2023-04-15T00:00:00+02:00

A few months ago I extensibly modify picowiki and creeated NanoWiki.

After using NanoWiki for a few months, the code became somewhat of an spaghetti mess. This has to do that picowiki was designed as a single file, single class application with Plugin extension. Since every change went to a single class this quickly became difficult to manage.

Additionally, I realized that "NanoWiki" was not a very good name as this name was used by several projects and organizations.

So, I went ahead and re-wrote the whole thing into NacoWiki. These are the changes:

Cleaner and more functional UI.
Initial REST-API support.
Added a CLI interface.
Off-tree installation, with the option of co-existing multiple instances.
Code modularization,
- nacowiki main class that integrates everything together.
- Core: main WIKI functionality
- Cli: CLI interface
- PluginCollection: plugin support
CodeMirror support now in the Core (instead of depending of Plugin implementations.
Re-organized CSS files
Raw/source code display.

As plugins:

Page handlers:
- HTML
- Markdown
- NEW source code
NEW YouTube Links
NEW Static site generator
Emojis
File includes
Var snippets
WikiLinks (NEW: search article names)

Features/improvements over picowiki:

file management: create, delete, rename, modify, attach, etc.
hooks for access control
meta data support
Disabled code execution. This can be considered a "security" feature.
Support for byte ranges. This lets you stream video files directly from the wiki.
toggable, folder or document views.
theme support
Multiple file type handling

Documentation

Documentation is now handled by phpDocumentor plus HTML generated using NacoWiki's own SiteGen plugin. You can find that here:

Python Development 2023

2023-04-25T00:00:00+02:00

Python development on Windows
Distributing Python scripts as single EXE or Directory
Installing netifaces on Windows
Documentation generation
- Using [sphinx][sphinx] for documentation
  - Example docstring
Passing Reserved Keywords as Keyword arguments

Python development on Windows

For windows, you can use WinPython. I prefer to use this instead of the official distribution because:

It is a portable distro.
You can choose for a batteries included, or just the dot release which only contains Python and Pip.

This way I can have multiple versions available. This is particularly useful for me because of my OTC development which uses OpenStack SDK. In my Windows system, I was only able to make it work with Python v3.8.10 (due to some ancient dependancies). Because I usually don't have a compiler, I can only use binary distros and I am unable to find it for a newer Python version.

Distributing Python scripts as single EXE or Directory

When distributing I had good results with pyinstaller. You can create a single folder or a single exe distribution. The results work suprisingly well (if they work). Some hints when using pyinstaller:

Cross packaging is not possible. If you need a Windows package you need to run it on Windows.
Dependancies not always work correctly. You may need to use these options:
- --hidden-import module
- --collect-data module
- --copy-metadata module
- --collect-all package
In some cases, you may need to force the inclusion of non-python files. Use:
- set sitedir=%WINPYDIR%\Lib\site-packages
- And in the command line:
- --add-data %sitedir%\path\to\data\file;path\to\data
- I use the %sitedir% variable to find things in the Python packages directory.
It is best to create a batch file to issue the pyinstaller command.

Because the command-line could become quite long, you can use the ^ escape. Example:

pyinstaller %buildtype% ^
  --hidden-import keystoneauth1 ^
  --collect-data keystoneauth1 ^
  --copy-metadata keystoneauth1 ^
  --hidden-import os_service_types ^
  --collect-data os_service_types ^
  --copy-metadata os_service_types ^
  --collect-all openstacksdk ^
  --copy-metadata openstacksdk ^
  --add-data %sitedir%\openstack\config\defaults.json;openstack\config ^
  --hidden-import keystoneauth1.loading._plugins ^
  --hidden-import keystoneauth1.loading._plugins.identity ^
  --hidden-import keystoneauth1.loading._plugins.identity.generic ^
urotc.py

Installing `netifaces` on Windows

NOTE: I tested this on Feb 2023. See Original article here

netifaces is a OpenStack SDK dependancy. Under version 3.8.10 I am able to install using --only-binary=netifaces option.

For newer versions it will fail with Microsoft Visual C++ 14.0 is required error message.

C:\RH>pip install netifaces
Collecting netifaces
  Downloading https://files.pythonhosted.org/packages/81/39/4e9a026265ba944ddf1fea176dbb29e0fe50c43717ba4fcf3646d099fe38/netifaces-0.10.7.tar.gz
Installing collected packages: netifaces
  Running setup.py install for netifaces ... error
    Complete output from command c:\users\rh\appdata\local\programs\python\python37\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\RH\\AppData\\Local\\Temp\\pip-install-wbfanly3\\netifaces\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\RONALD~1.HEI\AppData\Local\Temp\pip-record-m26yfbyt\install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'netifaces' extension
        error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

Since the suggested URL doesn't work. You need to do the following:

Go to the Microsoft-Repository Tools for Visual Studio 2017 or use the direct link to vs_buildtools.exe.
- ... it’s about 1.2MB
run „vs_buildtools.exe“
- it downloads ~ 70 MB
Select Workloads => Windows => [x] Visual C++ Build Tools“ => [Install]
- it downloads 1.12 GB
- and installs
Re-boot (I don't know if it is required, but I did it just in case)

Now netifaces can get installed:

C:\RH>pip install netifaces
Collecting netifaces
  Using cached https://files.pythonhosted.org/packages/81/39/4e9a026265ba944ddf1fea176dbb29e0fe50c43717ba4fcf3646d099fe38/netifaces-0.10.7.tar.gz
Installing collected packages: netifaces
  Running setup.py install for netifaces ... done
Successfully installed netifaces-0.10.7

Documentation generation

When programming documentation is important, allthough very often it takes a back seat.

To help keep it up to date, it is good to make it so it is easier to maintain and update. One way to do that is with keeping documentation and code together, and automating the way documentation is generated.

There is a number of solutions to do this. The one I looked at the most were:

mkdocs with mkdocstrings : which is nice because it uses markdown, however, because it is essentially a static site generator, a lot of things needed to be done manually.
sphinx : At the end, sphinx was the option that I liked the most. It uses RST for markup which is different from markdown, but it is close enough. Also, sphinx can also support markdown via some extensions but I did not try that.

Using sphinx for documentation

Prepare your environment:

pip install sphinx sphinx-argparse

In your project directory, I have two folders:

docs : where the documentation source resides
src : where the python code resides

Also, I ignore:

public : where the generated documentation is created. This can then be added into a CI pipeline to publish documentation.

Run:

sphinx-quickstart

In the docs folder, to initialize things. This will create the files:

conf.py
Makefile
index.rst

Modify conf.py to:

include the source:

sys.path.insert(0, os.path.abspath('../src'))

Customize project meta data
Include enable desired extensions. For max automation I enable:
- sphinx.ext.autodoc
- sphinx.ext.autosummary

Modify Makefile to run:

sphinx-apidoc -o apidoc ../src

This command extracts from python docstrings and creates the relevant rst files.

For command line arguments, I use arparse extension. Create a cli.rst like so:

CLI
===

.. argparse::
   :filename: ../src/cli.py
   :func: cli_args
   :prog: cli.py

Where cli.py contains a function cli_args that returns an ArgumentParser object.

Example `docstring`

This is an example doc string to add to your code, after the element declaration:

   '''
   Summary text

   Description of the function

   :param str argname: argument passed
   :returns bool: Returns a boolean True on success, False on failure
   '''

No Need to make it too complicated.

Passing Reserved Keywords as Keyword arguments

Very often when using wrapped APIs, that used functions with Keyword arguments that you would need to pass reserved keywords (such as class, or import) as function keywords.

Of course, this is NOT allowed in python. And you will get an error like:

SyntaxError: Invalid syntax

To work around that, you need to place those keywords in a dictionary and use ** notation. So instead of:

response = client.service.SendSMS( toNum = '0666666666666',
                pass = '123456'}
           )

you would:

response = client.service.SendSMS( toNum = '0666666666666',
                **{'pass': '123456'}
           )

Raspberry Pi emulation with Qemu

2024-02-02T00:00:00+01:00

The idea here is that we use a Desktop PC for developing/debugging Raspberry Pi set-ups using qemu for emulating Rasperrby Pi.

qemu currently supports the following configurations:

Raspberry Pi Zero and 1A+ (armhf)
Raspberry Pi 2B (armv7)
Raspberry Pi 3A+ (aarch64)
Raspberry Pi 3B (aarch64)
- This is the version I am targetting in this article. I already recycled all my older boards.
- Actually I tried emulating the other configurations but they did not work. Either they failed to boot, or the graphic display wouldn't work.
- So raspi3b with 64-bit run-time is the only configuration I was able to succesfully boot.
NOTE that Raspberry Pi 4 is not supported at the moment.

So, unfortunately the state of things is far from perfect.

Missing display bug

During my tests on the raspi3b configuration, I was not able to get a working console. This has to do with this commit, which disables the Frame Buffer driver because qemu doesn't seem to report the display properly. This causes the error to show up on the kernel log:

bcm2708_fb soc:fb: Unable to determine number of FBs. Disabling driver.

Before the commit, the kernel would assume that there was always one display. On the other hand, this only affects use-cases that require display. For headless development, using the serial port works just fine.

For Alpine Linux, the last working Frame Buffer version seems to be 3.16.3-aarch64. The display did not work for 3.17.0-aarch64. The 32 bit 3.16.3 would display the Disabling driver. message but I wasn't able to boot further than that.

Emulated hardware

According to the qemu documentation, the following is impleted:

ARM1176JZF-S, Cortex-A7 or Cortex-A53 CPU. I only tested the Cortex-A53 CPU for the raspi3b configuration.
Interrupt controller
DMA controller
Clock and reset controller (CPRMAN)
System Timer
GPIO controller
Serial ports (BCM2835 AUX - 16550 based - and PL011)
Random Number Generator (RNG)
Frame Buffer : However the Linux kernel does not seem to find it.
USB host (USBH)
GPIO controller
SD/MMC host controller
SoC thermal sensor
USB2 host controller (DWC2 and MPHI)
MailBox controller (MBOX)
VideoCore firmware (property)

As you can see, no network interface is implemented, so you must use a USB network.

Getting started

The basic command line I am using is:

  qemu-system-aarch64 \
     -machine raspi3b -cpu cortex-a53 -m 1G -smp 4 -dtb bcm2710-rpi-3-b-plus.dtb \
     -kernel $linux_kernel -initrd $linux_initrd -append "$cmdline" \
     -sd $sd_image \
     -serial stdio \
     -usb \
     -device usb-mouse -device usb-kbd \
     -device usb-net,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22

Command explanation:

qemu-system-aarch64 : emulate a 64-bit ARM system
-machine raspi3b -cpu cortex-a53 -m 1G -smp 4 -dtb bcm2710-rpi-3-b-plus.dtb : Matches the Raspberry Pi model 3B configuration. The dtb is a file from the Raspberry Pi boot partition that is normally loaded by the Firmware.
-kernel $linux_kernel -initrd $linux_initrd -append "$cmdline" : Linux related boot configuration. You must provide a kernel and optional initrd files. Usually you would extract them from your sdcard image. The append is used for the kernel command line. If you want a serial console make sure you include:
- console=ttyAMA0,115200
sd $sd_image : Image for the sdcard storage
-usb : Enable USB bus. Needed for the emulated console mouse/keyboard and usb network.
-serial stdio : enable a serial console (if you are using the emulated framebuffer. Note that Ctrl+C are not caught and would kill the emulation.
-device usb-mouse -device usb-kbd : these are used with the virtual framebuffer for providing keyboard and mouse.
-device usb-net,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22 : Enable virtual networking using slirp.

If you wish to run a headless (only serial console) configuration, you should remove the -serial stdio -device usb-mouse -device usb-kbd options and just use:

-nographic

This would automatically enable -serial stdio and remove the framebuffer. In this configuration Ctrl-C is handled properly.

Tested OS

I tested the following images, with these results:

Operating System	Status
2020-08-20-raspios-buster-arm64-lite.zip	fully working
2022-09-22-raspios-bullseye-arm64-lite.img.xz	Only works headless. Default user is not set properly, so the image needs to be modified to inject login credentials
alpine-rpi-3.16.3-aarch64.tar.gz	fully working
alpine-rpi-3.17.0-aarch64.tar.gz	Only works headless

raspi-emu

For convenicne, I wrote the raspi-emu script:

rasi-emu

This can be used to prepare images and run emulation sessions.

Usage:

Preparing base image

raspi-emu prep [options] src

Prepares the downloaded image so it can be used as a base for a qemu thin-provisioned image.

Options:

--sz=size : Set the base image to the given size.
-c | --compress : For qcow2 images, create a compressed image.
--qcow2 : Create a qcow2 format image. This is the default.
--raw : Create a raw image.
--volume=name : When creating AlpineLinux images, use name as the volume name. Otherwise a random name is generated.

Formatting SDCARD image

raspi-emu format [options] base dest

Create an SDCARD image to be used for qemu emulation. It will create a thin-provisioned image when possible.

Options:

--resize=size : Set the SDCARD image to the given size.

Running Emulation

raspi-emu run [options] sdimg

Will boot qemu emulation with the specified SDCARD image. Configuration when possible is read from the boot partition of the SDCARD.

Options:

--vfb-only : Enable the virtual framebuffer and disables the serial console.
--vfb : Enables the virtual framebuffer. The serial console is kept enabled.
--no-vfb : Disables virtual framebuffer. This is the default.
--ttycon : Enables the serial console for Linux logins. (Default)
--no-ttycon : Disables the serial console for Linux logins.
--vnet : Enable virtual network. (Default)
--no-vnet : Disables hte virtual network.
--portfwd : Enables virtual network. Forwards port 5555 on host to port 22 on VM.
--portfwd=rule : Adds the given port forwarding rule.
Example rule: tcp::5555-:22
--no-portfwd : Dsiables port forwarding. (Default)
--raspi3b : Emulate a Raspberry Pi Model 3B. (Default)

The default is running headless (only serial console) with networking enabled.

Home Assistant Wall Panel

2023-01-11T00:00:00+01:00

For a while I was using TabletClock with old tablets. But this has not been updated in a while. I was thinking of writing my own version until I found WallPanel.

Essentially it is purposely built web-browser with special features which makes it possible to use it to replaces TabletClock. Essentially, I could replace TabletClock with a web-page showing the time and a weather forecast and have WallPanel point to it. Easy and simple, and can be customized in multiple ways.

Currently I am using WallPanel with HomeAssistant. I am using these features:

Date/time screen saver with auto off by face recognition
Web Cam functionality

I normally have it on a HomeAssistant panel showing time, weather forecaset and a few choice controls.

RDP vs VNC

2023-01-11T00:00:00+01:00

For years I have been using VNC for my remote desktop needs. This works usually well enough. The features that I like are:

Basic set-up is easy
Desktop sessions are persistent
Can be used to view an actual X11.org desktop.
Browser based clients via noVNC or Guacamole

On the other hand, a number of features are either not implemented or are not easily implementable.

On-demand desktop sessions. Usually you can hack scripts to do this. Or you can use inetd mode to create a session on-demand, however, this loses session persistence.
While in theory, because most Desktops use pulseaudio which would let you redirect audio to a remote, this is another protocol so it is not simple to set-up in practice.

So I have now found xrdp which provides:

Easy basic set-up
On-demand desktop sessions (session management) with persistent session support.
Sound redirection (however I have not been able to make this work)
Browser based clients via Guacamole

For the client side, you can either use:

RDP Client : which comes with MS-Windows or
freerdp : For Linux.

Still I have not been able to:

Enable sound re-direction
- Configure sound
- Essentially compile module
Use to view an actuall X11.org desktop. For this I am simply using x11vnc and just using the vncviewer.

QNAP Snapshots

2024-02-02T00:00:00+01:00

I wrote a small tool to access QNAP snapshots from the Linux command line.

Pre-requistes:

Snapshots have to be enabled
You need a /share/netcfg containing the file:
- In my case, I set this share as read-only with root-squash.
- `admin.yaml' : contains the private/public keys and the configuration of the forced command. For access control, it is only readable to group and the file is owned by the UNIX group that can do snapshot operations.
- registry.yaml : this is optional if you are changing the admin username.

Afterwards run:

install_key.sh server-name

This installs the public key into the authorized keys. You will need ssh access for this to work.

You need to do this on all the QNAP servers that offer snapshots.

Copy qsnap to somewhere in your path.

Usage

Listing snapshots

qsnap

List snapshots for the current directory

bash qsnap ls file-path

List snapshots for the given file-path. file-path can be provided multiple times.

Reading snapshot files

qsnap cat [--snap=snapid] file1 [file2 file3 ...]

Would display the given file(s) from the snapshot. If snapid is not specified will use the latest available snapshot.

Dumping snapshots

``bash qsnap tar [--snap=snapid] [options] path



Will dump the given `path` as a tarball.  If `snapid` is not
specified will use the latest available snapshot.

The `path` can be either a file or directory.

Additional options:

- `--base64` : Data will be dumped using MIME Base64 encoding
- '--no-compress' : Default is to compress.  This disables compression
- '-v' : Pass `v` flag to `tar` command.

All this can be found on [github](https://github.com/alejandroliu/0ink.net/blob/main/snippets/2023/qsnap).

Docker on Void

2025-04-01T00:00:00+02:00

This is a quick recipe to run Docker on void:

Make sure your system is up-to-date:
- sudo xbps-install -Syu
Install docker executables:
- sudo xbps-install -S docker
- Additional recommended packaged: docker-compose, docker-buildx.
Check if docker was installed properly:
- docker --version
Enable services:
- sudo ln -s /etc/sv/containerd /var/service
- sudo ln -s /etc/sv/docker /var/service
Add your user to the docker group:
- sudo usermod -a -G docker $(whoami)

You can start using docker. You may need to logout and login again for the group membership to be updated.

Here is a beginners tutorial

Home Assistant sensors

2023-01-11T00:00:00+01:00

I finished migrating my VeraEdge to Home Assistant. I think after using it for some time, I find Home Assistant far superior to the VeraEdge in every way.

So, I took the time to mostly standardise my sensors which make things simpler to manage/mantain.

As such, essentially I am only using 4 types of sensors:

Neo Coolcam Door/Window Sensor DS01Z : This is a door/window sensor. I still have a couple of these sensors from before. They are EOL now, being replaced by the DS07Z.
New Coolcam Door/Window Sensor DS07Z : This is a door/window sensor with integrated temperature and humidty sensors. I was standardirising to this sensor type for door/window and also for temperature and humidity until they became hard to source. The last couple of sensors of this type I had to buy from AliExpress. For the moment, I have enough sensors, but in the future, I may need to find a new source.

Since this sensor can be charged using a USB port, I have an additional sensor to be used as a "charging" sensor. That way I can replace batteries, and the low batteries can be charged in the "charging" sensor. This is due to the fact usually the sensors are in hard to reach places, where USB power is not easily available.
Fibaro Smoke Sensor 2 : This is a Smoke Sensor with integrated temperature meter. I am using this to replace my "dumb" smoke sensors. These sensors only detect smoke and do not detect CO₂. This is not a problem because on one hand₂se sensors are mounted on the ceiling₂ch is detrimental for CO₂ detection as CO₂ tends to accumulate on the floor first. The other is that CO₂ detection is more important for smokeless fires (i.e. gas burning). Since we are only using this for the water heater, it is less of a priority.
New Coolcam Water Leak sensor : This is a water leak sensor. Unfortunately is already EOL.

I used to have other sensors types:

Philio Tech Door/Window Sensor
Philio Tech multi sensors : While these sensors were good in the sense that they paired easily with my VeraEdge and gave accurate reading, they were (at least to me) not easy to open. So every time I would want to replace the battery I would accidentally break the latches.
Other door sensors : Also, a number of sensors that I tried, made it difficult to stock up on spare batteries. Furthermore, specially for the door/window sensors, some had awkward shapes.

Happy New Year 2023

2023-01-11T00:00:00+01:00

Best wishes for 2023!

This website is now 10 Years Old.

Home Assistant RFXCOM Integration

2022-12-15T00:00:00+01:00

RFXCOM RFXtrx

This integration is to control RFXtrx devices. I am using to control Somfy blinds and KlikAanKlikUit remotes.

Add integration: RFXCOM RFXtrx
Conection type: Serial
Select device: RFXtrx433XL - RFXtrx433XL, s/n: * - RFXCOM

Remember to configure the RFXCOM unit before using it with Home Assistant as Home Assistant integration has limited control of it.

Specifically, you need to use the RFXCOM rfxmngr tool to enable the relevant protocols and in the case of Somfy blinds, pair them.

For my home I enabled the AC protocol for use with KAKU devices.

KlikAanKlikUit

KAKU are inexpensive home automation devices that use a wireless protocol very similar to X-10.

The simplest way is to:

List integrations
Configure RFXTRX
Enable Automatic Add
Submit

Afterwards, just use the lights and devices, and they will be added automatically.

Somfy

For pairing the Somfy blinds, you can refer to this article and ths one.

In my configuration I am using:

remoteID: 010E1 >  0 : 10 : E1 : 010E1
          123456
          ABCDEF
remoteID: 69121  > 0 : 16 : 225 : 4321
unitCode: 01

1234567890123456
071a000000000000
071a00000010e101

Tweaks

Because for some reason my Skylight cover says "Open" when is "Closed" and viceversa.

To clean that up we use the:

To create a "template" cover that shows things properly. In configuration.yaml we have this:

cover:
  - platform: template
    covers:
      study_skylight_p:
        unique_id: 13d7a089-c536-4dc0-b2c0-e5dae6521460
        open_cover:
          service: cover.close_cover
          target:
            entity_id: cover.rfy_0010e1_1
        close_cover:
          service: cover.open_cover
          target:
            entity_id: cover.rfy_0010e1_1
        stop_cover:
          service: cover.stop_cover
          target:
            entity_id: cover.rfy_0010e1_1

Home Assistant Behind Reverse Proxy

2022-12-15T00:00:00+01:00

To set-up a reverse proxy I took the following steps:

configure DNS
get Letsencrypt certificates
Configure NGINX
Configure Home Assistant to trust the proxy

At the time of this writing I can't really confirm if the reverse proxy configuration for home assistant is working as I can't tell what IP address is using for the trustednetworks authenticator.

Sample NGINX configuration:


map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    # Update this line to be your domain
    server_name example.com;

    # These shouldn't need to be changed
    listen [::]:80 default_server ipv6only=off;
    return 301 https://$host$request_uri;
}

server {
    # Update this line to be your domain
    server_name example.com;

    # Ensure these lines point to your SSL certificate and key
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    # Use these lines instead if you created a self-signed certificate
    # ssl_certificate /etc/nginx/ssl/cert.pem;
    # ssl_certificate_key /etc/nginx/ssl/key.pem;

    # Ensure this line points to your dhparams file
    ssl_dhparam /etc/nginx/ssl/dhparams.pem;

    # These shouldn't need to be changed
    listen [::]:443 ssl default_server ipv6only=off; # if your nginx version is >= 1.9.5 you can also add the "http2" flag here
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    # ssl on; # Uncomment if you are using nginx < 1.15.0
    ssl_protocols TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;

    proxy_buffering off;

    location / {
        proxy_pass http://127.0.0.1:8123;
        proxy_set_header Host $host;
        proxy_redirect http:// https://;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Home Assistant configuration.yaml entries:

http:
  # For extra security set this to only accept connections on localhost if NGINX is on the same machine
  # Uncommenting this will mean that you can only reach Home Assistant using the proxy, not directly via IP from other clients.
  # server_host: 127.0.0.1
  use_x_forwarded_for: true
  # You must set the trusted proxy IP address so that Home Assistant will properly accept connections
  # Set this to your NGINX machine IP, or localhost if hosted on the same machine.
  trusted_proxies: <NGINX IP address here, or 127.0.0.1 if hosted on the same machine>

Looking up docker image tags

2022-12-15T00:00:00+01:00

This recipe is to check the tags defined for a specific Docker image in docker.hub.

The basic API is at https://registry.hub.docker.com/v2

So the format is as follows:

https://registry.hub.docker.com/v2/repositories/{namespace}/{image}/tags/

Where:

namespace : usually is the user account posting the image. For official images set the namespace to library.
image : Image name.

Examples:

Docker Official alpine image
- namespace: library
- image : alpine
photoprism/photoprism
- namespace : photoprism
- image : photoprism

So, you can then use curl and jq to access the relevant data. For example, to get the tags, last updated time and digest as a tsv:

curl -s -L $URL | jq -r '(.results[] | select(.tag_status == "active") | [.name, .last_updated
, .digest]) | @tsv'

v1 API

You could also use the v1 API while it is still available:

https://registry.hub.docker.com/api/content/v1/repositories/public/{namespace}/{image}/tags/

Home Assistant Large Clock

2024-02-02T00:00:00+01:00

This recipe is my version of providing a "large clock" face in the home assistant dashboard.

Enable serving local static files:

Create directory www in your config directory.
Restart home assistant.
Static files are now available as http://homeassistant.local:8123/local/.

Place the HTML with your clock in a file i.e. $config/www/clock.html. I am using this:

Then add a webpage card:

type: iframe
url: /local/clock.html
aspect_ratio: 45%

Obviously you can fully exercise your HTML to get your clock to look exactly like you want.

I wrote this because I couldn't get the markdown card to style properly. Also, I wasn't keen on installing the time and date sensor which is required for the clock examples based on the picture elements card.

Other implementations:

Home Assistant HTTP Based Authentication Backend

2024-02-02T00:00:00+01:00

This recipe is to authenticate users using a web server providing Basic HTTP authentication for it users.

This is useful if you want to consolidate users/passwords in a single system. So instead of managing users on Home Assistant you can have all users managed from a central location.

It uses the Home Assistant command line authentication provider and the curl command.

To make it work is quite simple. Copy this script to your /config directory as curl_auth.sh:

Add the following lines to your configure.yaml:

homeassistant:
  auth_providers:
    - type: command_line
      command: /config/curl_auth.sh
      args: [ "http://your-web-site-url/" ]
      meta: true

Make sure that you modify the URL in the configuration to a web server that is doing Basic HTTP authentication. It uses curl for checking URLs, so http and https protocols would work.

If using https with self-signed certificates, you need to pass the -k option which is then passed to curl. See curl(1).

Example:

args: [ "-k", "https://your-web-site-url/" ]

Moving to Home Assistant

2024-02-02T00:00:00+01:00

I am busy moving away from my VeraEdge installation to a Home Assistant running on a Raspberry Pi 4. This is because it looks like the maker of the VeraEdge was bought and it is slowly being phased out.

For this I am using the following parts:

Geekwork X728 18650 UPS + X728-C1 case : This provides with a case (with cooling fan), UPS and RTC functionality.
Aeotec Z-Stick Gen5+ : For Z-Wave compatibility.
ConBee 2 : For ZigBee compatibility.
A Raspberry Pi 4 - 4GB
64GB SD Card. Actually I wanted to use a 32GB SD card, but the 64GB had better specs and was only a couple os bucks more expensive.

I will be reusing these components:

Hardware build

Building the case is simple and straight forward. You can follow this video on youtube.

The steps are:

Open the case.
Install the fan.
Install the additional battery holder.
Install the power button.
Screw the spacers to the Raspberry Pi.
Insert the X728 UPS hat on top and screw in place.
Install batteries.
Plug connectors.
Screw the Raspberry Pi to the case.
Test that everything is in working order.
Optional: Set the jumper selector to auto power-on.
Close the case.

Case software

I did not like the software that comes with the case, so I rolled-up my own. The RTC uses the standard rtc-ds1307 which is in the Linux kernel. For GPIO programming I am using the /sysfs interface. The only component that requires custom programming was the battery charge and voltage readings. For that I wrote a small C program.

x728batt

This is tied to systemd through these files:

/etc/systemd/system/x728clock.service executes clock.sh This is used to load the RTC kernel modules and activate the RTC in the i²c bus.
/etc/systemd/system/x728ups.service executes upsmon.sh This is used to monitor the Push Button, the A/C power status and if the A/C power is lost, the battery status. It will trigger a graceful shutdown if the button is pressed or if the battery charge is insufficient.
/lib/systemd/system-shutdown/gpio-poweroff This is used to turn off the UPS power if the user issues the poweroff command.

In addition to this, I created a script to inject the Home Assistant Raspberry Pi image with the relevant files and also adds a RAUC post-install handler. so that OTA upgrades will keep these customizations.

As a bonus I am adding a munin-node systemd unit for system monitoring. And yes, I am old-fashioned.

Also, I am enabling ssh to the underlying Operating System.

Home Assistant installation

Following the raspberry pi installation guide is quite straight forward.

I chose to use the HAOS image install as it gives a more consumer device experience.

Download the 64bit image for Raspberry Pi 4 from the releases page
Use the haos-x728.sh to customize the image to include my X728 files.
Write the modified image to SD card.

Boot the raspberry pi from the new SD card and do the GUI installation. For my case, I needed to login to my router to look-up the IP address. It is configured via DHCP. I also statically assign an IP address and DNS name based on MAC address. To make sure that the DNS name and the host name match, I modified it on the configuration:

Settings -> System -> network

Modify hostname.

Initial configuration

Set up home areas. I set-up one area per room, following a naming convention:

F floor-number Room-name

For example:

F0 Kitchen
F2 Attic

Also, create additional areas for:

External : External items
System : System related entities and devices

For devices, I am using this naming convention:

Room-name room-section device optional

The idea is to make it simple to guess the name for voice recognition.

Room-name : this should match the area.
room-section : optional, section of the room this applies to.
device : device type.
- Chromecast : google chromecast device
- Display : google nest hub
- TV : with optional casting, upnp or api
- Skylight
- Light
- Switch
- Double Switch
- Remote: Remote control
- Window Sensor
- Door Sensor
optional : used for when multiple device of the same type are in the same room.

Add-Ons

I installed the followng add-ons:

Home Assistant Community Add-ons
- Studio Code Server : for editing files. Press F1 and start typing home assistant to view available integration commands. This is needed because (unfortunately) not everything can be configured through the UI.
- Z-Wave JS UI : Instead of Official add-ons. The community add-on gives you a control panel with more detailed control featires. Specifically you can set group associations.

Also, adding my own repository: https://github.com/iliu-net/hassio-addons

rsync-folders : save data and backups to remote server using rsync
watchdogdev : watchdog timer

Further Configuration

These configurations require modifying files. So usually I would do them after installing Studio Code Server.

Modifying authentications

To simplify login in local networks (specially to support physical control panels) I configured the trusted_networks by adding the following lines to your configuration.yaml.

homeassistant:
  auth_providers:
    - type: trusted_networks
      trusted_networks:
      - 192.168.2.0/24

Essentially what this does is that devices connecting from the trusted networks do not to login with username/password.

System temperature

For fun I also configured a CPU temperature sensor. Add this to configuration.yaml:

sensor:
  ### command line
  - platform: command_line
    name: CPU Temperature
    command: "cat /sys/class/thermal/thermal_zone0/temp" # RPi
    # command: "cat /sys/class/thermal/thermal_zone2/temp" # NUC
    # If errors occur, remove degree symbol below
    unit_of_measurement: "°C"
    value_template: "{{ '%.1f' | format(value | multiply(0.001)) }}" # RPi & NUC
    unique_id: sys_cpu_temp

Integrations

DSMR Slimme Meter

This integration is to read energy consumption as provided by NL smart meters. Just make sure that you get the right cable. I am using a cable from ROBBshop. Just plug and add to the integrations.

To include:

Add Integration : DSMR Slimme Meter
Serial : connection
Select device: Select the right serial port (should be easy to identify).
DSMR Version : 5

This is very easy to add and configure.

ZigBee Home Automation

This integration is automatically discovered for supported coordinators. I am using a ConBee II ZigBee coordinator. As long as the device is supported (see compatibility list ) things are fairly easy and simple.

There are multiple options for ZigBee support. I opted for ZHA because it has fairly good device support and is easy to use and set-up.

The alternatives are:

Zibgee2MQTT
- Good for power users, execellent configuratbility and the best device support
- It can be complicated.
deCONZ Add-On
- Made by the ConBee2 developers. There are no real benefits to using this integration.

Z-Wave automation

For Z-Wave I am using the Z-Wave JS integration paired with a Aeotec Z-Stick Gen5.

I am using the Z-Wave JS UI from Community Add-ons because that gives you a control panel user interface that is handy when debugging obscure Z-Wave issues and also has support for creating direct node group associations.

In most day to day situations, I don't really use this control panel, as most operations can be done from the Home Assistant integration directly.

When adding this, it is not possible to do it from the auto-discovered z-stick gen5 as that will automatically install the core Z-Wave JS add-on. So just ignore it and use the Add Integration functionality and pick Z-Wave JS integration from the menu. This will let you skip the standard add-on installation and let you specify the right Z-Wave JS UI add-on instead.

Other integrations

Buienradar : Dutch weather data. Just add it and it works mostly out of the box.
DLNA media servers : these are discovered automatically as long as they are in the same Subnet.
Google Cast : These are also discovered automatically. Used for Android TV devices and Google Nest speakers/displays.
Printer : This was automatically discovered.
Waze Travel Time : Show the commute time between two points.
Netherlands Vehicle Authority : Yes, this can be done, but not sure its use. Maybe to remind you that the APK is due?
Philips TV : Installing this integration is quite straight forward. This enables automation options, but I don't know what to do with it yet. Also has the limitation that it can't be turned on using the API.
Jellyfin : Add a Jellyfin server as a media source. Note that only a single Jellyfin can be configured.

References

Markdown cheat sheet

2022-10-12T00:00:00+02:00

This is intended as a quick reference and showcase. For more complete info, see John Gruber's original spec and the Github-flavored Markdown info page.

Headers

Source:

# H1
## H2
### H3
#### H4
##### H5
###### H6

Alternatively, for H1 and H2, an underline-ish style:

Alt-H1
======

Alt-H2
------

Output:

H1

H2

H3

H4

H5

Alternatively, for H1 and H2, an underline-ish style:

Alt-H1

Alt-H2

Emphasis

Source:

Emphasis, aka italics, with *asterisks* or _underscores_.

Strong emphasis, aka bold, with **asterisks** or __underscores__.

Combined emphasis with **asterisks and _underscores_**.

Strikethrough uses two tildes. ~~Scratch this.~~

Output:

Emphasis, aka italics, with asterisks or underscores.

Strong emphasis, aka bold, with asterisks or underscores.

Combined emphasis with asterisks and underscores.

Strikethrough uses two tildes. ~~Scratch this.~~

Lists

(In this example, leading and trailing spaces are shown with with dots: ⋅)

Source:

1. First ordered list item
2. Another item
⋅⋅* Unordered sub-list.
1. Actual numbers don't matter, just that it's a number
⋅⋅1. Ordered sub-list
4. And another item.

⋅⋅⋅You can have properly indented paragraphs within list items. Notice the blank line above, and the leading spaces (at least one, but we'll use three here to also align the raw Markdown).

⋅⋅⋅To have a line break without a paragraph, you will need to use two trailing spaces.⋅⋅
⋅⋅⋅Note that this line is separate, but within the same paragraph.⋅⋅
⋅⋅⋅(This is contrary to the typical GFM line break behaviour, where trailing spaces are not required.)

* Unordered list can use asterisks
- Or minuses
+ Or pluses

Output:

First ordered list item
Another item

Unordered sub-list.

Actual numbers don't matter, just that it's a number
Ordered sub-list
And another item.

You can have properly indented paragraphs within list items. Notice the blank line above, and the leading spaces (at least one, but we'll use three here to also align the raw Markdown).

To have a line break without a paragraph, you will need to use two trailing spaces. Note that this line is separate, but within the same paragraph. (This is contrary to the typical GFM line break behaviour, where trailing spaces are not required.)

Unordered list can use asterisks

Or minuses

Or pluses

Links

There are two ways to create links.

Source:

[I'm an inline-style link](https://www.google.com)

[I'm an inline-style link with title](https://www.google.com "Google's Homepage")

[I'm a reference-style link][Arbitrary case-insensitive reference text]

[I'm a relative reference to a repository file](../blob/master/LICENSE)

[You can use numbers for reference-style link definitions][1]

Or leave it empty and use the [link text itself].

URLs and URLs in angle brackets will automatically get turned into links.
http://www.example.com or <http://www.example.com> and sometimes
example.com (but not on Github, for example).

Some text to show that the reference links can follow later.

[arbitrary case-insensitive reference text]: https://www.mozilla.org
[1]: http://slashdot.org
[link text itself]: http://www.reddit.com

Output:

I'm an inline-style link

I'm an inline-style link with title

I'm a reference-style link

I'm a relative reference to a repository file

You can use numbers for reference-style link definitions

Or leave it empty and use the link text itself.

URLs and URLs in angle brackets will automatically get turned into links. http://www.example.com or http://www.example.com and sometimes example.com (but not on Github, for example).

Some text to show that the reference links can follow later.

Images

Source:

Here's our logo (hover to see the title text):

Inline-style:
![alt text](https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png "Logo Title Text 1")

Reference-style:
![alt text][logo]

[logo]: https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png "Logo Title Text 2"

Output:

Here's our logo (hover to see the title text):

Inline-style:

Reference-style:

Code and Syntax Highlighting

Code blocks are part of the Markdown spec, but syntax highlighting isn't. However, many renderers support syntax highlighting. Which languages are supported and how those language names should be written will vary from renderer to renderer.

To see the complete list, and how to write the language names, see the highlight.js demo page.

Source:

Inline `code` has `back-ticks around` it.

Output:

Inline code has back-ticks around it.

Blocks of code are either fenced by lines with three back-ticks "```", or are indented with four spaces. I recommend only using the fenced code blocks -- they're easier and only they support syntax highlighting.

Source:

```javascript
var s = "JavaScript syntax highlighting";
alert(s);
```

```python
s = "Python syntax highlighting"
print s
```

```
No language indicated, so no syntax highlighting.
But let's throw in a <b>tag</b>.
```

Output:

var s = "JavaScript syntax highlighting";
alert(s);

s = "Python syntax highlighting"
print s

No language indicated, so no syntax highlighting in Markdown Here (varies on Github).
But let's throw in a <b>tag</b>.

Tables

Tables aren't part of the core Markdown spec, but they are part of GFM. They are an easy way of adding tables to your email -- a task that would otherwise require copy-pasting from another application.

Source:

Colons can be used to align columns.

| Tables        | Are           | Cool  |
| ------------- |:-------------:| -----:|
| col 3 is      | right-aligned | $1600 |
| col 2 is      | centered      |   $12 |
| zebra stripes | are neat      |    $1 |

There must be at least 3 dashes separating each header cell.
The outer pipes (|) are optional, and you don't need to make the
raw Markdown line up prettily. You can also use inline Markdown.

Markdown | Less | Pretty
--- | --- | ---
*Still* | `renders` | **nicely**
1 | 2 | 3

Output:

Colons can be used to align columns.

Tables	Are	Cool
col 3 is	right-aligned	$1600
col 2 is	centered	$12
zebra stripes	are neat	$1

There must be at least 3 dashes separating each header cell. The outer pipes (|) are optional, and you don't need to make the raw Markdown line up prettily. You can also use inline Markdown.

Markdown	Less	Pretty
Still	`renders`	nicely
1	2	3

Blockquotes

Source:

> Blockquotes are very handy in email to emulate reply text.
> This line is part of the same quote.

Quote break.

> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote.

Output:

Blockquotes are very handy in email to emulate reply text. This line is part of the same quote.

Quote break.

This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can put Markdown into a blockquote.

Inline HTML

You can also use raw HTML in your Markdown, and it'll mostly work pretty well.

Source:

<dl>
  <dt>Definition list</dt>
  <dd>Is something people use sometimes.</dd>

  <dt>Markdown in HTML</dt>
  <dd>Does *not* work **very** well. Use HTML <em>tags</em>.</dd>
</dl>

Output:

Definition list: Is something people use sometimes.
Markdown in HTML: Does *not* work **very** well. Use HTML tags.

Horizontal Rule

Source:

Three or more...

---

Hyphens

***

Asterisks

___

Underscores

Output:

Three or more...

Hyphens

Asterisks

Underscores

Line Breaks

My basic recommendation for learning how line breaks work is to experiment and discover -- hit <Enter> once (i.e., insert one newline), then hit it twice (i.e., insert two newlines), see what happens. You'll soon learn to get what you want. "Markdown Toggle" is your friend.

Here are some things to try out:

Source:

Here's a line for us to start with.

This line is separated from the one above by two newlines, so it will be a *separate paragraph*.

This line is also a separate paragraph, but...
This line is only separated by a single newline, so it's a separate line in the *same paragraph*.

Output:

Here's a line for us to start with.

This line is separated from the one above by two newlines, so it will be a separate paragraph.

This line is also begins a separate paragraph, but... This line is only separated by a single newline, so it's a separate line in the same paragraph.

(Technical note: Markdown Here uses GFM line breaks, so there's no need to use MD's two-space line breaks.)

Local extensions

Source:

- [x] ticked checkboxes
- [ ] unticked check box
- Use ++insert++ text
- This is ^^superscript^^ stuff.
- This is ,,subscript,, stuff.
- This is ~~striketrough~~ text.
- This is ??marked?? text.

Output:

ticked checkboxes
unticked check box
Use insert text
This is ^superscript stuff.
This is _subscript stuff.
This is ~~striketrough~~ text.
This is marked text.

Diagrams

Source:

dot {
  graph NET {
    layout=neato

    edge [weight=2.0 fontsize=7]
    node [style=filled shape=box]

    node [fillcolor=white] kpnmodem
    node [fillcolor=lightblue] ngs1 ngs2 ngs3
    node [fillcolor=lightgreen] cctv_sw
    node [fillcolor=silver] cn4 iptv1 veraedge1 nd2 nd3 philtv
    node [fillcolor=yellow] wac1 wac2 wac3 owap1

    kpnmodem -- cn4 [label="v2" taillabel="p1" headlabel="p2"]
    kpnmodem -- iptv1 [label="v2 (p#7)" taillabel="p2"]
    kpnmodem -- veraedge1 [label="v2" taillabel="p4"]

    ngs1 -- cn4 [label="v1,3" taillabel="p10" headlabel="p0"]
    ngs1 -- ngs2 [label="v1,3 (p#4)" taillabel="p1" headlabel="p1"]
    ngs1 -- ngs3 [label="v1,3" taillabel="p8,9" headlabel="p1,8" penwidth=2.0]
    ngs1 -- cctv_sw [label="v3" taillabel="p7" headlabel="p5"]

    ngs3 -- nd2 [label="v1,3" taillabel="p7"]
    ngs3 -- nd3 [label="v1,3" taillabel="p8"]

    cctv_sw -- ipcam1 [label="v3" taillabel="p1"]
    cctv_sw -- ipcam2 [label="v3 (p#1)" taillabel="p2"]
    cctv_sw -- ipcam3 [label="v3 (p#5)" taillabel="p3"]
  }
}

\```aafigure {"foreground": "#ff0000"}
      +-----+   ^
      |     |   |
  --->+     +---o--->
      |     |   |
      +-----+   V
```

Output:

dot { graph NET { layout=neato

edge [weight=2.0 fontsize=7]
node [style=filled shape=box]

node [fillcolor=white] kpnmodem
node [fillcolor=lightblue] ngs1 ngs2 ngs3
node [fillcolor=lightgreen] cctv_sw
node [fillcolor=silver] cn4 iptv1 veraedge1 nd2 nd3 philtv
node [fillcolor=yellow] wac1 wac2 wac3 owap1

kpnmodem -- cn4 [label="v2" taillabel="p1" headlabel="p2"]
kpnmodem -- iptv1 [label="v2 (p#7)" taillabel="p2"]
kpnmodem -- veraedge1 [label="v2" taillabel="p4"]

ngs1 -- cn4 [label="v1,3" taillabel="p10" headlabel="p0"]
ngs1 -- ngs2 [label="v1,3 (p#4)" taillabel="p1" headlabel="p1"]
ngs1 -- ngs3 [label="v1,3" taillabel="p8,9" headlabel="p1,8" penwidth=2.0]
ngs1 -- cctv_sw [label="v3" taillabel="p7" headlabel="p5"]

ngs3 -- nd2 [label="v1,3" taillabel="p7"]
ngs3 -- nd3 [label="v1,3" taillabel="p8"]

cctv_sw -- ipcam1 [label="v3" taillabel="p1"]
cctv_sw -- ipcam2 [label="v3 (p#1)" taillabel="p2"]
cctv_sw -- ipcam3 [label="v3 (p#5)" taillabel="p3"]

} }

Others

#++ and #-- for headown
$include: file.md $

Nanowiki

2022-10-12T00:00:00+02:00

NanoWiki is a Wiki implementation based on picowiki.

I have been using SimpleNote for a number of years. It works pretty well but somehow I was looking for:

Ability to include and render nice asciiart pictures
Organizes articles in a folder structure.

So I was looking for a Wiki package that could either do this or be extended to do this.

Other features that I was looking for:

Use of markdown for markup, and be able to tweak the format as needed.
Editor that would syntax highlight the markdown syntax
Store data as simple files
Written in a language I am familiar with.
software generated network graphs (graphviz)

So, after looking at a number of packages, I opted for one that allthough did not have all features, but it was small enough and easily extendable.

PicoWiki is a very small Wiki implementation with a plugin architecture, so it is quite easy to extend. The downside of this is that the functionality in PicoWiki is quite limited. So I added the following features:

file management: create, delete, rename, modify, attach, etc.
hooks for access control
meta data support
Disabled code execution. This can be considered a "security" feature.
Support for byte ranges. This lets you stream video files directly from the wiki.
toggable, folder or document views.
theme support
Multiple file type handling

The default installations has the following plugins:

Emoji : Render emojis
HTML : HTML content handler
MarkDown : Markdown content handler
Includes : Include Wiki documents in another
Vars : Expand variables. Either from document metadata or from the NanoWiki config file.
WikiLinks : short hand for wiki links.

X728 kit for Raspberry Pi 4

2024-02-02T00:00:00+01:00

As part of my small project of movng my Z-Wave Hub to a Raspberry PI, I got an X728 kit. This has:

UPS controller board
- RTC circuit
- Battery and Power control board
Case
- Button
- Cooling fan
- Additional Battery holder

The case has holes for wall-mounting.

The Geekworm X728 kit is very easy to build. There is a video to show how to do this:

Build video

Otherwise, refer to the hardware guide here.

In my case, before the build, I took the disassembled case to measure the holes needed for wall-mounting the case. You need fairly small screws for this. I actually had to bend the case slighly for my screws to work.

Also, I set the jumper to automatic Power-on and a few cable ties to fix a USB Hub to the case.

To test the hardware I downloaded a 64-bit Raspberry OS Lite image from Raspberrypi.com and image an micro-SD card.

Boot the Raspberry OS. The first boot will resize the filesystems, so please wait. Also, it will let you configure the default user and password.
Enable the i2c function:
- sudo raspi-config
- Go to Interfacing Options -> I2C - Enable/Disable automatic loading.
- While you are at-it, you may also enable SSH.
Alternatively, you can a manual install by:
- Modify the config.txt in the /boot partition:
- Add at the end:
- [all]
- dtparam=i2c_arm=on
Install pre-requisites:
- sudo apt-get update
- sudo apt-get upgrade
- sudo apt-get -y install i2c-tools
- This is only needed for i2cdetect.
Reboot the system.
Check if the hardware is detected:
- sudo i2cdetect -y 1
- #36 - the address of the battery fuel gauging chip
- #68 - the address of the RTC chip
- Different x728 versions may have different values. Mine used these values.

I personally did not like the example software. This can be found in github. Specifically, the shutdown functionality seemed to have race-conditions.

However, using it is not that complicated. So I wrote my own software, but you can do your own thing:

RTC functionality

The RTC functionality is supported by the Raspberry OS kernel. You need to enable the i2c functionality in /boot/config.txt by adding the line:

dtparam=i2c_arm=on

With that enabled, you need to add the kernel modules:

i2c-dev
rtc-ds1307

You need to enable it in the bus:

echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device

From then on, you can use the standard hwclock command.

These can be added to rc.local which is how the sample code does. I prefer to do this from a script run from systemd unit file:

# file: /etc/systemd/system/x728clock.service

[Unit]
Description=Restore / save X728 clock
DefaultDependencies=no
Before=sysinit.target shutdown.target
Conflicts=shutdown.target

[Service]
ExecStart=/etc/x728/clock.sh start
ExecStop=/etc/x728/clock.sh stop
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=sysinit.target

After saving this file and creating a /etc/x728/clock.sh script you can:

systemctl daemon-reload
systemctl enable x728clock
systemctl start x728clock
systemctl disable fake-hwclock
systemctl stop fake-hwclock

GPIO assignments

Pin	Function	Direction	Comment
#6	PLD	in	1: A/C lost, 0: A/C OK
#5	5hutdown	in	Sense button press
#12	Boot	out	Control SW/HW controlled button
#20	Buzzer	out
#26	Button	out	Simulate power button press

Reading PLD detects if the A/C power is available or not. If it reads 1 A/C power was lost. 0 if A/C power is available.
Buzzer if set to 1 it will sound a rather loud beep. 0 for off.
Button simulates pressing the hardware Off button. If you set Button to 1 for 6 seconds, the system will poweroff.
Shutdown is used to read the status of the hardware button. This works only if Boot is set to 1. Otherwise, Shutdown doesn't seem to work. When Boot is set to 1, it would read 1 if pressed, 0 if released. Weirdly enough, the Shutdown button is not very sensitive. It takes about 3 seconds to register the button press. The button release takes bout 50 seconds to detect.

GPIO programming

Programming GPIO is quite easy. It can be done from shell scripting using the sys file-system.

gpioIO() {
  local pin=$1
  if [ $# -eq 1 ] ; then
    cat /sys/class/gpio/gpio$pin/value
  else
    echo "$2" > /sys/class/gpio/gpio$pin/value
  fi
}

gpioInit() {
  local name="$1" pin="$2" dir="$3"

  [ ! -d /sys/class/gpio/gpio$pin ] && echo "$pin" > /sys/class/gpio/export
  echo $dir > /sys/class/gpio/gpio$pin/direction

  eval "gpio${name}() { gpioIO $pin \"\$@\" ; }"
}
ticks() {
  echo $(date +%s)$(date +%N | cut -c-2)
}

beep() {
  local len="$1" ; shift

  gpioBUZZER 1
  sleep "$len"
  gpioBUZZER 0

  [ $# -eq 0 ] && return

  local repeat="$1" idle
  [ $# -gt 1 ] && idle="$2" || idle="$len"

  while [ $repeat -gt 1 ]
  do
    repeat=$(expr $repeat - 1)
    sleep "$idle"
    gpioBUZZER 1
    sleep "$len"
    gpioBUZZER 0
  done
}

gpioInit SHUTDOWN 5 in
gpioInit PLD 6 in
gpioInit BOOT 12 out
gpioInit BUZZER 20 out
gpioInit BUTTON 26 out

Afterwards, you can just:

gpio[PIN] to read, i.e:
- gpioSHUTDOWN
- gpioPLD
gpio[PIN] {1|0} to write i.e.:
- gpioBOOT 1
- gpioBUZER 0

Reading Battery status

You can read battery voltage and battery charge from the smbus. To read the smbus I am using an example from [rpi-examples]((https://github.com/leon-anavi/rpi-examples/tree/master/BMP180/c). Specifically, I am only using the files smbus.c and smbus.h from that repository.

The code outline is:

open /dev/i2c-1 in read/write mode.
ioctl(fd, I2C_SLAVE, I2C_ADDRESS) where I2C_ADDRESS = 0x36.
From smbus.c read i2c_smbus_read_word_data(fd, address), and byte swap.
Voltage can be read from address 2:
- Voltage = (swapped) * 1.25 / 1000 / 16
Battery charge can be read from address 4:
- Battery = (swapped) / 256

The code to do this can be found on github.

A precompiled 64bit static binary can be found there too.

Power down

The x728 will turn off power by holding down the power button for around 6 seconds. Doing this will skip the shutdown process. Also, if you execute the poweroff command, the Raspberry Pi will shutdown but power will not go OFF until you hold the power button for 6 seconds.

For this to work properly, I am adding this small script to /lib/systemd/system-shutdown/gpio-poweroff:

#!/bin/sh
#
# file: /lib/systemd/system-shutdown/gpio-poweroff
# $1 will be either "halt", "poweroff", "reboot" or "kexec"
#

BUTTON=26

op_poweroff() {
  echo $BUTTON > /sys/class/gpio/export
  echo out > /sys/class/gpio/gpio$BUTTON/direction
  echo 1 > /sys/class/gpio/gpio$BUTTON/value
  sync;sync;sync
  sleep 7
  echo 0 > /sys/class/gpio/gpio$BUTTON/value
  sleep 3
}

case "$1" in
  poweroff) op_poweroff ;;
esac

This hooks into systemd's shutdown target and uses the BUTTON pin to simulate holding the button for 6 seconds to force the UPS board to power off.

script

UPS management

In addition, I wrote a small script to:

graceful shutdown when power button is pressed.
- hold the power button, after approximately 3 seconds, you will hear 2 beeps. YOu can release the power button then. The system will do a graceful shutdown and power down.
When A/C power is lost:
- If battery status can not be determined, the system will do a graceful powerdown.
- If battery status can be read, it will beep once every 60 seconds until power is restored.
- If battery is low, it will do a graceful powerdown.
Events are written to /dev/kmsg, so they could be forwarded to a syslog server.

The files to do this are:

Home Assistant

I am using the X728 kit for creating a Home Assistant installation. Home Assistant has a "managed Operating System" called Home Assistant OS which is a mostly read-only installation. This makes it complicated to add your own "low-level" customizations. To include these scripts and making them persistant accross upgrades I am hooking into the RAUC OTA upgrade subsystem.

For that, I hook up to the System handlers which makes use of a Handler Interface.

With these scripts, I am able to move the customizations from a previous image to the new upgraded image.

For this, I have a script haos-x728 that injects the customizations into a new installation image. This script also modifies /etc/rauc/system.conf so that the customization handler is called during an upgrade.

The post-install handler re-adds the handler to /etc/rauc/system.conf and copies the necessary files to the updated image.

The customization scripts are not X728 specific, and essentially lets you copy all the files in a directory to the custom image. As such, I am using to not only inject these X728 scripts, but also the vcgencmd and a muninlite agent. Also, the dependant binaries for the post-install handler are injected in the same way.

For the post-install handler to work properly you need to copy binaries for:

gensquashfs
sqfs2tar

And dependant shared libraries (that are not part of the Home Assistant OS image:

liblz4.so.1
liblz4.so.1.9.3
liblzma.so.5
liblzma.so.5.2.5
liblzo2.so.2
liblzo2.so.2.0.0
libselinux.so.1
libsquashfs.so.1
libsquashfs.so.1.1.0
libzstd.so.1
libzstd.so.1.4.8

The simplest way to get these is to install squashfs-tools-ng on standard Raspberry PI OS and copy those files from there.

The full set of customization files that I am using can be found here.

It contains:

RAUC handler:

lib/rauc/post-install

squashfs-tools-ng (dependancy to RAUC handler)

bin/gensquashfs
bin/sqfs2tar
lib/liblz4.so.1
lib/liblz4.so.1.9.3
lib/liblzma.so.5
lib/liblzma.so.5.2.5
lib/liblzo2.so.2
lib/liblzo2.so.2.0.0
lib/libselinux.so.1
lib/libsquashfs.so.1
lib/libsquashfs.so.1.1.0
lib/libzstd.so.1
lib/libzstd.so.1.4.8

Actual X728 support scripts.

bin/x728batt
etc/x728/clock.sh
etc/x728/upsmon.sh
etc/systemd/system/x728clock.service
etc/systemd/system/x728ups.service
etc/systemd/system/sysinit.target.wants/x728clock.service
etc/systemd/system/multi-user.target.wants/x728ups.service
lib/systemd/system-shutdown/gpio-poweroff

Munin node:

bin/munin-node
etc/systemd/system/sockets.target.wants/munin-node.socket
etc/systemd/system/munin-node.socket
etc/systemd/system/munin-node@.service
etc/muninlite.conf

cuylib

2024-02-02T00:00:00+01:00

This is a tiny library to implement Web server embedded editor.

You can find it in github.

Can be used either from haserl or directly from a shell script.

Features:

Uses codemirror
Escaped HTML entities (html_enc)
Decode URL escaping (url_decode)
Read POST form data (post_data)
Parse QUERY_STRING (query_string and query_string_raw)
Render HTML and Markdown documents with pre-processing (cuy_render)

Support functions:

codemirror_link : Configured URL to where you can find codemirror files.
html_msg : generate a HTML response
html_enc : Encode special HTML characters
url_decode : Decode URL encoded strings
post_data : Read data posted using a HTML POST request.
query_string_raw : parse HTML query parameters.
query_string : part HTML query paramters and also decodes URL encoding.

Editor components:

cuy_header : snippet of code for the html document header.
cuy_editform : snippet of code for genereting the html editor form
cuy_editarea : snippet of code to bind codemirror to a text area
cuy_savecb : snippet for save command callback
cuy_render : convert content into suitable HTML markup

Main editing entry point

cuy_editapp : A full editing page

Photoprism

2022-09-20T00:00:00+02:00

photoprism is a web based photo management application.

From its website:

PhotoPrism® is an AI-Powered Photos App for the Decentralized Web. It makes use of the latest technologies to tag and find pictures automatically without getting in your way. You can run it at home, on a private server, or in the cloud.

Features:

Browse all your photos and videos without worrying about RAW conversion, duplicates or video formats
search: Easily find specific pictures using powerful search filters
places: Includes four high-resolution world maps to bring back the memories of your favorite trips
Play Live Photos™ by hovering over them in albums and search results
people: Recognizes the faces of your family and friends
Automatic classification of pictures based on their content and location

What I found is that it does most things automatically.

My implementation:

I use a docker instance for photoprism linked to a mysql database. (actually mariadb).
- database setup
This photoprism instance is set as AUTH=public, so no authentication. Users on my home network can connect directly. From the Internet, I am using an nginx reverse proxy. This reverse proxy requires authentication.
For photo sharing, a different photoprism instance is used pointing to the same file system and mysql as the main instance. This instance however has AUTH=password enabled, so only shared links can be visited.
For uploading photos from my iPhone, I am using PhotoSync. This can be configured to upload directly to photoprism using WebDav. (See instructions for syncing with mobile devices.) On the other hand, it is complicated for me due to permission problems in my home network. For that reason, I am using a small container running a sshd daemon and I sync to that small container using sftp protocol.

Some issues I found:

face recognition is a bit wonky. Specially for kid's faces.
some tweaks may be needed to get things to display just right.
running using the embedded sqlite database won't scale beyond a handful of photos.
I have a photo library of 400GB. It took a long time to index.

Before I would copy files from my camera to my server. In the case of video files, a re-encoding step was needed. With todays smartphones this is not needed. The smartphone can sync directly to the server and files are already compressed with a good enough CODEC.

SupervisorUI MF

2022-09-01T00:00:00+02:00

In a previous article, I updated a supervisorui project to work for me.

This updated version supervisorui-redone is essentially a PHP application which is a different approach from the original supervisorui project which was more of a JavaScript application with some helper functionality implemented in PHP.

As such, I figured that I could probably fix the supervisorui code base eliminating the Silex library, but keeping most of the JavaScript framework in place. This resulted in my SupervisorUI-mf dashboard.

This maintains JavaScript framework and simply removes the Silex dependancy.

As before, the Incutio XML-RPC Library was updated so that it works with PHP8.

In addition I added/fixed the following functionality:

Added links to supervisor built-in web UI.
Added links to restart and reload config supervisor daemons.
Fixed the updateServers functionality, so that services status gets updated every 30 seconds without having to reload the page. Similarly, starting/stopping services do not need to reload the page as with supervisorui-redone.
Added the ability to configure port's in the server IP specifications.

So the result is an updated supervisorui which works at least for me.

Note that I am still keeping supervisorui-redone because it is more static than SupervisorUI-mf which means that supervisorui-redone works better with High Latency (high ping time) connections.

Supervisorui REDONE

2022-08-25T00:00:00+02:00

Currently I am using docker containers to deploy applications. A number of those containers make use of supervisord for managing processes. While supervisord itself comes with a UI, it is unhandy for me because each container is its own supervisord instance.

So I was interested in some software that would let me manage multiple supervisord instances in a single page. Turns out that there are several tools lsted here. Unfortunately none of these worked for me.

So with the power of open source I rolled my own. I based mine on supervisorui. Unfortunately, supervisorui hasn't been updated in 10 years. What is worse it depends on a Silex library that doesn't seem to exist anymore.

So I ripped out a bunch of complex functionality and recreated it as supervisorui-redone.

It is indeed a quick and dirty implementation, unlike the original supervisorui that has heavier JavaScript dependencies. supervisorui is mostly a JavaScript applications and simply uses php for backend access to the supervisord. I assume that it would make it more interactive.

My re-worked version is basically a PHP application.

As such, I removed the dependancies on

Silex
Twitter Bootstrap javascript, only CSS is in use.
jQuery
Backbone.js

Also fixed Incutio XML-RPC Library so that it works with PHP8.

voidlinux virtualization

2024-02-02T00:00:00+01:00

This recipe is for setting virtualization on a voidlinux desktop.

Use this setup script to set things up on void linux.

Connecting to libvirtd

Note that virsh and virt-manager commands connect to different libvirtd sessions by defauult.

virsh defaults to qemu:///session while virt-manager to qemu:///system.

It is better to use qemu:///system as qemu:///session does not seem to see all available resources.

To force virsh to connect to the right session you can use commands such as:

virsh --connect qemu:///system net-list
virsh --connect qemu:///system pool-list

Create network

Set-up a bridge for internal networking using NetworkManager. See this article for reference.

Go to NetworkManager menu and use the Edit network connections applet:

add new bridge connection
give a suitable name
disable IPv4 and IPv6
everything can be left as default.

Import images

This needs to be done on CLI (as the GUI doesn't seem to allow this)

virsh --connect qemu:///system vol-create-as $pool $vol 32k --format $format
virsh --connect qemu:///system vol-upload $vol $file

For format, use raw for iso, qcow2 for actual drives.

Setup VM

Just create VM as normal.

virt-manager create Vm wizard

manual install
alpine linux
mem: depends
Create storage (4GB is enough)
name. Customize configuration. Network use NAT.
Add CDROM, make readonly and shareable
Add Network connected to internal bridge.
Add boot device.
Add shared filesystem:

driver: virtio-9p
source path: /var/lib/libvirt/filesystems/shared
target path: /shared

Prepare system

create filesystem

mkfs.vfat /dev/vda
apk add syslinux
syslinux /dev/vda

copy media

mount -t vfat /dev/vda /mnt
cp -av /media/cdrom/. /mnt
edit /mnt/boot/syslinux/syslinux.cfg add: dom0_mem=1024M
umount /mnt

remove cdrom

power off
remove cdrom
change boot options

re-start system
setup-alpine

enter fqdn
set interface eth0 to dhcp

Mount shared fs:

mount -t 9p -o trans=virtio /shared /shared
fstab
/sharepoint /share 9p trans=virtio,version=9p2000.L,rw 0 0

Thin provisioning

Create an overlay file like so:

qemu-img create -b ubuntu-20.04-server-cloudimg-amd64-disk-kvm.img -F qcow2 -f qcow2 guest-1.qcow2

A couple of useful sites for development

2023-04-03T00:00:00+02:00

Unicode

Can be useful for looking up unicode code points. Particularly useful for looking up accented characters. Another interesting use is for UI graphics characters.

Unicode search

Another site to search for unicode characters.

gist-hkan

This can be used to look-up emojis and shortcode for emojis.

color picker

Let you visually select a color so that you can paste its definition in your HTML or CSS documents

color names

This is a list of "standard" color names.

Keyboard Mouse control

2022-03-03T00:00:00+01:00

This comes in handy when working at a colo or someplace where you don't have a mouse and then find yourself needing to use X11. Press the following key combo:

Ctrl-Shift-Numlock

Now you can control the mouse pointer using the number pad. The key bindings are:

Move the mouse pointer

7, 8, 9 are the up directions
4, 6 are left and right
1, 2, 3 are the down directions

To control the mouse buttons

/ selects the left mouse button
* selects the middle mouse button
- selects the right mouse button

This only selects the mouse button but does not press it. To actually use the mouse button:

5 mouse click
+ double mouse click
0 to press and hold the mouse button (e.g. for dragging)
. to release the currently mouse button.

For example, to do a mouse-click with the middle button you first press the * key and then 5 to click.

flatpak

2022-03-03T00:00:00+01:00

Flatpak is a utility for software deployment and package management for Linux. It is advertised as offering a sandbox environment in which users can run application software in isolation from the rest of the system. Flatpak was developed as part of the freedesktop.org project and was originally called xdg-app.

Snap vs Flatpak

Snaps and Flatpaks are often compared to each other because they both make it super easy for Linux users to get the latest versions of desktop applications. If a Linux user wants to install the latest version of apps like Slack, Krita or Blender, either tool will work just fine. There is one fundamental difference between Snaps and Flatpaks, however. While both are systems for distributing Linux apps, snap is also a tool to build Linux Distributions.

Flatpak is designed to install and update “apps”; user-facing software such as video editors, chat programs and more. Your operating system, however, contains a lot more software than apps. It contains a kernel, printer drivers, audio subsystems and more. While Flatpak assumes this software is installed using a traditional package manager, snaps can install anything. These are some examples.

There is current work ongoing to put the entire Linux printing stack inside of a snap. This has the advantage that printer drivers can be updated independently from the operating system. Once this work is complete, every single Ubuntu version will be able to use the latest printer drivers. Trying to use new printers on old Linux distributions can be very frustrating, and installing newer printer drivers can be risky. Having the printing stack in a snap will solve this issue.
A few years ago, Ubuntu drastically changed the system theme. When the "CommuniTheme" initiative started, they wanted an easy way to make the latest updates of the theme available to users immediately. Normally, a system theme is shipped together with the distro, so users do not get theme updates after the distro releases. For “CommuniTheme”, however, they fixed this by putting the system theme inside of a snap. Because of this, users got updates to their theme every day, instead of every 6 months. This is again not something Flatpak was built for. Flatpak applications can update their own theme, but it is not possible to ship the system theme as a Flatpak. This is because Flatpak was designed for distributing apps, not building an entire Linux distribution.
Even the Linux kernel, the most fundamental part of a Linux distribution, can be put in a snap. This is used a lot for IoT devices such as routers and satellites. The impact of a broken kernel update is catastrophic if you require a rocket in order to plug a USB stick into the device. Snaps allow these devices to safely update their kernel and automatically roll back if something goes wrong during the process.

As a result, it’s possible to build an entire operating system using only snaps, which is exactly what Ubuntu Core is.

Flatpak was designed to give developers an easy way to bring their apps directly to users, and it does that job very well. The focused approach of Flatpak even has a big advantage: it’s a lot easier for a distribution to integrate with Flatpak because it does a lot less. The tradeoff is that it only provides app distribution; it doesn’t solve the issues of distributing entire operating systems. Fedora Silverblue, for example, creates an immutable desktop operating system by using Flatpak for app distribution and OSTree for distributing the OS itself.

Unfortunately, for my choice Linux distro (void linux), only Flatpak is available.

Install Flatpak

To install Flatpak, run the following:

sudo xbps-install -S flatpak

After installation

Once flatpak is installed, you can choose to install software system wide or on an per-user basis. The default is to install system wide which requires root priviledge (or sudo). You may also specify --system if you want to force system wide install.

To manage software on a per-user basis, use the --user option.

The best place to find software for Flatpak is flathub. You can browse the catalogue in https://flathub.org.

If you want to start installing software you need to first add the remote:

sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Or for per-user installation:

flatpak --user remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Afterwards you can install software:

sudo flatpak install pkg-id ...

Or per-user:

flatpak --user install pkg-id ...

Interesting flatpaks:

com.spotify.Client : Spotify desktop client
dev.alextren.Spot : Alternative spotify client
org.chromium.Chromium : Open source chrome.
org.mozilla.firefox : Firefox Web browser
com.valvesoftware.Steam : Steam Launcher
com.simplenote.Simplenote : Simple Note
com.mojang.Minecraft : Minecraft launcher

Basic commands

Search:

flatpak search gimp

Install:

sudo flatpak install flathub org.gimp.GIMP

Or:

flatpak --user install flathub org.gimp.GIMP

Note that the current version of flatpak will do a search, so you don't have to specify the ID.

Running applications:

flatpak run org.gimp.GIMP

This will automatically determine if the application was installed system-wide or on per-user basis. You can use the --system or --user flags to specify which to use if it was installed multiple times, otherwise the system-wide installation will be used.

Updating:

sudo flatpak update

Or:

flatpak --user update

This will update all installed applications.

List installed applications:

flatpak list

This will list applications and runtimes indicating if it was installed system wide or per-user. To list only applications:

flatpak list --app

Removing applications:

sudo flatpak uninstall org.gimp.GIMP

flatpak --user uninstall org.gimp.GIMP

Keep in mind that uninstalling applications will not delete files in your $HOME directory. These are in $HOME/.var/app/$PKGID.

You need to manually delete these files.

Managing repositories

List remotes:

flatpak remotes

This gives a list of the existing remotes that have been added. The list indicates whether each remote has been added per-user or system-wide.

Add a remote:

sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Or:

flatpak --user remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Remove a remote:

sudo flatpak remote-delete flathub

Or:

flatpak --user remote-delete flathub

Troubleshooting

Flatpak has a few commands that can help you to get things working again when something goes wrong.

To remove runtimes and extensions that are not used by installed applications, use:

sudo flatpak uninstall --unused

Or:

flatpak --user uninstall --unused

To fix inconsistencies with your local installation, use:

sudo flatpak repair

Or:

flatpak --user repair

Flatpak also has a number of commands to manage the portal permissions of installed apps. To reset all portal permissions for an app, use flatpak permission-reset:

sudo flatpak permission-reset org.gimp.GIMP

flatpak --user permission-reset org.gimp.GIMP

To find out what changes have been made to your Flatpak installation over time, you can take a look at the logs (since 1.2):

flatpak history

However, this requires libsystemd which is not used in void linux.

My git release script

2022-01-16T00:00:00+01:00

I always had issues remembering how to create releases. So in order to standardise things, I wrote this script:

ghrelease

So whenever I am ready to release I would then just issue the command:

./ghrelease vX.Y.Z

Pre-requisistes:

You obviously need git. But also you would need github-cli.

Your repository must also be clean, without any pending commits.

You must be on the default branch (usually main or master), unless doing a pre-release.

Optionally, you may have a wfscripts/checks directory containing checking scripts.

What happens on release

remote --tags will be synchronised with local tags.
if a tag of the same name already exists then, release will be stopped.
check if we are on the default branch (unless pre-release)
check if there are any uncomitted changes.
If available, wfscripts/checks is run using run-parts
Will create release notes based on log entries since the last release (previous annotated tagged commit)
VERSION or version.h is updated and comitted.
A new annotated tag is created.
Commits and tags are push to remote (origin).
Release is created using github-cli.

Pre-releases

You can create pre-releases. These do not have to be on the default branch. To do this use the --rc (Release candidate) option:

./ghrelease vX.Y.Z-rcN

This will create a release in github but tag it as pre-release.

After release, you may delete all pre-release candidates:

./ghrelease --purge

Meta Database

2022-01-10T00:00:00+01:00

So I was looking for a way to version control database schemas, but I never found something that worked for me. I found all these options that never seem to match what I wanted:

At the end, I finally began doing the following.

The first schema release, I would create a file called:

init-1.0.sql

This would contain all the sql statements needed to initialize the database. To indicate this is the current version I would create a symlink to it:

init.sql -> init-1.0.sql

The next schema update I would create file with the commands to transform the schema. i.e. additional CREATE DATABASE or ALTER TABLE etc.

upgrade-1.1.sql

So actually, the upgrade file would be the working version used for development. For release then I would create a new init file. Either doing it manually or by doing:

cat init-1.0.sql upgrade-1.1.sql > init-1.1.sql

To indicate that this is the current version, the symlink would be updated:

init.sql -> init-1.1.sql

Final notes

This would end-up with a history of schema changes and scripts that could be used to upgrade the database schema from any version to any later version.

lnbin

2024-06-03T00:00:00+02:00

This is my lnbin script.

This is a program for managing symlink in a /usr/local/bin directory. It is similar to stow, lndir, cleanlinks and others.

The approach used by lnbin is based on Stow, and it is to install each into its own tree, then use symbolic links to make its bin directory, so that the command can be in the executable path.

When run, lnbin examines packages in pkgs-dir and the target directory (see OPTIONS), adding or removing links as needed.

Sample usage:

pkg installation

The standard way to use lnbin is:

download source package
build and install package

# extract archive
tar zxvf archive-x.x.tar.gz
cd archive-x.x
# GNU autoconf
./configure --prefix="/usr/local/pkgs/archive-x.x"
make
# Package installation
make install
# ... or ...
make install DESTDIR=/usr/local/pkgs/archive-x.x

update symlinks in /usr/local/bin

cd /usr/local/bin
lnbin -v -x ../pkgs

This will add the new links (and also remove/update obsolete/changed links)

Removing packages

rm -rf /usr/local/pkgs/archive-x.x
cd /usr/local/bin
lnbin -v -x ../pkgs

Updating symlinks (after upgrade)

cd /usr/local/bin
lnbin -v -x ../pkgs

This will add new links and/or remove obsolete links

Handling additional files

For packages that install additional files like man pages or desktop files you can use the commands:

lnbin -v -x -s share/man/man1 -t /usr/local/share/man/man1 ../../../pkgs
lnbin -v -x -s share/applications -t /usr/local/share/applications ../../pkgs

References

There are a number of packages that do similar things. The main attractiveness of this one is that it is a /bin/sh script intended to have low dependancies.

Other options:

I chose not to use lt because while written in /bin/sh, I wanted a script that could use relative links instead of absolute links.

Linux Icons

2022-01-16T00:00:00+01:00

A quick note on how to add icons to menus in a Linux desktop.

Create the icon image in: /usr/share/pixmaps.
- png and svg (and maybe others) are supported.
- 24x24 seems to be a good size for menus.
You need to create a .desktop file in /usr/share/applications.
- Desktop menu specification
- Registered categories

User desktop files:

These are located in:
- $HOME/.local/share/applications
Icons can be found here:
- $HOME/.local/icons
- I am not sure about this one
Autostart files:
- $HOME/.config/autostart.

nas ops cmd

2024-02-02T00:00:00+01:00

This is my op script.

This is stupidly simple script to elevate priviledges in order to manage NFS shares on my QNAP NAS.

The idea is that NFS shares do squash-root so admin access is disallowed through NFS. This gives a convenient way to issue root level commands without using NFS but instead use ssh (and ssh authentication) to do this, which should provide stronger security.

This script makes the following assumptions:

the user has a ssh-key with admin access on the NAS.
NFS is mounted using autofs and is on the /net virtual folder.

The way it works is that it uses the current working directory when the command is launch. It then resolves any symlink path and check is the directory is in the /net/ virtual folder so the NFS server is the second component of the path.

Linux stuff

2022-01-10T00:00:00+01:00

Sudoers

Since sudo v1.9, it is possible to use the following statements:

#includedir
@includedir

This is useful better for adding sudo rules rather than modifying the /etc/sudoers file.

Make sure that the includedir statement is the LAST entry in /etc/sudoers and the files in the directory:

names do not contain . (dots)
ownership to root:root and permission is set to 0440.

Graphviz markdown extensions

2022-01-10T00:00:00+01:00

I have enabled several extensions to my pelican website.

One that I wanted to include was graphviz. So, I searched for one and while I found a few, they somehow, did not work for me.

So I wrote my own: mdx_graphviz.

It is quite straight forward. You just need to create blocks:

dot {
    digraph G {
        rankdir=LR
        Earth [peripheries=2]
        Mars
        Earth -> Mars
    }
}

You can use this Graphviz Visual Editor for a more interactive approach.

Pelican Test page

2024-02-02T00:00:00+01:00

shortcodes
mytags
Drawings
- aafigure
- blockdiag
mdx_include
GFM style check lists
my mdx variables

This page is used for testing some pelican and markdown extensions I added.

shortcodes

OK, this is awkward... I am not sure if this is needed.

mytags

Using ~~del~~ and ins.

test mark tags. How about E=mc² and H₂O.

Drawings

aafigure

blockdiag

Block diagram

blockdiag { A -> B -> C -> D; A -> E -> F -> G; }

mdx_include

{! nginx_mod_authrequest/auth1.py !}

GFM style check lists

my mdx variables

We use snippets as an example.

How we handle missing ${VARS}.

DVTM

2022-03-03T00:00:00+01:00

The other day I found dvtm. Looking at it, it looks very nice. It appeals to me because I am particularly fond of text user interfaces.

At the end I choose not to use it because:

terminal support was less than 100% useful.
At the end of the day using the mouse is just more convenient.
It is not as ubiquitous as for example screen. So it is easier to just use screen that can be set-up much more easily.
most of the time I am already on a window session, so there is not that many opportunities to use this.

dvtm Cheat Sheet

This is a simple cheat sheet for dvtm.

This uses the default mod key: C-g.

Key Seq	Function
C-g M	Toggle mouse mode
C-g Enter	Zoom current window to master area
C-g h	Shrink master area
C-g l	Enlarged master area
C-g Spc	Toggle layout (vertical stack, bottom stack, grid, full screen)
C-g f	Vertical stack
C-g b	Bottom stack
C-g g	Grid layout
C-g m	Full screen
C-g 0	view all windows
C-g c	Create window
C-g j	Focus on next window
C-g k	Focus on previous window
C-g m	Minimize window
C-g s	Toggle status bar
C-g 1-9	Focus on window
C-g TAB	Toggle focus (last window)
C-g q	Quit
C-g C-l	Redraw
C-g r	Redraw
C-g PgUp	Scroll back
C-g PgDn	Scroll Fwd
C-g C-g	Send C-g

Key Seq	Additional functions
C-g C	Create window with current directory
C-g J	Focus on next window "m"
C-g K	Focus on prev window ?
C-g i	Increase # windows in master area
C-g d	decrease # windows in master area
C-g s	Toggle status bar position (top or bottom)

Other functions that I don't understand or haven't configured:

tagging
copymode
status bar

Migration to Pelican

2021-12-25T00:00:00+01:00

Finally got fedup with github pages and its jekyll static site generator. Essentially things would break without any particular reason and there would be nearly no way to tell what went wrong. I addition, it was not easy to test changes before making them public.

So I switched to pelican, essentially because it was a static site generator that is part of the void software repository.

I don't really like it that much as its documentation is not very good. But eventually I got it to work.

I was able to get one of their public templates to work and tweaked to match my preferences.

I also was able to add:

automatic tag generation
- this is done before processing input files. i.e. a script reads existing content and modifies files as needed with automatic tags.
sitemap generator
- this is done as post-processing stage. A script reads generated html files and generates the sitemap accordingly.

The most useful feature I found is its that I can preview changes before commiting them.

Storing secrets in git

2022-01-04T00:00:00+01:00

So I gave into the temptation to store "secret" data into a git repository. Of course, to keep things safer, I chose to use an encryption tool. So I tested:

git-secret

So, I tested git-secret. It seems to work but was in my opinion cumbersome.

Furthermore, the version I tested, which was the one from the void has a bug whereby adding files for encryption would update the .gitignore file but would forget to put an EOL at the end of the file.

The main issue I have is that you need to explicitly issue the command:

git-secret hide

To hide files. Alternatively you can include this in your pre-commit hook, but that brings its own issues along.

Overall it was not the best experience.

git-crypt

At the end I opted for git-crypt, which is more seamless and requires less user interaction for it to work.

git-crypt mini howto

I installed git-crypt using my distro package installation command.

Initialize an existing git repo and export encryption key:

cd repo
git-crypt init
git-crypt export /path/to/key

The exported key now needs to be shared between all the repo users. For example can be saved into a secret variable in the CI/CD pipeline system.

Select the files that need to be protected by creating a .gitattributes file:

secretfile filter=git-crypt diff=git-crypt
*.key filter=git-crypt diff=git-crypt
secretdir/** filter=git-crypt diff=git-crypt

Like a .gitignore file, it can match wildcards and should be checked into the repository. Make sure you don't accidentally encrypt the .gitattributes file itself (or other git files like .gitignore or .gitmodules).

NOTE Make sure your .gitattributes rules are in place before you add sensitive files, or those files won't be encrypted!

After cloning a repository with encrypted files, unlock with the secret key:

git-crypt unlock /path/to/key

That's all you need to do - after git-crypt is set up (either with git-crypt init or git-crypt unlock), you can use git normally - encryption and decryption happen transparently.

Verifying that git-crypt is working

The simplest way:
- git crypt status
The native way:
- git check-attr -a -- <path>
Checking object hashes (these shouldn't match):
- git hash-object <path>
- cat <path> | git hash-object --stdin

Enable syslog with void

2021-12-25T00:00:00+01:00

In void Linux, the default is without logging. Most cases it is OK for desktop use.

If you want to enable syslog service in void, you need to install:

socklog-void

Also to let your user have access to the logs, use:

usermod -aG socklog <your-username>

Because I like to have just a single directory for everything and use grep, I do the following:

rm -rf /var/log/socklog/?*
mkdir /var/log/socklog/everything
ln -s socklog/everything/current /var/log/messages.log

Create the file /var/log/socklog/everything/config with these contents:

+*
u<syslog-server-ip>:514

Enable daemons...

ln -s /etc/sv/socklog-unix /var/service/
ln -s /etc/sv/nanoklogd /var/service/

Reload svlogd (if it was already running)

killall -1 svlogd

Reference:

voidlinux logging

Stupid SSL tricks

2021-12-25T00:00:00+01:00

Some hints and tips foor doing SSL related things:

Netcat for SSL

This command lets you connect to a SSL server (a-la netcat):

cat request.txt | openssl s_client -connect server:443

Creating self-signed certificates

This is a single command to generate a self-signed certificate:

openssl req -new \
      -newkey rsa:4096 \
      -days 365 \
      -nodes -x509 \
      -subj "/C=NL/ST=ZH/L=Den Haag/O=HomeBase/CN=$fqdn" \
      -keyout $ca_root/$fqdn/$fqdn.key \
      -out $ca_root/$fqdn/$fqdn.cer

This is unlike other recipes where you create a csr and key first and then create the certificate.

Checking and verifying certificates

Check certificate
- openssl x509 -in server.crt -text -noout
Check SSL key and verify consistency
- openssl rsa -in server.key -check
Check CSR and print CSR data
- openssl req -text -noout -verify -in server.csr
Verify that certificate and key matches:
- openssl x509 -noout -modulus -in server.crt| openssl md5
- openssl rsa -noout -modulus -in server.key| openssl md5
Check SSL Certificate expiration date
- openssl x509 -dates -noout -in hydssl.cer

Check SSL connection

Tests connectivity to an HTTPS service:
- openssl s_client -connect <hostname>:443
Prints all certificates in the certificate chain presented by the SSL service. Useful when troubleshooting missing intermediate CA certificate issues.
- openssl s_client -connect <hostname>:<port> -showcerts
Forces TLSv1 and DTLSv1.
- openssl s_client -connect <hostname>:<port> -tls1
- openssl s_client -connect <hostname>:<port> -dtls1
Forces a specific cipher. This option is useful in testing enabled SSL ciphers. Use the openssl ciphers command to see a list of available ciphers for OpenSSL.
- openssl s_client -connect <hostname>:<port> -cipher DHE-RSA-AES256-SHA

For troubleshooting connection and SSL handshake problems, see the following:

If there is a connection problem reaching the domain, the OpenSSL s_client -connect command waits until a timeout occurs and prints an error, such as connect: Operation timed out.
If you use the OpenSSL client to connect to a non-SSL service, the client connects but the SSL handshake doesn't happen. CONNECTED (00000003) prints as soon as a socket opens, but the client waits until a timeout occurs and prints an error message, such as 44356:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/SourceCache/OpenSSL098/OpenSSL098-47.1/src/ssl/s23_lib.c:182:.

After disabling a weak cipher, you can verify if it has been disabled or not with the following command.

openssl s_client -connect google.com:443 -cipher EXP-RC4-MD5

Check SSL certificates on a remote server:

Check who has issued the SSL certificate:
- echo | openssl s_client -servername howtouselinux.com -connect howtouselinux.com:443 2>/dev/null | openssl x509 -noout -issuer
Check whom the SSL certificate is issued to:
- echo | openssl s_client -servername howtouselinux.com -connect howtouselinux.com:443 2>/dev/null | openssl x509 -noout -subject
Check for what dates the SSL certificate is valid:
- echo | openssl s_client -servername howtouselinux.com -connect howtouselinux.com:443 2>/dev/null | openssl x509 -noout -dates
Show the SHA1 fingerprint of the SSL certificate:
- echo | openssl s_client -servername www.howtouselinux.com -connect www.howtouselinux.com:443 2>/dev/null | openssl x509 -noout -fingerprint
Extract all information from the SSL certificate (decoded)
- echo | openssl s_client -servername www.howtouselinux.com -connect www.howtouselinux.com:443 2>/dev/null | openssl x509 -noout -text
Show the SSL certificate itself (encoded):
- echo | openssl s_client -servername howtouselinux.com -connect howtouselinux.com:443 2>/dev/null | openssl x509

Becoming your own `CA`

The easiest way is to use mkcert. mkcert is a command line tool that automates most of the activities related a CA.

Otherwise, this article by Brad Touesnard explains the process fully.

Linux Post Install tasks

2021-12-26T00:00:00+01:00

These tips are for void linux as that is the distro I am using nowadays.

mate tricks: change background from cli
- dconf write /org/mate/desktop/background/picture-filename "'PATH-TO-JPEG'"
web page to check if your browser is html5 compliant:
- https://www.youtube.com/html5

HotKeys

Install xbindkeys
Add to startup? $HOME/.xprofile
Create default config with xbindkeys -d > $HOME/.xbindkeysrc
Lookup key combinations:
- xbindkeys --multikey or
- xbindkeys --key
update bindings xbindkeys --poll-rc
rofi
- A good addition to this is rofi. Which is a dynamic menu.
- Create a xbindkey shortcut to run rofi with:
- rofi -show-icons -modi drun,window -show drun

Additional software

Additional software (Tested on void linux):
- Tip: use sed -e 's/#.*$//' to strip comments!

{! void-installation/swlist-extras.txt !}

My tale of IPv6 blues

2024-02-02T00:00:00+01:00

My ISP provider is KPN. They recently enabled IPv6 in my street. I was using before a IPv6 Tunnel Broker, starting with SixXS and after they went out, with Hurricane Electric. So naturally, I decided to switch to KPN's native IPv6 service.

They provide a /64 prefix, which is reasonable. Would be better if they provided a /48, but /64 is better than other providers.

So to start using KPN as the IPv6 turned out very easy. Their default configuration works right out of the box if you have single flat network.

I used to have a router/FW between the KPN modem and my network, but at some point I decided to go for a flat network design. With this, (without having to do anything) once KPN enabled IPv6, all my equipment that was IPv6 capable started using IPv6. It was like magic.

I run a number of server systems in my home network, using Alpine Linux as its operating system.

For some reason, these servers would be able to use IPv6 at first (either via static configuration or auto-configuration), but stop working after a few minutes (often after 65 seconds).

Things worked fine for my void-linux systems. These use NetworkManager so I guess this helps.

Even googling around I was not able to find a solution. Apparently doing this would re-enable things:

ip -6 a del $ip6_addr dev $IFACE
ip -6 a add $ip6_addr dev $IFACE

So what I did is, I wrote a small little script that would run every 45 seconds, and do this:

ip -6 address save dev $IFACE scope global > $savefile
ip -6 address flush dev $IFACE scope global
ip -6 address restore < $savefile 2>&1 | grep -v 'RTNETLINK answers: File exists' || :

Again, I have no idea what is going on.

Eventually I changed my set-up to have something like this:

The router in between does Network Address Translation and Firewalling. The reasons I chose this is:

More natural way of handling incoming connections
Makes it possible to switch ISP's easier, down the line. Alternatively, would make it possible to load-balance between two ISPs.
Can use iptables for firewally. I recognize that this is only good for a geek like me though.

This causes problems with my IPv6 set-up because now I have two segments.

The KPN modem, assumes a flat network (with /64). Since I can't create routes in the KPN modem then the only option would have been to NAT. However the general concensus is NOT to NAT IPv6. See this article for example.

An alternative would have been to split the /64 into /80 segments. Unfortunately, that doesn't work as a lot of the software out there assumes that the network part of the IPv6 address is at most 64 bits.

Linux has a feature built-in to the kernel called proxy_ndp. For example,

The problem is that this does not scale well as the proxy address needs to be statically configured.

There are daemons that claim to proxy NDP for ranges:

These however did not work for me.

So I wrote my own script to manage kernel proxy_ndp entries myself. Essentially it does the following:

listen on ip monitor for IPv6 neighbor messages
add and remove kernel data

The whole script can be found here.

Another approach is to use dnsmasq and can be found here.

Replacing emacs...

2021-12-26T00:00:00+01:00

So after over 30 years of using GNU emacs I have switched to a more modern options.

So I am using:

Geany: For use in a Window environment (both X11 and MSWIN)
micro: For command line use
vi: For small environments

Geany

Geany is a nice programer's text editor. I like that it has syntax highlighting and has a modern UI. It runs on Windows and Linux.

Geany has most of the features that I would like without much need of customization. Most of the customization that I have done is around the area of getting indentation right and the use of spaces versus tab when indenting. Specially when writing python or yaml files where the indentation is important.

A couple of featues that I thought were important but I am not using much are:

macro recording
split screens

These features are available in geany as plugins, but I am not using them.

micro

A micro is a modern, intuitive terminal based text editor. It follows modern key bindings and is fairly customizable.

Actually it works well straight out of the box, but I did some customizations in order to look similar to geany, and so that it would work well with putty.

This is my bindings.json file:

{! https://github.com/alejandroliu/dotfiles/raw/rcm-style/config/micro/bindings.json !}

Links about micro

Linux HDMI hotplug

2024-07-12T00:00:00+02:00

Now in 2024 this is usually no longer needed.

A more current solution is:

https://github.com/phillipberndt/autorandr

The point of this article is to document I workaround that I came up with to handle a HDMI KVM switch.

What happens is that if my Linux PC is turned on while the KVM switch is selecting the other PC, it fails to initialize the display, so when you switch back to the Linux PC, no display is shown.

The trick for this to work is to the use of udev and xrandr.

We use udev to detect the monitor being plugged in, and we use xrandr to tell X windows to update the display.

I don't think this happens very often, as the Linux defaults will handle things properly most of the time. In my case there were a number of configurations where this makes sense:

Using the old X display manager (XDM). Which doesn't seem to care for display change events.
Personally, I like using XDM because is very minimalistic.
Using a 4K HDMI EDID editor but connecting a monitor that does not support 4K.

Figuring out udev

First in the agenda is to figure out what kind of event we should be looking at. For that, we use the command:

udevadm monitor

udevadm monitor --property

With that we can determine what kind of udev events to look for (if any).

Next we need to figure out what keys we need to match. Unfortunately there is some guess work required as you need to figure out the /dev device path, whereas udevadm monitor shows a /devices/ path.

However, you manage, you need to use the following command:

udevadm info --query=all --name=/dev/dri/card0 --attribute-walk

This will show possible attributes in the udev rules key format.

Once we know the keys to use, we can know create the rules files.

Rules are located in two locations:

/usr/lib/udev/rules.d/ : for system default rules
/etc/udev/rules.d/ : for local specific rules

Essentially, we are waiting for the monitor configuration to change and when that happens we will run a script. This is accomplish with the following rules file (99-xwin-hotplug.rules):

Running xrandr

The script that is kicked off by udev is called xwin-houtplug and does the following:

Check if DISPLAY is set. (i.e. running from Xorg session)
Check if Xorg is running.
Determine the DISPLAY and a suitable XAUTHORITY file.
Run xrandr to find connected monitors and relevant display modes.
Check if the user has configured a preferred video mode.
Use the preferred video mode or auto-detect:
- Preferred: xrandr --output $monitor --mode $mode
- auto-detect: xrandr --output $monitor --auto
Run xrefresh for good measure.

See script:

To configure the preferred video mode, create a file /etc/X11/vmode.prefs with the format:

output:mode

Example:

HDMI-A-0:1920x1080
HDMI-A-1:2560x1440
DisplayPort-0:1280x720

Refer to the output of xrandr for the connection names.

monitor warmplug

Run the xwin-hotplug script when Xorg starts. This will make sure that the selected preferred video mode is used when the X session starts.

Configuring Kernel video modes

You can select a linux console preferred video mode by adding the video option to the command line.

For example:

video=HDMI-A-0:1280x720

NOTE: The output specification here follows Linux kernel conventions which are different from xrandr.

To get the name and current status of connectors, you can use the following shell oneliner:

$ for p in /sys/class/drm/*/status; do con=${p%/status}; echo -n "${con#*/card?-}: "; cat $p; done

DVI-I-1: connected
HDMI-A-1: disconnected
VGA-1: disconnected

Alpine Boot switcher

2024-02-02T00:00:00+01:00

I boot from a USB boot drive using UEFI. Because of the UEFI boot, it just a matter of copying the files from the alpine ISO to a USB thumbdrive VFAT32 partition. Partition may be set to EFI (but this doesn't seem to be required).

Since I would like to switch between different alpine versions, I wrote a script to let me have multiple alpine versions and switch between them. The boot partition can be kept ro as the script will automatically remount rw.

In your boot/EFI partition, you need to have these two scripts:

The select.sh script is the main script. fixup.sh is called from select.sh to tweak the boot command line parameters. I use it to add dom0_mem=2048M arguments to the boot command line so that xen reserves memory for guests.

Usage:

Install an ISO image

Download a iso image from the alpine repository and enter:

sh select.sh --install <iso-file>

This will extract the contents of the <iso-file> and use fixup.sh to apply any necessary tweaks. The new ISO file will be placed in a directory named according to the alpine version.

Enabling a version

sh select.sh <directory>

Makes the alpine version in <directory> the current active version for boot.

PulseAudio hints and tricks

2024-02-02T00:00:00+01:00

PulseAudio is nowadays the default sound system in many Linux distributions. It lets you do a number of useful things.

PulseAudio comes with a handy command line utility pacmd to do a number of things.

`pacmd` commands

pacmd exit
pacmd help
pacmd list-(modules|sinks|sources|clients|cards|samples)
pacmd list-(sink-inputs|source-outputs)
pacmd stat
pacmd info
pacmd load-module NAME [ARGS ...]
pacmd unload-module NAME|#N
pacmd describe-module NAME
pacmd set-(sink|source)-volume NAME|#N VOLUME
pacmd set-(sink-input|source-output)-volume #N VOLUME
pacmd set-(sink|source)-mute NAME|#N 1|0
pacmd set-(sink-input|source-output)-mute #N 1|0
pacmd update-(sink|source)-proplist NAME|#N KEY=VALUE
pacmd update-(sink-input|source-output)-proplist #N KEY=VALUE
pacmd set-default-(sink|source) NAME|#N
pacmd kill-(client|sink-input|source-output) #N
pacmd play-sample NAME SINK|#N
pacmd remove-sample NAME
pacmd load-sample NAME FILENAME
pacmd load-sample-lazy NAME FILENAME
pacmd load-sample-dir-lazy PATHNAME
pacmd play-file FILENAME SINK|#N
pacmd dump
pacmd move-(sink-input|source-output) #N SINK|SOURCE
pacmd suspend-(sink|source) NAME|#N 1|0
pacmd suspend 1|0
pacmd set-card-profile CARD PROFILE
pacmd set-(sink|source)-port NAME|#N PORT
pacmd set-port-latency-offset CARD-NAME|CARD-#N PORT OFFSET
pacmd set-log-target TARGET
pacmd set-log-level NUMERIC-LEVEL
pacmd set-log-meta 1|0
pacmd set-log-time 1|0
pacmd set-log-backtrace FRAMES

Changing audio output from the command line

For this I use the pacmd utility and manipulate the sink inputs. For already running streams, the move-sink-input needs to be used.

I have a PC with a weird configuration and requires me to switch profiles instead.

Get the current active profile:

pacmd list-cards | grep 'active profile'

Set the active profile:

pacmd set-card-profile #card #profile#

Example commands:

pacmd set-card-profile 0 output:analog-stereo+input:analog-stereo
pacmd set-card-profile 0 output:hdmi-stereo+input:analog-stereo

All that logic is in a script here or download from this link.

MATE control crashing status icon

For some reason the sound control icon in the notification bar gets lost for me. To make it re-appear use this command:

mate-volume-control-status-icon

Getting the current proxy pac configuration

2024-11-22T00:00:00+01:00

This is done using tcl for convenience. If you do not have it installed you can download freewrap executable and rename freewrap.exe to wish.exe or freewrapTCLSH.exe to tclsh.exe.

Registry Key : HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\
REG_SZ AutoConfigURL = https://<your url>/proxy.pac
REG_DWORD ProxyEnable = 0

This is the tcl script:

package require http

set pacURL [registry get {HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings} AutoConfigURL]
puts $pacURL

set conn [::http::geturl $pacURL]
::http::wait $conn
puts [::http::data $conn]

Definiton of maturity

2024-11-22T00:00:00+01:00

Maturity is:

The ability to stick with a job until it’s finished.

The ability to do a job without being supervised.

The ability to carry money without spending it.

And the ability to bear an injustice without wanting to get even.

Using XScreenSaver Hacks with mate-screensaver

2024-02-02T00:00:00+01:00

Here we explain how to use XScreenSaver EXCELLENT screensaver hack collection with the MATE screensaver applet.

Install xscreensaver and mate-screensaver
On my linux distribution this creates the following directories:
- /usr/libexec/xscreensaver: contains the screensaver hacks executables
- /usr/libexec/mate-screensaver : contains the mate-screensaver executables
- /usr/share/applications/screensavers : containes the dekstop files
Create a small script that will call the screensaver hack with the right arguments. Make sure this script is in the /usr/libexec/mate-screensaver directory, as the mate-screensaver preferences will not accept any executables that are not in the right places.
Create a desktop file to call the screensaver hack. Verify that the Exec property contains the application with the right arguments and the TryExec only contains a path to the script that you created in the previous step. The mate-screensaver preferences applet will test if the file specified in TryExec is indeed executable.
Restart mate-screensaver. I usually logout and log back in.

For my computers I use this script:

This simplifies the full process. Just run the script (you may need to sudo) with the following options:

$0 hacks [-e\|-d]
- shows the list of hacks and its enabled or disabled status.
- the -e option will only show enabled hacks.
- the -d option will only show disabled hacks.
`$0 enable [--all|hacks]
- enable the specified hacks.
- Use --all to enable all available hacks (excluding blacklisted hacks)
`$0 disable [--all|hacks]
- disable the specified hacks.
- Use --all to disable all available hacks

If you like XScreenSaver and would like to see the same software on Windows, you should read this article from the XScreenSaver author.

nginx's auth_request_module howto

2024-02-02T00:00:00+01:00

This article tries to supplement the nginx documentations regarding the auth_request module and how to configure it. In my opinion, that documentation is a bit incomplete.

What is the nginx's auth_request module

The documentation for this module says, it implements client authorization based on the result of a subrequest.

This means that when you make an HTTP request to a protected URL, nginx performs an internal subrequest to a defined authorization URL. If the result of the subrequest is HTTP 2xx, nginx proxies the original HTTP request to the backend server. If the result of the subrequest is HTTP 401 or 403, access to the backend server is denied.

By configuring nginx, you can redirect those 401s or 403s to a login page where the user is authenticated and then redirected to the original destination.

The entire authorization subrequest process is then repeated, but because the user is now authenticated the subrequest returns HTTP 200 and the original HTTP request is proxied to the backend server.

Configuring nginx

In your nginx configuration...

This block configures the web server area that will be protected:

        location /hello {
          error_page 401 = @error401;   # Specific login page to use
          auth_request /auth;       # The sub-request to use
          auth_request_set $username $upstream_http_x_username; # Make the sub request data available
          auth_request_set $sid $upstream_http_x_session;   # send what is needed

          proxy_pass http://sample.com:8080/hello;  # actual location of protected data
          proxy_set_header X-Forwarded-Host $host;  # Custom headers with authentication related data
          proxy_set_header X-Remote-User $username;
          proxy_set_header X-Remote-SID $sid;
        }

    location @error401 {
          return 302 /login/?url=http://$http_host$request_uri;
        }

error_page 401 defines the custom login page to use (if any). In theory, for a REST API, would be possible to authenticate using a provided token header which makes the login page unnecesary.
auth_request /auth defines that this location needs authentication and defines the sub-request location to use.
auth_request_set can be used to get data from the subrequest headers and make it available later (like for instance, to a backend content server using custom headers.
proxy_pass defines the actual backend content server.
proxy_set_header defines custom header used to pass information to the content backend server.

This block configures the authentication sub-request server:

        location = /auth {
          proxy_pass http://auth-server.sample.com:8080/auth;   # authentication server
          proxy_pass_request_body off;              # no data is being transferred...
          proxy_set_header Content-Length '0';
          proxy_set_header Host $host;              # Custom headers with authentication related data
          proxy_set_header X-Origin-URI $request_uri;
          proxy_set_header X-Forwarded-Host $host;
        }

proxy_pass where the sub request should be handled.
proxy_pass_request_body off and proxy_set_header Content-Length 0 are used to supress the content body and only sends the headers to the authentication server.
proxy_set_header additional details being send to the sub request. For example, the X-Origin-URI.

This implements the login pager URL

        # If the user is not logged in, redirect them to login URL
        location @error401 {
          return 302 https://$host/login/?url=https://$http_host$request_uri;
        }

In this example, the login page is on the same reverse proxy, but it doesn't have to be that way.

The actual login page:

        location /login/ {
          proxy_pass http://auth-server.sample.com:8080/login/; # Where the login happens
          proxy_set_header X-My-Real-IP $remote_addr;       # Additional parameters to send to login page
          proxy_set_header X-My-Real-Port $remote_port;
          proxy_set_header X-My-Server-Port $server_port;
        }

proxy_pass points to where the login script runs
proxy_set_header can be used to pass additional fields that may be needed by the login script. To implement for example, $remote_addr based access rules.

So in this particular example, we are referring to a server with TWO locations:

http://auth-server.sample.com:8080/auth - The sub-request URI which is not visible outside but handles the sub-request.
http://auth-server.sample.com:8080/login/ - The login URI which handles the login conversation.

The Authentication Server

This is where the nginx documentation falls a bit short, there is no actual authentication server example to refer to.

In my example, we have a simple authentication workflow. When an unauthenticated user hits the server, the sub-request is called and checks (and fails) for a session cookie.

The user is then re-directed to the login page, where the actual login takes place. If succesful, a session cookie is set and the user is redirected to the original URL.

This is implemented using the following script:

This makes uses of the bottle micro framework.

It implements four routes:

GET /hello This is just a demo URL used for testing. Only shows the request headers.
GET /login/ This is the login page entry point.
POST /login/ This is the handler for the login page.
GET /auth This is the sub-request handler.

For the demo, we are not really doing any login handling. You only need to make the username the same as the password to login. Anything else is a login failure.

When the user succesfully logs in, we set a cookie. Because the login URL and the protected resource (/hello URL) are in the same cookie scope, we can use cookie set by the login page as the verification token in the sub-request.

Note that the login page can be as simple or as complex as it is needed. For example, it is possible to implement a SAML, OpenID Connect, or any Single-Sign-On workflow available.

Alternatively, two factor authentication could be implemented here. The possibilities are endless.

An interesting use of the auth_request module would be to delegate Basic Authentication to a different server or to even implement authentications not supported by nginx like for example a simple Token-Bearer header or Digest authentication

basic authentication

This may seem silly since nginx supports basic authentication out of the box. The use case for this is when you have a cluster of nginx front ends, and you want all of them to authenticate against a central identity server. Furthermore, since the URI can be passed, a more sophisticated access control can be implemented. Finally, additional values can be passed through headers, such as group names, tokens, etc.

nginx configuration

Protected Resource:

        location /hello {
          auth_request /auth;       # The sub-request to use
          auth_request_set $username $upstream_http_x_username; # Make the sub request data available

          proxy_pass http://sample.com:8080/hello;  # actual location of protected data
          proxy_set_header X-Forwarded-Host $host;  # Custom headers with authentication related data
          proxy_set_header X-Remote-User $username;
        }

NOTE: unlike the previous example, we do not need to provide a @error401 page.

Sub-request configuration:

        location = /auth {
          proxy_pass http://auth-server.sample.com:8080/auth;   # authentication server
          proxy_pass_request_body off;              # no data is being transferred...
          proxy_set_header Content-Length '0';
          proxy_set_header Host $host;              # Custom headers with authentication related data
          proxy_set_header X-Origin-URI $request_uri;
          proxy_set_header X-Forwarded-Host $host;
        }

authentication server

The python implementation (again, using bottle):

Like in the previous example, we are not doing any user/password verification. We are only checking if username and password are matching.

Unlike the previous example, all the authentication is handled by a single route (/auth). It returns 'WWW-Authenticate' to prompt the user for a password. And if it sees an Authorization header, it would validate it.

digest authentication

This implements digest authentication for nginx using the auth request module. The nginx configuration is the same as in the Basic authentication.

The implentation in python (using bottle framework):

Python Virtual Environments

2021-12-25T00:00:00+01:00

This is the least you need to know to get to use a Python virtual environment.

What is a Virtual Environment

At its core, the main purpose of Python virtual environments is to create an isolated environment for Python projects. This means that each project can have its own dependencies, regardless of what dependencies every other project has.

The great thing about this is that there are no limits to the number of environments you can have since they're just directories containing a few scripts.

Pre-requisites

While venv is part of python3, for python2 you need to install virtualenv.

void-linux: python-virtualenv

Create

To create a new virtual environment:

Python2

mkdir folder
virtualenv folder

If you want to inherit system global packages in your virtual environment use this instead:

mkdir folder
virtualenv --system-site-packages folder

Python3

mkdir folder
python3 -m venv folder

If you want to inherit system global packages in your virtual environment use this instead:

mkdir folder
python3 -m venv --system-site-packages folder

I prefer to use the --system-site-packages option, that way I can have binary modules using the host's package manager. This is in order to avoid having a compiler in the host system.

Activate

To activate a virtual environment:

. <folder>/bin/activate

Pay attention that we are using . to source the script in the current interpreter.

De-Activate

To de-activate:

deactivate

Run from a script

To use the virtual environment from a script (i.e. running as a background daemon) you need to add these to the beginning of your python script:

activate_this = '/path/to/virtualenv/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

Or from a shell cript:

#!/bin/sh
source name_Env/bin/activate
# virtualenv is now active.
exec python script.py "$@"

References

For more information see:

Secure erase of disc drives

2021-12-25T00:00:00+01:00

This article is about erasing disc drives securely. Specially for SSD drives, writing zeros or random data to discs is not good enough and counterproductive.

One way to do secure erase (for disposal) is to begin with an encrypted disc. However, after the fact the following options are possible:

ATA Secure Erase

You should use the drive's security erase feature.

Make sure the drive Security is not frozen. If it is, it may help to suspend and resume the computer.

$ sudo hdparm -I /dev/sdX | grep frozen
       not     frozen

The (filtered) command output means that this drive is ”not frozen” and you can continue.

Set a User Password (this password is cleared too, the exact choice does not matter).

sudo hdparm --user-master u --security-set-pass Eins /dev/sdX

Issue the ATA Secure Erase command

sudo hdparm --user-master u --security-erase Eins /dev/sdX

Notes:

/dev/sdX is the SSD as a block device that you want to erase.
Eins is the password chosen in this example.

See the ATA Secure Erase article in the Linux kernel wiki for complete instructions including troubleshooting.

If for some reason you need to remove the password use:

sudo hdparm --security-disable Eins

blkdiscard

util-linux 2.23 offers blkdiscard which discards data without secure-wiping them. This has been tested to work over SATA and mmcblk but not USB.

An excerpt from the manual page of blkdiscard(8):

NAME

blkdiscard - discard sectors on a device

SYNOPSIS

blkdiscard [-o offset] [-l length] [-s] [-v] device

DESCRIPTION

blkdiscard is used to discard device sectors. This is useful for solid-state drivers (SSDs) and thinly-provisioned storage. Unlike fstrim(8) this command is used directly on the block device.

By default, blkdiscard will discard all blocks on the device. Options may be used to modify this behavior based on range or size, as explained below.

The device argument is the pathname of the block device.

WARNING: All data in the discarded region on the device will be lost!

Use TRIM

To enable TRIM:

sudo vi /etc/fstab

Change ext4 errors=remount-ro 0" into "ext4 discard,errors=remount-ro 0. (Add discard)

Save and reboot, TRIM should now be enabled.

Check if TRIM is enabled:

sudo dd if=/dev/urandom of=tempfile count=100 bs=512k oflag=direct
sudo hdparm --fibmap tempfile

Use the first begin_LBA address.

hdparm --read-sector [begin_LBA] /dev/sda

Now it should return numbers and characters. Remove the file and sync.

rm tempfile
sync

Now, run the following command again. If it returns zeros TRIM is enabled.

hdparm --read-sector [begin_LBA] /dev/sda

Another option is to use the fstrim command.

Old fashioned writes

This is what I used to do for magnetic discs. Note, that this is discouraged for SSD devices:

First I create some random data to use:

dd if=/dev/urandom of=/var/tmp/random bs=1M count=128

Then we write random data to disc:

(while : ; do dd if=/var/tmp/random bs=4k ; done ) | pv | dd of=/dev/sdX bs=4k

The pv part of the pipe is optional.

Afterwards:

dd if=/dev/zero of=/dev/sdX bs=4k

Co-existing GLIBC binaries with Void-Linux MUSL edition

2025-01-01T00:00:00+01:00

I am running void-linux at home with musl as the standard C library.

While most things work well, there is a number of programs that do not and must be using glibc counterparts.

To enable this I followed this guide here: Live switching Void Linux from glibc to musl.

To set-up:

sudo mkdir -p /glibc
sudo env XBPS_ARCH=x86_64 xbps-install --repository=https://repo-default.voidlinux.org/current -r /glibc -S base-voidstrap

sudo : yes, we need root
env : Needed because we are using sudo.
XBPS_ARCH=x86_64 : architecture to use. Since we are using musl, we point to the glibc version. It should be possible to create a 32 bit root here.
--repository=https://repo-default.voidlinux.org/current : the repository to use. Feel free to replace to something closer.
-r /glibc : directory tree where the glibc executables will live
base-voidstrap : unlike base-system, this meta-package is normally used for containers.

To keep this tree up to date:

sudo env XBPS_ARCH=x86_64 xbps-install --repository=https://repo-default.voidlinux.org/current -r /glibc -Su

To add software to the tree:

sudo env XBPS_ARCH=x86_64 xbps-install --repository=https://repo-default.voidlinux.org/current  -r /glibc -S pkg

Once this is set-up you need a small program to kick off the glibc executables. I copied this one:

To compile and install:

gcc -s -o glibc glibc.c
sudo cp glibc /usr/bin
sudo chown root:root /usr/bin/glibc
sudo chmod +sx /usr/bin/glibc

Then you can just run:

glibc cmd args

The following software I have found doesn't work well using musl:

Calibre
building buildroot (Because of compilation of fakeroot)

Calculate system availability

2024-11-22T00:00:00+01:00

To calculate the availability of redundant systems you can use this formula:

total_avail = 1-(1 - single_avail) ^ (number_of_nodes)

Ad-Hoc rsync daemons

2024-02-02T00:00:00+01:00

The other day I needed to copy a bunch of files between to servers in my home network. Because of the volume I wanted to copy the files without having to go through ssh's encryption overhead. So I figured I could use netcat for the data transport.

To do that I wrote these short scripts.

Remote scripts

Copy these scripts on the remote server. Make sure they are executable.

Remote CLI

Remote Helper script

Local scripts

Copy these scripts on the local server. Make sure they are executable

Local CLI

Local Helper Script

Usage

Usage is fairly straight forward, on the remote server enter the command:

./recv-nc

This will make remote listen for new client connects. On the local server, enter the command:

./send -avzr --delete --stats SRC/ remote:DST

Actually, just use whatever rsync options you need to use. The send script will include the --rsh option to make sure the helper script gets executed.

Issues

Unfortunately the local helper script does not detect that the transfer has completed. The remote helper script would finish correctly and exit when the transfer is done.

You can simply press Ctrl+C to quit, or if you want to see any summary stats, I would kill the running cat command.

Example:

$ bg
[2]+ ./send -avzr --stats SRC localhost:DST &
$ pidof cat
13143
$ kill 13143
$ Terminated

Number of files: 6 (reg: 5, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 5,869 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 202
Total bytes received: 17

sent 202 bytes  received 17 bytes  438.00 bytes/sec
total size is 5,869  speedup is 26.80
rsync error: syntax or usage error (code 1) at main.c(1189) [sender=3.1.3]
$

Resizing Virtual Disks with virsh

2021-12-25T00:00:00+01:00

I am currently using libvirt for managing my VMs. For virtual discs I am using LVM2 volumes. On a regular basis I need to resize these virtual discs, but not that often that I can do this from memory. This is a short procedure to do this:

ls -l /dev/vgX/lvX      # note down the major/minor numbers for later
lvextend -L +50G /dev/vgX/lvX   # adding 50GB to this volume
cat /proc/partitions        # look up the size (in blocks) using major/minor numbers
virsh blockresize --path /dev/vgX/lvX --size SIZE_FROM_PROC_PARTITIONS vmname

Then on the running system you can do:

cat /proc/partitions        # Make sure that size is right
xfs_growfs /mount/point     # On-line partition re-size

Z-Wave Associations with With Vera UI

2021-12-25T00:00:00+01:00

I couldn't find any to the point documentation on how to do this, so I am writing this.

The way I understand Z-Wave associations work is that once devices are in the same Z-Wave network, a device can directly send a command to another device without intervention of the Hub or controller.

For this to really work the master device sending commands must support this functionality. This varies from device to device, so you must look up the documentation of the master device and find the supported associations groups. In a nutshell, you look-up what each association group is for, and then you add to that group the slave devices that will receive the Z-wave commands from the master.

So for example, a TKB-Home TZ55D Wall mounted dimmer/switch with two buttons has the following association groups:

Group 1: Control device using the left button.
Group 2: Control device using the right button.
Group 3: Control device using the right button after a double tap.
Group 4: Control device so that it follows the switch state.

So this dimmer/switch has two buttons, the left button is used for local control. The right button can be used to control devices associated to groups 2 and 3.

In the Vera UI7 this is done as follows:

Go to the Devices list.
Locate the master device in the list and click it settings button.
Click Device Options.
Under Associations, enter the Group ID from the documentation as explained above, and click Add group. You may need to click Back and enter Device Options again for the group to be visible.
Once the desired Group ID is available, click on the Set button.
Click the Z-Wave devices checkbox to add them to the group. Leave the entry field next to the device is to enter sub-device ids. This is used for multichannel devices. For example, an RGBW controller may have multiple channels to control the different color LEDs. Check the slave device documentation for the valid sub-channel IDs to use.

Encrypting FileSystem in Void Linux

2025-01-01T00:00:00+01:00

The idea is that the boot/root filesystem containing the encryption keys are stored in a different device as the encrypted file system.

Generate a passphrase and save it a safe place for later.

Create block devices

xbps-install -S lvm2 cryptsetup
cryptsetup luksFormat /dev/xda2
cryptsetup luksOpen /dev/xda2 crypt-pool

Alternate commands

dd if=/dev/urandom of=/crypto_keyfile.bin bs=1024 count=4
chmod 000 /crypto_keyfile.bin
cryptsetup luksFormat /dev/xda2 /crypto_keyfile.bin
cryptsetup luksOpen --key-file=/crypto_keyfile.bin /dev/xda2 crypt-pool

Add rd.luks.crypttab=1 rd.luks=1 to the kernel command line.

Create a decryption key

Create the key file in the unencrypted / partition

dd if=/dev/urandom of=/crypto_keyfile.bin bs=1024 count=4
chmod 000 /crypto_keyfile.bin
cryptsetup -v luksAddKey /dev/xda2 /crypto_keyfile.bin

Look up the UUID

blkid /dev/xda2

Create entry in /etc/crypttab:

crypt-pool  UUID=xxxxxxxxxxxxxxxx /crypto_keyfile.bin luks

If you want to enable discard option in flash sorage use this instead:

crypt-pool  UUID=xxxxxxxxxxxxxxxx /crypto_keyfile.bin luks,discard

Create /etc/dracut.conf.d/10-crypt.conf

install_items+="/etc/crypttab /crypto_keyfile.bin"

Update initrd:

xbps-reconfigure -f linux4.19

Update boot menu entries:

bash /boot/mkmenu.sh

At this point it would be good to save:

/etc/crypttab
/crypto_keyfile.bin
Optionally, passphrase

Reboot and make sure that the block device gets created on start-up.

Create your file-system and add it to /etc/fstab.

vgcreate pool /dev/mapper/crypt-pool
lvcreate --name home0 -L 20G pool

Re-using an existing fs in a new OS install

Usually this procedure would be used on a fresh install when the root filesystem was destroyed. It requires to have a backup of the /crypto_keyfile.bin and optionally the /etc/crypttab.

Add rd.luks.crypttab=1 rd.luks=1 to the kernel command line.
Restore the /crypto_keyfile.bin. Make sure it is in / and permissions are chmod 000 /crypto_keyfile.bin.
If available, restore the /etc/crypttab otherwise look up the block device UUID and re-create the /etc/crypttab entry:
- Look-up the UUID:
- blkid /dev/xda2
- Add the entry in /etc/crypttab
- crypt-pool UUID=xxxxxxxxxxxxxxxx /crypto_keyfile.bin luks
Create /etc/dracut.conf.d/10-crypt.conf
- install_items+="/etc/crypttab /crypto_keyfile.bin"
Update initrd:
- xbps-reconfigure -f linux4.19
Update boot menu entries:

bash /boot/mkmenu.sh

Installing Void Linux

2025-07-09T00:00:00+02:00

I made the switch to void linux. Except for compatibility issues around glibc, it works quite well. Most compatibility I have worked around with a combination of Flatpaks, chroots and namespaces.

The highlights of void linux:

musl build - which is very lightweigth
Does not depend on systemd
a reasonable selection of software packages

I have tweaked the installation on my computers to use UEFI and thus I am using rEFInd instead of grub. This is because it makes doing bare metal backups and restore just a simple file copy.

My installation process roughly follows the void linux UEFI chroot install.

This process is implemented in a script and can be found here:

install.sh

Script usage:

    Usage: installer.sh _/dev/sdx_ _hostname_ [options]

    - _sdx_: Block device to install to or
      - --image=filepath[:size] to create a virtual disc image
      - --imgset=filebase[:size] to create a virtual filesystem image set
      - --dir=dirpath to create a directory
    - _hostname_: Hostname to use

    Options:
    - swap=kbs : swap size, defaults computed from /proc/meminfo, uses numfmt to parse values
    - glibc : Do a glibc install
    - noxwin : do not insall X11 related packages
    - nodesktop ; do not install desktop environment
    - desktop=mate : Install MATE dekstop environment
    - passwd=password : root password (prompt if not specified)
    - enc-passwd=encrypted : encrypted root password.
    - ovl=tar.gz : tarball containing additional files
    - post=script : run a post install script
    - pkgs=file : text file containing additional software to install
    - bios : create a BIOS boot system (needs syslinux)
    - cache=path : use the file path for download cache
    - xen : do some xen specific tweaks
    - xdm-candy : Enable xdm candy
    - noxdm : disable graphical login

Command line examples

sudo sh install.sh --dir=$HOME/vx9 vx9 swap=4G glibc passwd=1234567890 cache=$HOME/void-cache xen
sudo sh install.sh --dir=$HOME/vx1 vx1 swap=4G glibc passwd=1234567890 cache=$HOME/void-cache xen
sudo sh install.sh --dir=$HOME/vx11 vx11 swap=4G passwd=1234567890 cache=$HOME/void-cache xen

Initial set-up

Boot using the void live CD and partition the target disk:

cfdisk -z /dev/xda

Make sure you use gpt label type (for UEFI boot). I am creating the following partitions:

800MB EFI System
RAM Size x 1.5 Linux swap, Mainly used for Hibernate.
Rest of drive Linux filesystem, Root file system

When I first published this article back in 2019, I was using 500MB for the EFI filesystem. Now I am using 800MB. This is because the size of the Linux kernel including all the drivers have been growing over time. With 800MB you can only have about one or two spare kernels.

This is on a USB thumb drive. The data I keep on an internal disk.

Now we create the filesystems:

mkfs.vfat -F 32 -n EFI /dev/xda1
mkswap -L swp0 /dev/xda2
mkfs.xfs -f -L voidlinux /dev/xda3

We're now ready to mount the volumes, making any necessary mount point directories along the way (the sequence is important, yes):

mount /dev/xda3 /mnt
mkdir /mnt/boot
mount /dev/xda1 /mnt/boot

Installing Void

So we do a targeted install:

For musl-libc

env XBPS_ARCH=x86_64-musl xbps-install -S :##     -R http://repo-default.voidlinux.org/current/musl :##     -r /mnt :##     base-system

For glibc

env XBPS_ARCH=x86_64 xbps-install -S :##     -R http://repo-default.repo.voidlinux.org/current :##     -r /mnt :##     base-system

But actually, for the package list I have been using these lists:

base list:

x-windows:

Software selection notes

For time synchronisation (ntp) we are choosing chrony as it is reputed to be more secure that ntpd and more compliant than openntpd.
We are using the default configuration, which should be OK. Uses pool.ntp.org for the time server which would use a suitable default.
For cron we are using dcron. It is full featured (i.e. compatibnle with cron and it can handle power-off situations, while being the most light-weight option available. See: VoidLinux FAQ: Cron
Includes autofs and nfs-utils for network filesystems and automount support.

nonfree software and other repositories

Additional repositories are available to support either non-free software and in the case of glibc, multilib (32 bit) binaries.

To enable under the musl version:

env XBPS_ARCH="$arch" xbps-install -y -S -R "$voidurl" -r /mnt :##     void-repo-nonfree

For glibc:

env XBPS_ARCH="$arch" xbps-install -y -S -R "$voidurl" -r /mnt :##     void-repo-nonfree void-repo-multilib void-repo-multilib-nonfree

Then you can install non-free software, like:

Enter the void chroot

Upon completion of the install, we set up our chroot jail, and chroot into our mounted filesystem:

mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -o bind /dev /mnt/dev
mount -t devpts pts /mnt/dev/pts
cp -L /etc/resolv.conf /mnt/etc/
chroot /mnt bash -il

In order to verify our install, we can have a look at the directory structure:

 ls -la

The output should look something akin to the following:

total 12
drwxr-xr-x 16 root root 4096 Jan 17 15:27 .
drwxr-xr-x  3 root root 4096 Jan 17 15:16 ..
lrwxrwxrwx  1 root root    7 Jan 17 15:26 bin -> usr/bin
drwxr-xr-x  4 root root  127 Jan 17 15:37 boot
drwxr-xr-x  2 root root   17 Jan 17 15:26 dev
drwxr-xr-x 26 root root 4096 Jan 17 15:27 etc
drwxr-xr-x  2 root root    6 Jan 17 15:26 home
lrwxrwxrwx  1 root root    7 Jan 17 15:26 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Jan 17 15:26 lib32 -> usr/lib32
lrwxrwxrwx  1 root root    7 Jan 17 15:26 lib64 -> usr/lib
drwxr-xr-x  2 root root    6 Jan 17 15:26 media
drwxr-xr-x  2 root root    6 Jan 17 15:26 mnt
drwxr-xr-x  2 root root    6 Jan 17 15:26 opt
drwxr-xr-x  2 root root    6 Jan 17 15:26 proc
drwxr-x---  2 root root   26 Jan 17 15:39 root
drwxr-xr-x  3 root root   17 Jan 17 15:26 run
lrwxrwxrwx  1 root root    8 Jan 17 15:26 sbin -> usr/sbin
drwxr-xr-x  2 root root    6 Jan 17 15:26 sys
drwxrwxrwt  2 root root    6 Jan 17 15:15 tmp
drwxr-xr-x 11 root root  123 Jan 17 15:26 usr
drwxr-xr-x 11 root root  150 Jan 17 15:26 var

While chrooted, we create the password for the root user, and set root access permissions:

 passwd root
 chown root:root /
 chmod 755 /

Since I am a bash convert, I would do this:

 xbps-alternatives --set bash

Create the hostname for the new install:

echo <HOSTNAME> > /etc/hostname

Edit our /etc/rc.conf file, like so:

HOSTNAME="<HOSTNAME>"

# Set RTC to UTC or localtime.
HARDWARECLOCK="UTC"

# Set timezone, availables timezones at /usr/share/zoneinfo.
TIMEZONE="Europe/Amsterdam"

# Keymap to load, see loadkeys(8).
KEYMAP="us-acentos"

# Console font to load, see setfont(8).
#FONT="lat9w-16"

# Console map to load, see setfont(8).
#FONT_MAP=

# Font unimap to load, see setfont(8).
#FONT_UNIMAP=

# Kernel modules to load, delimited by blanks.
#MODULES=""

Also, modify the /etc/fstab:

#
# See fstab(5).

# <file system> <dir>   <type>  <options>       <dump>  <pass>
tmpfs       /tmp    tmpfs   defaults,nosuid,nodev   0       0
LABEL=EFI   /boot   vfat    rw,fmask=0133,dmask=0022,noatime,discard  0 2
LABEL=voidlinux /   xfs rw,relatime,discard 0 1
LABEL=swp0  swap    swap    defaults        0 0

For a removable drive I include the line:

LABEL=volume    /media/blahblah xfs rw,relatime,nofail 0 0

The important setting here is nofail. When the drive is available it gets mounted. If not, the nofail prevents this to cause the boot sequence to stop.

If using glibc you can modify /etc/default/libc-locales and uncomment:

en_US.UTF-8 UTF-8

Or whatever locale you want to use. And run:

 xbps-reconfigure -f glibc-locales

Set-up UEFI boot

Download the rEFInd zip binary from:

rEFInd download

Set-up the boot partition:

mkdir /boot/EFI
mkdir /boot/EFI/BOOT

Copy from the zip file the file refind-bin-{version}/refind/refind_x64.efi to /boot/EFI/BOOT/BOOTX64.EFI.

The version I am using right now can be found here: v0.14.2 BOOTX64.EFI

Create kernel options files /boot/cmdline:

root=LABEL=voidlinux ro quiet

Depending of your hardware you may add additional options. For example, at one time I hadd to use:

intel_iommu=igfx_off
- To work around some strange bug.
i915.enable_ips=0
- fixes a power saving mode problem on 4.1-rc6+

Create the following script as /boot/mkmenu.sh

Add the following scripts to:

/etc/kernel.d/post-install/99-refind
/etc/kernel.d/post-remove/99-refind

Make sure they are executable. This is supposed to re-create menu entries whenever the kernel gets upgraded.

We need to have a look at /lib/modules to get our Linux kernel version

ls -la /lib/modules

Which should return something akin to:

drwxr-xr-x  3 root root   21 Jan 31 15:22 .
drwxr-xr-x 23 root root 8192 Jan 31 15:22 ..
drwxr-xr-x  3 root root 4096 Jan 31 15:22 5.2.13_1

And this script to create boot files:

xbps-reconfigure -f linux5.2

If you need to manually prepare boot files:

# update dracut
dracut --force --kver 4.19.4_1
# update refind menu
bash /boot/mkmenu.sh

We are now ready to boot into Void.

exit
umount -R /mnt
reboot

Post install

After the first boot, we need to activate services:

Command line set-up:

ln -s /etc/sv/dhcpcd /var/service
ln -s /etc/sv/sshd /var/service
ln -s /etc/sv/{acpid,chronyd,crond,uuidd,statd,rcpbind,autofs,socklog-unix,nanoklogd} /var/service


Creating new users:

```bash
useradd -m -s /bin/bash -U -G wheel,users,audio,video,cdrom,input newuser
passwd newuser

Note: The wheel user group allows the user to escalate to root.

Configure sudo:

visudo

Uncomment:

# %wheel ALL=(ALL) ALL

Configure keyboard

Create configuration file: /etc/X11/xorg.conf.d/30-keyboard.conf

Section "InputClass"
    Identifier "keyboard-all"
    Option "XkbLayout" "us"
    # Option "XkbModel" "pc105"
    # Option "XkbVariant" "altgr-intl"
    Option "XkbVariant" "intl"
    # MatchIsKeyboard "on"
EndSection

This makes the intl for the XkbVariant the system-wide default.

Since, as a programmer I prefer the altgr-intl variant, then I run this in my de desktop environment startup to override the default:

setxkbmap -rules evdev -model evdev -layout us -variant altgr-intl

I don't normally use a display manager. To start the GUI enviornment, you can add a file in /etc/profile.d to start X at login if on tty1.

I am using the session script, which is a modified version of an earlier Xsession script that I was using for xdm to launch a desktop session.

The script zzdm.sh is used to startx on login.

Tweaks and Bug-fixes

/etc/machine-id or /var/lib/dbus/machine-id

Because we don't use systemd, we need to create /etc/machine-id and /var/lib/dbus/machine-id. manually. This is only needed for desktop systems.

See [this article][machineid] for more info.

  dbus-uuidgen | tee /etc/machine-id /var/lib/dbus/machine-id

power button handling

This patch prevents the /etc/acpi/handler.sh to handle the power button instead, letting the Desktop Environment handle the event.

It does it by checking if a X session is running. In the /etc/rc.local script, we create a file called /run/xsession.pid which is made writeable by all. The system is configured so that xdm or /etc/profile/zzdm.sh (when login as normal user on tty1) will start an X session and will use the scripts /etc/X11/xinit/session or /etc/X11/xdm/Xsession to start the session. From these scripts, the current X session information is saved to /run/xsession.pid.

When /etc/acpi/handler.sh starts, it will check /run/session.pid if it contains a running session. It will also check if a Desktop Environment power manager (in this case mate-power-manager) is running. If it is, then it will exit.

xen tweaks

For xen we need to make some adjustments...

Tweak block device references.
- /etc/fstab : mount xvda and other devices
- /boot/cmdline : get the right xvda root device
Enable disable services
- Disable: slim, agetty-ttyX
- Enable: agetty-hvc0
- Decide if you want to use NetworkManager or dhcpcd.

Normally, I would create a tarball image to transfer over, in order for the image to work properly you need to save capabilities.

My Linux Keyboard Shortcuts

2021-12-25T00:00:00+01:00

In general we try to be similar to MS-Windows shortcuts.

Default bindings (in MATE)

Key	Action
Alt + F4	Close the active item, or exit the active program
Alt + Tab	Switch between open items
Ctrl + Alt + Tab	Use the arrow keys to switch between open items
Alt + Esc	Cycle through items in the order in which they were opened
Alt + Spacebar	Open the shortcut menu for the active window

Custom bindings (Window Manager)

Key	Action
Ctrl + Esc	Open the Start menu
SuperKey + Up arrow	Toggle Maximize the window.
SuperKey + Down arrow	Remove current app from screen or minimize the desktop window.
SuperKey + Left arrow	Maximize the app or desktop window to the left side of the screen.
SuperKey + Right arrow	Maximize the app or desktop window to the right side of the screen.
SuperKey + D	Display or hide the desktop.

This is configured using this script: keybindings.sh

Xbindkeys

Key	Action
Ctrl + Shift + Esc	Open Task Manager
SuperKey + L	Lock your PC
SuperKey + E	Open File Manager.
SuperKey + F	Open search.
SuperKey + R	Open the Run dialog box.
SuperKey + KP_Mult	Volume Up
SuperKey + KP_Minus	Volume Down
SuperKey + KP_Div	Mute Toggle
SukerKey + KP_Add	Switch PA output

This makes use of this xbindkeysrc file.

TODO: SuperKey + V | Open the clipboard.

Global Windows Keyboard Shorcuts

2022-03-03T00:00:00+01:00

Common Window Management Shortcuts

Key	Action
Alt + F4	Close the active item, or exit the active program
Alt + Tab	Switch between open items
Ctrl + Alt + Tab	Use the arrow keys to switch between open items
Alt + Esc	Cycle through items in the order in which they were opened
Ctrl + Esc	Open the Start menu
Alt + Spacebar	Open the shortcut menu for the active window
Ctrl + Shift + Esc	Open Task Manager
WinKey	Open or Close Start button
WinKey + D	Display or hide the desktop.
WinKey + E	Open File Explorer.
WinKey + F	Open File Explorer Search (Find).
WinKey + L	Lock your PC or switch accounts.
WinKey + M	Minimize all windows.
WinKey + Shift + M	Restore minimized windows on the desktop.
WinKey + R	Open the Run dialog box.
WinKey + V	Open the clipboard.
WinKey + Up arrow	Maximize the window.
WinKey + Down arrow	Remove current app from screen or minimize the desktop window.
WinKey + Left arrow	Maximize the app or desktop window to the left side of the screen.
WinKey + Right arrow	Maximize the app or desktop window to the right side of the screen.
WinKey + Home	Minimize all except the active desktop window (restores all windows on second stroke).

Additional Window Management Shortcuts

Key	Action
WinKey + Tab	Cycle through programs on the taskbar by using Aero Flip 3-D
Ctrl+WinKey + Tab	Use the arrow keys to cycle through programs on the taskbar by using Aero Flip 3-D
Left Alt + Shift	Switch the input language when multiple input languages are enabled

Rare Window Management Shortcuts

Key	Action
Ctrl+Shift	Switch the keyboard layout when multiple keyboard layouts are enabled
Right or Left Ctrl + Shift	Change the reading direction of text in right-to-left reading languages
WinKey + B	Set focus in the notification area.
WinKey + P	Choose a presentation display mode.
WinKey + T	Cycle through apps on the taskbar.
WinKey + U	Open Ease of Access Center.
WinKey + Pause	Display the System Properties dialog box.
WinKey + Ctrl + Enter	Turn on Narrator.
WinKey + Plus (+)	Open Magnifier.

CUA Shortcuts

Key	Action
F1	Display Help
F2	Rename the selected item
F3	Search
Ctrl + F4	Close the active document (in programs that allow you to have multiple documents open simultaneously)
F5 (or Ctrl + R)	Refresh the active window
F6	Cycle through screen elements in a window or on the desktop
F10	Activate the menu bar in the active program
Shift + F10	Display the shortcut menu for the selected item
-	-
Esc	Cancel the current task
Delete (or Ctrl + D)	Delete current item
Shift + Delete	Delete current item (without undo?)
Alt + Enter	Display properties for the selected item
-	-
Ctrl + C (or Ctrl + Insert)	Copy the selected item
Ctrl + X	Cut the selected item
Ctrl + V (or Shift + Insert)	Paste the selected item
Ctrl + Z	Undo an action
Ctrl + Y	Redo an action
Ctrl + A	Select all items in a document or window
Ctrl + Mouse scroll wheel	Change zoom factor
Alt + underlined letter	Display the corresponding menu
Alt + underlined letter	Perform the menu command (or other underlined command)
-	-
Ctrl + Right Arrow	Move the cursor to the beginning of the next word
Ctrl + Left Arrow	Move the cursor to the beginning of the previous word
Ctrl + Down Arrow	Move the cursor to the beginning of the next paragraph
Ctrl + Up Arrow	Move the cursor to the beginning of the previous paragraph
Ctrl + Shift with an arrow key	Select a block of text

Third Party SimpleNote clients

2024-11-22T00:00:00+01:00

An inventory of simplenote clients:

SimpleNote clients:

Current plan is to use a webapp wrapper for the actual simplenote web site.

Docker on Alpine Linux

2022-01-04T00:00:00+01:00

Alpine Linux Quick installation

See wiki For Alpine Linux > 3.8

Un-comment community repo from /etc/apk/repositories
apk add docker
rc-update add docker boot
service docker start

Optional: (docker compose)

apk add docker-compose

Note 2021-03-21: When I tested this, the daemon.json did not work! Your mileage may vary.

Recommended for user namespace isolation (not sure if this works)

Also is good to use data mode (persistent /var) as most docker data is stored there.

adduser -SDHs /sbin/nologin dockremap
addgroup -S dockremap
echo dockremap:100000:65535 | tee /etc/subuid
echo dockremap:100000:65535 | tee /etc/subgid

In /etc/docker/daemon.json:

{
        "userns-remap": "dockremap"
}

For more info docker docs

Test docker:

docker version
docker info
docker run hello-world
docker image ls
docker container ls
docker container ls --all
docker container ls --aq

Mounting NFS

From docker 17.06, you can mount NFS shares to the container directly when you run it, without the need of extra capabilities

docker run --mount 'type=volume,src=VOL_NAME,volume-driver=local,dst=/LOCAL-MNT,volume-opt=type=nfs,volume-opt=device=:/NFS-SHARE,"volume-opt=o=addr=NFS-SERVER,vers=4,hard,timeo=600,rsize=1048576,wsize=1048576,retrans=2"' -d -it --name mycontainer ubuntu

Useful options for docker

docker run -d : Run as a daemon (runs in the background).
NFS mounting (pre 17.06)
- you@host > mount server:/dir /path/to/mount/point
- you@host > docker run -v /path/to/mount/point:/path/to/mount/point
docker run -p 4000:80 : Forward port 4000 to 80. So host listens on port 4000 and everything is forwarded to port 80 on the container.

Alpine Linux relocating /var/lib/docker

In the file /etc/conf.d/docker you can add additional command line options in:

DOCKER_OPTS

In particular you can use the -g option.

See linuxconfig.org article

Make your own docker image, quick example

Run the app

Making changes to an existing image

docker run -i -t [--name guest] image_name /bin/bash|/bin/sh
... make changes to it ...
docker stop name|container_id
`docker commit -m 'change name' -a 'A N Other' container_id image_name
- container id: $(docker ps -l -q)
docker rm guest|container_id

Better way to create container images:

Updating containers

Just use them...

Alpine on OTC

2022-01-04T00:00:00+01:00

These are just random thoughts nothing really was implemented.

Alpine Linux image

preparation: jq and other deps to /apks/x86_64
/etc/local.d/ cloud-init-lite

if /etc/network/intefaces exists we abort
apk add --force-non-repository /path oniguruma,jq .. restore /etc/apk/world
udhcpc -b -p /var/run/udhcpc.eth0.pid -i eth0
install openssh and start it
wget meta data and create /root/.ssh/authorized_keys

wget -O- http://169.254.169.254/openstack/YYYY-MM-DD/meta_data.json

Windows Account Lockouts

2021-12-25T00:00:00+01:00

To prevent windows lockouts the following can be done:

Delete Internet Explorer browsing history
Run the following:
- Open Start --> Search filed--> Type in Run --> rundll32.exe keymgr.dll, KRShowKeyMgr --> Delete
Disconnect network shares
Change password

Skipping grep when using AWK

2021-12-25T00:00:00+01:00

Over the years, We've seen many people use this pattern (filter-map):

$ [data is generated] | grep something | awk '{print $2}'

but it can be shortened to:

$ [data is generated] | awk '/something/ {print $2}'

You (probably) don't need grep

Following this logic, you can replace a simple grep with:

$ [data is generated] | awk '/something/'

This will implicitly print lines that match the regular expression.

If you feel lost, Here are a series of posts about awk for you:

Why would you want to do this?

There are a number of reasons:

it's shorter to type
it spawns one less process
awk uses modern (read "Perl") regular expressions, by default - like grep -E
it's ready to "augment" with more awk

What about `grep -v`?

grep -v can be done with:

$ [data is generated] | awk '! /something/'

Reference: jpalardy.com

Naming Schemes

2021-12-25T00:00:00+01:00

This web site contains list of names of different topics. This can be used for naming schemes:

Naming Schemes

Build a VR app in 15 minutes

2021-12-25T00:00:00+01:00

In 15 minutes, you can develop a virtual reality application and run it in a web browser, on a VR headset, or with Google Daydream. The key is A-Frame, an open source toolkit built by the Mozilla VR Team.

Test It

Open this link using Chrome or Firefox on your mobile phone.

Put your phone into Google Cardboard and stare at a menu square to switch the 360-degree scene.

Fork it

Fork the sample repository from GitHub. Change directory into the repo.

If you have 360-degree images, you can drop them into the img/ sub-directory. If you don't have 360-degree images, you can get started with the open source Hugin panorama photo stitcher. The boilerplate app includes RICOH THETA media I took at a meetup in San Francisco.

Create thumbnails

The menus in the headset are standard images that are 240x240 pixels. A-Frame handles the perspective shifts for you automatically.

Edit code

If you use the same image file names and overwrite 1.jpg in /img, you do not need to edit the code at all. If you want to extend the program or modify it with your own filenames, change the id and the src in index.html to match your files.

<body>
  <a-scene>
    <a-assets>
      <img id="kieran" src="img/1.jpg">
      <img id="kieran-thumb" crossorigin="anonymous" src="img/kieran-thumb.png">
      <img id="christian-thumb" crossorigin="anonymous" src="img/christian-thumb.png">
      <img id="eddie-thumb" crossorigin="anonymous" src="img/eddie-thumb.png">
      <audio id="click-sound" crossorigin="anonymous" src="https://cdn.aframe.io/360-image-gallery-boilerplate/audio/click.ogg"></audio>
      <img id="christian" crossorigin="anonymous" src="img/2.jpg">
      <img id="eddie" crossorigin="anonymous" src="img/4.jpg">

Scroll down and edit the section for the menu links.

<!-- 360-degree image. -->
<a-sky id="image-360" radius="10" src="#kieran"></a-sky>

<!-- Image links. -->
<a-entity id="links" layout="type: line; margin: 1.5" position="0 -1 -4">
  <a-entity template="src: #link" data-src="#christian" data-thumb="#christian-thumb"></a-entity>
  <a-entity template="src: #link" data-src="#kieran" data-thumb="#kieran-thumb"></a-entity>
  <a-entity template="src: #link" data-src="#eddie" data-thumb="#eddie-thumb"></a-entity>
</a-entity>

Upload to GitHub pages

Add and commit your changes:

git add *
git commit -a -m ?changed images'
git push

Open your app on a mobile phone at http://username.github.io/360gallery.

Next steps

This is a brief taste of A-Frame to illustrate that WebVR is easy and accessible to web developers. Go to aframe.io to see more demos. Although the display of 360 images is not true VR, it is easy, fun, and accessible today. Using 360 images is also a great way to start to understand the basics of augmented reality.

Take your own pictures with a standard camera and stitch them together or buy or borrow a 360-degree camera. The camera I used supports 360-degree video files and live streaming.

Troubleshooting

The application won't run from a local file that you open in your browser. You must either run a local webserver like Apache2 or upload it to an external site like GitHub Pages for testing.

If you're using an Oculus Rift or HTC Vive, you may need to install Firefox Nightly or experimental Chromium builds. See the current status of your browser at Is WebVR Ready?

360-degree video works on desktop browsers. I've experienced some glitches on mobile devices. The technology is improving quickly.

Source OpenSource

Set your google account to automatically delete

2021-12-25T00:00:00+01:00

Want to share your family photos after your death, but take your search history to the grave? All that and more is possible with Google's Inactive Account Manager.

How You Can Control Your Information After Death

It's not nice to think about, but one day, you will die, along with the keys to your online kingdom. And these days, those online accounts can hold a lot of stuff you may want to pass on.

Your Google account has a feature tucked deep in the bowels of your account settings called "Inactive Account Manager". Although the feature is several years old now, it's practically unknown among Google users–in a casual survey of people outside our office who had Google accounts, not one of them was aware of the feature.

Inactive Account Manager is what fans of old spy movies and psychological thrillers will immediately recognize as a "dead man's switch". Once activated if you do not interact with your Google account in X amount of time, then Google's servers will automatically either notify your trusted contacts and/or share specified data with those select contacts. Or, at your instruction, it can wipe your account.

In this way, you can ensure that things like family photos stored in Google Photos are available to your family, that your spouse can will have full access to your contacts to manage your business affairs, or that anyone you wish to share your account with upon your demise or incapacitation can access it legitimately and without resorting to masquerading themselves as you.

Setting Up the Inactive Account Manager

To set up Inactive Account Manager, make sure you're logged into your Google account and visit this page.

You can supply a contact phone number for the person (don't worry, they won't get an immediate text indicating that you've selected them, so there won't be any awkward conversations about death triggered by this process). Then you need to specify what Google data you want to share with them. We'd encourage you to apply this step selectively, and not simply check "Select all". Most of us would happily share our Google Photo collections with our next of kin, after all, but would prefer to take our search histories private.

The final step is the weightiest one: selecting whether or not your Google account will be wiped upon the completion of the timeout period.

There is no option to partially delete the data, so make this very binary decision with care. You cannot, for example, wipe your search history and email but leave your YouTube content and Blogger posts intact for posterity. Once the countdown is complete, like a real dead man's switch, the account data is gone forever.

Source HowToGee

HTML Entities

2021-12-25T00:00:00+01:00

(remember the ampersand at the start and the semi-colon at the end of each "tag")

| Á | Á | | á | á | | À | À | | Â | Â | | à | à | | Â | Â | | â | â | | Ä | Ä | | ä | ä | | Ã | Ã | | ã | ã | | Å | Å | | å | å | | Æ | Æ | | æ | æ | | Ç | Ç | | ç | ç | | Ð | Ð | | ð | ð | | É | É | | é | é | | È | È | | è | è | | Ê | Ê | | ê | ê | | Ë | Ë | | ë | ë | | Í | Í | | í | í | | Ì | Ì | | ì | ì | | Î | Î | | Ï | Ï | | ï | ï | | ñ | ñ | | Ó | Ó | | ó | ó | | Ò | Ò | | ò | ò | | Ô | Ô | | ô | ô | | Ö | Ö | | ö | ö | | Õ | Õ | | õ | õ | | Ø | Ø | | ø | ø | | ß | ß | | Þ | Þ | | þ | þ | | Ú | Ú | | ú | ú | | Ù | Ù | | ù | ù | | Û | Û | | û | û | | Ü | Ü | | ü | ü | | Ý | Ý | | ý | ý | | ÿ | ÿ | | © | © | | ® | ® | | ™ | ™ | | & | & | | < | < | | > | > | | € | € | | ¢ | ¢ | | £ | £ | | " | " | | ‘ | ‘ | | ’ | ’ | | “ | “ | | ” | ” | | « | « | | » | » | | — | — | | – | – | | ° | ° | | ± | ± | | ¼ | ¼ | | ½ | ½ | | ¾ | ¾ | | × | × | | ÷ | ÷ | | α | α | | β | β | | ∞ | ∞ | | |   |

Reference:

3 Open Source Password Managers

2021-12-25T00:00:00+01:00

Keep your data and accounts safe by using a secure open source password manager to store unique, complex passwords.

Maintaining complex, unique passwords for each site and service you use is among the most common pieces of advice that security professionals provide to the public every year.

Yet no matter how many times it is said, it seems like a week doesn't go by where a high-profile hacking story hits the news, revealing that users of the service in question more often than not had such secure passwords as "12345" or "password" as the only wall of protection on their account.

Or perhaps a user offers up just enough variation on the classic password selection to get past the minimal rules of the service. Unfortunately, "Pa$$w0rd!" isn't secure in any meaningful way, either. At this point, almost every variation of words and phrases strung together with a few numbers or substitutions is simply too easy for a password cracking tool to make its way through, and the shorter the password, the easier.

The best passwords are long, random or pseudo-random combinations of every possible character allowed, with a different password for each unique use. But how could a normal person remember the hundreds or even thousands of individual passwords associated with each account they've ever created? The short answer is: they can't. And don't even think about writing a password down in plain text, whether in the physical world or the digital.

Perhaps the easiest way to keep track of these complex, unique passwords is with a password manager, which provides easy access to strong encryption. While proprietary commercial solutions like LastPass are popular, there are several open source solutions as well. And with passwords, being able to audit the source code of your password manager is especially important, as it helps ensure that your passwords are encrypted properly and are not vulnerable to backdoors.

So without further ado, here are a few open source password managers we hope you will consider.

KeePass

KeePass is a GPLv2-licensed password manager, primarily designed for Windows but also running elsewhere. KeePass offers multiple strong encryption options, easy exports, multiple user keys, advanced searching features, and more. Designed for desktop use, there are plugins that allow direct use from your web browser, and it can run from a USB stick if you'd prefer to physically carry your passwords from machine to machine.

KeePassX, which started as a Linux port of KeePass, is another project you may consider. KeyPassX is compatible with KeePass 2 password files, and has also been ported to run on different operating systems.

Padlock

Padlock is a very new entrant into the world of open source password managers. Currently available for Windows, Mac, iOS, and Android, with a Linux client in the works, Padlock is designed as a "minimalist" password manager. Its source is available on GitHub. The project is also developing a cloud backend, also open source, which will be a welcomed addition to anyone tired of managing password files or setting up syncing across multiple computers.

Passbolt

Passbolt is another relatively new option, with plugins available for Firefox and Chrome and mobile and command-line options on the way. Based on OpenPGP, you can check out its online demo which shows off some of the features (you'll need to install the plugin for your browser, though). You can check out the source code on GitHub.

Using a password manager that you trust alongside complex passwords is not a substitute for taking other security precautions, nor is it foolproof. But for many users, it can be an important part of keeping your digital life secured. These definitely aren't the only options out there. There are some older options, like Clipperz and Password Safe, and web-based tools like RatticDB that I would be interested to try out.

Source opensource.com

How to encrypt linux partitions with LUKS

2023-09-29T00:00:00+02:00

There are plenty of reasons why people would need to encrypt a partition. Whether they're rooted it in privacy, security, or confidentiality, setting up a basic encrypted partition on a Linux system is fairly easy. This is especially true when using LUKS, since its functionality is built directly into the kernel.

Installing Cryptsetup

Debian/Ubuntu

On both Debian and Ubuntu, the cryptsetup utility is easily available in the repositories. The same should be true for Mint or any of their other derivatives.

$ sudo apt-get install cryptsetup

CentOS/Fedora

Again, the required tools are easily available in both CentOS and Fedora. These distributions break them down into multiple packages, but they can still be easily installed using yum and dnf respectively.

CentOS

# yum install crypto-utils cryptsetup-luks cryptsetup-luks-devel cryptsetup-luks-libs

Fedora

# dnf install crypto-utils cryptsetup cryptsetup-luks

OpenSUSE

OpenSUSE is more like the Debian based distributions, including everything that you need with cryptsetup.

# zypper in cryptsetup

Arch Linux

Arch stays true to its "keep it simple" philosophy here as well.

# pacman -S cryptsetup

Gentoo

The main concern that Gentoo users should have when installing the tools necessary for using LUKS is whether or not their kernel has support. This guide is not going to cover that part, but just be aware that kernel support is a factor. If your kernel does support LUKS, you can just emerge the package.

# emerge --ask cryptsetup

Setting Up The Partition

WARNING: The following will erase all data on the partition being used and will make it unrecoverable. Proceed with caution. From here on, none of this is distribution specific. It will all work well with any distribution.The defaults provided are actually quite good, but they can easily be customized. If you really aren't comfortable playing with them, don't worry. If you do know what you want to do, feel free.

The basic options are as follows:

--cypher: This determines the cryptographic cypher used on the partition. The default option is aes-xts-plain64
--key-size: The length of the key used. The default is 256
--hash: Chooses the hash algorithm used to derive the key. The default is sha256.
--time: The time used for passphrase processing. The default is 2000 milliseconds.
--use-random/--use-urandom: Determines the random number generator used. The default is --use-random.

So, a basic command with no options would look like the line below.

# cryptsetup luksFormat /dev/sdb1

Obviously, you'd want to use the path to whichever partition that you're encrypting. If you do want to use options, it would look like the following.

# cryptsetup -c aes-xts-plain64 --key-size 512 --hash sha512 --time 5000 --use-urandom /dev/sdb1

Cryptsetup will ask for a passphrase. Choose one that is both secure and memorable. If you forget it, your data will be lost. That will probably take a few seconds to complete, but when it's done, it will have successfully converted your partition into an encrypted LUKS volume.

Next, you have to open the volume onto the device mapper. This is the stage at which you will be prompted for your passphrase. You can choose the name that you want your partition mapped under. It doesn't really matter what it is, so just pick something that will be easy to remember and use.

# cryptsetup open /dev/sdb1 encrypted

Once the drive is mapped, you'll have to choose a filesystem type for you partition. Creating that filesystem is the same as it would be on a regular partition.

# mkfs.ext4 /dev/mapper/encrypted

The one difference between creating the filesystem on a regular partition and an encrypted one is that you will use the path to the mapped name instead of the actual partition location. Wait for the filesystem to be created. Then, the drive will be ready for use.

Mounting and Unmounting

Manually mounting and unmounting encrypted partitions is almost the same as doing so with normal partitions. There is one more step in each direction, though. First, to manually mount an encrypted partition, run the command below.

# cryptsetup --type luks open /dev/sdb1 encrypted
# mount -t ext4 /dev/mapper/encrypted /place/to/mount

Unmounting the partition is the same as a normal one, but you have to close the mapped device too.

# umount /place/to/mount
# cryptsetup close encrypted

Closing

There's plenty more, but when talking about security and encryption, things run rather deep. This guide provides the basis for encrypting and using encrypted partitions, which is an important first step that shouldn't be discounted. There will definitely be more coming in this area, so be sure to check back, if you're interested in going a bit deeper.

Source linuxconfig

Open Source Alternatives to Visio

2021-12-25T00:00:00+01:00

Need to create diagrams, flowcharts, circuits, or other kinds of entity-relationship models? Microsoft Visio is without a doubt the best software for that, but that doesn't mean it's the best choice for you.

Visio may be the industry standard in the corporate world, but it comes with a huge drawback: it's expensive ($299 for the standard version as of this writing). Can't afford that? Then you 'll be happy to know that several open source alternatives exist for the low, low price of FREE.

We're going to highlight the two best ones here, but if you don't like them for whatever reason, you can scroll down to the bottom of the article for even more options to explore.

Diagram Creation with Dia

Dia has been the go-to Visio alternative for many years. What I like most about it is the first impression that you get when it launches: clean, simple, with an interface that's familiar and easy to navigate. Quite reminiscent of Visio, in fact:

You'll be able to create your first diagram in mere minutes. Drag-and-drop a few symbols onto the canvas, then connect them using the various types available in the toolbox: lines, zigzags, arcs, circles, curves, etc.

Dia also supports layers, making it a lot easier to manage complex charts, and moving elements between layers is as simple as hitting a hotkey.

Snap to grid, easy resizing, text labels, image insertions -- Dia has it all. Anything you can do in Visio can be done in Dia as well. The only real downside is that Dia can't open Visio VSD files, but it can handle most other diagramming formats like XML, EPS, and SVG.

Download -- Dia (Free)

Diagram Creation with LibreOffice Draw

Have you heard of LibreOffice? As far as open source competitors to Microsoft Office go, you won't find a more solid and robust alternative.

LibreOffice is far from perfect, but it's a respectable option for fans of open source software. The app that should interest you is LibreOffice Draw, the Visio counterpart in this office suite.

LibreOffice Draw supplies two things for you: shapes and lines. You use the shapes to represent diagram entities, and you use the lines to connect them according to the entity relationships. It's perfect for creating flowcharts, but you can do more with it if you want (like desktop publishing or PDF editing).

First you have to open the Drawing toolbar, which you can do through View > Toolbars > Drawing. Grid snapping is on by default, but you'll want to change the snapping sensitivity by going to Tools > Options, navigate to LibreOffice Draw > Grid, change the values under Resolution to your intended grid size, and change the values under Subdivision to 1.

LibreOffice is surprisingly easy to use once it's set up properly. You can draw shapes, connectors, lines, curves, symbols, arrows, thought bubbles, and even 3D objects. If you're already using LibreOffice as your main office suite, forget Dia and learn to use Draw instead. The learning curve isn't much worse at all, and you can use it for more than just diagrams.

Download -- LibreOffice (Free)

Other Alternatives to Visio

Dia and Draw may be the best available right now, but a quick web search will turn up plenty of competitors that are just as good in many ways. Keep in mind that these are NOT open source unless specifically noted in the description.

yEd Graph Editor - Very similar to Dia, except much more powerful and proportionally harder to use. It has an automatic layout feature that can instantly rearrange a diagram to be clutter-free and more readable, which is fantastic for big and complex flowcharts.
LucidChart - A very solid alternative to Visio in a lot of ways. It's web-based, so you can access it from anywhere, and packed full of features that make diagramming easy.
draw.io - A no-login-required web-based diagramming tool that may not be the slickest in appearance, but can certainly get the job done. Diagrams can be saved to Dropbox, Google Drive, OneDrive, or locally. The interface is clean, the results are acceptable, and it's open source.

Reference: makeuseof

10 tips for making documentation crystal clear

2024-10-22T00:00:00+02:00

So you've some written excellent documentation. Now what? Now it's time to go back and edit it. When you first sit down to write your documentation, you want to focus on what you're trying to say instead of how you're saying it, but once that first draft is done it's time to go back and polish it up a little.

One of my favorite ways to edit is to read what I've written aloud. That's the best way to catch awkward phrasing or sentence structure that might not stand out when you're reading it to yourself. If it sounds good when you read it aloud, it probably is. If your documentation happens to include instructions, you can watch someone try to follow them. This provides good feedback on what steps are missing or unclear, particularly if the person is unfamiliar with the subject.

Active vs. passive voice

You should prefer the active voice in most cases. It's okay to be direct. How do you check for passive voice? Insert the words "by zombies". For example, it is much clearer to say "If you click 'yes', you will delete your data" instead of "If you click 'yes', data will be deleted." Apply the zombie test to these two examples:

"If you click 'yes', you will delete your data by zombies."
"If you click 'yes', data will be deleted by zombies."

In the first example there is no doubt that you are the actor. The second example shows that there is room for misinterpretation. Make it very clear what actors perform actions.

Eliminate jargon

Some jargon terms are unavoidable, but for the sake of clarity you should avoid them as much as possible. Linking to the definition of a term the first time you use it is acceptable, and you should also write your own brief definition. You don't want to rely on the availability of external sites, or make your readers jump through too many hoops to understand your documentation.

Check for common mistakes

Question everything you think you know, and take advantage of having the world at your fingertips and look up everything. For example, "e.g." means "for example" and "i.e." means "in other words". "Effect" is a noun (except when it isn't, as the comic xkcd demonstrates), and "affect" is a verb. Use "which" when a clause can be removed from the sentence without a change in meaning, and "that" when it cannot.

Remove dangling modifiers

You have created a dangling modifier when it is unclear which object is being modified by a word or phrase. A classic example is "Hungry, the leftover food was devoured". Is Hungry the name of the food? Add a comma after "food" if that is your meaning. If not, rewrite it to make your meaning unambiguous: "Your author was hungry and devoured the leftover food." Organize your sentences carefully; don't force your readers to guess your meaning.

Check your style guide

If your project or company has a style guide for documentation, check that what you've written conforms to it. One common mistake is the inappropriate abbreviation of company and project names.

Avoid unclear words

Do you know how many times I deleted words like "often" and "some" while writing this article? I don't know exactly how many, but I know it's a non-zero number. Use words with specific meanings. This is particularly important when you're trying to convince your reader that what you're telling them is important. If I say "Following these tips will make your writing better", that's less convincing than "Following these tips will increase financial contributions to your project by 45%." If you catch yourself using vague words, ask yourself if you really understand your topic, or if perhaps you're trying to hide something.

Check the word order

The English language does not have a formally-defined ordering structure for modifying nouns, but there is an informal structure which is described in this Tweet by Matthew Anderson: Opinion-size-age-shape-color-origin-material-purpose Noun. It's something native English speakers know, but don't know we know. Peter Sokolowski replied with advice to Put the "nounier" words closer to the noun. If that doesn't help you read the discussion to his reply, which has numerous examples and explanations.

Remove words like "just" and "simply"

Technology is not as simple as we like to pretend it is. If you tell your readers that something is simple and then they can't do it, what are they to think of themselves? Unless you're writing an infomercial for the latest must-have kitchen gadget, leave out those words.

Check your pronouns

When you say "we", who do you really mean? I have seen documentation written in what I call "cooking show style", where "Next we click the whatchamadoozit to fribble the wozulator." When you're writing in a support context it is especially important to be clear who does what. If you tell someone "We can change that setting", they expect that you will do it for them, and not that they can do it with your guidance. As a general rule, I avoid using first person (I/we) except when I am talking about myself as the author or the organization I am representing. When in doubt, refer to yourself in the third person (e.g. "The author suggests you refer to yourself in the third person"). It might sound overly formal, but it is clear.

Remove split infinitives

Don't put words in between "to" and a verb. Your documentation is on a mission to go boldly where no docs have done before. (Starship captains are excused from this rule).

Reference: opensource life

Project Requirements

2022-11-02T00:00:00+01:00

This is sometimes so true!

Ascii Art Tools

2021-12-25T00:00:00+01:00

Here are some resources dealling with ASCII art...

AsciiToSVG - PHP code to convert ascii art into SVG.
AsciiFlow - Web App implement an ascii art editor.
Asciio A perl application allows you to draw ASCII diagrams in a modern (but simple) graphical interface.
ditaa - Java based ascii art to PNG converter.
shaape - Shaape is an ascii art to image converter designed to be used with asciidoc.
blockdiag - blockdiag and its family generate diagram images from simple text files. (Python)

A few more added ones:

6 Cloud-Based Tools To Help You Build A Web App With Ease

2021-12-25T00:00:00+01:00

In just a relatively short amount of time, building mobile apps has transformed from a process that included lots of knowledge in developing into something that almost anyone can do. Cloud-based tools are quickly becoming the norm for app developers, and these are some of the highest recommended tools, each one being ideal for certain developers.

Source: 6 Cloud-Based Tools To Help You Build A Web App With Ease

So the server crashed

2021-12-25T00:00:00+01:00

Now we have to re-create things...

custom desktop ideas

2021-12-25T00:00:00+01:00

fast boot

Window Manager

Snap windows to border/windows

File Manager

Next style file manager
TkDesk : Note, it use [incr tcl]
TkMC : Basic file browser functionality
TOXFile: Another file manager
TkWm

Notification and Tray areas

Launcher

Applications

browser
media player
photo/video manager (or webapp)
office apps (open365.io)
ah photo album app
tkpaint
retrolook is a Window Manager

TCL/TK

libwm

Alt-desktop

Use assist to do a basic install
Set-up xorg

pacman -S xorg-server xorg-server-utils xorg-xinit xorg-utils mesa
xterm xorg-twm xorg-xclock

Drivers

nvidia
xf86-video-ati
xf86-video-intel

Common

xf86-input-synaptics : for laptops
ttf-dejavu ttf-droid ttf-freefont ttf-liberation ttf-opensans
aur ttf-ms-fonts

budgie: budgie-desktop

Retropie

2021-12-25T00:00:00+01:00

DVD player
Bluetooth receiver
keyboard
good keyboard bindings
how to exit games
convert probox into binding keys
where are key codes saved

Write image:
- gunzip < retropie-4.3-rpi2_rpi3.img.gz | sudo dd of=/dev/sde bs=4M
Configure config.txt
- hdmi_force_hotplug=1
- hdmi_drive=2
Boot and configure keyboard
- D-Pad => D-Pad
- start : mike (F5)
- select : menu key
- A : OK (enter)
- B : back
- X : vol-
- Y : vol+ ... skip all ...
- HotKey : power
Configure SSH
- sudo raspi-config
  - Interfacing Options
  - Enable SSH
Install Kodi
- ?

NFS
- nfs-common should be installed.
- sudo vi /etc/default/nfs-common
- activate statd
- Add systemctl start rpcbind|nfs-common to /etc/rc.local
- Doing systemctl enable resulted in a messed up system...
- sudo apt-get install autofs
- Enable /net hosts from /etc/auto.master ... pointing to /etc/auto.net
  - alvm1-xvdb1d -> /net/alvm1/media/xvdb1/d
  - alvm1-xvdb1m -> /net/alvm1/media/xvdb1/m
  - alvm1-xvdb1p -> /net/alvm1/media/xvdb1/p
  - alvm1-xvdc1p -> /net/alvm1/media/xvdc1/p
  - alvm1-xvdd1p -> /net/alvm1/media/xvdd1/p
  - ow1-v1 -> /net/ow1/data/v1
  - vs1-xvdb1 -> /net/vs1/media/xvdb1
RTC
- sudo raspi-config
  - Interfacing options
  - I2C
- Edit /boot/config.txt
  - dtoverlay=i2c-rtc,pcf2127
Power control:mausberry
- git clone https://github.com/t-richards/mausberry-switch.git
- autoreconf -i -f
- ./configure
- make
- sudo make install
- Add also mausberry-switch & to /etc/rc.local
CEC control
- sudo apt-get install cec-utils
- sudo apt-get install python-pip
- sudo pip install python-uinput
- git clone https://github.com/dillbyrne/es-cec-input.git
input

cron backup kodi
document cec input
re-do keyboard bindings

using cachefiles on an Linux NFS share

2021-12-25T00:00:00+01:00

If you often mount and access a remote NFS share on your system, you will probably want to know how to improve NFS file access performance. One possibility is using file caching. In Linux, there is a caching filesystem called FS-Cache which enables file caching for network file systems such as NFS. FS-Cache is built into the Linux kernel 2.6.30 and higher.

In order for FS-Cache to operate, it needs cache back-end which provides actual storage for caching. One such cache back-end is cachefiles. Therefore, once you set up cachefiles, it will automatically enable file caching for NFS shares.

In this tutorial, I will describe how to enable local file caching for NFS shares by using cachefiles.

Requirements for Setting Up CacheFiles

One requirement for setting up cachefiles is that local filesystem support user-defined extended file attributes (i.e., xattr), because cachefiles use xattr to store extra information for cache maintenance.

If your local filesystem is ext4-type, you don't need to worry about this since xattr is enabled in ext4 by default.

However, if you are using ext3 filesystem, then you need to mount the local filesystem with "user_xattr" option. To do so, edit /etc/mtab to add "user_xattr" mount option to the disk partition that will be used by cachefiles for file caching. For example, assuming that /dev/hda1 is such a partition:

/dev/hda1 / ext3 rw,user_xattr  0 0

After modifying /etc/fstab, reload it by running:

sudo mount -o remount /

Set Up CacheFiles

In order to set up cache back-end using cachefiles, you need to install cachefilesd, a userspace daemon for managing cachefiles.

To install cachefilesd on Ubuntu or Debian:

sudo apt-get install cachefilesd

To install cachefilesd on CentOS, Fedora or RedHat:

sudo yum install cachefilesd
sudo chkconfig cachefilesd on

After installation, enable cachefilesd by editing its configuration file as follows.

sudo vi /etc/default/cachefilesd

RUN=yes

Next, mount a remote NFS share with fsc option:

sudo vi /etc/fstab

192.168.1.13:/home/xmodulo /mnt nfs rw,hard,intr,fsc

Alternatively, if you mount the remote NFS share from the command line, specify fsc as a command-line option:

sudo mount -t nfs 192.168.1.13:/home/xmodulo /mnt -o fsc

Finally, restart cachefilesd:

sudo service cachefilesd restart

At this point, file caching should be enabled for the mounted NFS share, which means that previously accessed files in the mounted NFS share will be retrieved from local file cache.

If you want to flush NFS file cache for any reason, simply restart cachefilesd.

sudo service cachefilesd restart

Reference: xmodule.com

VNC desktop

2021-12-25T00:00:00+01:00

IDEA:

Client connects >
        < server sends version string (Use 3.3 only)
Client replies with actual verison string >
        < server sends security type; NONE
Client send ClientInit (shared flag) > 
        < sever sens ServerInit (server details) WxHxD Name
=== standard stuff ===

2 VERSIONS

kiosk
- unmodified vncviewer connects to a multiplexer screen
- server (in inetd mode) first spawns a Xvnc (in inetd mode) which does a login authentication and finds an existing desktop or spawns a new one saves the port and exists
- server then connects to the new desktop port and does the VNC handshake. Sends client Desktop change and name change messages
- Forwards everything...
command
- User points command to a server.
- Script selects a new port.
- Ssh to server, look for vnc session, or spawn new one.
- netcat to vnc session.
- Listen to to new port, and netcat to ssh.
- vncviewer to netcat port.

We use only v3.3 because we don't want to mess with security types. Security should be handled by SSH tunnel.

Check if user can login

SimpleNote/Markdown editor

2021-12-25T00:00:00+01:00

We just load SimpleNote as a Desktop WebApp

Create first a basic markdown editor
- styling

Hack retext
- add stuff for multiple views
nvpy remixed with retext?
Wrapper python for Windows

Another option:

SimpleNote native app + editor

Features:

Search tags from a menu
Search within document
Poll website for changes

Or just use: mistune

Basic editors

Syntax highligthing:

Interesting wordpress plugins

2021-12-25T00:00:00+01:00

Rollback with YUM History Command

2021-12-25T00:00:00+01:00

From 2daygeek.com

Server patching is one of the important task of Linux system administrator to make the system more stable and better performance. All the vendors used to release security/vulnerabilities patches very often, the affected package must be updated in order to limit any potential security risks.

Yum (Yellowdog Update Modified) is RPM Package Management utility for CentOS and Red Hat systems, Yum history command allows administrator to rollback the system to a previous state but due to some limitations, rollbacks do not work in all situations, or The yum command may simply do nothing, or it may remove packages you do not expect.

I advise you to take a full system backup prior to performing any update/upgrade is always recommended, and yum history is NOT meant to replace systems backups. This will help you to restore the system to previous state at any point of time.

n some cases, the hosted applications might not work properly or through some error due to recent patch updates (It could be some library incompatibility or package upgrade), what will be the solution in this case?

Get in touch with App Dev team and figure it out an issue creating library' and packages then do the rollback with help of yum history command.

Note:

Rollback of selinux, selinux-policy-*, kernel, glibc (dependencies of glibc such as gcc) packages to older version is not supported. Downgrading a system to minor version is not recommended (CentOS 6.9 to CentOS 6.8) which leads to make the system in undesired state Let' first verify an available updates on system and pick any of the package for this experiment.

# yum update
Loaded plugins: fastestmirror, security
Setting up Update Process
Loading mirror speeds from cached hostfile
epel/metalink                                                             |  12 kB     00:00
 * epel: mirror.csclub.uwaterloo.ca
base                                                                      | 3.7 kB     00:00
dockerrepo                                                                | 2.9 kB     00:00
draios                                                                    | 2.9 kB     00:00
draios/primary_db                                                         |  13 kB     00:00
epel                                                                      | 4.3 kB     00:00
epel/primary_db                                                           | 5.9 MB     00:00
extras                                                                    | 3.4 kB     00:00
updates                                                                   | 3.4 kB     00:00
updates/primary_db                                                        | 2.5 MB     00:00
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.7.1-8.el6 will be updated
---> Package git.x86_64 0:1.7.1-9.el6_9 will be an update
---> Package httpd.x86_64 0:2.2.15-60.el6.centos.4 will be updated
---> Package httpd.x86_64 0:2.2.15-60.el6.centos.5 will be an update
---> Package httpd-tools.x86_64 0:2.2.15-60.el6.centos.4 will be updated
---> Package httpd-tools.x86_64 0:2.2.15-60.el6.centos.5 will be an update
---> Package perl-Git.noarch 0:1.7.1-8.el6 will be updated
---> Package perl-Git.noarch 0:1.7.1-9.el6_9 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================
 Package               Arch             Version                          Repository         Size
=================================================================================================
Updating:
 git                   x86_64           1.7.1-9.el6_9                    updates           4.6 M
 httpd                 x86_64           2.2.15-60.el6.centos.5           updates           836 k
 httpd-tools           x86_64           2.2.15-60.el6.centos.5           updates            80 k
 perl-Git              noarch           1.7.1-9.el6_9                    updates            29 k

Transaction Summary
=================================================================================================
Upgrade       4 Package(s)

Total download size: 5.5 M
Is this ok [y/N]: n

As you can see in the above output git package update is available, so we are going to take that. Run the following command to know the version information about the package (current installed version and available update version).

# yum list git
Loaded plugins: fastestmirror, security
Setting up Update Process
Loading mirror speeds from cached hostfile
 * epel: mirror.csclub.uwaterloo.ca
Installed Packages
git.x86_64                                 1.7.1-8.el6                                    @base
Available Packages
git.x86_64                                 1.7.1-9.el6_9                                  updates

Run the following command to update git package from 1.7.1-8 to 1.7.1-9.

# yum update git
Loaded plugins: fastestmirror, presto
Setting up Update Process
Loading mirror speeds from cached hostfile
 * base: repos.lax.quadranet.com
 * epel: fedora.mirrors.pair.com
 * extras: mirrors.seas.harvard.edu
 * updates: mirror.sesp.northwestern.edu
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.7.1-8.el6 will be updated
--> Processing Dependency: git = 1.7.1-8.el6 for package: perl-Git-1.7.1-8.el6.noarch
---> Package git.x86_64 0:1.7.1-9.el6_9 will be an update
--> Running transaction check
---> Package perl-Git.noarch 0:1.7.1-8.el6 will be updated
---> Package perl-Git.noarch 0:1.7.1-9.el6_9 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================
 Package               Arch                Version                    Repository            Size
=================================================================================================
Updating:
 git                   x86_64              1.7.1-9.el6_9              updates              4.6 M
Updating for dependencies:
 perl-Git              noarch              1.7.1-9.el6_9              updates               29 k

Transaction Summary
=================================================================================================
Upgrade       2 Package(s)

Total download size: 4.6 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 4.6 M
(1/2): git-1.7.1-9.el6_9.x86_64.rpm                                       | 4.6 MB     00:00
(2/2): perl-Git-1.7.1-9.el6_9.noarch.rpm                                  |  29 kB     00:00
-------------------------------------------------------------------------------------------------
Total                                                            5.8 MB/s | 4.6 MB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : perl-Git-1.7.1-9.el6_9.noarch                                                 1/4
  Updating   : git-1.7.1-9.el6_9.x86_64                                                      2/4
  Cleanup    : perl-Git-1.7.1-8.el6.noarch                                                   3/4
  Cleanup    : git-1.7.1-8.el6.x86_64                                                        4/4
  Verifying  : git-1.7.1-9.el6_9.x86_64                                                      1/4
  Verifying  : perl-Git-1.7.1-9.el6_9.noarch                                                 2/4
  Verifying  : git-1.7.1-8.el6.x86_64                                                        3/4
  Verifying  : perl-Git-1.7.1-8.el6.noarch                                                   4/4

Updated:
  git.x86_64 0:1.7.1-9.el6_9

Dependency Updated:
  perl-Git.noarch 0:1.7.1-9.el6_9

Complete!

Verify updated version of git package.

# yum list git
Installed Packages
git.x86_64                                 1.7.1-9.el6_9                                 @updates

or
# rpm -q git
git-1.7.1-9.el6_9.x86_64

As of now, we have successfully completed package update and got a package for rollback. Just follow below steps for rollback mechanism.

First get the yum transaction id using following command. The below output clearly shows all the required information such transaction id, who done the transaction (i mean username), date and time, Actions (Install or update), how many packages altered in this transaction.

# yum history
or
# yum history list all
Loaded plugins: fastestmirror, presto
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    13 | root               | 2017-08-18 13:30 | Update         |    2
    12 | root               | 2017-08-10 07:46 | Install        |    1
    11 | root               | 2017-07-28 17:10 | E, I, U        |   28 EE
    10 | root               | 2017-04-21 09:16 | E, I, U        |  162 EE
     9 | root               | 2017-02-09 17:09 | E, I, U        |   20 EE
     8 | root               | 2017-02-02 10:45 | Install        |    1
     7 | root               | 2016-12-15 06:48 | Update         |    1
     6 | root               | 2016-12-15 06:43 | Install        |    1
     5 | root               | 2016-12-02 10:28 | E, I, U        |   23 EE
     4 | root               | 2016-10-28 05:37 | E, I, U        |   13 EE
     3 | root               | 2016-10-18 12:53 | Install        |    1
     2 | root               | 2016-09-30 10:28 | E, I, U        |   31 EE
     1 | root               | 2016-07-26 11:40 | E, I, U        |  160 EE

The above command shows two packages has been altered because git updated it' dependence package too perl-Git. Run the following command to view detailed information about the transaction.

# yum history info 13
Loaded plugins: fastestmirror, presto
Transaction ID : 13
Begin time     : Fri Aug 18 13:30:52 2017
Begin rpmdb    : 420:f5c5f9184f44cf317de64d3a35199e894ad71188
End time       :            13:30:54 2017 (2 seconds)
End rpmdb      : 420:d04a95c25d4526ef87598f0dcaec66d3f99b98d4
User           : root 
Return-Code    : Success
Command Line   : update git
Transaction performed with:
    Installed     rpm-4.8.0-55.el6.x86_64                       @base
    Installed     yum-3.2.29-81.el6.centos.noarch               @base
    Installed     yum-plugin-fastestmirror-1.1.30-40.el6.noarch @base
    Installed     yum-presto-0.6.2-1.el6.noarch                 @anaconda-CentOS-201207061011.x86_64/6.3
Packages Altered:
    Updated git-1.7.1-8.el6.x86_64        @base
    Update      1.7.1-9.el6_9.x86_64      @updates
    Updated perl-Git-1.7.1-8.el6.noarch   @base
    Update           1.7.1-9.el6_9.noarch @updates
history info

Fire the following command to Rollback the git package to the previous version.

# yum history undo 13
Loaded plugins: fastestmirror, presto
Undoing transaction 53, from Fri Aug 18 13:30:52 2017
    Updated git-1.7.1-8.el6.x86_64        @base
    Update      1.7.1-9.el6_9.x86_64      @updates
    Updated perl-Git-1.7.1-8.el6.noarch   @base
    Update           1.7.1-9.el6_9.noarch @updates
Loading mirror speeds from cached hostfile
 * base: repos.lax.quadranet.com
 * epel: fedora.mirrors.pair.com
 * extras: repo1.dal.innoscale.net
 * updates: mirror.vtti.vt.edu
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.7.1-8.el6 will be a downgrade
---> Package git.x86_64 0:1.7.1-9.el6_9 will be erased
---> Package perl-Git.noarch 0:1.7.1-8.el6 will be a downgrade
---> Package perl-Git.noarch 0:1.7.1-9.el6_9 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================
 Package                Arch                 Version                    Repository          Size
=================================================================================================
Downgrading:
 git                    x86_64               1.7.1-8.el6                base               4.6 M
 perl-Git               noarch               1.7.1-8.el6                base                29 k

Transaction Summary
=================================================================================================
Downgrade     2 Package(s)

Total download size: 4.6 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 4.6 M
(1/2): git-1.7.1-8.el6.x86_64.rpm                                         | 4.6 MB     00:00
(2/2): perl-Git-1.7.1-8.el6.noarch.rpm                                    |  29 kB     00:00
-------------------------------------------------------------------------------------------------
Total                                                            3.4 MB/s | 4.6 MB     00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : perl-Git-1.7.1-8.el6.noarch                                                   1/4
  Installing : git-1.7.1-8.el6.x86_64                                                        2/4
  Cleanup    : perl-Git-1.7.1-9.el6_9.noarch                                                 3/4
  Cleanup    : git-1.7.1-9.el6_9.x86_64                                                      4/4
  Verifying  : git-1.7.1-8.el6.x86_64                                                        1/4
  Verifying  : perl-Git-1.7.1-8.el6.noarch                                                   2/4
  Verifying  : git-1.7.1-9.el6_9.x86_64                                                      3/4
  Verifying  : perl-Git-1.7.1-9.el6_9.noarch                                                 4/4

Removed:
  git.x86_64 0:1.7.1-9.el6_9                   perl-Git.noarch 0:1.7.1-9.el6_9

Installed:
  git.x86_64 0:1.7.1-8.el6                     perl-Git.noarch 0:1.7.1-8.el6

Complete!

After rollback, use the following command to re-check the downgraded package version.

# yum list git
or
# rpm -q git
git-1.7.1-8.el6.x86_64

Rollback Updates using YUM downgrade command

Alternatively we can rollback an updates using YUM downgrade command.

# yum downgrade git-1.7.1-8.el6 perl-Git-1.7.1-8.el6
Loaded plugins: search-disabled-repos, security, ulninfo
Setting up Downgrade Process
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.7.1-8.el6 will be a downgrade
---> Package git.x86_64 0:1.7.1-9.el6_9 will be erased
---> Package perl-Git.noarch 0:1.7.1-8.el6 will be a downgrade
---> Package perl-Git.noarch 0:1.7.1-9.el6_9 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================
 Package                Arch                 Version                    Repository          Size
=================================================================================================
Downgrading:
 git                    x86_64               1.7.1-8.el6                base               4.6 M
 perl-Git               noarch               1.7.1-8.el6                base                29 k

Transaction Summary
=================================================================================================
Downgrade     2 Package(s)

Total download size: 4.6 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): git-1.7.1-8.el6.x86_64.rpm                                         | 4.6 MB     00:00
(2/2): perl-Git-1.7.1-8.el6.noarch.rpm                                    |  28 kB     00:00
-------------------------------------------------------------------------------------------------
Total                                                            3.7 MB/s | 4.6 MB     00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : perl-Git-1.7.1-8.el6.noarch                                                   1/4
  Installing : git-1.7.1-8.el6.x86_64                                                        2/4
  Cleanup    : perl-Git-1.7.1-9.el6_9.noarch                                                 3/4
  Cleanup    : git-1.7.1-9.el6_9.x86_64                                                      4/4
  Verifying  : git-1.7.1-8.el6.x86_64                                                        1/4
  Verifying  : perl-Git-1.7.1-8.el6.noarch                                                   2/4
  Verifying  : git-1.7.1-9.el6_9.x86_64                                                      3/4
  Verifying  : perl-Git-1.7.1-9.el6_9.noarch                                                 4/4

Removed:
  git.x86_64 0:1.7.1-9.el6_9                   perl-Git.noarch 0:1.7.1-9.el6_9

Installed:
  git.x86_64 0:1.7.1-8.el6                     perl-Git.noarch 0:1.7.1-8.el6

Complete!

Note : You have to downgrade a dependence packages too, otherwise this will remove the current version of dependency packages instead of downgrade because the downgrade command cannot satisfy the dependency.

For Fedora Users

Use the same above commands and change the package manager command to DNF instead of YUM.

# dnf list git
# dnf history
# dnf history info
# dnf history undo
# dnf list git
# dnf downgrade git-1.7.1-8.el6 perl-Git-1.7.1-8.el6

SeedBoxes

2021-12-25T00:00:00+01:00

A seedbox is a dedicated server at a high speed datacenter with a public IP address for the downloading and seeding of bittorrent files. Persons who have access to a seedbox can download these files to their personal computers at any time and from any place that has an internet connection.

References:

Centos Install notes

2022-01-04T00:00:00+01:00

Set-up local.repo

yum installs:

nfs-utils autofs
@x11
@xfce
wget
dejavu-sans-fonts dejavu-sans-mono-fonts dejavu-serif-fonts
xorg-x11-fonts-{Type1,misc,75dpi,100dpi}
bitmap-console-fonts bitmap-fixed-fonts bitmap-fonts-compat bitmap-lucida-typewriter-fonts
ucs-miscfixed-fonts urw-fonts
open-sans-fonts
webcore-fonts webcore-fonts-vista
liberation-mono-fonts liberation-sans-fonts liberation-serif-fonts
bitstream-vera-sans-fonts bitstream-vera-serif-fonts
gnu-free-{mono,sans,serif}-fonts
tk
firefox
mplayer ffmpeg alsa-utils
xsensors xfce4-sensors-plugin
keepassx
git

Building RPM packages with mock and Centos mock overview

Using mock with fedora

packaging rpms with mock

Adding trusted certs… or rehat solution

php using ssl

Chrome on Centos7 from google repos.

travis ci installation

Install ruby (must be greater than 1.9.3, 2.0.0 recomended)
Additional dependencies (through yum)
ruby-ffi

As normal user:

gem install travis -v 1.8.8 --no-rdoc --no-ri

For a new application...

travis enable
travis settings builds_only_with_travis_yml -t

PRoot

project -module: centos/alpine -module: proot

travis ci custom build
install: install any dependencies required
script: run the build script

Nameing conventions

Naming tmXXXXYYYYRRRR

tmc7r1 : centos 7 template
tmwin7r1 : not using these, I think it is better to do fresh install...
tmal3r1 : alpine linux template.
- after boot:
  1. mount xvda1 on /media/xvda1 and run setup-alpine
  2. modify /etc/inittab /etc/securetty to allow ttyS0 (console) login
  3. may need to switch cdrom as needed.

Sample vm names:

winvm1 : windows vm
cvm2 : centos vm
alvm3 : alpine linux vm

More complete guides:

Second wifi on OpenWrt

android development

2021-12-25T00:00:00+01:00

Android devs

Install JAVA

yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

Install SDK Tools:

Download the sdk-tools zip from here

mkdir /opt/android
cd /opt/android

The sdk should go under /opt/android/tools

unzip sdk-tools.zip
sudo chmod a+x $(sudo find . -type f -executable )

Create a /etc/profile.d or just modify PATH directly

ANDROID_HOME=/opt/android
$ANDROID_HOME/tools/bin:$ANDROID_HOME/platform-tools:$ANDROID_HOME/tools

Then run:

sudo /opt/android/tools/bin/sdkmanager tools

This refreshes the tools and makes sure some basic stuff is there.

SDK Manager packages

sdkmanager --list

TO check the list and install (use sudo...)

build-tools;<version>
platforms;android-<version>
 'system-images;android-19;google_apis;x86'

Install latest, and others as required

This is needed by cordova and some applications:

Gradle

Choose binary only releases

cd /opt
unzip gradle-bin.zip
ln -s gradle-<version> gradle

Add gradle bin to PATH

When creating AVDs need to tweak config.ini (inside $HOME/.android/avd/.avd

hw.gpu.enabled=yes
hw.gpu.mode=host

CORDOVA INSTALL

sudo yum install nodejs npm (From EPEL)
sudo npm install -g cordova
sudo npm install -g typescript (Optional)

Securing rsync on ssh

2021-12-25T00:00:00+01:00

Reference: positon.org

You have 2 systems and you want to set up a secure backup with rsync + SSH of one system to the other.

Very simply, you can use:

backup.example.com# rsync -avz --numeric-ids --delete root@myserver.example.com:/path/ /backup/myserver/

To do the backup, you have to be root on the remote server, because some files are only root readable.

Problem: you will allow backup.example.com to do anything on myserver.example.com, where just read only access on the directory is sufficient.

To solve it, you can use the command="" directive in the authorized_keys file to filter the command.

To find this command, start rsync adding the -e'ssh -v' option:

rsync -avz -e'ssh -v' --numeric-ids --delete root@myserver.example.com:/path/ /backup/myserver/ 2>&1 | grep "Sending command"

You get a result like:

debug1: Sending command: rsync --server --sender -vlogDtprze.iLsf --numeric-ids . /path/

Now, just add the command before the key in /root/.ssh/authorized_keys:

command="rsync --server --sender -vlogDtprze.iLsf --numeric-ids . /path/" ssh-rsa AAAAB3NzaC1in2EAAAABIwAAABio......

And for even more security, you can add an IP filter, and other options:

from="backup.example.com",command="rsync --server --sender -vlogDtprze.iLsf --numeric-ids . /path/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB3NzaC1in2EAAAABIwAAABio......

Now try to open a ssh shell on the remote server.. and try some unauthorized rsync commands... Notes:

Beware that if you change rsync command options, change also the authorized_keys file.
No need for complex chroot anymore.

Desktop environments on Centos 7

2021-12-25T00:00:00+01:00

These are commands to install different Desktop environments on Centos7

Gnome

yum groupinstall 'GNOME Desktop'

KDE

yum groupinstall "KDE Plasma Workspaces"

Cinnamon

yum install epel-release
yum --enablerepo=epel install cinnamon

MATE

yum install epel-release
yum --enablerepo=epel groupinstall "MATE Desktop"

XFCE

yum install epel-release
yum --enablerepo=epel groupinstall XFCE

2021-12-25T00:00:00+01:00

Telegram is a messenger designed to overcome the limitations of other messengers like WhatsApp or similar ones. It is different and better than other messengers on more than one level. A few of the important features that make it stand out among other messengers are:

Open API. This enables the users to develop their own versions of the telegram.
Open Protocol. This protocol is highly secure and will protect important messages from hackers
Free. As is freedom as well as subscription. There will be no ads and the developers intend to keep it that way forever.
It has unlimited cloud storage. There is no limit on the size of media or chats.

Available on all devices like tablets, smartphones, desktop ( Mac, Windows, and Linux as well) and there is web version too... Telegram is not here for profits but instead to provide users an alternative to all those other messengers which don' value privacy. We will now look at setting up of the desktop telegram on Linux.

I used arch with deepin desktop, but telegram should work in any Linux based PC.

First, Visit the telegram website and choose telegram for PC/Mac/Linux.

Then, the website should automatically detect your Linux and show you the download link. Or you can select the one that suits your PC. Keep in mind the CPU architecture 32 bit or 64 bit. Click on Download.

From linuxandubuntu

Free Clipart sites

2022-01-10T00:00:00+01:00

In 2014, Microsoft killed and buried Clipart in the digital graveyard.

Clipart had outlived its usefulness as users relied more on search engines than Microsoft' somewhat limited supply through the Office suite.

Today' clipart needs to be modern, colorful, and less cartoonish. An online search for clip art images will net you files that are of far better quality and relevance. But you need shortcuts when searching for the right image is a daily workout.

These are 13 of the top websites for free clipart downloads. Browse through them and bookmark the ones that meet your needs.

clker.com : This site is among the more neatly designed ones you can expect to find for clipart hunting. The images are free to download and use. All images are in the public domain, so feel free to use them anywhere after agreeing to the site' terms of use policy.
Vecteezy : Vecteezy covers the gamut of vector art -from vector icons to vector patterns. Both free and premium image files are available for use. You can visit the small section for free vector clipart downloads which is still choc-full of nearly 75000 assets. The community contributions keep the artwork fresh and updated.
ClipArt Etc. : Start with any of the 71,500 clip art images on this simple site. The site keeps its focus on classroom friendly images that are appropriate for school websites, class projects, student reports, homework assignments, presentations, posters, art projects, picture books, bulletin boards, and creating teaching aids. For instance, go over to the Fractions collection that has nearly 500 files to help demonstrate math concepts in class.
Webweaver' Free Clipart : Free images and animations without the annoyance of pop ups or registration. That' how this simple site announces itself, and it lives up to its name. Clip art categories include Holidays, Animals & Nature, Celebrations, Historical, and even a dedicated one for Fantasy. No Game of Thrones here, but I spotted a few that could be passed off as Gandalf from LOTR.
Clipart Of : Clipart Of is a stock image website offering royalty-free vector, cartoon and 3D files, illustrations and clipart from artists around the world. One glance at the home page tells you that the quality is better than the average. But you have to pay for them.
ArtVex : The free clip art database gives you more than 10,000 original images to choose from. The files are neatly categorized and you can also use the Google custom search engine to go through the vast collection. Some useful categories include Shapes Signs & Symbols, Math, Callouts, and Stickmen & Figures.
Clipart Lord : There' nothing remarkable about the site apart from the fact that someone has gone to the pains of collecting the best clip art found over the web. If you are a clip art buff then the simple site is worth a bookmark. I usually find some excellent files here and use them in my projects.
Vector Portal : The site creates and showcases free stock vectors which designers can use in commercial projects. The library includes a good collection of stock vectors and clip art images which you can use with an attribution. Vector Portal allows you non-exclusive, non-transferable right to use and modify its images which carry a "or commercial use" license. Most of the files are in the EPS and AI formats (supported by Adobe Illustrator).
Free PNG IMG : You can you can download free PNG images, pictures, icons and clip arts in high resolution quality. Categories covered include animals, artistic icons, cars, cartoons, clothing, electronics, games, fantasy, and more. It is effortless to drill down to the one you want with the detailed category breakdown. Some of the icons you might not find on most sites cover learning, internet, and entertainment related themes.
PD Clip Art : Public Domain Clip Art is a growing trove of clipart images which requires no sign-up to download and use.
All Free Download : More than 21000 clip art choices organized in 700 pages should be enough to keep you busy. You don' have to rummage around as all files are organized around tags. Most files are in the Adobe Illustrator (AI) and Encapsulated PostScript (EPS) format. As new files are added daily, sort them by newest first. Any download is free for commercial use with attribution.
Discovery Education : Discovery is one of the more kid-friendly websites you can visit on the web. Go straight to the space devoted for clip arts tucked away among their educational videos and curricular resources. The graphics are excellent and cover most of the categories (including animated clip arts) you would need to complete any school assignment. The designs are consistent because they are made by one designer.
Wikimedia Commons : This is arguably the largest collection of free images on the web. So far there are 38,205,390 freely usable media files and it is always open for contribution from anyone in the world. Use the search box at the top to ferret the files you want to use. As a regular user, you can take advantage of the syndicated feeds to grab the latest images that come into the archive.

From makeuseof

Anti Roboto skills

2022-11-02T00:00:00+01:00

Losing your job to robots is no longer a sci-fi fantasy.

Some estimates say, robots may take over more than five million jobs across 15 developed countries. Machines could account for more than half the workforce in places like Cambodia and Indonesia, particularly in the garment industry.

While such information has led many people to seek out higher-tech skills, others have said we need a stronger emphasis on trade skills to combat the high competition in tech fields. In one 2016 survey, 60 percent of respondents wanted more emphasis on Shop classes in high schools, while a 2015 Gallup poll found that 90 percent of parents want computer sciences emphasized in schools.

The good news. There are some skills robots can't embody, and if you have them, there's no need to worry about losing your job due to robotic advancements. Better yet, many of them are transferrable, meaning they can help you advance your career, even if you need to change industries.

Here are eight skills that can keep your job from being handed off to a robot.

Complex Problem-Solving Skills
Project and Personnel Management Skills
Athletic Skills
Confidence and Leadership Skills
Critical Thinking and Judgment Skills
Empathy Skills
Listening Skills
Robotics and Hardware Repair

Source: makeuseof

stop procrastinating

2022-11-02T00:00:00+01:00

We are all guilty of procrastinating from time to time--here's always something more interesting than the work in hand. We usually think it's no big deal, since deadline is our biggest inspiration, and we do our best work when we're inspired. We may even joke about it.

However, procrastination is a massive waste of time as it turns out.

A survey in 2015 found that on average, a person loses over 55 days per year procrastinating, wasting around 218 minutes every day on doing unimportant things.

Here' the maths:

218 minutes/day x 365 = 79570 minutes = 55.3 days

That' a lot of time wasted!

If you think you need to have a lot of willpower to get productive, you're wrong.

We're human beings, we all have limited willpower. Our brain is wired to instant gratification. Temporary rewards are always more tempting to us.

When you make plans, you're making plans for your future self. You'll only experience the benefits in future. But most of the time, the present moment can give you the immediate reward you want, making you want to delay the plans and just enjoy the moment.

This is why relying on our willpower to stop procrastination will never be effective. What we should do is to look into the root causes of procrastination and start with the small things we can do every day and build a habit of staying productive.

Basically there'e 5 common reasons why we procrastinate.

Identify the real reason and find out how to stop procrastination accordingly:

Type 1: The Perfectionist

They are the ones who pay too much attention to the minor details. The perfectionist is afraid to start a task because they get stressed out about getting every detail right. They can also get stuck in the process even when they've started since they're just too scared to move on.

Advice for the Perfectionist:

Instead of letting your obsession with details take up all your time, be clear about the purpose of your tasks and assign a time limit to each task. This will force you to stay focused and finish your task within the time frame.

For example,

If you're going to write a report, be clear about the purpose of the report first.

If the goal of having the report is to clearly present the changes in data over the past few months, don't sweat too much about writing up a lot of dainty words; rather, focus more on the figures and charts. Just make sure the goal can be reached, and there's really no need to work on things that don't help you achieve the ultimate goal.

Type 2: The Dreamer

This is someone who enjoys making the ideal plan more than taking actions. They are highly creative, but find it hard to actually finish a task.

Advice for the Dreamer

To stop yourself from being carried away by your endless imagination, get your feet back on the ground by setting specific (and achievable) goals for each day based on the SMART framework. Set a goal and break down the plan into small tasks that you can take actions right away.

For example,

If you dream about waking up earlier every day, set a clear goal about it - "In 3 weeks, I will wake up at 6:30am every day."

Then, break this goal down into smaller tasks:

From tonight onwards, I will go to sleep before 11:00pm.
Set alarm to remind me to go to sleep
Schedule earlier friends gathering so I can go to sleep early
For the 1st week, I will wake up at 7:30am even for non-working days
Go jogging or swimming in the morning for weekends

... and the task list goes on.

Also, you should reflect on your progress while you work. Track your input and output for each task, so you can easily tell which tasks are only a waste of time with little importance.6 This can help you focus on doing the things that bring positive results, which will improve productivity.

Type 3: The Avoider

The worrier are scared to take on tasks that they think they can't manage. They would rather put off work than be judged by others when they end up making mistakes.

Advice for the Avoider

I know checking emails seems tempting, but don't make answering emails the first thing on your to-do list. More often than not, emails are unimportant. But they steal your time and mental energy before you even notice.

Instead, focus on the worst first. Spend your morning working on what you find the most challenging. This will give you a sense of achievement, and helps you build momentum for a productive day ahead.

Try to break down your tasks into smaller sub-tasks. Understand how much time and energy is really needed for a given task. Make realistic calculations.

For example,

A 2000-word report does seem to take a lot of time and effort, it does seem scary to just start working on it. But is there anyway to break this down into smaller pieces so it'll seem less scary? What about this:

Introduction: around 100 words (15 min)
Table of content (5 min)
Report on the financial status: a chart with 100 supporting text (20 min)
Case study: 3 cases based on the new business model with around 400 words each (around 40 min each)
Conclusion: around 800 words (30 min)

Does it look a lot more easier now?

Type 4: The Crisis-maker

Now the crisis-maker deliberately pushes back work until the last minute. They find deadlines (the crises) exciting, and believe that they work best when being forced to rush it.

Advice for the Crisis-maker

Being forced to rush the work will perform better is just an illusion because it actually leaves you no room for reviewing the work to make it better afterwards.

If you always leave work until the last minute, try using the Pomodoro technique. Literally the "tomato technique" developed by Italian entrepreneur Francesco Cirillo.

It focuses on working in short, intensely focused bursts, and then giving yourself a brief break to recover and start over.

For example,

Use a timer and divide your complex work into small manageable sessions. In between the small sessions, give yourself a break to recover.

While giving your brain a regular break can highly boost your performance by recharging your brain's energy;10 having completed the tasks earlier allows you to have plenty of time to go through your work again to make it even better.

Type 5: The Busy Procrastinator

This type of procrastinators are the fussy ones. They have trouble prioritizing tasks because they either have too many of them or refuse to work on what they see as unworthy of their effort. They don't know how to choose the task that's best for them and simply postpone making any decisions.

Advice for the Busy Procrastinator

You have to get your priorities straight. Important tasks should take priority over urgent ones because "urgent" doesn't always mean important. You only have so much time and energy, and you don't want to waste that on things that don' matter.

Identify the purpose of your task and the expected outcome. Important tasks are the ones that add value in the long run.

Replying an email that's written "please get back to me asap" seems to be urgent, but before you reply that email, think about how important it is compared to other tasks.

For example,

Imagine the email is sent by a client asking about the progress of a project and she wants you to reply her as soon as possible; at the same time you have another task about fixing the logistics problem that is affecting all the projects on hand. Which one should you handle first?

The time cost for replying an email is as low as just around 5 minutes but the benefit is also very low because you're just satisfying one client request. Fixing the logistic problem probably takes a lot more time but it's also a lot more worth it because by fixing the problem, you're saving all the projects on hands, benefiting the whole company.

Be smart about every small choice you make because...

You may notice most of the characteristics of procrastinators have to do with their mindset. They keep delaying work because of some sort of fear. This is exactly why tweaking our attitude towards work can help us stop procrastinating and become more productive.

Changing our mindset may seem a lot of work. But by doing the smallest things every day, you're getting used to the way you handle works - from setting goals, to breaking down tasks, to evaluating each task's values.

Source lifehack.org

manage busyness

2022-11-02T00:00:00+01:00

Former United States President Dwight Eisenhower was responsible for putting together one of the most important yet fundamentally simple to understand concepts in time management. Eisenhower's Urgent/Important Principle is a tool to help decipher what tasks need to be addressed more immediately than others. Anyone who uses the principle will be better able to organize and orchestrate their daily tasks. This skill is especially imperative for busy people who find themselves working too hard and still not getting everything done.

Eisenhower's Urgent/Important Principle places tasks into four categories:

Important and Urgent
Important but Not Urgent
Not Important but Urgent
Not Important and Not Urgent

These four categories are used to label and organize which tasks need to be addressed first and which ones can be approached last. By asserting something's importance and its urgency, we are better able to identify what comes first:

What these quadrants reveal is that identifying which tasks are either important or urgent boils down to time management and what makes us most efficient. For example, President Obama' former campaign manager said in an article by WebMD that Obama valued his time to exercise and that it helped fuel him for the rest of his day. According to Obama, "The rest of my time will be more productive if you give me my workout time." The article goes on in detail about his routine and how he values its importance.

James Clear, a behavioral psychology writer, noted in a blog post that "too often, we use productivity, time management, and optimization as an excuse to avoid the really difficult question: 'Do I actually need to be doing this?' It is much easier to remain busy and tell yourself that you just need to be a little more efficient or to 'work a little later tonight' than to endure the pain of eliminating a task that you are comfortable with doing, but that isn' the highest and best use of your time."

Let' take a deeper look at each quadrant, what it means, and how we should approach all of our tasks with either urgency or importance (or both).

Urgent And Important

For Urgent/Important tasks, they can arise unexpectedly or may have been left for the last minute. These tasks need to be managed ahead of time. Make plans to address these tasks so that they do not become stressful activities when it comes close to deadlines. It's also a good idea to leave some wiggle room in your daily schedule just in case unexpected tasks come about.

Assess your deadlines. Are you moving at an appropriate pace to meet that deadline?

Emergencies happen. Whether they are unexpected meetings or sickness or injuries, they can't be put off until later.

This will force you to reconsider your task list and how much time you have to apply to each quadrant.

Important But Not Urgent

Not Urgent/Important tasks are integral to personal growth, building relationships, and accomplishing long-term professional goals. If these tasks are given the proper amount of time, they will not become urgent. This will prevent unexpected and last-minute tasks from unexpectedly cluttering up your time later on, keeping stress and frustration at bay. You'll be able to complete work efficiently and effectively.

Exercise is an example of this. Personal growth through exercise is not an overnight progress. Training for a run or any other sort of competition doesn't begin just days before. Plan your goals ahead of time, but leave room for urgent, unexpected tasks.

Maintaining your relationships is also important. Keep up with friends and family and partners, but be mindful of how much time you're alotting here. There is such a thing as putting too much time into relationships. Your goals are important, too. If you keep putting them off, they'll soon become urgent and you'll become stressed. This may affect your relationships in the long run.

Urgent But Not Important

Urgent/Not Important tasks are cumbersome and get in the way of your goals. Responding to phone calls or emails that are not pertinent to your goals or attending meetings with people who don't bring any value to completing your activities can be wasted time. Avoid these if possible and delegate the activities if you can. Something to keep in mind: you're saying yes to the person, but no to the task.

If someone or something requires that you do things for them frequently, then it might be best to arrange time for them in one larger block of time. This will allow you to focus your energy and time on multiple things.

Respond to time-sensitive correspondence as needed. Don't wait until after a deadline to inform someone when that deadline is:

You: "Hey, the class will be starting at noon today."

Colleague: "Really? Because it' already 2 P.M.!"

Not Urgent And Not Important

Not Urgent/Not Important tasks should also be avoided. Spending time on Facebook or Twitter, watching TV, and shopping (when it's not important to completing your tasks to have the things you're shopping for) can significantly drain your time. Limit these tasks as much as possible. It's not always going to be easy saying no to these mostly leisure activities, but it is important to remain mindful of how much of that time you're using here.

Yes, everyone is talking about the new show on Netflix. They watched it this past weekend and are already posting memes and gifs on Facebook. This doesn't mean you have to do the same.

Complete tasks first and then assess if you have time to participate in leisure. Otherwise, you're procrastinating, and that affects all the other quadrants.

In Conclusion

Eisenhower's Principles can be vital in developing skills to effectively and consistently complete tasks, delegate properly, and work efficiently. Take time to look over your tasks to determine which quadrant they belong.

Is there a deadline? If yes, then it is important.
Is the deadline soon? If yes, then it is urgent.
Is the task necessary to completing the other tasks? If yes, then it is important.
Can I delegate the task to someone else? If yes, then it is not important.
What does it have to do with your personal growth?
What does it have to do with your professional growth?

Ask yourself these questions when you need to determine a task's importance and urgency. Make a quadrant table of your own somewhere to help you visualize all your tasks. This is an excellent exercise for time management, and it could be the foundation of healthy work habits that stick around for a long time.

Source lifehack.org

Kivy

2022-03-03T00:00:00+01:00

Today I want to briefly write about kivy. kivy is an Python library intended for developing Mobile Apps. It is a cross-platform library that runs on Android, iOS, Linux, OS X and Windows. It is licensed under the MIT, so it is free and open source.

Kivy is the main framework developed by the Kivy organisation, alongside with Python for Android, Kivy iOS and other libraries meant to be used in all platforms. It is compatible with Python2 and Python3, as well as having support for the Raspberry Pi.

The framework contains all the elements for building an application such as:

extensive input support for mouse, keyboard, TUIO, and OS-specific multitouch events,
a graphic library using only OpenGL ES 2, and based on Vertex Buffer Object and shaders,
a wide range of Widgets that support multitouch,
an intermediate language kv used to easily design custom Widgets.

Mail Archiver ideas

2021-12-25T00:00:00+01:00

We use it for receiving junk e-mails (i.e. for those times where we need an e-mail address for sign-up to a service).

E-mails are of the form:

ar-XXXX@0ink.net

TODO:

Extend postie:

Before posting we insert all the header information into a table.

Automatically delete postings. If we want to keep the post we change its category.

https://wordpress.org/plugins/auto-prune-posts/

MAYBE: Markdownify it...

https://github.com/Elephant418/Markdownify

EXTRA ARCHIVER:

Check how gmail keeps folders (http://php.net/manual/en/function.imap-open.php) And then see if we can hack it into Postie.
https://www.electrictoolbox.com/open-mailbox-other-than-inbox-php-imap/ Maybe we do by e-mail@something/folder

http://postieplugin.com/forcing-an-email-check/

Would be the call back to MailGun

Check if we can add MailGun to Postie
- save transient when we start
- Use event API to get message lists (since last transient)
- Use message API to retrieve and delete messages

E-mail archiving alternatives
- lurker
- Enkive
- mboxpurge.pl
- archivemail
- Open Mail Archiva
- GYB
- gmvault
- Mail Piler

Fixed drive letters for removable USB sticks

2021-12-25T00:00:00+01:00

If you use multiple USB drives, you've probably noticed that the drive letter can be different each time you plug one in. If you'd like to assign a static letter to a drive that's the same every time you plug it in, read on.

Windows assigns drive letters to whatever type of drive is available. This can be annoying especially if you use backup tools or portable apps that prefer to have the same drive letter every time.

To work with drive letters, you'll use the Disk Management tool built into Windows. In Windows 7, 8, or 10, click Start, typecreate and format, and then click Create and format hard disk partitions. Don't worry. You're not going to be formatting or creating anything. That's just the Start menu entry for the Disk Management tool. This procedure works the same in pretty much any version of Windows (though in Windows XP and Vista, you'd need to launch Disk Management through the Administrative Tools item in the Control Panel).

Windows will scan and then display all the drives connected to your PC in the Disk Management window. Right-click the USB drive to which you want to assign a persistent drive letter and then click Change Drive Letter and Paths.

The Change Drive Letter and Paths window the selected drive's current drive letter. To change the drive letter, click Change.

In the Change Drive Letter or Path window that opens, make sure the Assign the following drive letter option is selected and then use the drop-down menu to select a new drive letter. When you're done, click OK.

NOTE: We suggest picking a drive letter between M and Z, because earlier drive letters may still get assigned to drives that don't always show up in File Explorer-like optical and removable card drives. M through Z are almost never used on most Windows systems.

Windows will display a warning letting you know that some apps might rely on drive letters to run properly. For the most part, you won't have to worry about this. But if you do have any apps in which you've specified another drive letter for this drive, you may need to change them. Click Yes to continue.

Back in the main Disk Management window, you should see the new drive letter assigned to the drive. You can now close the Disk Management window.

From now on, when you disconnect and reconnect the drive, that new drive letter should persist. You can also now use fixed paths for that drive in apps such as back up apps that may require them.

Source: howtogeek

Portable Console

2021-12-25T00:00:00+01:00

portable console

Set scrolling region:

printf "\033[1;24r"

Reset scrolling region:

printf "\033[r"

However, it is easier/better to do:

stty rows 24 cols 80

Xnest

2021-12-25T00:00:00+01:00

This trick lets you run X-Windows within an X-Windows session.

This is kinda like running VNC. It is useful for testing scenarios.

#!/bin/sh
Xnest :1 -name "Bla" -ac -geometry 800x600 &amp;
sleep 1
export DISPLAY=:1
exec xterm

CyberWorld 2017.1

2022-01-04T00:00:00+01:00

Development

travis cordova build
travis ionic build
owx
- common
  - muninlite (can it support plugins?)
  - flock, pwgen, ifstat
- ow1
  - diags&tools: usbutils, netstat-nat
  - sniffer: tcpdump[-mini] 317K/617K, libpcap 191K
- owX
  - FW/NAT
  - DNSMASQ: DHCP + DNS
  - NTP server
  - Dynamic DNS updating (mushu porker)
  - NFS
  - IPv6 tunnel
  - Provisioning server: (PXE, TFTP, NFS, HTTP, rpmgot, syslog?)
  - TLR server: HTTP, file manipulation, HTTPS?
  - USB storage
owx - switches
cn1
- data scrubbing
- backups
- boot cd mirroring
- config backup to alvm1
- NFS mounting installable iso images
- alvm1 : Main file store
  - file sharing (NFS, Samba, http)
  - rsync backup target
  - undup, backup puller
- alvm2 : Backup file store
  - snapshot server (NFS)
  - backuper
- alvm3 : Transmission
  - Implemented as its own server because of the VPN
- cvm1 : Main APP server
- alvm4 : X10 server
  - Implemented as its own server because VM only runs if HW is available
- alvm5: DMZ Server
  - reverse proxy
  - PocketMine
    - Muirfield
    - Niños
  - asterisk
- alvm6 : Scan&Print server
  - Spin-off cvm1, because SELINUX exception. Shouldn't connect to DMZ, nor X10

DMZ Server Basic Alpine Linux install

Create dos partition on the data drive
mkdosfs on partition and mount
setup-alpine
apk update
lbu ci

Reverse Proxy

install nginx

apk add nginx ?php-fpm?
configure in /etc/nginx/nginx.conf (Reference)
apk apache2-utils : for htpasswd command
Add a proxy command:

    location / {
      proxy_pass http://$server/;
      auth_basic "Restricted";
      auth_basic_user_file /etc/nginx/_htpasswd;
      proxy_set_header X-Remote-User $remote_user;
      proxy_pass_request_headers      on;
    }

Variable Reference: NGINX Docs

WebServer

PHP checks headers (passed by the reverse proxy), otherwise,
use authd:
If using selinux we need to set this boolean:
- setsebool -P httpd_can_network_connect on
PHP Function on server to determine user:

define('IDENT_PORT',113);

function identd_query($remote_ip,$remote_port,$local_port,$tout=3) {
  $remote_ip = 'localhost';
  $sock = @fsockopen($remote_ip,IDENT_PORT,$errno,$errstr,$tout);
  //print_r([$sock,$errno,$errstr]);
  if (!$sock) return FALSE;
  @fwrite($sock,$remote_port.','.$local_port."\r\n");
  $line = @fgets($sock,1000); // 1000 octets according to RFC1413
  fclose($socket);
  if (preg_match('/^\s*(\d+)\s*,\s*(\d+)\s*:\s*(\S+)\s*:\s*(\S+)\s*:\s*(\S+)\s*$/', $line,$mv)) {
    if ($mv[1] == $remote_port && $mv[2] == $local_port &&
    $mv[3] == 'USERID') {
      return $mv[5];
    }
  }
  return FALSE;
}

Web Browser

For archlinux, install oidentd

yum install authd
check firewall port
systemctl start authd.socket
Enable authd.socket
Add Override:
- /etc/systemd/system/auth@.service.d/override.conf
  - [Service]
  - ExecStart=
  - ExecStart=-/usr/sbin/in.authd -t60 --xerror

https://wiki.alpinelinux.org/wiki/LXC https://wiki.alpinelinux.org/wiki/Setting_up_a_basic_vserver

browser -> guac -> xinetd|vncserver|x2go-client -> x2go-server browser -> revproxy -> guac -> xinetd|vncserver|x2go-client -> x2go-server

Check what Thin client software Tiny Core Linux supports otherwise Browser with Guacamole

Server script (haserl) on OW1 Show version and last update Options: Delete Entry Post update : Using wget

Create a local pastebin (to add notes from SONY PRS-T2) https://wiki.alpinelinux.org/wiki/Pastebin

Configure a Windows VM

./mxt.sh \
vmcfg \
vm=winvm1 \
rem="win7 system" \
-serial \
viridian=1 \
boot=d \
hd=1,16G \
cdrom=3,/xendat/installers/Win7AIO.x32-x64.preact.iso

Centos 7

Template preparation

Configure serial console:

Modify /etc/default/grub
- GRUB_TERMINAL_OUTPUT=serial
- GRUB_CMDLINE_LINUX=console=ttyS0 --rhgb
Run grub2-mkconfig -o $d/grub.cfg either on /boot/efi/EFI or /boot/grub2

Stop ssh and remove all ssh keys.

Modify rc.local to run something once:

change hostname (if possible?)
remove all SSH keys (and reboot)

Create a centos/xen template prep script. We pass it as a custom tar in xvdh. Another option:

Use a serial port (connected to UNIX socket)
Use a Xen PV channel

We need to pass vm name.

rhel7 and systemd

Cfg script /etc/xen

After block devices stanza
Check if tar is there
Append to the list

Better to do this

Notes

munin

plugin writing
Monitored data using xentop
- CPU is done, what about vbd I/O or network I/O
xen wiki on xentop

Serial xen configuration

serial=/dev/ttyS0
[Linux only] Use host tty, e.g. ‘/dev/ttyS0’. The host serial port parameters are set according to the emulated ones.
serial=unix:path[,server][,nowait]
A unix domain socket is used instead of a tcp socket. The option works the same as if you had specified -serial tcp except the unix domain socket path is used for connections.
The TCP Net Console has two modes of operation. It can send the serial I/O to a location or wait for a connection from a location. By default the TCP Net Console is sent to host at the port. If you use the server option QEMU will wait for a client socket application to connect to the port before continuing, unless the nowait option was specified. The nodelay option disables the Nagle buffering algorithm. If host is omitted, 0.0.0.0 is assumed. Only one TCP connection at a time is accepted. You can use telnet to connect to the corresponding character device.

Side Load apps on Android TV

2021-12-25T00:00:00+01:00

So we bough a Philips 50PFK6540. This is a 50" TV with Ambi Ligh and Android TV.

One of the things I wanted to do from the very start was to load my own APKs. This was not possible until a recent (2016) update that enabled the "Install from Unknown Sources" setting option.

This made it possible to side load applications. However, things are not as easy as I initially thought. Because while installing from unknown sources was possible, you can not do this from the built-in browser. So the procedure is as folows:

Go to settings to enable Install from Unknown sources, which should be under Security & Restrictions.
Download ES File Explorer from the Play Store. A bit of a warning: on phones and tablets, ES File Explorer isn't something that is generally recommended as it used to be a reliable file manager that was one of the most valuable Android apps, but recently it became riddled with ads many of which are highly intrusive) leading many users to uninstall it and websites to remove it from their must have* lists. Fortunately, the Android TV app seems to have gone largely untouched by this, so it still is recommended it for the purpose of this tutorial.
Use ES File Explorer to download the APK you want to side load. There is a number of ways to do this. I used the built-in FTP server. But you could use any method (i.e Thumb drive, Cloud Storage, etc...)
Open the APK from ES File Explorer and install it.

Building Signed APKs

2021-12-25T00:00:00+01:00

Building signed APK's for Android is easy if you know what you are doing.

This article goes over the preparation steps and the additional build instructions needed to created signed APKs.

Preparation

First you need to have a keystore. Use this command:

#!/bin/bash
keystore_file="my_key_store.keystore"
key_name="john_doe"
secret='fake_password'
name='John Doe'
dept='Engineering'
org='TLabs Inc'
place='New York'
province='NY'
country='US'

keytool -genkey -v -keystore "$keystore_file" -alias "key_name" -keyalg "RSA" -validity 10000 -storepass "$secret" -keypass "$secret" &lt;&lt;EOF
$name
$dept
$org
$place
$province
$country
yes
EOF

Remember the keystore file and passwords.

Build instructions

In your build.gradle you need the following:


android {
  signingConfigs {
    release {
      storeFile file("my_keystore.keystore")
      storePassword "{password}"
      keyAlias "Key_Alias"
      keyPassword "{password}"
    }
  }
  buildTypes {
    release {
      signingConfig signingConfigs.release
    }
  }
}

Archiving DVDs and CDs

2021-12-26T00:00:00+01:00

Since now I have a Android TV I put away my HTPC and with that the capability to view DVDs or listen CDs directly.

So I converted my entire CD and DVD library to media files and stored in my home NAS.

Since we are talking hundreds of DVDs and CDs, I was using some tools.

CD ripping

For CD ripping, pretty much everything can be done with abcde. I would use the following command:

abcde -G -k -o mp3 -x

Options:

-G : Get album art.
-k : Keep wav after encoding. This is not really necessary.
-o mp3 : Output to mp3.
-x : Eject the CD after all tracks have been read.

Afterwards I would use eyeD3 to embed the cover art and tweak things. (Note under archlinux, eyeD3 is installed from the python2-eyed3 package).

To add cover art:

eyeD3 --add-image="$cover_file":FRONT_COVER \*.mp3

DVD Ripping

For DVD Ripping I was using a couple of homegrown scripts. These can be found on github.

I started using vobcopy, but if I were to do this again I would use dvdbackup with the -M option. vobcopy is quite old and probably is orphaned by now.

Scripts for archiving media

Scripts:

archive-dvd : Create an iso image from a DVD.
alltitles : Extract titles/chapters from a DVD.
auto.sh : Used to transcode titles/chapters extracted by alltitles

archive-dvd

This script uses vobcopy and mkisofs to create an ISO file. Just run the script and insert a DVD, you will get an ISO file in return.

alltitles

Usage:

[option_vars] sh alltitles [chapter]

Option vars:

drive=[device-path] defaults to /dev/sr0
titles="01 02 03 ..." defaults to all titles in DVD (as listed by lsdvd) You can also specify titles as: title="01,1-4 01,5-8" This will create two files, one with track 1, chaptes one trough four (inclusive) and another one with track 1, chapters five through eigth (inclusive)

Command options:

chapter: Leave blank for all chapters, otherwise:

-chapter [$start-$end]

Will dump starting from $start until $end. (or end) If you only want to extract chapter 7 by itself, use -chapter 7-7

auto.sh

Usage:

sh $0 [options]

vob files must be the ones extracted from alltitles.

Options:

--preview|-p : Only encode 30 seconds from 4 minutes in
--copy|-c : Do only copy
--interlace|-i : Force interlace filter
--no-interlace|+i : Disables interlace filter

Dependancies

libdvdcss (or equivalent). This is used by the dvdread library to decode CSS protected DVDs.
libdvdread This is used to read DVD by a number of binaries.
vobcopy Used by archive-dvd to extract the data that will be used to create the ISO image. Uses libdvdread.
udisks or udisks2 Used by the scripts to detect when a CD/DVD is inserted.
cdrkit Used to create the iso images by archive-dvd.
lsdvd Used by alltitles.sh to get track information.
mplayer Used by alltitles.sh to extract DVD titles/chapters.
ffmpeg Used by alltitles.sh to encode video.

Some useful commands

Using mplayer to play extract:

mplayer -dvd-device /dev/sr0 dvd://$title -chapter $chapter-$chapter -dumpstream -dumpfile ~/$title.VOB

Writing Safe Shell scripts

2024-11-22T00:00:00+01:00

Writing shell scripts leaves a lot of room to make mistakes, in ways that will cause your scripts to break on certain input, or (if some input is untrusted) open up security vulnerabilities. Here are some tips on how to make your shell scripts safer.

Don't

The simplest step is to avoid using shell at all. Many higher-level languages are both easier to write the code in in the first place, and avoid some of the issues that shell has. For example, Python will automatically error out if you try to read from an uninitialized variable (though not if you try to write to one), or if some function call you make produces an error.

One of shell's chief advantages is that it's easy to call out to the huge variety of command-line utilities available. Much of that functionality will be available through libraries in Python or other languages. For the handful of things that aren't, you can still call external programs. In Python, the subprocess module is very useful for this. You should try to avoid passing shell=True to subprocess (or using os.system or similar functions at all), since that will run a shell, exposing you to many of the same issues as plain shell has. It also has two big advantages over shell: it's a lot easier to avoid word-splitting or similar issues, and since calls to subprocess will tend to be relatively uncommon, it's easy to scrutinize them especially hard. When using subprocess or similar tools, you should still be aware of the suggestions in "Passing filenames or other positional arguments to commands" below.

Shell settings

POSIX sh and especially bash have a number of settings that can help write safe shell scripts.

I recommend the following in bash scripts:

set -euf -o pipefail

In dash, set -o doesn't exist, so use only set -euf.

What do those do?

set -e

If a command fails, set -e will make the whole script exit, instead of just resuming on the next line. If you have commands that can fail without it being an issue, you can append || true or || : to suppress this behavior - for example set -e followed by false || : will not cause your script to terminate.

set -u

Treat unset variables as an error, and immediately exit.

set -f

Disable filename expansion (globbing) upon seeing *, ?, etc..

If your script depends on globbing, you obviously shouldn't set this. Instead, you may find shopt -s failglob useful, which causes globs that don't get expanded to cause errors, rather than getting passed to the command with the * intact.

set -o pipefail

set -o pipefail causes a pipeline (for example, curl -s http://sipb.mit.edu/ | grep foo) to produce a failure return code if any command errors. Normally, pipelines only return a failure if the last command errors. In combination with set -e, this will make your script exit if any command in a pipeline errors.

Quote liberally

Whenever you pass a variable to a command, you should probably quote it. Otherwise, the shell will perform word-splitting and globbing, which is likely not what you want.

For example, consider the following:

alex@kronborg tmp [15:23] $ dir="foo bar"
alex@kronborg tmp [15:23] $ ls $dir
ls: cannot access foo: No such file or directory
ls: cannot access bar: No such file or directory
alex@kronborg tmp [15:23] $ cd "$dir"
alex@kronborg foo bar [15:25] $ file=*.txt
alex@kronborg foo bar [15:26] $ echo $file
bar.txt foo.txt
alex@kronborg foo bar [15:26] $ echo "$file"
*.txt

Depending on what you are doing in your script, it is likely that the word-splitting and globbing shown above are not what you expected to have happen. By using "$foo" to access the contents of the foo variable instead of just $foo, this problem does not arise.

When writing a wrapper script, you may wish pass along all the arguments your script received. Do that with:

wrapped-command "$@"

See "Special Parameters" in the bash manual for details on the distinction between $*, $@, and "$@" - the first and second are rarely what you want in a safe shell script.

Passing filenames or other positional arguments to commands

If you get filenames from the user or from shell globbing, or any other kind of positional arguments, you should be aware that those could start with a "-". Even if you quote correctly, this may still act differently from what you intended. For example, consider a script that allows somebody to run commands as nobody (exposed over remctl, perhaps), consisting of just sudo -u nobody "$@". The quoting is fine, but if a user passes -u root reboot, sudo will catch the second -u and run it as root.

Fixing this depends on what command you're running.

For many commands, however, -- is accepted to indicate that any options are done, and future arguments should be parsed as positional parameters - even if they look like options. In the sudo example above, sudo -u nobody -- "$@" would avoid this attack (though obviously specifying in the sudo configuration that commands can only be run as nobody is also a good idea).

Another approach is to prefix each filename with ./, if the filenames are expected to be in the current directory.

Temporary files

A common convention to create temporary file names is to use something.$$. This is not safe. It is better to use mktemp.

Other resources

Google has a Shell Style Guide. As the name suggests, it primarily focuses on good style, but some items are safety/security-relevant.

Conclusion

When possible, instead of writing a "safe" shell script, use a higher-level language like Python. If you can't do that, the shell has several options that you can enable that will reduce your chances of having bugs, and you should be sure to quote liberally.

Source Writing Safe Shell.

editor to replace emacs

2021-12-25T00:00:00+01:00

At the end, I switched to geany

GUI

Console

TCL/TK

Windows only

nodepad++
Crimson or Emerald Editors
Macros
Split Views
Interactive search
File Browser?
Smart indent
Parenthesis matching
Syntax: PHP, Markdown, C, Java, JavaScript, HTML, C++
UTF8

Key Bindings: bindings

Other CUA stuff: ergoemacs

Others:

scite Download It now has a single file exe for Windows.
editra
notepadqq
Geany
Scintilla
- curses based: scinterm
  includes jinx which is an example for it.
- SciTE - the default for Win and Linux.
- Others
http://tke.sourceforge.net/index.html
- TCL based. Can we use cdk? Can it be use in linux and windows?
dex

Notes

GUI and TUI, Linux and Windows
Modeless
Syntax highlighting
"Compact"?
Key recording macros
Split windows

Emacs tips

Mote ideas:

LinkdMode Paired with "deft"?
iMenu: M-x imenu or:
(add-hook 'c-mode-hook 'imenu-add-menubar-index)
Start typing or use TAB completion to find function defintions. See imenuMode
Predictive Mode
Record, play, re-play:
(global-set-key [f10] 'start-kbd-macro)
(global-set-key [f11] 'end-kbd-macro)
(global-set-key [f12] 'call-last-kbd-macro)
Selective display:
M-1 C-x $ to activate
C-x $ to go back
Or create shortcuts:

(defun jao-toggle-selective-display () (interactive) (set-selective-display (if selective-display nil 1))) (global-set-key [f1] 'jao-toggle-selective-display)
- (setq cua-enable-cua-keys nil) (setq cua-highlight-region-shift-only t) ;; no transient mark mode (setq cua-toggle-set-mark nil) ;; original set-mark behavior, i.e. no transient-mark-mode (cua-mode)

MariaDB Quickest Quick start

2022-01-10T00:00:00+01:00

This article outlines the bare minimum to get a MariaDB or MySQL database up and running.

It covers a CentOS/RHEL and an ArchLinux installs.

Make sure your system is up to date:

CentOS/RHEL	ArchLinux
`yum update -y`	`pacman -Syu`

Install the software:

CentOS/RHEL	ArchLinux
`yum install mariadb-server`	`pacman -S mariadb`
	`mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/mysql`

Start the database service:

 systemctl start mariadb

Check if it is running:

 systemctl is-active mariadb.service
 systemctl status mariadb

The following step is optional but highly recommended:

 mysql_secure_installation

Enable database to start on start-up:

 systemctl enable mariadb

Enter SQL:

 mysql -u root -p

Creating database:

 create database bugzilla;
 FLUSH PRIVILEGES;

Create user:

 GRANT ALL PRIVILEGES ON bugzilla.* TO 'warren'@'localhost' IDENTIFIED BY 'mypass';
 GRANT ALL PRIVILEGES ON killrate.* TO 'pocketmine'@'%' IDENTIFIED BY 'mypass';
 FLUSH PRIVILEGES;

Jaxon: Call PHP classes from JavaScript using AJAX

2021-12-25T00:00:00+01:00

Jaxon is an open source PHP library for easily creating Ajax web applications. It allows into a web page to make direct Ajax calls to PHP classes that will in turn update its content, without reloading the entire page.

Jaxon implements a complete set of PHP functions to define the contents and properties of the web page. Several plugins exist to extend its functionalities and provide integration with various PHP frameworks and CMS.

Building chroots with yum

2021-12-25T00:00:00+01:00

Building CHROOTs with Yum in a single command:

yum --releasever=7 --installroot=/chroot/jail2 -y install httpd

Will install httpd with all its dependancies. If you are on x86_64 and want a 32 bit chroot:

setarch i386 yum --releasever=6 --installroot=/chroot/jail32 -y install httpd

My WordPress plugins

2021-12-25T00:00:00+01:00

For my own purposes I have written a number of WordPress plugins.

S3Copy - Makes backup copies of your pictures to an S3 Compatible server. I use sirv.com myself. It also mangles tags so files are server from the S3 bucket.
wptools - A collection of WordPress related functionality.
auto-content - A basic post from template plugin.
simple-members-only - A fork of a defunct plugin.

vr starting points

2021-12-25T00:00:00+01:00

WebGL frameworks:

Could it be converted to WebVR?

Hosting WordPress on OpenShift

2022-01-10T00:00:00+01:00

So I finally moved my WordPress web sites to OpenShift.

OpenShift is a cloud based Platform-as-a-Service offering from RedHat. And while there is a learning curve I would say that so far it works great.

My implementation is a fully cloud based solution. Makes use of the following services:

GitHub for code hosting
Travis-CI for continuous integration.
OpenShift (with autoscaling) for the database and web server
CloudFlare
Sirv.com for image hosting
Facebook and G+ integration
Google drive for cloud backups

All the code can be examined on GitHub.

For the WordPress hosting I started with the OpenShift WordPress QuickStart and added scripts to deploy directly from Github to OpenShift via Travis.

Actually Travis has that functionality built in but it was a little quirky for my use cases so I wrote my own.

On the OpenShift side, I added code to download add-ons (plugins and themes) automatically and to deploy from the same repo to multiple apps.

The rationale for this is to get addons installed automatically in the event of autoscaling while keeping the github commit log fairly tidy.

Also created a couple of Wordpress plugins to:

Misc shortcodes and stuff
Automatically upload pictures to an S3 cloud storage (sivr.com in my case but this is configurable)

CSR ideas

2021-12-25T00:00:00+01:00

Work improvements

NFR

Javascript single page application
- JS GUI
Retargettable back-end:
- Local
- Remote
- Synchronization utility
Multi-user (authentication?)
Output Excel
Output changes

FR

requirements
roadmap objects
- release time lines
- indicators
- descriptive text
Meta data
- attributes (i.e owner, reviewers, etc)
- versioning
Milestone data
Detail data
- release details
- line items

game lists

2021-12-25T00:00:00+01:00

Cybernator
Darius Twin
Another World | Out of this World
Front Mission Series
Strike Gunner
The Legend ...

Super Bomberman for about 59.99 (dollars) but later it was also sold alone for approximately 29.95.

Multitap compatible games:

Barkley: Shut Up and Jam!, Bill Walsh College Football, College Slam, Elite Soccer, ESPN National Hockey Night, FIFA International Soccer, FIFA '96, Firestriker, Hammerlock Wrestling, Head On Soccer, J-League Soccer, Looney Toons B-Ball, Lord of the Rings, Madden '94, Madden '95, Madden '96, Madden '97, Micro Machines, Natsume Championship Wrestling, NBA Give 'n Go, NBA Jam, NBA Jam TE, NBA Live 95, NBA Live 96, NBA Live 97, NCAA Final Four, NCAA Football, NHL '94, NHL '95, NHL '96, NHL '97, Olympic Summer Games, Peace Keepers, Pieces, Rap Jam Vol. 1, Saturday Night Slam Masters, Secret of Mana, Slam Dunk TV Animation (Japanese), Soccer Shootout, Sporting News: Power Baseball, Sterling Silver: End 2 End, Street Hockey '95, Street Racer, Super Bomberman 1, Super Bomberman 2, Super Bomberman 3, Super Bomberman 4 (Japanese), Super Bomberman 5 (Japanese), Super Tetris 3 (Japanese), Tiny Toons Wacky Sports, Virtual Soccer (Japanese), Top Gear 3000, WWF Raw. (Puh!).
Peacekeepers
Secret of Mana
Firestriker
Bakukyuu Renpatsu!! Super B-Daman: Port 2
Bakutou Dochers: Port 2
Barkley Shut Up and Jam!: Port 2
Battle Cross: Port 2
Battle Jockey: Port 2
Bill Walsh College Football: Port 2
Capcom's Soccer Shootout: Port 2
College Slam: Port 2
Crystal Beans From Dungeon Explorer: Port 2
Dragon - The Bruce Lee Story: Port 2
Dream Basketball - Dunk and Hoop: Port 2
Dynamic Stadium: Port 2
ESPN National Hockey Night: Port 2
FIFA 98: Port 2
FIFA International Soccer: Port 2
FIFA Soccer 96: Port 2
FIFA Soccer 97: Port 2
Final Set: Port 2
Fire Striker: Port 2
From TV Animation Slam Dunk - SD Heat Up!!: Port 2
Go! Go! Dodge League: Port 2
Hammerlock Wrestling: Port 2
Hat Trick Hero 2: Port 2
Head-On Soccer: Port 2
Hebereke no Oishii Puzzle ha Irimasenka: Port 2
Human Grand Prix III - F1 Triple Battle: Port 2
Human Grand Prix IV - F1 Dream Battle: Port 2
Hungry Dinosaurs: Port 2
International Superstar Soccer Deluxe: Port 2
J. League Excite Stage '94: Port 2
J. League Excite Stage '95: Port 2
J. League Excite Stage '96: Port 2
J. League Super Soccer '95: Port 2
J. League Super Soccer: Port 2
JWP Joshi Pro Wrestling - Pure Wrestle Queens: Port 2
Jikkyou Power Pro Wrestling '96: Port 2
Jimmy Connors Pro Tennis Tour: Port 2
Kunio-kun no Dodge Ball dayo Zenin Shuugou!: Port 2
Looney Tunes Basketball: Port 2
Madden NFL '94: Port 2
Madden NFL '95: Port 2
Madden NFL '96: Port 2
Madden NFL '97: Port 2
Madden NFL '98: Port 2
Micro Machines 2 - Turbo Tournament: Port 2
Micro Machines: Port 2
Mizuki Shigeru no Youkai Hyakkiyakou: Port 2
Multi Play Volleyball: Port 2
NBA Give 'N Go: Port 2
NBA Hang Time: Port 2
NBA Jam - Tournament Edition: Port 2
NBA Jam: Port 2
NBA Live 95: Port 2
NBA Live 96: Port 2
NBA Live 97: Port 2
NBA Live 98: Port 2
NCAA Final Four Basketball: Port 2
NCAA Football: Port 2
NFL Quarterback Club 96: Port 2
NFL Quarterback Club: Port 2
NHL '94: Port 2
NHL '98: Port 2
NHL Pro Hockey '94: Port 2
Natsume Championship Wrestling: Port 2
Peace Keepers, The: Port 2
Pieces: Port 2
Rap Jam - Volume One: Port 2
Saturday Night Slam Masters: Port 2
Secret of Mana: Port 2
Shin Nippon Pro Wrestling '94 - Battlefield in Tokyo Dome: Port 2
Shin Nippon Pro Wrestling - Chou Senshi in Tokyo Dome: Port 2
Shin Nippon Pro Wrestling Kounin '95 - Tokyo Dome Battle 7: Port 2
Smash Tennis: Port 2
Sporting News, The - Power Baseball: Port 2
Sterling Sharpe End 2 End: Port 2
Street Hockey '95: Port 2
Street Racer: Port 2
Sugoi Hebereke: Port 2
Sugoro Quest ++ Dicenics: Port 2
Super Bomberman - Panic Bomber W: Port 2
Super Bomberman 2: Port 2
Super Bomberman 3: Port 2
Super Bomberman 4: Port 2
Super Bomberman 5: Port 2
Super Bomberman: Port 2
Super Fire Pro Wrestling - Queen's Special: Port 2
Super Fire Pro Wrestling Special: Port 2
Super Fire Pro Wrestling X Premium: Port 2
Super Fire Pro Wrestling X: Port 2
Super Formation Soccer 94 - World Cup Final Data: Port 2
Super Formation Soccer 94: Port 2
Super Formation Soccer 95 della Serie A - UCC Xaqua: Port 2
Super Formation Soccer 95 della Serie A: Port 2
Super Formation Soccer 96: Port 2
Super Formation Soccer II: Port 2
Super Ice Hockey: Port 2
Super Kyousouba - Kaze no Sylphid: Port 2
Super Power League: Port 2
Super Tekkyuu Fight!: Port 2
Super Tetris 3: Port 2
Syndicate: Port 2
Tenryu Genichiro no Pro Wrestling Revolution: Port 2
Tiny Toon Adventures - Wild & Wacky Sports: Port 2
Top Gear 3000: Port 2
Turbo Toons: Port 2
Virtual Soccer: Port 2
Vs. Collection: Port 2
WWF Raw: Port 2
Yuujin no Furi Furi Girls: Port 2
Zero 4 Champ RR-Z: Port 2
Zero 4 Champ RR: Port 2

C64 notes

c64 wiki

VICE with SDL

Coop games

The Goonies
Realm of Impossibility
ACE Air Combat Emulator
Alien Syndrome
Armalyte
Bubble Bobble
Katakis
Mario Bros (Atari)
Mega Apocalypse
Castle s of Doctor Creep
Wizball

Head to head

MULE
Batty
Bomb Squad
Highlander
International Soccer
Pitstop II
Spy vs Spy
The way of the exploding fist
Trailblazer
Wizard of Wor

JavaScript resources

2021-12-25T00:00:00+01:00

More powerful github web pages

Use JavaScript XHR to get data from github API (CORS is enabled).

Show:

Download counts for a project
Latest release tag
typescript tutorial
intro to typescript

Programming 2016

2021-12-25T00:00:00+01:00

Programming 2016

GWT and GWT on Mobile and Java servlets
- Generate Excel http://www.gwtproject.org/overview.html http://www.m-gwt.com/

Java based:

Game Api libgdx
Other Game lib JMonkeyEngine
MultiOS
j2objc
RoboVM forks:
- FlexoVM
- BugVM
Swift?
D status

Programming 2015

Cross-Platform: Linux, Windows, Android, iOS, WebApp?
Run-Time: >100MB?
ease of deployment (wrap app and drops)
gui programming
object classes and types
memory management
speed
skills marketability

Development

http://hyperpolyglot.org/web - comparison between TypeScript, Dart, Hack (php like)

ANGULAR

TypeScript

Headers: http://definitelytyped.org/
TypeScript - compiled, optionally typed language that compiles to JavaScript
node-webkit - Desktop apps
ionic framework - deploy to phone

Frameworks:

https://angular.io/ - JavaScript framework for web apps
jQuery
"app.js" : This is a UI library for writing mobile apps
TypeScript?
http://blog.scottlogic.com/2014/09/10/node-webkit.html
Angular.JS? https://angular.io/
React Native https://facebook.github.io/react-native/
http://appjs.com/ -
Enyo? http://enyojs.com/
http://noeticforce.com/best-hybrid-mobile-app-ui-frameworks-html5-js-css

RunTimes:

NW.js
electron (https://github.com/atom/electron)

Facebook's React Native

JavaScript supersets:

TypeScript
Dart
CoffeeScript

Translateable:

Google Web Toolkit (Java to JavaScript)
Pyjamas (Python to Javascript)
HaXe

Dev Notes

Replacment for Make and Autoconf: MakeMe

(If you don't have root but have Android 4+ you can use the command-line program adb from the Android SDK platform tools to make backups via a desktop computer)

http://www.chromebookhq.com/five-best-online-ides-making-the-switch-to-a-chromebook/

Dev Tools

Alternative languages:

D : better than C, but not over-the-top like C++? Covers only Win and Linux
Vala : Kinda like C# but for Gnome. Covers Win and Linux. (Android maybe through NDK).
Java: Kinda over the top and heavy. Covers Win and Linux. Android yes, but different GUI library. iOS probably yes.
Python: scripting language. Win, Linux. Android maybe... iOS maybe...
Javascript: scripting language. ALL PLATFORMS.

Other options:

Python with Kivy
Haxe

Build Tools

MakeKit - autotools look & feel but lighter
mobs: autoconf workalike.

Resources

http://www.dervishd.net/libre-software-projects syslogd in perl, mobom perl modules.

My own Notes App

JumpNote + OI Notpad (Background (Tags support) Sync) V Simple Note backend V Tags UI (Filter, modify tags) V Task UI V Widget

WebApp + Mobile Dev:

http://demux.vektorsoft.com/demux/ A Java framework that works on multiple platforms.
http://asterclick.drclue.net/WBEA.html Allows for webapps on desktops
PhoneGap
http://www.mobilexweb.com/emulators Test mobile apps on desktop
Javascript optimizer: https://developers.google.com/closure/ https://github.com/mishoo/UglifyJS
JS Compiler: https://developer.mozilla.org/en/Rhino_JavaScript_Compiler
Java 2 JS Toolkits: http://code.google.com/webtoolkit/ http://j2s.sourceforge.net/
Python 2 JS Toolkigs: http://pyjs.org/
JS Compiler for command line: https://developers.google.com/v8/ http://en.wikipedia.org/wiki/Nodejs
http://this-voice.org/alchemy/pride.html Compiling Android stuff

Documentation around Syncing...

Other Notes:

Perki replacement that runs on Android.
Use WebKit/PhoneGap + Javascript and HTML5
Markdown library for Javascript
Markdown editor for javscript
TXGR converted to HTML5 Canvas
How do we do background sync?

More example code:

We want to have it for Android, Linux and Windows.

http://libreplanet.org/wiki/Group:Hardware/Howto_have_a_free_android_sdk

We need to research:

* Alternative to freewrap

    * http://jsmooth.sourceforge.net/
    * http://launch4j.sourceforge.net/
    * http://www.thisiscool.com/gcc_mingw.htm

        * http://vertis.github.com/2007/06/24/native-java-with-gcj-and-swt.html

    * http://winrun4j.sourceforge.net/

* Alternative to Canvas

    * http://www.piccolo2d.org/
    * http://www.jhotdraw.org/
    * http://www.manageability.org/blog/stuff/open-source-structured-graphics-libraries-in-java

Contains an overview of options...

http://jean-philippe.leboeuf.name/notebook/archives/000315.html Another overview of options
Which Toolkit to use (SWT, Swing, AWT, etc)

An alternative to Eclipse for Android Development:

http://freecode.com/projects/pride

A freewrap like tool for pythonL

http://freecode.com/projects/pyinstaller

More Android Dev options:

PhoneGAP
http://kivy.org/ Python, multi platform
https://code.google.com/p/android-python27/w/list - Python on android

Managing our personal finances

2022-01-10T00:00:00+01:00

During my last vacation I wanted to move how we manage our personal finances away from the ad-hoc spreadsheet that we had been using for the past few years. I envisioned something server side, so I wouldn't need to add software on my wife's computer. And initial quick run through of server side software did not yield anything that interested me. In general I could only find full accounting applications, which would have been an over kill for personal finance/expense tracking. So then I checked through some desktop applications. I looked at the following:

I found many others, but I only tried these two. The most common suggestion among Open Source advocates is GNU Cash, but I did not try that because it was too big for my modest requirements. I installed these two but I was not able to get it to do what I wanted. Which was be able to import transactions from my back and entered into the the application. So I went back searching to the web this time looking for "personal finance" instead of "accounting" and found this web application (amongst others):

PFMGR

So I installed it and was able to run it on my home server. (This was the first of these types of application that I managed to run, so I was initially happy). So, running it, it looked OK, had an AJAX based user interface, etc. Did not have anything in the way to import the data files from my bank, but since it is Open Source, I could easily make up something for it. So I modified it to include a page to import my bank data. This seemed to work OK. That's when the annoyances started. PFMGR author had an specific use case in mind, so it can track not only money accounts but share accounts. While nice, I did not have such investments, so that feature was unused, but it will show on the forms (annoying). A lot of the functionality of the software was around check reconciliation. Since I don't use checks, that is not useful for me at all. Finally, I couldn't get the reports to work at all, and the times that they did work, they did not give me the information that I wanted. I figured, since this is all open source I could just add/remove the features the way I wanted. Which curiously turned out I would remove all the features and just keep PFMGR as a simple CRUD application. So I figure I might as well toss it all out and find a small PHP Framework that could do CRUD. So I came across this tutorial:

FatFree :: CRUD with MVC

So it gives a gentle intro to the FatFreeFramework. This was just what I was looking for. I can say for simple applications, this is perfect. I was able to get started into my own personal finance application. Did run into a few problems. Most of it around the fact that Fat-Free (aka as F3), although has a very gentle learning curve, and one can get things started very quickly, it did a few things that I was not expecting. Well, actually, for a novice programmer, it did things right. For somebody used to using PHP directly, I would add some code to escape and protect against invalid inputs, F3 was doing it automatically which caused me a few headaches until I realized what F3 was doing. My main problem with F3 is that my schema required a wide varchar column and that seemed to cause problem with its ORM mapper. Later I will write a simple test case and see if I can track down the issue.

Starting with 3D Printing

2022-03-03T00:00:00+01:00

So I finally tried my hand at 3D printing. Obviously I did not buy at 3D printer. These are either quite expensive or you need to assemble them yourself, which I don't think is in my capacity level.

To get started, you first need a 3D model to print. There are several 3D models available in Thingieverse, however I actually wanted to make my own model. After all, that is the whole point of 3D printing. Custom made parts/objects that can be printed as needed.

To create a 3D model you need some 3D modelling software. For my very first model I opted for TinkerCAD. This is software that runs on the cloud that lets you create your own 3D models. This is particularly interesting because you don't need to install anything on your computer and it would essentially run on anything where a web browser runs.

For a web based application, it is quite responsive and feature-full. You can use (like me) a facebook account to sign-in.

Models can then be downloaded as an ".stl" file (the format used by 3D printers) or send directly to 3D printing service such as 3D Hubs.

3D Hubs, is an online 3D Printing service which facilitates transactions betwen 3D Printer owners (Hubs) and people who want to make 3D prints. Printer owners can join the platform to offer 3D printing services while customers can locate printer owners to get their 3D models printed nearby.

So what I did myself is design a soap dish. The one we had in our shower was glass which fell and broke. I tried looking for a replacement in several hardware stores but came up empty. So, this was a perfect use case for 3D printing.

This was the end result:

Some learning points for this:

While support material can be used to create complex shapes, the results is not as smooth as I would hope for.
The soap dish was a very thigh fit in the holder in the shower. This is good, because you can make a very precision part, it also means that accurate measurements (in some cases less than 1 millimeter) are very important.

So, allthough TinkerCAD is quite usable, it is very much an entry level tool. I have since switched to using Art of Illusion which is harder to use, but allows for larger, more complex models. Also, it allows to create shapes by specifying coordinates and sizes by typing a floating point value. This is important because you can get very accurate measurements that way (as opposed to dragging shapes with the mouse, which can't do that accurately).

OpenShift notes

2021-12-25T00:00:00+01:00

THIS IS FOR ARCHIVAL PURPOSES. THIS IS OUT-OF-DATE

backup OpenShift

openshift getenv(USER) from OpenShift php
ssh to {user}@{app-domain} gear snapshot  > file

Run gear app

OpenShift migration further notes

Encrypt a file using a supplied password :

$ openssl enc -aes-256-cbc -salt -in file.txt -out file.txt.enc -k PASS

Decrypt a file using a supplied password :

$ openssl enc -aes-256-cbc -d -in file.txt.enc -out file.txt -k PASS

Add script to tar plugin, then rhc a SSH key into account, ssh to account and tar x data...

get target directory
clean-up target directory
extract new contents

Probably can use deploy as an example...

need to list images that have been uploaded to S3.
need to convert imgsrc references...

? wordpress filters vs hooks ?

Sample stuff:

Skeleton

Mangle data when saving

post types ... perhaps we add an S3 flag to the post type: Attachment

WordPress

Standard Customizations:

Appearance
- Set theme
- Site Identity
- Header & Background iamge
- Menus: Set-up top bar?
Settings
- General Settings
  1. Membership: Not anyone can register
  2. Timezone UTC
  3. Date/Time Format
- Reading
  1. For each article in the feed: Show full text
- Discussion
  1. Users must be registered to comment. Not fill out name+email
  2. Comments author must have previously approved comment
- Permalinks
  1. Month and name

Plugins

Front Page Category
- Customizer, Front Page Categories, select what to show
Collapsing category list
- Customizer, Widgets, Categories, customize...
bbPress
- NO anonymous posting
WP Social Login
- Bouncer
- Allow Username change
Rich Revies
- Integrate user accounts

OpenShift Recipe

The official deploy tool dpl does not seem to work with secondary branches.

Pre-requisistes

Install git
Install RHC command line
- yum install epel-release
- yum install rubygem-rhc
Install Travis command line
- yum install epel-release
- yum install ruby-devel rubygem-ffi (maybe others)
- gem install travis -v 1.8.2 --no-rdoc --no-ri
A github, travis-ci and opens

Preparing Repo

This section can be skipped if we already have a github repo.

Fork wordpress-example
Create any additional branches as needed.
Configure travis-ci by creating a basic .travis.yml
- language: php
- php:
- - '5.4'
- script: true
Since travis setup openshift doesn't work, we need to use the DIY deploy script. So make a copy of it. And configure:
- env:
- _ global:
- __ OPENSHIFT_USER=$username
- __ OPENSHIFT_SECRET=$secret
  1. Obviously the secret should be encrypted using:
    - travis encrypt OPENSHIFT_SECRET=$secret [--add env.global]
- script:
- - sh deploy.sh
- diydeploy:
- - deploy $branch:$openshift_app ... initially empty...

Deploying Repo to OpenShift App

Create a new Application from the Openshift console.
- Use (WordPress 4)
- Just leave initial repo to the default
- Decide on scaling options.
- DO NOT GO THROUGH SITE INSTALL YET!
Add the $branch:$openshift_app to the .travis.yml, and push so travis-ci will deploy.
Tweak configuration:
- force https through .htaccess.
- Enable MULTISITE (if needed)
Enable custom domain
- Create Domain Name (on DNS) and add custom domain in OpenShift
- Add Certificate to OpenShift (self-signed or maybe CloudFlare)
  - openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 3650 -nodes
- Wait for DNS to propagate
Log-on to the site and go through installation.
- Verify that URLs use https:
- Dashboard -> Settings -> General
- Verify in Permalinks that https is used.

Fork syncing

   - Clone repo
   - Configure a remote fork
     1. git remote -v
     2. git remote add upstream https://github.com/openshift/wordpress-example.git
   - Syncing a fork
     1. git fetch upstream
     2. git checkout master
     3. git merge upstream/master

      //define('DOMAIN_CURRENT_SITE', 'dev.iliu.net');
      if ($_SERVER['SERVER_NAME'] != 'dev.iliu.net') {
         define('DOMAIN_CURRENT_SITE', 'iliu.net');
      } else {
         define('DOMAIN_CURRENT_SITE', 'dev.iliu.net');
      }

openshift mailgun

Success! You're signed up and we just created your sandbox server sandboxf9dbaaa2f22a49a693955138381837e7.mailgun.org

Include the Autoloader (see "Libraries" for install instructions)

require 'vendor/autoload.php';
use Mailgun\Mailgun;

# Instantiate the client.
$mgClient = new Mailgun('key-xxxxxxxxxxxxxxxxxxxxxxxxx');
$domain = "sandboxf9dbaaa2f22a49a693955138381837e7.mailgun.org";

# Make the call to the client.
$result = $mgClient->sendMessage("$domain",
                  array('from'    => 'Mailgun Sandbox <postmaster@sandboxf9dbaaa2f22a49a693955138381837e7.mailgun.org>',
                        'to'      => 'Alejandro Liu <alejandro_liu@hotmail.com>',
                        'subject' => 'Hello Alejandro Liu',
                        'text'    => 'Congratulations Alejandro Liu, you just sent an email with Mailgun!  You are truly awesome!  You can see a record of this email in your logs: https://mailgun.com/cp/log .  You can send up to 300 emails/day from this sandbox server.  Next, you should add your own domain so you can send 10,000 emails/month for free.'));

Windows administration from the command line

2022-03-03T00:00:00+01:00

Windows system administration is very mouse driven and to reach all tools you need to browse through Windows explorer.

If you are like me and prefer to log on a limited privilege account and use Runas to perform admin tasks, you can open these consoles with the .msc file names.

Here is a list of admin tools with their .msc file names.

domain.msc: AD Domains and Trusts
admgmt.msc: Active Directory Management
dssite.msc: AD Sites and Serrvices
dsa.msc: AD Users and Computers
adsiedit.msc: ADSI Edit
azman.msc: Authorization manager
certsrv.msc: Certification Authority Management
certtmpl.msc: Certificate Templates
cluadmin.exe: Cluster Administrator
compmgmt.msc: Computer Management
comexp.msc: Component Services
cys.exe: Configure Your Server
devmgmt.msc: Device Manager
dhcpmgmt.msc: DHCP Managment
dfrg.msc: Disk Defragmenter
diskmgmt.msc: Disk Manager
dfsgui.msc: Distributed File System
dnsmgmt.msc: DNS Managment
eventvwr.msc: Event Viewer
ciadv.msc: Indexing Service Management
ipaddrmgmt.msc: IP Address Management
llsmgr.exe: Licensing Manager
certmgr.msc: Local Certificates Management
gpedit.msc: Local Group Policy Editor
secpol.msc: Local Security Settings Manager
lusrmgr.msc: Local Users and Groups Manager
nlbmgr.exe: Network Load balancing
perfmon.msc: Performance Monitor
pkiview.msc: PKI Viewer
pkmgmt.msc: Public Key Managment
acssnap.msc: QoS Control Management
tsmmc.msc: Remote Desktops
rsadmin.msc: Remote Storage Administration
ntmsmgr.msc: Removable Storage
ntmsoprq.msc: Removable Storage Operator Requests
rrasmgmt.msc: Routing and Remote Access Manager
rsop.msc: Resultant Set of Policy
schmmgmt.msc: Schema management
services.msc: Services Management
fsmgmt.msc: Shared Folders
sidwalk.msc: SID Security Migration
tapimgmt.msc: Telephony Management
tscc.msc: Terminal Server Configuration
licmgr.exe: Terminal Server Licensing
tsadmin.exe: Terminal Server Manager
uddi.msc: UDDI Services Managment
wmimgmt.msc: Windows Mangement Instumentation
winsmgmt.msc: WINS Server manager

Deploying Kerberos based SSO

2021-12-25T00:00:00+01:00

This article goes over how to implement Single-Sign-On on Linux. It goes over the integration around the Kerberos service and the applications, like for example FireFox.

Pre-requisites

Kerberos Domain Controller (KDC)
User accounts in the KDC
KDC based logins

To make sure that this is working, login to your workstation using your kerberos password and use the command:

klist

This should show your principals assigned to you.

Ticket cache: FILE:/tmp/krb5cc_XXXX_ErVb5X
Default principal: zzzz@LOCALNET

Valid starting       Expires              Service principal
01/11/2016 15:51:35  01/12/2016 15:51:34  krbtgt/LOCALNET@LOCALNET

Configuring Apache

Install any necessary modules on the server:
- yum install mod_auth_kerb
Create a service principal for the web server (this needs to be done on the KDC.
- kadmin.local -q "addprinc -randkey HTTP/www.example.com
Export the encpryption keys to a keytab:
- kadmin.local -q "ktadd -k /tmp/http.keytab HTTP/www.example.com
Copy /tmp/http.keytab to the webserver at /etc/httpd/http.keytab.
Set ownership and permissions:
- chmod 600 /etc/httpd/http.keytab
- chown apache /etc/httpd/http.keytab
Enable authentication, configure this:
- AuthType Kerberos
- AuthName "Acme Corporation"
- KrbMethodNegotiate on
- KrbMethodK5Passwd off
- Krb5Keytab /etc/httpd/http.keytab
- require valid-user
Re-start apache

Configure FireFox

Navigate to about:config
Search for: negotiate-auth
Double click on network.negotiate-auth.trusted-uris.
Enter hostname's, URL prefixes, etc, separated by commas. Examples:
- www.example.com
- http://www.example.com/
- .example.com

It is possible to configure this setting for all users by creating a global config file:

Find configuration directory:
- rpm -q firefox -l | grep preferences
Create a javascript file in that directory. (by convention, autoconfig.js; other file names will work, but for best results it should be early in the alphabet.)
Add the following line:
- pref("network.negotiate-auth.trusted-uris",".example.com");

Configure OpenSSH server

Create a service principal for the host (this needs to be done on the KDC.
- kadmin.local -q "addprinc -randkey host/shell.example.com
Export the encpryption keys to a keytab:
- kadmin.local -q "ktadd -k /tmp/krb5.keytab host/shell.example.com
Copy /tmp/krb5.keytab to the host at: /etc/krb5.keytab.
Set ownership and permissions:
- chmod 600 /etc/krb5.keytab
- chown root /etc/krb5.keytab
Enable authentication, change these settings in /etc/ssh/sshd_config:
- KerberosAuthentication yes
- GSSAPIAuthentication yes
- GSSAPICleanupCredentials yes
- UsePAM no # This is not supported by RHEL7 and should be left as yes
Restart sshd.

Configure OpenSSH clients

Configure /etc/ssh_config or ~/ssh/ssh_config:

Host *.localnet
  GSSAPIAuthentication yes
  GSSAPIDelegateCredentials yes

clipping ideas

2021-12-25T00:00:00+01:00

Divide into
- Work : Only visible to company and clients
- Personal: Public/Private areas

Features:

Send e-mail to address => creates entry
handle attachments
Rich text support?
Markdown through short codes (maybe)
Searchable
Auto Tag/Auto Categorize
Can create entries through UI

Options:

MHonArc
WordPress + WebMail posting

Let's Encrypt

2021-12-25T00:00:00+01:00

This is a service that let's you get SSL certificates for HTTPS. These certificates are trusted by major browsers. See Let's Encrypt This is a barebones howto to get SSL certificates:

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

This contains the client software for let's encrypt.

./letsencrypt-auto certonly --manual

This will start by updating and getting any needed dependencies and then jump to a wizard like configuration to get this done. Follow the prompts and pay special attention on the prompt used to validate your domain. (You need to create a couple of folders and a file with the right content). Afterwards your certificates will be in:

/etc/letsencrypt/live/mydomain.tld

Then go to your CPanel configuration, then upload:

privkey.pem to Private Keys
cert.pem to Certificates

Then you go to Manage SSL Hosts -> Browse Certificates, pick the right certificate. Then paste chain.pem (from /etc/letsencrypt/live/mydomain.tld) to the CA Bundle box.

undup

2021-12-25T00:00:00+01:00

So, after a long while, I wrote a new C language program. As usual, the same things that I dislike about C programming popped up, specifically the need for low level data structures and manual memory management.

I did learn some new things:

uthash : I have used this library before, but there were a few new features that I did not know before, specifically it not only includes the hash library, but also some other high level structures that were quite handy.
Unit testing : So I started using cu, a C unit testing library. Frst time I write a program with integrated unit-testing. I can see its usefulness, but it does feel like a lot of work. For a casual programmer like myself, does feel like an over-kill.
Continuous integration with Travis-CI : For this project I tried using a CI tool. I chose Travis-CI because it integrates with GitHub. This only makes sense with unit testing. Once again, for a casual programmer like myself, it feels like a bit too much, but I can see how it would be useful if you have multiple contributors to the same project repository.
Creating binaries for a Zyxel NSA 325 v2 : So I got the NSA 325v2 SDK, and I am cross compiling for it. Quite straightforward, but still, something new.
An interesting feature of this code, is that, when possible, it is object oriented.

Anyway, this project can be found in github:

Undup github repository

Markdown Javascript editors

2021-12-25T00:00:00+01:00

VUE JS: Includes a Markdown editor example that allows edit with online preview next to it

Embeddable JS Markdown editor : Has a button to preview

Editors that edit in preview-like mode

Picade Todo

2021-12-25T00:00:00+01:00

key mappings
- look up and label default mappings

  { KEY_UP_ARROW,    UP     },
  { KEY_DOWN_ARROW,  DOWN   },
  { KEY_LEFT_ARROW,  LEFT   },
  { KEY_RIGHT_ARROW, RIGHT  },

  { KEY_LEFT_CTRL,   BTN_1  },
  { KEY_LEFT_ALT,    BTN_2  },
  { ' ',             BTN_3  },
  { KEY_LEFT_SHIFT,  BTN_4  },
  { 'z',             BTN_5  },
  { 'x',             BTN_6  },

  { 's',             START  },
  { 'c',             COIN   },
  { KEY_RETURN,      ENTER  },
  { KEY_ESC,         ESCAPE },

  /* Change these lines to set key bindings for VOL_UP and VOL_DN */
   { 'u',      VOL_UP  },
   { 'd',      VOL_DN },

Properly secure pi to case
Properly configure MAME
SSH to picade
Add roms to picade
Add a external port to plugin controllers
scrapping games How?
netplay
setup battery power
Change the art work
Properly install power button
minecraft server
Install Java and how to run minecraft on PC
Add a HD for PS1 games sata adapter

Wiring...

GPIO --- 220Ohm --- +LED- ---> GND

Python

import RPi.GPIO as GPIO

GPIO.setmode(GPIO.BCM)
GPIO.setup(25, GPIO.OUT)
GPIO.output(25, 1)
GPIO.cleanup()

GPIO.setup(22, GPIO.IN)
GPIO.input(22) == bool

ASCIART

                            +--- 10 kOhm --- GND
                            |
                            |
                            |      _-v
GPIO -- 1 kOhm --+---+    +------ 3.3V

button

GND -- 10K Ohm --+---+ SW +--- 5V
                 |
GPIO----------------+

joystic

Centos7/RHEL7 FirewallD -- the least you need to know

2021-12-25T00:00:00+01:00

This post is just a simple hints-tips to get something going with FirewallD without going into too much detail.

Checking if you are using firewalld:
- firewall-cmd --state
Check your zones (needed later when opening ports):
- firewall-cmd --get-default-zone
- firewall-cmd --get-active-zones
Checking what is active:
- firewall-cmd --zone=public --list-all
Opening services:
- firewall-cmd --zone=public --add-service=http Or alternatively:
- firewall-cmd --permanent --zone=public --add-service=http
- firewall-cmd --reload Services are defined in /usr/lib/firewalld/services and /etc/firewalld/services.
Opening ports:
- firewall-cmd --permanent --zone=public --add-port=443/tcp
- firewall-cmd --reload

Raspberry Pi Thin Client

2021-12-25T00:00:00+01:00

Thin Client project want to create a very low price thin client over Raspberry Pi board! Microsoft RDC, Citrix ICA, VMWare View, OpenNX & SPICE

RPITC

Raspberry pi notes

2021-12-25T00:00:00+01:00

raspberry pi shops

Hardware to attach/secure Raspberry Pi boards

4 M2 16mm bolts, + nuts, + spacers?

Creating a Read-Only root for Raspbian:

Hack/cycle something: rapsberry gear

3D Printing Cases:

Replacing Emacs with Atom

2021-12-25T00:00:00+01:00

As an old UNIX guy I have been using emacs for years. So in a way, I am very comfortable with using it and most of keyboard shortcuts. But, it really is an old animal and I have been thinking that I should be moving to a more modern replacement to it for quite a while.

My latest attempt (and the most serious attempt to date) has been trying atom. Atom is a new editor from the makers of github which claims to have been inspired by emacs and also supports the latest web technologies.

After using it for some time, I found the following conclusions:

I am really used to using CTRL + key to move around. And switching to the dedicated arrow and home/end keys feels like a step backwards. I might be old fashioned, but it really is about keeping your fingers in the keyboard "home" row. Also, while the arrow keys are easy to find, I have trouble with the home/end keys (which apparently I use a lot when programming). Specially because I switch between a laptop and full size keyboard all the time.
I really like the automatic programming style automation in Emacs.
I miss the record macro/execute macro facility of Emacs.
The automatic "(" inserts ")" really annoys me.
Browsing for extensions in "Atom" seems a bit non-intuitive to me.
I am used to using the command line and open the files in a running editor directly from there. I am able to configure emacs to do this, it is not clear to me how to do this with Atom yet.
I don't know why I am so used to the Emacs CTRL+S (search) functionality.

The following things I really like:

Syntax highlighting is quite solid
The project view pane is very useful.
The Markdown preview pane.

So up to know, looks promising but I am not convinced. I still using emacs specially because it sometimes feel that atom is slow to start.

Online IDEs

2021-12-25T00:00:00+01:00

If you want to move to the cloud and like to code like me, this is kinda of a basic necessity.

This applies in particular to Chromebook users.

5 Best online IDEs

Lifehacker App Guides

2021-12-25T00:00:00+01:00

These two hyperlinks from Lifehacker are quite useful:

Upload to OpenWRT

2021-12-25T00:00:00+01:00

Base 64 decoding: coreutils-base64

#!/usr/local/bin/haserl --upload-limit=4096 --upload-dir=/tmp 
content-type: text/html

<html><body>
<form action="<% echo -n $SCRIPT_NAME %>" method=POST enctype="multipart/form-data" >
<input type=file name=uploadfile>
<input type=submit value=GO>
<br>
<% if test -n "$HASERL_uploadfile_path"; then %>
        <p>
        You uploaded a file named <b><% echo -n $FORM_uploadfile_name %></b>, and it was
        temporarily stored on the server as <i><% echo $HASERL_uploadfile_path %></i>.  The
        file was <% cat $HASERL_uploadfile_path | wc -c %> bytes long.</p>
        <% rm -f $HASERL_uploadfile_path %><p>Don't worry, the file has just been deleted
        from the web server.</p>
<% else %>
        You haven't uploaded a file yet.
<% fi %>
</form>
</body></html>

haserl man page

Uploader tool: post

Disable/Relocate cgi-bin

organizing notes

2021-12-25T00:00:00+01:00

My Documents

DOCUMENTS

Project Folder
- old
- YYYY
- deliverables
- clips?
category folder
- expenses? expense reports and digital receipts
- regs - passwords, registrations, etc...
- nice notes - thank you letters, etc.
Personal folder
- info or important health account data, friends contacts, etc
- clips
- writing - personal writing, notes, letter, drafts,
- taxes folder
logs
- activity log
- travel log

Naming convention:

--. - initials : author initials or source initials - monthday : 2 digits each - type : type of document # TODO Lists - Work - Outlook based - Tickler File - Must Do List - E-mail to TODO - Personal - Google Tasks # Weekly Review Steps NOTE: Clean-up temp folders * Collect loose paper notes and materials. (business cards, receipts, etc. - put in in basket for processing) * Get IN to zero * Empty your head (write down any new projects, action items, etc.) * Review Action lists (Mark off completed actions & review for reminders of further action steps to capture) * Review Previous Calendar Data (review for remaining action items, reference information, etc.) * Review Upcoming Calendar Data * Review Waiting For List (Records appropriate actions for any needed follow-up & check off received items) * Review Project and Larger Outcome lists (ensure that at least one kick-start action is in your system for each) * Review Any relevant checklists * Review Someday/Maybe List (Check for any projects that may have become active and transfer them to "Projects" & delete items no longer of interest) * Review "Pending" and Support Files (Browse through all work-in-progress support material to trigger new actions, completions, and waiting-fors) ## Six Level Model for Reviewing Your Own Work 1. current actions 2. current projects 3. areas of responsibility 4. 1-2 year goals 5. 3-5 year vision 6. big picture view * * * * projects: clearly defined outcomes and the next * actions to move them towards closure * horizontal focus: reminders placed in a trusted system * that is reviewed regularly * vertical focus: informal back of the envelope planning # Task Lists - @ANYWHERE : Actions that can be done anywhere (rare?) - @CALLS : Phonecalls - @ERRANDS : Actions that I can do while going about - @HOME : Actions that can only be done at home - @HOME_PC : Actions that can be done at a PC (home PC?) about home. - @REVIEW : Items for review. Should be only text. Attachments should go to my Dropbox folder. - @WAITING_FOR : Tracker items that need to be follow-up later - @WORK : Actions that can only be done at the office. - @WORK_PC : Actions that can be done at a PC (work PC) about work. - @AGENDAS : Notes on what to discuss with different people - SOMEDAY_MAYBE : Idea parking lot

Another Markdown Editor

2021-12-25T00:00:00+01:00

This one is GitHubFlavored markdown...

markdown editor

Web Links

2021-12-25T00:00:00+01:00

Here a few web-links to interesting web apps.

It covers stuff about password security and checking if web sites are down, etc etc.

Down For Everyone or Just Me:

If you're getting an error when visiting a certain site, it could be down or something could be wrong on your end. To see which it is, head to and type in the web site's domain. It'll let you know if it's actually down or whether you need to do a little more troubleshooting. You can head there quicker by typing in .

If you're curious how fast your internet is for any reason, this is the site to check. It'll give you both and upload and download speed, so you can find out if you're getting what you pay for (or if you're just getting faster speeds than your friends). Just load it up and click "Begin Test" to get started.

How Secure Is My Password?:

Does what it says on the tin. Type in a password and it'll tell you how long it would take to crack.

What's My IP:

Whether you're setting up a home media server with Subsonic or you just need to SSH into a computer at home, sometimes you need to know a computer's IP address from outside of your network, and this site will tell you what it is.

Can You See Me: If you're having connection issues with a certain program, like email, IM, or BitTorrent, it could be because your firewall or ISP is blocking a certain port that program needs. Canyouseeme.org will let you type in a port and check if it's open- if it isn't, then that could be the source of your trouble. If it's open, then you know it's something else.

Fiddle Markdown Tool

2021-12-25T00:00:00+01:00

For a quick and simple Markdown Preview:

Fiddle

Code Kingdoms

2021-12-25T00:00:00+01:00

Code Kingdoms is targeted towards six- to 13-year olds and looks very much like your everyday puzzle adventure game. Choose an animal, walk around a kingdom saving animals through puzzles. The difference is most of the puzzles require kids to use code elements to solve the puzzles. At first this is through dragging-and-dropping code snippets, but as they progress, kids will be typing in code themselves.

Besides teaching actual JavaScript through play, Code Kingdoms also helps kids develop problem-solving skills and the encouragement to keep pushing on when they're faced with a challenge in the game-much like programmers often have to push through challenging walls.

Code Kingdoms

Kerberos Client

2021-12-25T00:00:00+01:00

This simple mini how-to goes over the configuration of a linux system so it can use a Kerberos Realm server for authentication.

Make sure you have the pam_krb5 rpm files installed. You can check this by running the rpm -qa | grep pam command and seeing whether the pam_krb5 rpm files are listed. If they aren't, you can typically download them in an update of the Linux or Unix operating system that you are running.
Add the line to the "/etc/pam.d/system-auth" part of the auth section of Kerberos. Add it after the "pam_unix.so" line:
```
auth sufficient /lib/security/pam_krb5.so use_first_pass forwardable
```
Add the line to the "/etc/pam.d/system-auth" part of the password section of Kerberos. Add it after the "pam_unix.so" line:
```
password sufficient /lib/security/pam_krb5.so use_authtok
```
Add the line to the "/etc/pam.d/system-auth" part of the session section of Kerberos. Add it after the "pam_unix.so" line:
```
session optional /lib/security/pam_krb5.so
```

HP Envy 4504 Set-up

2022-11-11T00:00:00+01:00

I bought a HP Envy 4504. Overall I am happy with it. This is how I configure it so I can use with Linux.

This mini howto applies to ArchLinux, void linux and Centos/RedHat distributions.

Installation

Archlinux:

cups, hplip, python2, sane

Centos:

cups, hplip, hplip-gui, sane

Some optional dependancies may be needed.

void linux:

hplip-gui

And for scanning, install:

simple-scan and/or xsane

Configuration Arch Linux and Centos/RedHat

sudo systemctl enable cups
sudo systemctl start cups
sudo hp-setup

hp-setup -i Select "Network", and "Advanced Options -> Manual Discovery" Printer: npr1 PPD File:

/usr/share/ppd/HP/hp-envy_4500_series.ppd

Uncoment hpaio from /etc/sane/dll.conf.

void linux configuration

These are void linux specific settings:

enable cups:

ln -s /etc/sv/cupsd /var/service

Add printer (run with sudo):

print_host=npr1
hp-setup $print_host

Tweaks

Some commands:

lpstat -p
cupsenable printer

Also since it is a WIFI printer no To prevent thisrmally it will go into sleep/power save mode. This means if you then try to print from cups it will fail (printer is asleep). Subsequent prints should work but now cupsd has flagged the printer as paused. To prevent this you should run this command as root:

lpadmin -p ENVY_4500 -o printer-error-policy=retry-job

More configuration commands:

Set default paper size:
- echo a4 > /etc/papersize

Updates

2022-11-06: Removed from voidlinux:
- uncompress PPD file (otherwise it is not recognized) so that it runs:
- removed OBSOLEtE patch
- For scanning, uncoment hpaio from /etc/sane/dll.conf.
2020-03-09 : Removed:
- To prevent this you should configure the default ErrorPolicy in /etc/cups/cupsd.conf by adding in the top scope: "ErrorPolicy retry-job"
- References: superuser.com
2019-02-19 : Added void linux instructions.

RPMGOT

2021-12-25T00:00:00+01:00

Software package download proxy

rpmgot is a simple/lightweight software package download proxy. It was designed to run on an OpenWRT router with some USB storage. So it is fully implemented as an ash script.

The basic idea has been implemented multiple times. For example refer to this article on a squid based implementation.

Unlike squid, which once you include all its dependencies can use up over 1MB of space just to install it, this software has very few dependencies.

The idea is for small developers running the same operating system version(s) would benefit from a local mirror of them, but they don't have so many systems that it's actually reasonable for them to run a full mirror, which would entail rsyncing a bunch of content daily, much of which may be packages would never be used.

rpmgot implements a lazy mirror something that would appear to its client systems as a full mirror, but would act more as a proxy. When a client installed a particular version of a particular package for the first time, it would go fetch them from a "real" mirror, and then cache it for a long time. Subsequent requests for the same package from the "mirror" would be served from cache.

The RPM files are cached for a very long time. Normally it is an awful, awful idea for proxy servers to do interfere with the Cache-Control / Expires headers that sites serve. But in the case of a mirror, we know that any updates to a package will necessarily bump the version number in the URL. Ergo, we can pretty safely cache RPMs indefinitely.

You can find this in Github.

SSL Certificates

2021-12-25T00:00:00+01:00

So it is is a more dangerous world out there. You can start securing web sites using self signed certificates. Another option is to:

Use CloudFlare. This will use a CF certificate from the CF CDN to the web site, while using a self-signed certificate between the CF CDN to your web server.
Use startssl

Convert HTML to Markdown

2021-12-25T00:00:00+01:00

These web sites convert to Markdown:

Mardownifier: Convert the given URL
to-markdown: Convert HTML snippets
turndown

Raspberry Pi - Low cost CCTV

2021-12-25T00:00:00+01:00

A good tutorial on creating a low cost surveillance camera using the raspberry Pi camera module and one of thos fake surveillance camera things.

Instructables has a good tutorial on creating a low cost surveillance camera.

Essentially makes use of a Pi, the Camera module and fitted into one of those inexpensive fake surveillance cameras.

It uses motion for the motion detection software.

Raspberry Pi as a Stratum-1 NTP Server

2021-12-25T00:00:00+01:00

This is something I found:

http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html

Essentially it requires pairing a Raspberry Pi with a NTPI Raspberry Pi GPS addon board

On the software side of things you need rpi_gpio_ntp

Incredible PBX for RasPBX

2021-12-25T00:00:00+01:00

This is a link to IncrediblePBX for RasPBX. Looks like bundles to run Asterisk PBX'es on a Raspberry Pi. Neat.

dev notes 2014

2021-12-25T00:00:00+01:00

Replacment for Make and Autoconf: MakeMe

(If you don't have root but have Android 4+ you can use the command-line program adb from the Android SDK platform tools to make backups via a desktop computer)

chromebook ides

Dev Tools

Alternative languages:

D : better than C, but not over-the-top like C++? Covers only Win and Linux
Vala : Kinda like C# but for Gnome. Covers Win and Linux. (Android maybe through NDK).
Java: Kinda over the top and heavy. Covers Win and Linux. Android yes, but different GUI library. iOS probably yes.
Python: scripting language. Win, Linux. Android maybe... iOS maybe...
Javascript: scripting language. ALL PLATFORMS.

Other options:

Python with Kivy
Haxe

Build Tools

MakeKit - autotools look & feel but lighter
mobs: autoconf workalike.

Resources

libre projects : syslogd in perl, mobom perl modules.

My own Notes App

JumpNote      +      OI Notpad
(Background          (Tags support)
Sync)
          V
     Simple Note backend
          V
        Tags UI
     (Filter, modify tags)
          V
        Task UI
          V
        Widget

WebApp + Mobile Dev:

A Java framework that works on multiple platforms
Allows for webapps on desktops
PhoneGap
Test mobile apps on desktop
Javascript optimizer:
- closure
- UglifyJS
JS Compiler
Java 2 JS Toolkits:
- WebToolKit
- J2S
Python 2 JS Toolkigs: PyJS
JS Interpretr for command line:
- v8
- NodeJS
Android Alternative IDE

Documentation around Syncing...

Other Notes:

Perki replacement that runs on Android.
Use WebKit/PhoneGap + Javascript and HTML5
Markdown library for Javascript
Markdown editor for javscript
TXGR converted to HTML5 Canvas
How do we do background sync?

More example code:

We want to have it for Android, Linux and Windows.

Free Android SDK

We need to research:

Alternative to freewrap
- jsmooth
- launch4j
- gcc mingw
- gcj+swt
- winrun4j
Alternative to Canvas
Which Toolkit to use (SWT, Swing, AWT, etc)

A freewrap like tool for python:

pyiinstaller

More Android Dev options:

Resizing a Linux RAID

2021-12-25T00:00:00+01:00

It is possible to migrate the whole array to larger drives (e.g. 250 GB to 1 TB) by replacing one by one. In the end the number of devices will be the same, the data will remain intact, and you will have more space available to you.

Extending an existing RAID array

In order to increase the usable size of the array, you must increase the size of all disks in that array. Depending on the size of your disks, this may take days to complete. It is also important to note that while the array undergoes the resync process, it is vulnerable to irrecoverable failure if another drive were to fail. It would (of course) be a wise idea to completely back up your data before continuing.

First, choose a drive and completely remove it from the array

mdadm -f /dev/md0 /dev/sdd1
mdadm -r /dev/md0 /dev/sdd1

Next, partition the new drive so that you are using the amount of space you will eventually use on all new disks. For example, if you are going from 100 GB drives to 250 GB drives, you will want to partition the new 250 GB drive to use 250 GB, not 100 GB. Also, remember to set the partition type to 0xDA - Non-fs data (or 0xFD, Linux raid autodetect if you are still using the deprecated autodetect).

fdisk /dev/sde

Now add the new disk to the array:

mdadm --add /dev/md0 /dev/sde1

Allow the resync to fully complete before continuing. You will now have to repeat the above steps for each\ disk in your array. Once all of the drives in your array have been replaced with larger drives, we can grow the space on the array by issuing:

mdadm --grow /dev/md0 --size=max

The array now represents one disk using all of the new available space.

If the array has a write-intent bitmap, it is strongly recommended that you remove the bitmap before increasing the size of the array. Failure to observe this precaution can lead to the destruction of the array if the existing bitmap is insufficiently large, especially if the increased array size necessitates a change to the bitmap's chunksize.

 mdadm --grow /dev/mdX --bitmap none
 mdadm --grow /dev/mdX --size max
 mdadm --wait /dev/mdX
 mdadm --grow /dev/mdX --bitmap internal

If the system relies on the disks in the array for booting the OS (a common approach is to keep /boot in a RAID 1 array, i.e. md0, across all the disks in the array) then you might need to manually reinstall the bootloader on each of the new disks, because the array synchronization does not sync the MBR. This should be done directly on each disk and not on the array itself (/dev/mdX), and is safe to do with the array online. For example, to re-install GRUB on the first disk:

grub
grub> root (hd0,0)
grub> setup (hd0)

You need to repeat this for each new disk that should contain the bootloader. If you forget to do so, and find that you cannot boot the system after replacing all the disks, you can boot from a rescue CD/DVD/USB in order to install the bootloader as instructed above.

Extending the filesystem

Now that you have expanded the underlying partition, you must now resize your filesystem to take advantage of it.

You may want to perform an fsck on the file system first to make sure there are no underlying issues before attempting to resize the file system

 fsck /dev/md0

For an ext2/ext3 filesystem:

resize2fs /dev/md0

For a reiserfs filesystem:

resize_reiserfs /dev/md0

Please see filesystem documentation for other filesystems.

LVM: Growing the PV

LVM (logical volume manager) abstracts a logical volume (that a filesystem sits on) from the physical disk. If you are used to LVM then you are likely used to growing LVs (logical volumes), but what we grow here is the PV (physical volume) that sits on the md device (RAID array).

For further LVM documentation, please see the Linux LVM HOWTO

Growing the physical volume is trivial:

pvresize /dev/md0

A before-and-after example is:

root@barcelona:~# pvdisplay
  \-\-\- Physical volume ---
  PV Name               /dev/md0
  VG Name               server1_vg
  PV Size               931.01 GB / not usable 558.43 GB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              95379
  Free PE               42849
  Allocated PE          52530
  PV UUID               BV0mGK-FRtQ-KTLv-aW3I-TllW-Pkiz-3yVPd1

root@barcelona:~# pvresize /dev/md0
  Physical volume "/dev/md0" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

root@barcelona:~# pvdisplay
  \-\-\- Physical volume ---
  PV Name               /dev/md0
  VG Name               server1_vg
  PV Size               931.01 GB / not usable 1.19 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              238337
  Free PE               185807
  Allocated PE          52530
  PV UUID               BV0mGK-FRtQ-KTLv-aW3I-TllW-Pkiz-3yVPd1

The above is the PV part after md0 was grown from ~400GB to ~930GB (a 400GB disk to a 1TB disk). Note the PV Size descriptions before and after.

Once the PV has been grown (and hence the size of the VG, volume group, will have increased), you can increase the size of an LV (logical volume), and then finally the filesystem, eg:

lvextend -L +50G -n home\_lv server1\_vg
resize2fs /dev/server1\_vg/home\_lv

The above grows the _homelv logical volume in the _server1vg volume group by 50GB. It then grows the ext2/ext3 filesystem on that LV to the full size of the LV, as per Extending the filesystem above.

Source: https://raid.wiki.kernel.org/index.php/Growing

Wi-Fi Sd Cards

2021-12-25T00:00:00+01:00

My latest weekend project. Making a normal digital camera WIFI enabled.

With the Transcend Wi-Fi SD Card you can convert any digital camera into a Wi-Fi enable camera.

What I did here is to set it up so that it would automatically upload photos whenever I turn the camera on while at home.

The nice thing about this camera is that it runs a fully functional Linux environment within the card. The manufacturer was also nice enough to give you the opportunity to customize the card by running arbitrary shell scripts from the SD card itself.

My code is in github.

Raspberry Pi Weekend project

2021-12-25T00:00:00+01:00

So finally took the time to try out a Raspberry Pi. For this weekend project wanted to do something relatively simple. Essentially, I wanted to recreate/enhance the functionality of a TL-WR702N.

The TL-WR702N Nano Router is a neat device but being closed, can not be customized to what I wanted. It can be used in the following modes:

AP
Client
Repeater
Router
Bridge

Specifically I was interested in the bridge mode. However, rather than bridging from one SSID to another SSID, I wanted to route/nat between the two. So in theory, should be simple to implement (as the hardware should have all the necessary components) but not allowed by the software.

Enter the Raspberry Pi.

So the Pi, is a mini computer that can be loaded with any software you want. The B-model, has a built-in Ethernet and USB ports to plug-in two WIFI adaptors. For this functionality I am using the following:

Raspberry-Pi Model-B
WIFI stick (2 units)

For the software I am using:

Raspberry Pi buildroot
hostapd-rtl8192cu from Realtek You need to get the RTL8188CUS package for Linux.

So you need two WIFI adaptors, as one will not work as master and slave at the same time. Essentially, one WIFI interface will act as the client WIFI station. The other WIFI interface acts as a WIFI hotspot. I chose to use buildroot instead of a normal Linux distro like Raspbian or Arch Linux Arm because I wanted to run it as an embedded system. Normal Linux distro's are supposed to be properly shutdown and would complain when you simply yank the power cord. The buildroot image I have is customized so that the file system is always mounted read-only. It will switch to read-write only to write persistent data and then switch back to read-only. The normal hostapd that comes with buildroot is the normal open source project and does not come with the rtl8192cu driver. You need to download and build the Realtek version. For this to work, I did the following:

Create a start-up scripts that set the whole thing up.
- sets-up the filesystem
- starts syslog, sshd, rngd
- sets-up eth0 and wlan0 to be configured by ifplugd
- starts wpa_supplicant on wlan0
- starts httpd
- start and configure wlan1 as an Access Point.
- start and configure dnsmasq for DNS and DHCP.
Wrote a small web UI to configure the WIFI client.

All this stuff can be found in github.

DVD archiving

2021-12-25T00:00:00+01:00

This is my simple procedure for backing up my DVD movies:

Examine the DVD:

dvdbackup -i /dev/sr0 -I

Create a full backup:

dvdbackup -i /dev/dvd -o ~ -M

Creating an ISO:

mkisofs -dvd-video -udf -o ~/dvd.iso ~/movie_name

Testing the newly created ISO:

mplayer dvd:// -dvd-device ~/dvd.iso

Private vs. Personal

2021-12-25T00:00:00+01:00

In Microsoft Outlook has the option to tag e-mails with a sensitivity tag. Technically this is fairly meaningless. However sometimes I like to use them.

The confidential tag is quite self explanatory. I always confuse what is the difference between private and personal. So here is one possibility...

Personal information are things like preferences, political association, likes and dislikes.
Private information are things like bank account numbers. Stuff that you probably would like to keep secret.

Cleaning-up Outlook Calendar

2021-12-25T00:00:00+01:00

This is a procedure I go through at the end of the year to clean-up my Outlook Calendar. Usually the Outlook Calendar gets full of junk over time. So this is something worth doing on a regular basis.

Procedure for Outlook 2007

Backup calendar folder
Select default calendar
Switch view to Inactive Appointments (non-recurrent)
Delete appointments
Switch view to Inactive Appointments (recurrent)
Delete appointments

Procedures for Previous versions of Outlook

This is my procedure for cleaning my Outlook calendar from old appointments and other assorted outdated stuff:

Backup your calendar folder (just in case)
Create a temporary Calendar folder
Select your default Calendar
Switch to All Appointments view:
- View -> Current View -> All Appointments
Select all the appointments and move them to the temporary Calendar folder
Select the temporary Calendar folder
Switch to Active Appointments vew:
- View -> Current View -> Active Appointments
Select all the visible appointments and move the m back to the default Calendar folder.
You can now dispose of the temporary folder (or backed it up for reference.

Chrome Kerberos Authentication

2021-12-25T00:00:00+01:00

To config chrome to use kerberos authentication you need to start the application the following parameter:

auth-server-whitelist - Allowed FQDN - Set the FQDN of the IdP Server. Example: chrome --auth-server-whitelist="*aai-logon.domain-a.com"
auth-negotiate-delegate-whitelist - For which FQDN credential delegation will be allowed.

References:

Deploying Chrome Extensions

2021-12-25T00:00:00+01:00

The following links outline how to deploy Chrome extensions in a enterprise manner:

My Must Have Android Apps

2021-12-25T00:00:00+01:00

This is a list of my favorite Android Apps:

Essentials

Barcode Scanner - Play Store F-Droid
Ghost Commander - F-Droid
F-Droid Alternative Application Manager. Usually Open source stuff with significantly less crap ware and ads.

Productivity

WordPress - PlayS tore
KeePassDroid - f-droid
GoTasks - Play Store
Dropbox - Play Store
Quickoffice - Play Store
SimpleNote - Play Store

Facebook - Play Store
Facebook Messenger Play Store
LinkedIn Play Store
Skype - Play store

Travel

KLM - Play Store
My Tracks - Play Store
TEST Wikivoyage offline - Travel guide. F-Droid

Tools

KPN HotSpots - Play Store
Timer - F-Droid
Yahoo Weather - Play Store

Special use

Searchlight - Flashlight App. F-Droid
Floating Image - Photo frame F-Droid
Worldclock - F-Droid

Diagnostics

TEST List my apps - F-Droid
TEST List Apps - F-Droid
TEST Internet Call settings - F-Droid

Root stuff

Root Verifier - F-Droid
No Frills CPU Control - F-Droid
Performance Control - F-Droid
oandbackup - F-Droid

Premium

GPS Test Plus - Play Store
Graffiti Pro for Android
TEST Signal Booster - Play Store

Kids

PlusMinusTimesDivide - F-Droid
MidiSheetMusic - F-Droid
Learn Music Notes - F-Droid

Stock Experience and Alternatives

Used to replace crapware for something closer to the Android experience or to complement incomplete ROMs.

Holo Locker Play Store
Holo Launcher Play Store
Contacts+ Play Store
AOSP Calendar - F-Droid
Stock Music Player F-Droid
Launcher3 F-Droid

Evaluate

RSS reader

Tiny Tiny RSS - Play Store
TTRSS-Reader - F-Droid

Drawing

Markers - F-Droid

Text editors

TED - F-Droid
Text Edit - F-Droid
Turbo Editor - F-Droid

Readers

need to support: Epub, Chm, Pdf, CBR/CBZ reader

Page Turner - Web site EPUB (Does page location synchronisation)
CoolReader: F-Droid epub, chm fb2, txt, rtf,tcr, html
APV PDF Viewer: F-Droid pdf
Document Viewer: F-Droid pdf, cbz djvu, xps,fb2
VuDroid: F-Droid pdf, djvu
ACV: F-Droid cbz, jpeg, png, bmp, folders

Multimedia

XBMC Remote - F-Droid
UPNP Player - F-Droid
AMPlayer - For an Ampache server F-Droid
ServeStream - F-Droid

Misc

Box - Play Store
Bump - Play Store
Daily Money - F-Droid
HotSpot Login - F-Droid
Linphone - F-Droid
LinConnect - Send notifications to desktop F-Droid
Serval Mesh - F-Droid
Read-it later poche - F-Droid
SSH client & Terminal Emulator - F-Droid
VNC client - F-Droid
Omnidroid automation - F-Droid
Wifi Analyzer - Play Store
Solitaire - Play Store F-Droid
ReGalAndroid - Client for G2/G3/Meanlto Gallery and Pwigo. F-Droid
DroidFish - F-Droid

Evaluate for Group Contacts/Calendar

aCal - F-Droid
CalDAV Sync Adapter - F-Droid
DAVdroid - F-Droid
Kolab Client- Dev Preview F-Droid

Wishlist?

Tetris
Sudoku
RPGs?
Tricorder
Push to talk
Emulators
Android remote control
wifi talkie or search 4 talkie

Alternative Keyboards

Remote Keyboard

For older Android versions:

Other things to check out:

wp-cron and cron

2021-12-25T00:00:00+01:00

Normal WordPress operation has a cron like functionality that runs scheduled tasks as users visit the blog.

It is possible to replace this with a standalone cron (like UNIX cron).

To disable the "webcron" (i.e. trigerring tasks as URLs are visited) add to your wp-config.php the following:

 define('DISABLE_WP_CRON', true);

Then call this from cron:

 curl http://example.com/wp-cron.php

Optionally you could call wp-cron.php using the php-cli executable.

Using wget with given IP/vhost

2021-12-25T00:00:00+01:00

This is one neat trick. For vhosts you can connect with an IP yet provide the right host name with the following:

 wget http://1.1.1.1/  --header 'Host: www.example.com'

Using a NAS200 as a Print server

2021-12-25T00:00:00+01:00

Last weekend I had a small weekend project to move my All-In-One Printer/Scanner from my Xen host server to a spare NAS200 I had lying around. Since the NAS200 has a i486 compatible CPU, and I had been able to run a CentOS 5 distro before, I figure it would make a good server with low power consumption.

For that I updated my NASCC firmware so that it would boot a USB key, and update my CentOS image creation script. This worked well, I was able to boot CentOS without that much effort altogether.

I myself have an Epson Stylus CX5500 which unfortunately only comes with binary drivers. This was not much of a problem since the NAS200 has a i486 compatible CPU. I find this is relatively unique among different NAS models.

Alas, the performance was quite disappointing. I should be used to the NAS200 underperforming. But really, this was truly sad. I did not bother to test the printing, but I did try scanning with it. Running scanimage to scan a single page was taking over 15 minutes before I hit Ctrl+C.

It was an idea, but the results were so sub par. The only take-aways of this are:

I was able to run open source as well as binary blobs on a NAS200 relatively easily.
I was able to use CentOS5 pretty much out-of-the box. No recompiles required. Did notice though that cups would seg-fault. My guess is that the i386 package some how got some i686 optimizations on it.
My Linux Ethernet Console made a very good network console. I was able to troubleshoot some very early boot problems with it.
NAS200 performance for scanning was abysmal.

UNIX find with dates

2021-12-25T00:00:00+01:00

-atime/-ctime/-mtime the last time a files's access time, file status and modification time, measured in days or minutes. Time interval in options -ctime, -mtime and -atime is an integer with optional sign.

n: If the integer n does not have sign this means exactly n days ago, 0 means today.
+n: if it has plus sing, then it means more then n days ago, or older then n,
-n: if it has the minus sign, then it means less than n days ago (-n), or younger then n. It's evident that -1 and 0 are the same and both mean today.

Examples:

Find everything in your home directory modified in the last 24 hours: $ find $HOME -mtime 0
Find everything in your home directory modified in the last 7 days: $ find $HOME -mtime -7
Find everything in your home directory that have NOT been modified in the last year: $ find $HOME -mtime +365
To find html files that have been modified in the last seven days, I can use -mtime with the argument -7 (include the hyphen): $ find . -mtime -7 -name "*.html" -print

If you use the number 7 (without a hyphen), find will match only html files that were modified exactly seven days ago:

 `$ find . -mtime 7 -name "*.html" -print`

To find those html files that I haven't touched for at least 7 days, I use +7:

$ find . -mtime +7 -name "*.html" -print

Enable local file caching for NFS share on Linux

2021-12-25T00:00:00+01:00

In Linux, there is a caching filesystem called FS-Cache which enables file caching for network file systems such as NFS. FS-Cache is built into the Linux kernel 2.6.30 and higher. In order for FS-Cache to operate, it needs cache back-end which provides actual storage for caching. One such cache back-end is cachefiles. Therefore, once you set up cachefiles, it will automatically enable file caching for NFS shares.

Requirements

One requirement for setting up cachefiles is that local filesystem support user-defined extended file attributes (i.e., xattr), because cachefiles use xattr to store extra information for cache maintenance. If your local filesystem is ext4-type, you don't need to worry about this since xattr is enabled in ext4 by default. However, if you are using ext3 filesystem, then you need to mount the local filesystem with "user_xattr" option. To do so, edit /etc/mtab to add "user_xattr" mount option to the disk partition that will be used by cachefiles for file caching. For example, assuming that /dev/hda1 is such a partition:

/dev/hda1 / ext3 rw,user_xattr  0 0

After modifying /etc/fstab, reload it by running:

$ sudo mount -o remount /

Configure CacheFiles

In order to set up cache back-end using cachefiles, you need to install cachefilesd, a userspace daemon for managing cachefiles. To install cachefilesd on Ubuntu or Debian:

$ sudo apt-get install cachefilesd

To install cachefilesd on CentOS, Fedora or RedHat:

$ sudo yum install cachefilesd
$ sudo chkconfig cachefilesd on

After installation, enable cachefilesd by editing its configuration file as follows.

$ sudo vi /etc/default/cachefilesd

RUN=yes

Next, mount a remote NFS share with fsc option:

 $ sudo vi /etc/fstab

 192.168.1.13:/home/xmodulo /mnt nfs rw,hard,intr,fsc

Alternatively, if you mount the remote NFS share from the command line, specify fsc as a command-line option:

$ sudo mount -t nfs 192.168.1.13:/home/xmodulo /mnt -o fsc

Finally, restart cachefilesd:

$ sudo service cachefilesd restart

At this point, file caching should be enabled for the mounted NFS share, which means that previously accessed files in the mounted NFS share will be retrieved from local file cache. If you want to flush NFS file cache for any reason, simply restart cachefilesd.

 $ sudo service cachefilesd restart

Source: xmodule.com

sdf.org

2021-12-25T00:00:00+01:00

sdf.org

This one is an interesting site.

The Super Dimension Fortress is a networked community of free software authors, teachers, librarians, students, researchers, hobbyists, computer enthusiasts, the aural and visually impaired. It is operated as a recognized non-profit 501(c)(7) and is supported by its members.

Our mission is to provide remotely accessible computing facilities for the advancement of public education, cultural enrichment, scientific research and recreation. Members can interact electronically with each other regardless of their location using passive or interactive forums. Further purposes include the recreational exchange of information concerning the Liberal and Fine Arts.

Members have UNIX shell access to games, email, usenet, chat, bboard, webspace, gopherspace, programming utilities, archivers, browsers, and more. The SDF community is made up of caring, highly skilled people who operate behind the scenes to maintain a non-commercial INTERNET.

Driving Continuous Integration from Git

2021-12-25T00:00:00+01:00

Testing, code coverage, style enforcement are all check-in and merge requirements that can be automated and driven from Git.

If you're among the rising number of Git users out there, you're in luck: You can automate pieces of your development workflow with Git hooks. Hooks are a native Git mechanism for firing off custom scripts before or after certain operations such as commit, merge, applypatch, and others. Think of them as henchmen for your Git repo. Pre-operation hooks act as bouncers, guarding your repo with a velvet rope. And post-operation hooks are your Man Friday, faithfully carrying out follow-up tasks on your behalf.

Installing hooks for a Git repository is fairly straightforward, and well-documented. In this article, we focus on using Git hooks to augment continuous integration practices, starting with an example that makes combining Git and continuous integration (CI) less painful. The code is written in Ruby. Fortunately, Ruby is a language that highly prizes readability, so even if you don't know Ruby, you can easily follow along.

Automate CI Configuration for Git Branches

One of the blessings of Git is how easy it is to branch off and develop in isolation. This means the master stays releasable, you get the freedom to experiment, and your teammates aren't derailed if code from the experimentation proves to be half-baked. One challenge of Git, however, is how many branches a team ends up with ? scores of active branches, most of which live for only a few days. Who is going to take the time to set up continuous integration for all those piddly little branches? Your henchmen, that's who.

To automatically apply CI to new development branches, you'll use the "post-receive" hook type. These are server-side hooks, triggered after pushes to the repository are completed. In such cases, you can use the post-receive hook to fire off a script that programmatically clones a master's CI configs and applies them to new branches using the CI server's exposed API. It might look something like this, when using the open-source and hugely popular Jenkins CI server:

#!/usr/bin/env ruby

 # Ref update hook for creating new Jenkins job 
 # configurations for newly pushed branches.
 #
 # requires Ruby 1.9.3+ 

 require 'yaml'
 require 'net/https'
 require 'uri'
 require 'rexml/document'
 include REXML

 # load ci-config.yml from hook directory
 def load_config
     hookDir = File.expand_path File.dirname(__FILE__)
     configPath = hookDir + "/ci-config.yml"
     puts configPath
     raise "No ci-config.yml found." unless File.exists? configPath
     YAML.load_file(configPath)
 end

 # Grab the configured Jenkins server
 config = load_config
 raise "ci-config.yml file is incomplete: missing jenkins_server" unless
     config["jenkins_server"]
 server = config["jenkins_server"]
 raise "ci-config.yml file is incomplete: username, password, url and
     default_job are required for jenkins_server" unless
         server['url'] and server['username'] and
         server['password'] and server['default_job']

 # iterate through updated refs looking for new branches
 ARGF.readlines.each { |line|
     args = line.split
     oldVal = args[0]
     newVal = args[1]
     ref = args[2]

     if /^0{40}$/.match(oldVal) and ref.start_with?("refs/heads/") 
         # new branch!
         # retrieve the jenkins job config
         # TODO only need to do this once!
         uri = URI.parse(
            "#{server['url']}/job/#{server['default_job']}/config.xml")
         req = Net::HTTP::Get.new(uri.to_s)
         req.basic_auth server['username'], server['password']
         http = Net::HTTP.new(uri.host, uri.port)
         http.verify_mode = OpenSSL::SSL::VERIFY_NONE
         http.use_ssl = uri.scheme.eql?("https")

         # execute the request
         response = http.start {|http| http.request(req)}

         raise "Bad response from jenkins, is your ci-config.yml correct?"
             unless response.is_a? Net::HTTPOK

         # parse the config.xml from the response
         doc = Document.new response.body
         doc.root.get_elements(
             "//branches/hudson.plugins.git.BranchSpec/name").each { 
                 # overwrite branch to be our new ref
                 |elem| elem.text = ref 
         }

         # create a new request to upload the modified config.xml
         newJob = ""
         doc.write newJob

         newJobName = ref["refs/heads/".length..-1].gsub("/", "-")
         uri = URI.parse("#{server['url']}/createItem?name=#{newJobName}")        
         req = Net::HTTP::Post.new(uri.to_s, 
         initheader = {'Content-Type' => 'application/xml'})

         req.basic_auth server['username'], server['password']
         req.body = newJob

         # upload the new job
         response = http.start {|http| http.request(req)}
         raise "Failed to post new job to jenkins" unless
             response.is_a? Net::HTTPOK
     end
 }

With this hook in place, you need only push a dev branch to the repo, and it will automatically be put under test. (It's possible to run CI builds against branches using a build parameter to represent the target branch, but that muddles the build history. The cloning approach provides a clean, clear history.) Applying every last facet of the CI scheme to branches isn't necessary ? for example, running each and every branch through the load test gamut might be overkill. But even if you skip the load and UI tests, and run just unit and API- or integration-level tests, these are huge wins.

The risk of introducing defects into master is greatly reduced by testing on the branch before merging. Developers can also work more efficiently and confidently because of the frequent feedback on changes (instead of the old merge-then-pray technique). And for teams who include testing as part of their definition of "done," managers and scrum master types catch a break. With the Git hook automatically putting branch code under test, the team's practices and values are being enforced without the need for nag-mails or raised eyebrows during stand-up.

Vet Merges to Master

Two hallmarks of coding craftsmanship are an affinity for automated tests, and adherence to stylistic rules (such as avoiding empty try/catch blocks or duplicated code). Despite best intentions, everyone neglects best practices from time to time. That's where Git hooks come in. Pre-receive hooks living in the central repository qualify incoming pushes, making sure they're good enough to get past the velvet rope. Let's look at three hooks designed to protect master from slip-ups made on development branches.

Require Passing Branch Builds

The whole point of working on a development branch is to isolate yourself and create a space to experiment (read: "break stuff"). So it's natural to see failing tests on the branch while development is in progress. When it's time to merge to master, however, things had better be tidied up. This can be enforced programmatically with a hook that checks to see whether the incoming push is a merge to master, and if so, verify that all tests are passing on the branch before processing the merge.

If you happen to be using Bamboo, you can cleanly fetch test results for a given commit. If you use Jenkins or its predecessor, Hudson, you can fetch a set of recent build results then parse through them to see which builds ran against the commit in question. (This hook, and those that follow are implemented for the Bamboo CI server, but they can be implemented in more or less the same way on all CI servers.)

#!/usr/bin/env ruby

 # Ref update hook for verifying the build status of 
 # a topic branch being merged into 
 # a protected branch (e.g. master) from a Bamboo server.
 #
 # requires Ruby 1.9.3+ 

 require_relative 'ci-util'
 require 'json'

 # parse args supplied by git: <ref_name> <old_sha> <new_sha>
 ref = simple_branch_name ARGV[0]
 prevCommit = ARGV[1]
 newCommit = ARGV[2]

 # test if the updated ref is one we want to enforce green 
 # builds for exit_if_not_protected_ref(ref)

 # get the tip of the most recently merged branch
 tip_of_merged_branch = 
    find_newest_non_merge_commit(prevCommit, newCommit)

 # parse our Bamboo server config
 bamboo = read_config("bamboo", ["url", "username", "password"])

 # query Bamboo for build results
 response = httpGet(
    bamboo, 
    "/rest/api/latest/result/byChangeset/#{tip_of_merged_branch}.json")
 body = JSON.parse(response.body)        
 # tally the results
 failed = successful = in_progress = 0
 body['results']['result'].collect { |result|
     case result['state']
     when "Failed"
       failed += 1
     when "Successful"
       successful += 1
     when "Unknown"
       if result['lifeCycleState'] == "InProgress"
         in_progress += 1
       end
     end
 }

 # display a short message describing the build status for 
 #the merged branch and abort if necessary
 if failed > 0
     # at least one red build - block the branch update
     abort "#{shortSha(tip_of_merged_branch)} has #{failed} 
     red #{pluralize(failed, 'build', 'builds')}."
 elsif in_progress > 0
     # at least one incomplete build - block the branch update
     abort "#{shortSha(tip_of_merged_branch)} has #{in_progress}
     #{pluralize(in_progress, 'build', 'builds')} that have not
     completed yet."
 else   
     # all green builds - allow the branch update
     puts "#{shortSha(tip_of_merged_branch)} has #{successful} 
         green #{pluralize(successful, 'build', 'builds')}."
 end

Enforce Code Coverage Requirements

Along with successful test runs, you want to make sure that new code added on development branches is tested as thoroughly as code already on master. This ensures that the overall test coverage level of the project doesn't drop when a development branch is merged back in. This, too, can be checked with Git hooks.

A simple Git hook can verify that coverage on the branch meets the minimum threshold. To enforce this, a hook can be created to compare the coverage rate on master with that of the branch, and reject the merge if the branch's coverage is inferior.

Most CI servers don't expose code coverage data through their remote APIs. But there's an easy work-around: pulling down the code coverage report. To do this, the build must be configured to publish the report as a shared artifact, both on master and on the branch build. (Notice how automatically cloning build configs for development branches comes in handy here: set it up for master, and get it on the branch for free!) Once published, you can get the latest coverage report from master by a call to the CI server. For branch coverage, you can fetch the coverage report either from the latest build, or for builds related to the reference (commit) being merged, as shown here for the code coverage tool Clover.

#!/usr/bin/env ruby

 # Ref update hook for asserting the code coverage of a 
 # topic branch being merged into a 
 # protected branch (e.g. master) is the same or better
 #
 # requires Ruby 1.9.3+ 

 require_relative 'ci-util'
 require 'rexml/document'

 include REXML

 # Determine the code coverage for a particular commit by 
 # parsing Clover artifacts
 def find_coverage(bamboo, commit)
     # grab the clover.xml artifact from the build. 
     # This (assumes a shared artifact named 
     # 'clover' with 'clover.xml' at the root).
     # Change this for your coverage tool?s report name.
     clover_xml = shared_artifact_for_commit(bamboo, commit,
         bamboo["coverage_key"], "clover/clover.xml")
     doc = Document.new clover_xml

     # parse out the project metrics element from the response
     metrics = XPath.first(doc, "coverage/project/metrics")

     # Use algorithm similar to Clover 
     # (https://confluence.atlassian.com/x/LoHEB) for 
     # determining coverage percentage
     covered_elements = 
         metrics.attribute("coveredconditionals").value.to_i
     covered_elements += 
         metrics.attribute("coveredmethods").value.to_i
     covered_elements += 
         metrics.attribute("coveredstatements").value.to_i

     elements = metrics.attribute("conditionals").value.to_i
     elements += metrics.attribute("methods").value.to_i
     elements += metrics.attribute("statements").value.to_i

     coverage = 0
     if (elements > 0)
         coverage = covered_elements / elements
     end
     coverage
 end

 # parse args supplied by git: <ref_name> <old_sha> <new_sha>
 ref = simple_branch_name ARGV[0]
 prevCommit = ARGV[1]
 newCommit = ARGV[2]

 # test if the updated ref is one we want to enforce 
 # green builds for  exit_if_not_protected_ref(ref)

 # get the tip of the most recently merged branch
 tip_of_merged_branch = 
     find_newest_non_merge_commit(prevCommit, newCommit)

 # parse our bamboo server config
 bamboo = read_config("bamboo", 
     ["url", "username", "password", "coverage_key"])

 # calculate code coverage for the old and new commits
 prev_coverage = find_coverage(bamboo, prevCommit)
 new_coverage = find_coverage(bamboo, tip_of_merged_branch)

 # if the coverage has dropped for the new commit, block the update
 if prev_coverage > new_coverage
     abort "Code coverage for #{shortSha(tip_of_merged_branch)} is 
         only #{new_coverage}! #{ref} is currently at #{prev_coverage}." 
 else
     # if the coverage has increased, TFCIT
     puts "Nice work! Code coverage for #{ref} has 
         increased by #{new_coverage - prev_coverage}."
 end

Enforce Good Coding Style

Tests are something no self-respecting software project can do without, but they only tell part of the story. Open source tools such as Checkstyle and Findbugs scour your codebase and provide reports on stylistic violations ? anything from duplicated code to excessively long methods to the use of deprecated methods. These are hard-won guidelines, and they exist for a reason: Ignoring them can result in code being harder to understand, harder to maintain, and more vulnerable to runtime problems.

As with code coverage, each team has a different level of tolerance for unstylish code. But introducing more style violations is almost universally agreed-upon as undesirable. In this, Git hooks come to the rescue. Build artifacts come into play here as well since you can easily retrieve the violations report. (No CI server we're aware of exposes static analysis data via remote access API.) So you can create another pre-receive hook that checks violations for master and the dev branch, and rejects the push if it would introduce additional errors into master.

#!/usr/bin/env ruby

 # Ref update hook for asserting that a topic branch 
 # being merged into a protected
 # branch (e.g. master) does not introduce an increase in 
 # checkstyle violations
 #
 # requires Ruby 1.9.3+ 

 require_relative 'ci-util'
 require 'rexml/document'

 include REXML

 # This example 
 def count_checkstyle_violations(bamboo, commit)
     # grab the checkstyle.xml artifact from the 
     # build (assumes a shared artifact named 
     # 'checkstyle' with 'checkstyle-result.xml' at the root)
     checkstyle_xml = 
         shared_artifact_for_commit(bamboo, commit, 
         bamboo["checkstyle_key"],
        "checkstyle/checkstyle-result.xml")
     doc = Document.new checkstyle_xml
     # could go to town on the comparison here - but let's just count  
     # the raw number of errors for the time being
     XPath.match(doc, "//error").length
 end

 # parse args supplied by git: <ref_name> <old_sha> <new_sha>
 ref = simple_branch_name ARGV[0]
 prevCommit = ARGV[1]
 newCommit = ARGV[2]

 # test if the updated ref is one we want to enforce green builds for
 exit_if_not_protected_ref(ref)

 # get the tip of the most recently merged branch
 tip_of_merged_branch = 
    find_newest_non_merge_commit(prevCommit, newCommit)

 # parse our bamboo server config
 bamboo = read_config("bamboo", 
    ["url", "username", "password", "checkstyle_key"])

 # calculate number of checkstyle violations for 
 #the old and new commits
 prev_violations = 
     count_checkstyle_violations(bamboo, prevCommit)
 new_violations = 
     count_checkstyle_violations(bamboo, tip_of_merged_branch)

 # if the number of checkstyle violations has increased, block the update
 if prev_violations > new_violations
     abort "#{shortSha(tip_of_merged_branch)} 
        has #{new_violations} checkstyle violations! #{ref} 
        currently has only #{prev_violations}." 
 else
     # if the number of checkstyle violations has 
     # decreased, send kudos to the dev
     puts "Nice work! #{ref} has #{new_violations - prev_violations} 
         fewer checkstyle violations than before."
 end

To get the original source code and surrounding config files for all the server-side hooks you've seen here, clone the repo at: bitbucket.org.

Think Globally, Hook Locally

We know that the sooner an issue is discovered, the easier (and faster and cheaper) it is to fix. That's why hooks that operate on local clones of a repository are so useful: They offer immediate feedback. Because we don't get the cmd prompt back until a hook completes, client-side hooks should be limited to operations that take only a few seconds, lest the development flow be interrupted. Let's look at two hooks that complete almost instantly.

Get Branch Build Status

Exposing branch build status in the terminal window with a post-checkout hook catches two fish with one worm: It provides actionable information, and eliminates the need to switch applications to get it. Upon checkout (and remember, in Git "checkout" means switching branches, not pulling down code as with SVN and Perforce), this hook grabs the branch's head revision number from the local copy. It then queries the CI server to see whether that revision has been built, and if so, whether the build succeeded.

#!/usr/bin/env ruby

 # post-checkout hook for determining the build status of the 
 # checked out ref from the CI server.
 #
 # Requires Ruby 1.9.3+ 

 require 'yaml'
 require 'json'
 require 'net/https'
 require 'uri'

 # utility for correctly pluralizing quantities
 def pluralize count, single, multiple
   count == 1 ? single : multiple
 end

 # parse args supplied by git
 ref = ARGV[1]       # ref being checked out
 isBranch = ARGV[2]  # 0 = file checkout, 1 = branch checkout

 # we only care about branch checkouts 
 if isBranch == "1"
   # initialise build status counts
   failed = successful = in_progress = 0

   # loop through each configured Stash server, retrieving build 
   # statuses for the checked out commit and 
   # counting the number of failed, successful and in progress builds
   hookDir = File.expand_path File.dirname(__FILE__)
   configPath = hookDir + "/bamboo-config.yml"
   raise "No bamboo-config.yml found." unless File.exists? configPath
   config = YAML.load_file(configPath)
   raise "bamboo-config.yml file is incomplete: 
       username, password & url are required" unless
       config['url'] and config['username'] and config['password']

   # normalize base url
   baseUrl = config['url']
   # assume https if no scheme spcified
   if not baseUrl.start_with? "http"
     baseUrl = "https://#{baseUrl}"
   end
   # strip trailing slashes
   while baseUrl.end_with? "/"
     baseUrl = baseUrl[0..-2]
   end

   # prepare a request to hit the build status REST end-point
   build_status_resource = 
       "#{baseUrl}/rest/api/latest/result/byChangeset"
   uri = URI.parse("#{build_status_resource}/#{ref}")
   req = Net::HTTP::Get.new(uri.to_s, initheader = 
       {'Content-Type' => 'application/json', 
           'Accept' => 'application/json'})
   req.basic_auth config['username'], config['password']
   http = Net::HTTP.new(uri.host, uri.port)
   http.verify_mode = OpenSSL::SSL::VERIFY_NONE
   http.use_ssl = uri.scheme.eql?("https")

   # execute the request
   response = http.start {|http| http.request(req)}

   if not response.is_a? Net::HTTPOK
     puts 'An unknown error occurred while querying 
         Bamboo for build results.'    
     exit    
   else 
     # if the request succeeded, count 
     # the statuses from the response
     body = JSON.parse(response.body)        
     body['results']['result'].collect { |result|
       case result['state']
       when "Failed"
           failed += 1
       when "Successful"
           successful += 1
       when "Unknown"
           if result['lifeCycleState'] == "InProgress"
             in_progress += 1
           end
       end
     }
   end   

   # display a short message describing the build status 
   # for the checked out commit
   shortRef = ref[0..7]
   if failed > 0
     puts "Warning! #{shortRef} has #{failed} 
          red #{pluralize(failed, 'build', 'builds')} 
          (plus #{successful} green and #{in_progress} 
          in progress).\nDetails: #{uri}"
   elsif successful == 0
       puts "#{shortRef} hasn't built yet."
   else
        puts "#{shortRef} has #{successful} green 
            #{pluralize(successful, 'build', 'builds')}."
   end

 end

If, for example, the hook tells you the head commit on the master has built successfully, then it's a "safe" commit to create a feature branch from. Or let's say the hook says the build for that revision failed, yet the team's wallboard shows a green build for that branch (or vice versa). That means the local copy is out-of-date. Whether to pull down the updates is determined on a case-by-case basis.

This hook and its config files can be found at bitbucket.

Sanity-Check Code Style

Checking for violations at merge time is great, but a pre-commit hook analyzing the changeset keeps the style police off your back entirely. Start by capturing the names of files being updated or added and concatenating them. That string of file names is then passed into the Checkstyle run command. If violations are found, the commit is rejected.

Note that despite variations between them, all static analysis tools can be used with this approach. Findbugs, for example, must be run on the entire project because it looks at methods referenced across classes. But that's not necessarily a deal-breaker. Small and medium-sized projects can be fully analyzed quickly, especially if a generous heap space is allocated to the process.

Come As You Are

All the ideas presented here are vendor-neutral. Git hooks may not revolutionize software development the way continuous integration has, but every time a task, practice or rule is automated, it's a win.

From Dr. Dobbs Journal

Off site backup options

2021-12-25T00:00:00+01:00

This is my working notes on doing off-site backups to the cloud. Still trying to figure out where to keep Offsite backups.

These are the candidates:

Site	Free Quota	100GB/Yr	Notes
AltDrive	30 day	USD 45	Unlimited, Linux binary
iDrive	5GB	USD 6	Starts at 1TB, Linux binary, API
pCloud	20GB	USD 10	Starts at 500GB, Binary, Rest API, WebDAV
DropBox	2GB	USD 12	Starts at 1TB, ZYPKG available
CopyCom	15GB	USD 20	starts at 250GB, binary
MEGA	50GB	USD 20	Linux client, Starts at 500GB.
Google	15GB	USD 24	ZYPKG available
SkyDrive	15GB	USD 24	WebDav
MemoPal	3GB	USD 25	WebDav, ZYPKG available
ADrive	60 days	USD 25	FTP or WebDAV
MemoPal	3GB	USD 25	starts at 200GB, Binary or WebDav, ZYPKG available
iDriveSync	5GB	USD 33	WebDav
Amazon S3	5GB/1yr	USD 36	pay-per-use, REST API
box.com	10GB	USD 48	Uses WebDAV
Crashplan	1 month	USD 48	Unlimited, Binary
OtherDrive	2GB	USD 55	Java client
4shared	15GB	USD 78	WebDAV + FTP, Max 100GB?
CloudMe	3GB	USD 96	WebDAV

Todo

checkout google drive and dropbox.

This is another tool to compare vendors CloudWards

iDrive API info:
- Sample code
- Reference

No Comment

2022-11-02T00:00:00+01:00

This should require no explanation...

Alarm Notification

2022-01-10T00:00:00+01:00

This tutorial describes how to use the alarm manager to set alarms and how to use the notification framework to display them. In short, the sequence goes like this:

In an Activity AlarmManager.set is called with a PendingIntent containing a Uri.
When the alarm goes off, the Uri is called triggering a BroadcastReceiver.
In the BroadcastReceiver NotificationManager.notify is called with a PendingIntent.
When the notification is clicked, the Activity in the PendingIntent is started.

Alarm Manager

The Alarm Manager is a SystemService so it should be gotten like this

AlarmManager am = (AlarmManager) getSystemService(Context.ALARM_SERVICE);

It's set method can take parameters to define when, how and what to set off for the alarm. To set an absolute time and have it go off even if the device is on stand-by, use the RTC_WAKEUP type. The PendingIntent parameter is what gets called when the alarm goes off. Unless you want an Activity to start when the alarm goes off, a broadcast-type intent should be used like this

PendingIntent pendingintent = PendingIntent.getBroadcast(Activity.this, 0, intent, Intent.FLAG_GRANT_READ_URI_PERMISSION);

The intent parameter can hold a Uri which can contain some information about what the alarm is all about.

BroadcastReceiver and NotificationManager

The BroadcastReceiver must be defined in the manifest.xml like this

    <receiver
        android:name="package.AlarmReceiver"
    >
        <intent-filter>
            <action
                android:name="intentname" />
            <data
                android:scheme="myscheme" />
        </intent-filter>
    </receiver>

The intentname and myscheme values must match the name used in the intent and the scheme used in the uri in the intent. In the onReceive method, the notification is started to show the user the alarm went off. The Notification Manager is also a SystemService, get it like this

NotificationManager nm = (NotificationManager) context.getSystemService(Context.NOTIFICATION_SERVICE);

Create a new Notification object with an icon, a title and the time (probably System.currentTimeMillis()) Set some flags into the defaults like this

notification.defaults |= Notification.DEFAULT_SOUND;
notification.defaults |= Notification.DEFAULT_VIBRATE;

Use the setLatestEventInfo method to set another PendingIntent into the notification. The Activity in this intent will be called when the user clicks the notification. Again you can include a Uri into the intent to pass data to the Activity about which notification was clicked. Finally, be sure to use a unique id when calling the Notification managers notify method to actually fire the notification.

Activity

When the user has clicked the notification is it probably safe to remove it. In the Activity which gets called by the notification, a call to the NotificationManagers cancel can be used to do this. The id which was used to fire the notification in the BroadcastReceiver will let the system know which notification to remove.

Reloading

Android's Alarm Manager does not remember alarms when the device reboots. In order to restore the alarms you need to take these steps:

In the Activity which sets the alarms, also save information about each alarm into a database.
Create an additional BroadcastReceiver which gets called at boot-up to re-install the alarms.

Add the android.permission.RECEIVE_BOOT_COMPLETED uses-permission and the following receiver to the manifest.xml

    <receiver
        android:name="package.AlarmSetter"
    >
        <intent-filter>
            <action
                android:name="android.intent.action.BOOT_COMPLETED" />
        </intent-filter>
    </receiver>

In the onReceive method, read the database and re-install the alarms in the same way as was done at the top.

Proximity Alerts

A similar mechanism can be used for proximity alerts. The LocationManager will fire an alert which is nearly identical to an alarm. The same mechanism to restore alerts can be used after rebooting the device.

DID vendors

2021-12-25T00:00:00+01:00

So I have been researching DID vendors with limited success. So far my leading candidates are:

Vendor	Country	Set-up fee	Monthly fee	Per-Minute
Sonetel	NL	EUR 1.40	EUR 1.40	EUR 0.01
Sonetel	Peru	EUR 5.50	EUR 5.50	EUR 0.01
Sonetel	USA	EUR 0.70	EUR 0.70	EUR 0.01
twilio	NL	USD 1.00	USD 0.01
twilio	Peru	USD 5.00	USD 0.01
twilio	USA	USD 1.00	USD 0.01
callcentric	USA	Free	Free	Free
callcentric	USA	USD 3.95	USD 1.95	USD 0.015
callcentric	NL	USD 7.95	USD 7.94	USD 0.00
callcentric	Peru	USD 9.95	USD11.95	USD 0.00

Note that Sonetel has a free trial number available so it would probably be better for initial set-up. The callcentric has free numbers (limited area coverage). It also has interesting phone rates for outgoing calls.

Parsing JSON in Shell scripts

2021-12-25T00:00:00+01:00

This can be simple by using jq.

This is a command line JSON processor. Here are a couple of examples of what can be done:

$ cat json.txt

{
        "name": "Google",
        "location":
                {
                        "street": "1600 Amphitheatre Parkway",
                        "city": "Mountain View",
                        "state": "California",
                        "country": "US"
                },
        "employees":
                [
                        {
                                "name": "Michael",
                                "division": "Engineering"
                        },
                        {
                                "name": "Laura",
                                "division": "HR"
                        },
                        {
                                "name": "Elise",
                                "division": "Marketing"
                        }
                ]
}

To parse a JSON object:

jq '.name' < json.txt

"Google"

To parse a nested JSON object:

$ jq '.location.city' < json.txt

"Mountain View"

To parse a JSON array:

$ jq '.employees[0].name' < json.txt

"Michael"

To extract specific fields from a JSON object:

$ jq '.location | {street, city}' < json.txt

{
  "city": "Mountain View",
  "street": "1600 Amphitheatre Parkway"
}

Yealink W52P

2021-12-25T00:00:00+01:00

Yealink W52P

So I was looking to replace my analog cordless phones mainly because I wanted to have a centralized way to maintain phonebooks. Right now I have two cordless phone that I have to manually enter phonebook entries on the two handsets independently.

Initially I was thinking of getting small/cheap Android tablet and load it with a SIP soft phone. Trying with a couple of tablets I had was not very successful. On one hand my network topology did not work very well, on the other hand, the integration of the SIP soft phone with the directory and the other phone functions did not work as well as I expected.

So when I came across the W52P, I was initially attracted to the low price. Grandstream had a cheaper phone, but it did not have remote phonebooks. After checking the documentation of the W52P, I confirmed that it did have a remote phonebook functionality. So bought it and tried it out.

As a phone itself, it is about the same as the analog phones that it was replacing. The voice quality was pretty good.

Configuring the remote phone book was not as straight forward as I would have hoped. I was reusing the same phonebook script that I had used for my Grandstream phone. But I was getting "CONNECT ERROR" when I tried to use the remote phone. This was not very useful trying to figure out what was wrong. Turns out, because I was using a dynamic script, the script was not setting the Content-Length HTTP header. This apparently caused the phonebook not to download. Calculating the Content-Length header and setting it made the system work like a charm.

Grandstream GXP1400

2021-12-25T00:00:00+01:00

Grandstream GXP1400

The other day I replaced an analog phone with a Grandstream GXP1400 IP phone. I think it is a great value phone. It is one of the cheapest I could find yet supports all the features I was looking.

Specifically I wanted a IP phone that could:

Have a remote phone directory
Speaker phone

Setting it up was simple. Creating an account on Asterisk and providing the details for it to the phone. Creating a phone directory was also quite simple. I wrote a small PHP script to manage that.

Backing up GMail

2021-12-25T00:00:00+01:00

The other day I found Gmvault.

Gmvault is an open source Gmail backup software written in Python.

This article provides a good overview on how it works (found it better than the Gmvault documentation):

How to back up and restore Gmail account on Linux

It uses IMAP to connect to Gmail and also stores files in .eml (plain text) formatted fails. It has a converter to export to mbox and Maildir formats.

Probably would be good for archiving and using mnoGoSearch for search front end.

BOX.com promotions

2021-12-25T00:00:00+01:00

This is a good link to keep an eye on: box.com promotions

Alternative to DynDNS

2021-12-25T00:00:00+01:00

linuxaria blog article This article has a script how to use Dynamic DNS on afraid.org.

assist

2021-12-25T00:00:00+01:00

Assist is my archlinux scripted installation script.

https://github.com/alejandroliu/assist

By default it gives you a menu driven archlinux installation with supposedly sensible defaults.

It has command line hooks so that you can perform automated installs using bash scripts to customize it.

It can be deployed from the CDROM by downloading and executing Assist directly from the Internet or by injecting it into the init ramdisk for deployment either from PXE or a custom boot CDROM.

SSH Tricks

2021-12-25T00:00:00+01:00

A bunch of stupid SSH tricks that can be useful somehow, somewhere...

Forcing either IPv4 or IPv6

This is for the scenario that you know which specific protocol works
to reach a particular host. Usually good to eliminate the delay
for SSH to figure out to switch IP protocols. For IPv4:

ssh -4 user@hostname.com

For IPv6

ssh -6 user@hostname.com

Reuse a SSH connection

Rather than start a new TCP connection to a remote host, simply multiplex over an existing connection: Add to your ~/.ssh/config the following lines:

Host *
    ControlMaster auto
    ControlPath /tmp/%r@%h:%p
    ControlPersist 4h
# Another option for Control Path
    ControlPath ~/.ssh/%r@%h:%p

Enable compression

Use the -C option. Or in the config file:

Compression yes

Using cheaper cyphers

Using less computation-heavy ciphers in SSH, so that less time is spent during encryption/decryption. The default AES cipher used by OpenSSH is known to be slow. An independent study shows that arcfour and blowfish ciphers are faster than AES. blowfish is a fast block cipher which is also very secure. Meanwhile, arcfour stream cipher is known to have vulnerabilities. So use caution when using arcfour. Use the -c blowfish-cbc,arcfour option or in the config file:

Ciphers blowfish-cbc,arcfour

Improve Session Persistence

ServerAliveInterval 60
ServerAliveCountMax 10
TCPKeepAlive no

Counterintuitively, setting this results in fewer disconnections from your host, as transient TCP problems can self-repair in ways that fly below SSH's radar. You may not want to apply this to scripts that work via SSH, as "parts of the SSH tunnel going non-responsive" may work in ways you neither want nor expect!

Running Windows on Linux for Free

2021-12-25T00:00:00+01:00

Microsoft is now making available Windows VM image for testing Internet Explorer for free. You can find them at: Modern IE testing Currently the following versions are available:

Windows XP Professional SP3 + IE 6 or 8
Windows Vista + IE 7
Windows 7 + IE 8, 9, 10 or 11
Windows 8 + IE 11
Windows 8.1 Preview + IE 11

If Administrator access is needed the password is:

Passw0rd!

Remote VirtualBox

2021-12-25T00:00:00+01:00

RemoteBox is a Remote VirtualBox UI. It is similar phpVirtualBox in that allows to manage VirtualBox remotely (on a potentially headless server). They differ in their requirements:

RemoteBox does not require much on the server, but you need to install it on the client.
phpVirtualBox only requires a browser and rdp viewer on the client, but requires a web server with PHP support on the server.

IPv6 testing

2021-12-25T00:00:00+01:00

When trying to get on-to the IPv6 Internet, here are a couple of links to do diagnostics:

http://www.subnetonline.com/pages/ipv6-network-tools/online-ipv6-ping.php
This actually contain generic network tools.
http://ds.testmyipv6.com/
Confirm if your browser is connecting through IPv6
http://test-ipv6.com/
Check also DNS

PingTool.org

2021-12-25T00:00:00+01:00

Another short and sweet. This web site provides a number of on-line tools. Useful for diagnosing problems when setting a home server.

http://pingtool.org/

Web Backups

2021-12-25T00:00:00+01:00

As usual with any IT system backups are important. This does not change when using a free shared hosting provider. Because it is free, one would argue it is even more important.

For my wordpress web site I used something called cli-exporter. It let's you create "Wordpress" export files from the command line so it can be run from cron. This is important because backups have to be automated.

In addition to that, I copy the backup files to an off-site location. I do this by copying files using WebDAV to a storage provider. I did this by writing a simple script and using the PHP library SabreDAV which makes writing DAV clients quite easy.

I myself don't mind using other people's Open Source code to do something. I was actually surprised that I was not able to find something that meet my criteria. However, thanks to the power of open source I was able to find something that fit the bill exactly.

To make things more interesting, because I wanted to keep backup files as compressed Zip archives, my backup scripts did not work in one of the web hosts that I was using. They did not have the zip extensions enabled. This is surprising considering is quite standard. Luckily I was able to find a pure PHP library pclzip.

Mini-Howto: Setup proxy on Ubuntu

2021-12-25T00:00:00+01:00

A quick and dirty mini-howto to setup a proxy on Ubuntu. This is meant mostly for doing quick setup of a proxy on a cloud environment.

Install Squid with the following command at the Linux command prompt: sudo apt-get install squid
Edit the Squid config file in /etc/squid adding these lines: http_access allow local_net acl local_net src 10.10.0.0/255.255.0.0
Save the file, exit the editor and restart Squid. You are now ready to configure your browser to use the proxy server.
Click "Tools," "Options," "Advanced," "Network" and "Settings" in Firefox, which is the normal Ubuntu Linux browser. Select "Manual Proxy Configuration," enter the IP address of your proxy server, enter port 3128 in the Port field and then click "OK."

References:

http://science.opposingviews.com/set-up-secure-proxy-server-ubuntu-linux-23184.html by Alan Hughes

Using CloudFlare

2021-12-25T00:00:00+01:00

So I have signed up 0ink.net to use the CloudFlare service.

CloudFlare is a reverse proxy service that is supposed to speed up and improve web server security.

This is done by:

globally distributed reverse proxy cache network
filters incoming request for attacks
optimize content (i.e. compressing, removing redundnat text, etc).
improving retrieving of web pages that have multiple components.

For it to work they need to take over your DNS service. That means that your DNS records resolve to CloudFlare servers. So when editing your DNS records, the CloudFlare DNS editor has an extra settings that allows you to control if that DNS entry would use the CloudFlare network or not.

So if you want to be able to access your web server (i.e. www) for ftp or ssh, then you need to create an additinional CNAME record that points to the web server but set to by-pass the CloudFlare network.

Some tips on what to do after install cloudflare can be found here.

This is handy command to test if your web-server is having problems but not CloudFlare

 curl -v -A firefox/4.0 -H 'Host: yourdomain.com' YourServerIP

Askozia Desktop Appliance

2021-12-25T00:00:00+01:00

So last weekend finally had some time to work with a Askozia Desktop Appliance.

It actually arrived much earlier but without a Power Supply. Initially I though, "this is strange; I didn't know this supported PoE". (Power Over Ethernet). It turns out it didn't and there was a shipping mistake. After contacting the vendor, they sent me the required Power Supply.

Overall I think the product is quite nice. It has a very nice User Interface that is quite easy to use. Simple configurations are indeed very easy to set-up.

My feeling is that, as with any GUI, it usually trades user-friendly with expressiveness. So while I could configure most of the things I wanted from the UI, it did not support my home network topology fully.

Initially, I had a DMZ vs Home-LAN configuration, with the Askozia box in the DMZ. Because the separation between the DMZ and the Home-LAN was through the router, it considered all the IP phones (in the Home-LAN) on the other side of the NAT, so things did not work properly.

After moving the appliance to the same LAN as the IP phones things started working properly. Normally under Asterisk this would be solved through the "localnets" settings. The UI obviously did not expose this setting.

Next time I will try to use the Integrator Panel as it is supposed to expose the Asterisk configuration files directly.

CipherUSB

2021-12-25T00:00:00+01:00

This is an interesting concept. Essentially is an Encryption dongle that encrypts stuff between your PC and your USB mass storage device.

Addonics Product: CipherUSB.

Upgrading pacman config files

2021-12-25T00:00:00+01:00

So when upgrading software packages sometimes you need to merge changes. My recipe in archlinux is as follows:

Look for *.pacnew files.
Retrieve the original version (from /var/cache/pacman) from the old source package.
Use a 3 way merge tool between old version, current file and the pacnew file.

These are my options for merging:

diff3 -m : Merges the changes into a single file (Use -m option)
diffuse
meld

libmspack

2021-12-25T00:00:00+01:00

Recently I found this Open Source project. Apparently it recently gained support to unpack Exchange Offline Address Book files. What I don't know is after you unpack it, how would you use such a file. Intriguing but apparently falls a little bit short. Probably would need to try it out for myself to see how it works.

libmspack.

atratus project

2021-12-25T00:00:00+01:00

The other day I came across this project. Looks an interesting idea. It is a project that lets you run unmodified Linux binaries on Windows. It is more similar to WINE than to for example coLinux. While I conceptually I understand how it would work at a low level, I am curious how it works with dynamically link executables. This is something I would like to test out when I have time.

Media Tips

2021-12-25T00:00:00+01:00

This is an article about different media (and more specifically) video files can be manipulated.

This is just for historical purposes as now almost everything can be done using ffmpeg and the right options.

libmp4v2 contains:
- mp4art - to extract a picture (or coverart from mp4)
- mp4info - to get meta data from mp4 streams
- mp4tags - to set metadata and picture.
qt-fastload to move index to the front and making mp4 streamable
When encoding:
- Change max GOP or IDR to around 5 seconds.
- 2-pass avg bitrate: 800 or even 500...

Concatenating files:

ffmpeg

ffmpeg has a feature concat, like

ffmpeg -i concat:"video1.ts|video2.ts"

There is also a "concat" video filter that may be useful. See http://ffmpeg.org/trac/ffmpeg/wiki/How%20to%20concatenate%20%28join,%20merge%29%20media%20files

gpac

An alternative is gpac. One command it includes is MP4Box to concatenate MP4s

mp4box -cat sbd0.mp4 -cat sbd1.mp4 -new sbd.mp4

AviDemux

Of course the avidemux GUI can append files.

Final notes

So far I have not been able to create a reliable media concat recipe.

Media Gain

mp3gain can be used to normalize volume levels (without re-encoding). Accomplish this by using ReplayGain that needs to be supported by player. (XBMC claims to supports this).

Diskless Archlinux

2021-12-25T00:00:00+01:00

I am still to test this recipe

Server Configuration

First of all, we must install the following components:

A DHCP server to assign IP addresses to our diskless nodes.
A TFTP server to transfer the boot image (a requirement of all PXE option roms).
A form of network storage (NFS or NBD) to export the Arch installation to the diskless node.

Note: dnsmasq is capable of simultaneously acting as both DHCP and TFTP server.

Network storage

The primary difference between using NFS and NBD is while with both you can in fact have multiple clients using the same installation, with NBD (by the nature of manipulating a filesystem directly) you'll need to use the copyonwrite mode to do so, which ends up discarding all writes on client disconnect. In some situations however, this might be highly desirable. Install nfs-utils on the server.

# pacman -Syu nfs-utils

NFSv4

You'll need to add the root of your arch installation to your NFS exports:

# vim /etc/exports
/srv/arch *(rw,fsid=0,no_root_squash,no_subtree_check)

Next, start NFS.

# systemctl start rpc-idmapd.service rpc-mountd.service

NFSv3

# vim /etc/exports
/srv/arch *(rw,no_root_squash,no_subtree_check,sync)

Next, start NFSv3.

# systemctl start rpc-mountd.service rpc-statd.service

Note: If you're not worried about data loss in the event of network and/or server failure, replace sync with async--additional options can be found in the NFS article.

NBD

Install nbd .

# pacman -Syu nbd

Configure nbd.

# vim /etc/nbd-server/config
[generic]
    user = nbd
    group = nbd
[arch]
    exportname = /srv/arch.img
    copyonwrite = false

Note: Set copyonwrite to true if you want to have multiple clients using the same NBD share simultaneously; refer to man 5 nbd-server for more details. Start nbd.

# systemctl start nbd.service

Client installation

Next we will create a full Arch Linux installation in a subdirectory on the server. During boot, the diskless client will get an IP address from the DHCP server, then boot from the host using PXE and mount this installation as its root.

Directory setup

NBD

Create a sparse file of at least 1 gigabyte, and create a btrfs filesystem on it (you can of course also use a real block device or LVM if you so desire).

# truncate -s 1G /srv/arch.img
# mkfs.btrfs /srv/arch.img
# export root=/srv/arch
# mkdir -p "$root"
# mount -o loop,discard,compress=lzo /srv/arch.img "$root"

Note: Creating a separate filesystem is required for NBD but optional for NFS and can be skipped/ignored.

Bootstrapping installation

Install devtools and arch-install-scripts , and run mkarchroot.

# pacman -Syu devtools arch-install-scripts
# mkarchroot -f "$root" base mkinitcpio-nfs-utils nfs-utils

Note: In all cases mkinitcpio-nfs-utils is still required--ipconfig used in early-boot is provided only by the latter. Now the initramfs needs to be constructed. The shortest configuration, #NFSv3, is presented as a "base" upon which all subsequent sections modify as-needed.

NFSv3

# vim "$root/etc/mkinitcpio.conf"
MODULES="nfsv3"
HOOKS="base udev autodetect net filesystems"
BINARIES=""

Note: You'll also need to add the appropriate module for your ethernet controller to the MODULES array. The initramfs now needs to be rebuilt; the easiest way to do this is via arch-chroot .

# arch-chroot "$root" /bin/bash
(chroot) # mkinitcpio -p linux
(chroot) # exit

NFSv4

Trivial modifications to the net hook are required in order for NFSv4 mounting to work (not supported by nfsmount--the default for the net hook).

# sed s/nfsmount/mount.nfs4/ "$root/usr/lib/initcpio/hooks/net" | tee "$root/usr/lib/initcpio/hooks/net_nfs4"
# cp "$root/usr/lib/initcpio/install/{net,net_nfs4}"

The copy of net is unfortunately needed so it does not get overwritten when mkinitcpio-nfs-utils is updated on the client installation. From the base mkinitcpio.conf, replace the nfsv3 module with nfsv4, replace net with net_nfs4, and add /sbin/mount.nfs4 to BINARIES.

NBD

The mkinitcpio-nbd package needs to be installed on the client.

# pacman --root "$root" --dbpath "$root/var/lib/pacman" -U mkinitcpio-nbd-0.4-1-any.pkg.tar

You will then need to append nbd to your HOOKS array after net; net will configure your networking for you, but not attempt a NFS mount if nfsroot is not specified in the kernel line.

Client configuration

In addition to the setup mentioned here, you should also set up your hostname, timezone, locale, and keymap , and follow any other relevant parts of the Installation Guide .

Bootloader

Pxelinux

Install syslinux .

# pacman -Syu syslinux

Copy the pxelinux bootloader (provided by the syslinux package) to the boot directory of the client.

# cp /usr/lib/syslinux/pxelinux.0 "$root/boot"
# mkdir "$root/boot/pxelinux.cfg"

We also created the pxelinux.cfg directory, which is where pxelinux searches for configuration files by default. Because we don't want to discriminate between different host MACs, we then create the default configuration.

# vim "$root/boot/pxelinux.cfg/default"

default linux

label linux
kernel vmlinuz-linux
append initrd=initramfs-linux.img ip=:::::eth0:dhcp nfsroot=10.0.0.1:/

NFSv3 mountpoints are relative to the root of the server, not fsid=0. If you're using NFSv3, you'll need to pass 10.0.0.1:/srv/arch to nfsroot. Or if you are using NBD, use the following append line:

append ro initrd=initramfs-linux.img ip=:::::eth0:dhcp nbd_host=10.0.0.1 nbd_name=arch root=/dev/nbd0

Note: You will need to change nbd_host and/or nfsroot, respectively, to match your network configuration (the address of the NFS/NBD server) The pxelinux configuration syntax identical to syslinux; refer to the upstream documentation for more information. The kernel and initramfs will be transferred via TFTP, so the paths to those are going to be relative to the TFTP root. Otherwise, the root filesystem is going to be the NFS mount itself, so those are relative to the root of the NFS server.

# vim "$root/etc/fstab"

/dev/nbd0  /  btrfs  rw,noatime,discard,compress=lzo  0 0

Program state directories

You could mount /var/log, for example, as tmpfs so that logs from multiple hosts don't mix unpredictably, and do the same with /var/spool/cups, so the 20 instances of cups using the same spool don't fight with each other and make 1,498 print jobs and eat an entire ream of paper (or worse: toner cartridge) overnight.

# vim "$root/etc/fstab"
tmpfs   /var/log        tmpfs     nodev,nosuid    0 0
tmpfs   /var/spool/cups tmpfs     nodev,nosuid    0 0

It would be best to configure software that has some sort of state/database to use unique state/database storage directories for each host. If you wanted to run puppet , for example, you could simply use the %H specifier in the puppet unit file:

# vim "$root/etc/systemd/system/puppetagent.service"
[Unit]
Description=Puppet agent
Wants=basic.target
After=basic.target network.target

[Service]
Type=forking
PIDFile=/run/puppet/agent.pid
ExecStartPre=/usr/bin/install -d -o puppet -m 755 /run/puppet
ExecStart=/usr/bin/puppet agent --vardir=/var/lib/puppet-%H --ssldir=/etc/puppet/ssl-%H

[Install]
WantedBy=multi-user.target

Puppet-agent creates vardir and ssldir if they do not exist. If neither of these approaches are appropriate, the last sane option would be to create a systemd generator that creates a mount unit specific to the current host (specifiers are not allowed in mount units, unfortunately).

OpenWRT web

2022-01-10T00:00:00+01:00

Some useful tidbits to use when using the OpenWRT embedded web server (uHTTPD).

Embedded Lua

uHTTPd supports running Lua in-process, which can speed up Lua CGI scripts. It is unclear whether LuCI supports running in this embedded interpreter. LuCI seems to work fine (if not better) with the embedded Lua interpreter.

root@OpenWrt:~# opkg install uhttpd-mod-lua
Installing uhttpd-mod-lua (18) to root...
Downloading http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages/uhttpd-mod-lua_18_ar71xx.ipk.
Configuring uhttpd-mod-lua.
root@OpenWrt:~# uci set uhttpd.main.lua_prefix=/lua
root@OpenWrt:~# uci set uhttpd.main.lua_handler=/root/test.lua
root@OpenWrt:~# cat /root/test.lua
function handle_request(env)
        uhttpd.send("HTTP/1.0 200 OKrn")
        uhttpd.send("Content-Type: text/plainrnrn")
        uhttpd.send("Hello world.n")
end
root@OpenWrt:~# /etc/init.d/uhttpd restart
root@OpenWrt:~# wget -qO- http://127.0.0.1/lua/
Hello world.
root@OpenWrt:~#

Tested on Backfire 10.03.1 with uHTTPd 28.

HTTPS Enable and Certificate Settings and Creation

First of all, you need to install the uhttpd-mod-tls package in order to pull into the system the 'TLS plugin which adds HTTPS support to uHTTPd'. Then if listen_https is defined in the server configuration, the certificate and private key is missing. In this case (as of 10.03.1) you'll need to install the luci-ssl meta-package which in turn will pull also the px5g script. With this utility the init script will generate the appropriate certifcate and key files when the server is started for the first time, either by reboot or by manual restart. The /etc/config/uhttpd file contains in the end a section detailing the certificate and key files creation parameters:

Name	Type	Required	Default	Description
days	integer	no	730	Validity time of the generated certificates in days
bits	integer	no	1024	Size of the generated RSA key in bits
country	string	no	DE	ISO country code of the certificate issuer
state	string	no	Berlin	State of the certificate issuer
location	string	no	Berlin	Location/city of the certificate issuer
commonname	string	no	OpenWrt	Common name covered by the certificate

Those will be needed only once, at the next restart.

Basic Authentication (httpd.conf)

For backward compatibility reasons, uhttpd uses the old Busybox httpd config file /etc/httpd.conf to define authentication areas and the associated usernames and passwords. This configuration file is not in UCI format and usually shipped or generated by external packages like webif (X-Wrt). Authentication realms are defined in the format prefix:username:password with one entry per line followed by a newline.

prefix is the URL part covered by the realm, e.g. /cgi-bin to request basic auth for any CGI program
username specifies the username a client has to login with
password defines the secret password required to authenticate

The password can be either in plain text format, MD5 encoded or in the form $p$user where user refers to an account in /etc/shadow or /etc/passwd. A plain text password can be converted to MD5 encoding by using the -m switch of the uhttpd executable:

root@OpenWrt:~# uhttpd -m secret
$1$$ysVNzQc4CTMkp5daOdZ.3/

If the $p$- format is used, uhttpd will compare the client provided password against the one stored in the shadow or passwd database.
URL decoding

Note that this creates a empty salt!

URL decoding

Like Busybox HTTPd, the URL decoding of strings on the command line is supported through the -d switch:

root@OpenWrt:/# uhttpd -d "An%20URL%20encoded%20String%21%0a"
An URL encoded String!

Program Documentation

2021-12-25T00:00:00+01:00

So these are my ideas on how to document projects. There are three types of documentation types:

User guides
Targetted and end-users of the software and people who want a brief overview.
Man pages
Again targetted at end-users but also sysadmins. Usually to address a specific feature.
API level documentation/reference guide.
Targetted at programmers enhancing, maintaining the software.

To generate I would use different tools. In general we would like to embed with source code.

User Guides

Either as a stand alone document or embedded in the code. My default tool is to use Markdown as can be easily be converted to HTML or PDF as needed.

Man pages

Use manify. Embedded in the source code.

API reference documentation

We need generation tools. So the candidates are:

C :
- zehdok
- mp_doccer
php : Multiples
- phpdoc: The main one
- peej's phpdoctor
- ApiGen : more modern alternative.
- PHP Markdown doc generator
perl : pod
python : pydoc
tcl : tcldoc or doctools
java : javadoc
javascript : jsdoc-toolkit
Shell script: shdoc

Multi languages:

doxygen: C, Objective-C, C#, PHP, Java, Python, IDL (Corba, Microsoft, and UNO/OpenOffice flavors), Fortran, VHDL, Tcl, and to some extent D.
ROBODoc: Virtually anything.

Git Tutorials

2021-12-25T00:00:00+01:00

Reference for Git tutorials

Remote Bridging

2024-10-23T00:00:00+02:00

Sometimes we need to connect two or more geographically distributed ethernet networks to one broadcast domain. There can be two different office networks of some company which uses smb protocol partially based on broadcast network messages. Another example of such situation is computer cafes: a couple of computer cafes can provide to users more convinient environment forr playing multiplayer computer games without dedicated servers. Both sample networks in this article need to have one *nix server for bridging. Our networks can be connected by any possible hardware that provides IP connection between them.

Connecting Two Remote Local Networks With Transparent Bridging Technique

Short description

In described configuration we are connecting two remote LANs to make them appearing as one network with 192.168.1.0/24 address space (however physically, presense of bridges in network configuration is not affecting IP protocol and is fully transparent for it, so you can freely select any address space). Both of the bridging servers has two network interfaces: one (as eth0 in our example) connested to the LAN, and second (eth1) is being used as transport to connect networks. When ethernet tunnel between gateways in both networks will be bringed up we will connect tunnel interfaces with appropriate LAN interfaces with bridge interfaces. Schematically this configuration can be following:

Setting Up Bridging Servers

Notice: This article describes Debian GNU/Linux servers setup. If you are using another distribution, there can be some differences in network configuration and package management, but the main idea of described actions will be the same. First of all, we need to check if tun and bridge modules is not included in current kernel. If they are not includen, we need to rebuild kernel with CONFIG_TUN and CONFIG_BRIDGE options. Next, we need to create tunnel device file for our tunnel:

# cd /dev
# ./MAKEDEV tun
# mkdir misc
# ln -s /dev/net /dev/misc/net

Notice: Last command is needed to make vtun work, because authors build for debian is looking for tunnel device driver at /dev/misc/net/tun. To create ethernet tunnel between bridging servers we will use vtun software. When vtun will be installed, we will need to select one of the bridging servers as master and second server will be slave and appropriately change vtund-start.conf and vtund.conf file in /etc/ on buth servers. Complete config files for master is following.

/etc/vtund-start.conf
----cut-here------------------------------------
--server-- 5000
----cut-here------------------------------------

/etc/vtund.conf
----cut-here------------------------------------
options {
    port 5000;            # Listen on this port.

    # Syslog facility
    syslog        daemon;

    # Path to various programs
    ifconfig      /sbin/ifconfig;
    route         /sbin/route;
    firewall      /sbin/iptables;
    ip            /sbin/ip;
}

default {
    compress no;
    encrypt no;
    speed 0;
}

rembridge {
    passwd Pa$$Wd;
    type ether;
    proto udp;
    keepalive yes;
    compress no;
    encrypt yes;

    up {
        # Connection is Up
        ifconfig "%% up";
        program "brctl addif br0 %%";
    };

    down {
        # Connection is Down
        ifconfig "%% down";
    };
}
----cut-here------------------------------------

Slave server config files is following:

/etc/vtund-start.conf
----cut-here------------------------------------
rembridge 10.1.1.1 -p
----cut-here------------------------------------

Notice: In this example 10.1.1.1 is transport address of master server.

/etc/vtund.conf
----cut-here------------------------------------
options {
  # Path to various programs
  ifconfig      /sbin/ifconfig;
  route         /sbin/route;
  firewall      /sbin/iptables;
}

korsar {
  pass  Pa$$Wd;         # Password
  type  ether;          # Ethernet tunnel
  up {
        # Connection is Up
        ifconfig "%% up";
        program "brctl addif br0 %%"
  };
  down {
        # Connection is Down
        ifconfig "%% down";
  };
}
----cut-here------------------------------------

To bring up bridge between LAN ethernet interface and our newly created tunnel interface we need to create bridge interface. To complete this task we will add br0 interface description to /etc/network/interfaces file:

auto br0
iface br0 inet static
    address 192.168.1.199
    netmask 255.255.255.0
    bridge_ports eth0

Notice: IP-addresses on both sides of our bridge must be unique in both networks. eth0 is LAN interface. Now, we need to bring this interface up:

# ifup br0

When br0 interface will be created, we will be able to start vtun. # /etc/init.d/vtund restart If everything was done correctly, we will see following results on both sersers (br0 and tap0 interfaces):

# ifconfig tap0
tap0  Link encap:Ethernet  HWaddr 00:FF:B2:91:CA:DE
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:701818 errors:0 dropped:0 overruns:0 frame:0
      TX packets:405939 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:975889241 (930.6 MiB)  TX bytes:44704104 (42.6 MiB)

# ifconfig br0
br0   Link encap:Ethernet  HWaddr 00:02:44:2A:03:30
      inet addr:192.168.1.199  Bcast:192.168.1.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:2660 errors:0 dropped:0 overruns:0 frame:0
      TX packets:42 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:239368 (233.7 KiB)  TX bytes:2338 (2.2 KiB)

#

If we need to see current state of bridge interface, we can use brctl tool:

# brctl show br0
bridge name     bridge id               STP enabled     interfaces
br0             8000.0002442a0330       no              eth0
                                                        tap0
#

When all of described steps will be completed, our computers in both networks will be able to communicate with each other. IP addresses on bridge interfaces can be used for troubleshooting network connection. And last, if you need, you can turn on compression or enrtyption of data within created tunnel.

Network wiring notes - 8P8C / RJ45

2021-12-25T00:00:00+01:00

What you were probably looking forT568A/B (10-BASE-T and 100-BASE-TX):

With pin positions are counted from left to right with the contacts facing you (clip on the back) and pointing up (cable coming out the bottom):

Color (568B)	Pin	Color(568A)
Orange-white	1	Green-white
Orange	2	Green
Green-white	3	Orange-white
Blue	4	Blue
Blue-white	5	Blue-white
Green	6	Orange
Brown-white	7	Brown-white
Brown	8	Brown

Cut the outer insulation and order the wires (right), cut for equal length (not shown), and insert into plug (left)

Check that the wires go to the end of the plug by seeing if you can see each wire at the end, preferably see its copper reflecting (left).

Then use the crimping tool (shoves the pins into the wires, and fixes the wire in the plug.

100Mbit ethernet (Fast Ethernet, FE) uses only two pairs - pins 1, 2, 3, and 6, which in the standard wiring are the orange and green wire pairs.

This means you could use the other wires for different things - e.g. one link and a phone (non-conflicting if you use the stand pins for each), or some more custom combination such as two links though one wire.

Gigabit ethernet uses all four pairs, so has no creative options.

On crossover cables

Given the two plug colorings in 568-type wiring, cables wired with these can be:

A straight cable, 568B-568B (or the functionally equivalent 568A-568A, but in terms of colors the B-B variant seems to be used everywhere, probably to avoid confusion)
A crossover cable (also patch cable) has 568A on one end and 568B on the other. (crossing is also effectively done by a switch or hub, so you can use straight cables except in cases where you don't use switches. Crossovers can be useful for direct computer-computer connections).

Gigabit ethernet doesn't need crossovers -- it decided to handle that case inside the NIC and switches rather than have you do it the cable. You use straight cables everywhere (NIC-switch-NIC and NIC-NIC).

Gigabit crossovers are rumored to exist (crossing blue and brown in addition to orange and green), but they are unnecessary.

On Loopbacks

Loopbacks connect a port to itself. This can be used to test whether a long cable and/or its wallplug is broken, and whether a switch/router port is broken (or perhaps dirty or corroded), both just by seeing whether the link light comes on.

Connect:

Pin 1 to 3
Pin 2 to 6
Pin 4 to 7 (for a gBit loopback)
Pin 5 to 8 (for a gBit loopback)

(If you're wiring a plug as a loopback, make sure you're not confused about which end is pin is pin 1). To create a loopback from a plug-with-cable you cut (that was wired according to 568A or 568B), this means:

Orange-white  to  Green-white
Orange        to  Green
Blue          to  Brown-white  (for a gBit loopback)
Brown         to  Blue-white   (for a gBit loopback)

gBit loopback is a limited concept:

Gigabit NICs have crosstalk detection (detects how much signal interferes onto other wires), and will likely decide that the loopback is an extreme amount of crosstalk - any may not show link. Meaning it's often only useful on NICs which let you disable crosstalk detection. Gigabit switches may behave differently (but I'm not sure what the spec says or the real-world variation is)

More notes on ethernet wiring

The wiring used on 10Mbit, 100Mbit (specifically 10-BASE-T and 100-BASE-TX) ethernet over 8P8C (informally RJ45) plugs is defined by TIA/EIA-568-B, which define two plug wiring alternatives, 568A and 568B. Notice the lack of dashes; 568-B is the standard they are part of, 568-A a completely different standard (yes, that naming is stupidly confusing).

Note that both 10Mbit and 100Mbit networking use only pair 2 and 3 (orange and green) in the standard (The blue pair is pair 1, orange is pair 2, green is pair 3, and brown is pair 4.)

This means that Americans or anyone else using 4P/6P-style phone connectors can use fully wired cables (most are fully wired - relatively few (cheaper) cables are only two-pair for only Ethernet) to wire their house/company and have the same sockets be usable to plug in phones, a computer (or both with a trivial splitter). Various companies can use use this to make their wiring simpler.

Gigabit ethernet

GBit ethernet can use cables wired 568-style, preferably rated Cat5e, or better.

Specifically, you want straight wiring and four-pair cable. Most older cables are, so can be used at gBit speeds, as 1000-BASE-T uses all four pairs instead of just two.

(If you press your own plugs, it is suggested that you keep the untwisted length as small as possible, to minimize near-end crosstalk)

Some implications:

you can't do the phone/networking split mentioned above
on-the-cheap two-pair cables will work, but only because the NICs fall back to 100mbit
you can mix 10/100/1000 in your network, by replacing switches (handy for partial/gradual upgrades), without having to wire about the cabling.

...as long as the cable is rated Cat5e (or better)

On cable standards (Cat5, etc.)

Cat5
- rarely seen - it has fallen out of favour and is barely sold
- (...but your company may still be wired with it)
- regularly and informally refers to Cat5a
Cat5a
- Currently still quite common
- 100mBit, 1gBit at <100m
Cat6
- 100mBit, 1gBit at <100m
- <55m for 10gBit (less if many are bundled)
Cat6a
- 10GBit at <100m
Cat7 - stricter about crosstalk (pairs individually insulated)

Any cabling that is not shielded will crosstalk, meaning permissible distances are lower when many are bundled (relevant to company wiring), or be likely to get a lot of outside interference.

The Shiny Special Expensive cables sold in the sort of computer shops that only sell things that come in plastic boxes are generally not necessary, particularly not on the few-meter cables for your home LAN.

For example, Cat6 was made for 10gBit, most single computers don't have a source of data to actually use that speed, and even if they had, it's hard to get a 10gBit switch.

Companies may want Cat6 (or perhaps Cat6a) for future compatibility, to be able to use gBit now and assuming that nothing replaces copper before the next update.

Cat7 can go beyond 10gBit at short-ish distances, but chances are you won't be needing that any time soon. Only data centers might care.

Naming pendantics and telephony

When we say RJ45, we often mean something like "Ethernet wiring on an 8P8C plug."

RJ is the group of plugs that can be described by their positions and connectors, such as 8P8C in Ethernet, while RJ45 actually refers to a specific telephone wiring on the 8P-style plug (probably the most common one among several), while 8P8C refers to that plug itself and no specific wiring. Regardless, most people call the plug RJ45, regardless of wiring.

Plugs may have fewer actually present conductors than they have positions, so 8P2C, 8P4C, 8P6C, 8P8C, 6P2C, 6P4C, 6P6C, 4P2C, 4P4C all exist.

When there are less connectors than positions, they are in the middle positions; RJ-style wiring is from the middle out.

For most of us, the is interesting only in that you can plug a phone with 6P plugs into a 8P (ethernet-plus-phone) socket and have the phone work - the clip aligns the plug in the middle.

If more more than the middle two wires are used in telephone wiring, they carry either power, or a second (RJ14) or even third (RJ25) telephone line on the same wire, but consumers rarely see this type of phone wiring)}}

There are exceptions to the 'always start in the middle', but they tend to be intentionally working around RJ-style wiring.

The most common use of 6P outside of the US is probably phone wiring according to RJ11, which often use just a single pair in the middle. In the US, 8P connectors with the RJ45 phone wiring is common.

Native Kerberos Authentication with SSH

2021-12-25T00:00:00+01:00

This article is about integrating OpenSSH in a kerberos environment. Allthough OpenSSH can provide passwordless logins (through Public/Private keys), it is not a true SSO set-up. This article makes use of Kerberos TGT service to implement a true SSO configuration for OpenSSH.

Pre-requisites

First off, you'll need to make sure that the OpenSSH server's Kerberos configuration (in /etc/krb5.conf) is correct and works, and that the server's keytab (typically /etc/krb5.keytab) contains an entry for host/fqdn@REALM (case-sensitive). I won't go into details on how this is done again; instead, I'll refer you to any one of the recent Kerberos-related articles (like this one, this one, or even this one). Just be sure that you can issue a kinit -k host/fqdn@REALM and get back a Kerberos ticket without having specify a password. (This tells you that the keytab is working as expected.)

Configuring the SSH Server

Configure `/etc/ssh/sshd_config with the following:

 KerberosAuthentication yes
 KerberosTicketCleanup yes
 GSSAPIAuthentication yes
 GSSAPICleanupCredentials yes
 UseDNS yes
 UsePAM no

If UseDNS is set to Yes, the ssh server does a reverse host lookup to find the name of the connecting client. This is necessary when host-based authentication is used or when you want last login information to display host names rather than IP addresses. Note: Some ssh sessions stall when performing reverse name lookups because the DNS servers are unreachable. If this happens, you can skip the DNS lookups by setting UseDNS to no. If UseDNS is not explicitly set in the /etc/ssh/sshd_config file, the default value is UseDNS yes.

Configuring the SSH Client

Edit /etc/ssh/ssh_config, and change the file accordingly. For example, we want to enable Kerberos mechanism for all Hosts:

 Host *
      ....
      GSSAPIAuthentication yes
      GSSAPIDelegateCredentials yes

or to enable to specific domains:

Host *.example.com
  GSSAPIAuthentication yes
  GSSAPIDelegateCredentials yes

This limits GSSAPI authentication to only those hosts in the example.com domain. Modify the domain to be the appropriate domain for your network.

Testing the Configuration

Obtain a valid Kerberos ticket kinit username from the command line. Once you have a ticket, you should be able to simply ssh fqdn.of.server and you will get logged in, without getting prompted for a password. If you get prompted for a password, go back and double-check your keytab, your SSH daemon configuration, and the time configuration on your OpenSSH server. Because Kerberos requires time synchronization, differences of greater than 5 minutes will cause the authentication to fail.

My new 0ink.net site

2021-12-25T00:00:00+01:00

So one weekend went by and managed to finish up my 0ink.net web site. So now I have:

wordpress
For main content.
tt-rss
This is my answer to Google's shutdown of the Reader service.
Automated backups
Through my own custom scripts.
New e-mail server
sitecopy
To manage the web software updates.
CloudFlare
Reverse proxy and web accelerator.

Mirroring a Gitorious repository to GitHub

2021-12-25T00:00:00+01:00

There is nothing special with GitHub and Gitorious here. This technique would work exactly the same the other way around or with other servers.

In a nutshell

# Inital setup
git clone --mirror git://gitorious.org/weasyprint/weasyprint.git weasyprint
GIT_DIR=weasyprint git remote add github git@github.com:SimonSapin/WeasyPrint.git

# In cron
cd /path/to/project && git fetch -q && git push -q --mirror github

How it works

Mirroring with Git is pretty easy: just pull from or push to another repository. GitHub and Gitorious allow you to push to them or pull from them, but you can not make them push to somewhere else. You need something in the middle. Digging a bit in the man pages tells you that the magic option is --mirror. First, clone your "source" repository:

git clone --mirror git://gitorious.org/weasyprint/weasyprint.git weasyprint

--mirror implies --bare. This repository is not for working, you don't want it to have a working directory. More importantly, --mirror sets up the origin remote so that git fetch will directly fetch into local branches without doing any merge. It will force the update if the remote history has diverged from the local one.

git fetch

Now our local repository is an exact mirror of what we have on Gitorious. Let's push it to GitHub:

git remote add github git@github.com:SimonSapin/WeasyPrint.git
git push --mirror github

The --mirror option for git push is similar to that for git clone: instead of pushing just a branch, it says that all references (branches, tags, -) should be the same on the remote end as they are here, even if it means forced updates or removing. Now our GitHub repository also is a mirror. Let's update it every hour with cron. The -q option says to suppress normal output but keep error messages, which cron should send you by email if your server is properly configured.

42 *    * * *   cd /path/to/weasyprint && git fetch -q && git push -q --mirror github

Warning: `--mirror` is like `--force`

Both --mirror options are kind of like --force in that you can lose data if you're not careful. It will make exact mirrors, no question asked. If you push changes to the mirror's destination, they will be overwritten/removed on the next update if they are not in the mirror's source.

Original article by Simon Sapin: http://exyr.org/2011/git-mirrors/

Kerberos howtos

2021-12-25T00:00:00+01:00

Kerberos is a network authentication protocol which works on the basis of "tickets" to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner. (Source Kerberos_(protocol) )

Backups

Create backup:

kdb5_util dump _dump_file_

Restore from dump file:

kdb5_util load _dump_file_

Master/Slave replication

Initial set-up:

(master)# kdb5_util dump _dump_file_
(master)# kprop -d -f _dump_file_ _slave_

In crontab on master:

krb5_util dump _dump_file_
kprop -f _dump_file_ _slave_

kadmin command

From command line:

kadmin.local -q 'cmd'

listprincs - list principals
ank principal - new principal (input: password)
delprinc principal - delete principal (input: yes/no)
ank -randkey host/fqdn@REALM - create a service key
ktadd -k filename host/fqdn@REALM export key to keytab file.

Save keytab in /etc/krb5.keytab and

chown root:root /etc/krb5.keytab
chmod 400 /etc/krb5.keytab

Logging

To turn logging on, add this section to /etc/krb5.conf (adapt the file paths to your likings):

 [logging]
     default = FILE:/var/log/krb5.log
     kdc = FILE:/var/log/krb5kdc.log
     admin_server = FILE:/var/log/kadmind.log

Merging (or editing) a keytab file

Merging or editing keytabs is done through the ktutil command. Suppose we have two keytabs, keytab1 and keytab2, each having their own set of keys, and we would like to merge the two keytabs in one (or create a new keytab containing specific keys). The operation is done through the ktutil shell, with rkt and write_kt commands, and optionally delent if you want to delete some entities. Example:

 # ktutil

Read content of keytab1:

 ktutil: rkt keytab1
 ktutil: list
 slot KVNO Principal
 ---- ---- -------------------------------------------------------------
 1 3 <principal and key of keytab1>
 2 3 <principal and key of keytab1>

Now, we will read the content of keytab2:

 ktutil: rkt keytab2
 ktutil: list
 slot KVNO Principal
 ---- ---- -----------------------------------------------------------
 1 3 <principal and key of keytab1>
 2 3 <principal and key of keytab1>
 3 2 <principal and key of keytab2>
 4 2 <principal and key of keytab2>

Save this content in a temporary keytab:

 ktutil: write_kt /tmp/krb5.keytab

This utility is used to duplicate and tweak keytab entries (as its name implies), and remove the need of exporting the keys out of the KDC twice or more (simultaneously avoiding KVNO's increment).

OpenWRT recipes

Packages

Server

krb5-server
- krb5-libs (dependency of krb5-server)

Client

krb5-client

Configuration

Create the file /etc/krb5.conf with the following credentials. Example:

[libdefaults]
    default_realm = YOURDOMAIN.ORG
    dns_lookup_realm = false
    dns_lookup_kdc = false
    ticket_lifetime = 24h
    forwardable = yes

[realms]
    YOURDOMAIN.ORG = {
        kdc = server_address_of_this_machine:88
        admin_server = server_address_of_this_machine:749
        default_domain = yourdomain.org
    }

[domain_realm]
    .yourdomain.org = YOURDOMAIN.ORG
    yourdomain.org = YOURDOMAIN.ORG

Replace YOURDOMAIN.ORG / yourdomain.org with the domain name of your domain the server should act for (names must be specified in UPPER- / lowercase as shown above). Replace server_address_of_this_machine with the host name/IP adress of this server you're setting up.

Starting the server

Start the server by issuing

/etc/init.d/krb5kdc start

This should create the /etc/krb5kdc/ directory with the following files

-rw-------    1 root     root         8192 Feb 13 11:17 principal
-rw-------    1 root     root         8192 Feb 13 09:12 principal.kadm5
-rw-------    1 root     root            0 Feb 13 09:12 principal.kadm5.lock
-rw-------    1 root     root            0 Feb 13 11:17 principal.ok

In case you don't get any error messages check your server by logging on with kadmin.local In case everything works well you will see the following message

root@bridge:~# kadmin.local
Authenticating as principal xxxxxxx/admin@YOURDOMAIN.ORG with password.
kadmin.local:

Start on boot

To enable/disable automatic start on boot:

/etc/init.d/krb5kdc enable

this simply creates a symlink: /etc/rc.d/S60krb5kdc ? /etc/init.d/krb5kdc

/etc/init.d/krb5kdc disable

this removes the symlink again

References

Emacs Cheat Sheet

2021-12-25T00:00:00+01:00

Quick reference article for how to use Emacs. Yes, it is really old skool!

Cursor Motion

Key	Cursor Motion
C-f	Forward one character
C-b	Backward one character
C-n	Next line
C-p	Previous line
C-a	Beginning of line
C-e	End of line
C-v	Next screenful
M-v	Previous screenful
M-<	Beginning of buffer
M->	End of buffer
C-s	Search forward incrementally
C-r	Reverse search incrementally
C-u C-s	Reg-Exp Search forward incrementally
C-u C-r	Reg-Exp Reverse search incrementally
C-x C-x	Swap mark and cursor
C-Space	Set mark
C-l	Cursor to the middle of the screen
M-}	Forward one paragraph
M-{	Backward one paragraph

Text editing

Key	Editing
C-q	Literal command
C-d	Delete next character
Backspc	Delete previous character
M-%	Query string replacement
M-d	Delete next word
M-Bcksp	Delete previous word
C-k	Kill to end of line (delete to end of line)
C-w	Cut region
M-w	Copy region
C-y	Yank most recent cut/copy (paste command)
M-y	Replace yanked text with previously cut/copy text (only works immediatly after C-y or another M-y)
C-x u	undo
C-u ##	Repeat the next command

File Commands

Key	Files
C-x C-f	Open a file
C-x C-s	Save buffer to file
C-x C-w	Write buffer to file (Save As)
C-x C-c	Exit Emacs
C-x s	Save all buffers
C-x i	Insert file
C-g	Cancel current command
C-z	Suspend/Minimize Emacs

Buffers

Key	Buffers
C-x b	Switch to buffer
C-x 1	Close all other buffers
C-x 2	Split current buffer in tow
C-x 3	Split current buffer horizontally
C-x 0	Close current buffer
C-x o	Switch to other buffer
C-x C-b	List buffers
C-x k	Kill buffer
C-x ^	Grow window vertically; prefix is number of lines

Help

Key	Help
C-h C-h	Help menu
C-h i	Info
C-h a	Apropos
C-h b	Key bindings
C-h m	Mode help
C-h k	Show command documentation; prompts for keystrokes
C-h c	Show command name on message line; prompts for keystrokes
C-h f	Describe function; prompts for command or function name, shows documentation in other window
C-h i	Info browser; gives access to online documentation for emacs and more

Misc

Key	Other
M-/	Abbreviation
M-q	Autoformat current text region
C-M-	Re-indent current region
C-x (	Start defining macro
C-x )	Stop macro defintion
C-x e	Execute macro

C-Mode Commands

Key	C-Mode
C-j	Insert a newline and indent the next line.
C-c C-q	Fix indentation of current function
C-c C-a	Toggle the auto-newline-insertion mode. (If it was off, it will now be on and vice versa.)
C-c C-d	Toggle the hungry delete mode

Extended commands

Enter ESC + [ and enter this text:

M-x	Commands
c-set-style	Change the indentation style
replace-string	Global string replacement
revert-buffer	Throw out all changes and revert to the last saved version of the file.
gdb	Start GNU debugger
shell	Start shell in new buffer
print-buffer	Send the contents of the current buffer to the printer
compile	Compile a program
set-variable	Change the value of an Emacs variable to customize Emacs
artist-mode	Start artist mode
artist-mode-off	Exit artist mode
tabify	...
untabify	...

ArchLinux tips

2022-03-03T00:00:00+01:00

A bunch of recipes useful for an ArchLinux system environment.

Mostly around system administration.

Custom Repos and Packages

In the repo directory, put all the packages in there.

repo-add ./custom.db.tar.gz ./*

Add to pacman.conf:

[custom]
SigLevel = [Package|Databse]Never|Optional|Required
Server = path-to-repo

See also repo-remove. A package database is a tar file, optionally compressed. Valid extensions are .db or .files followed by an archive extension of .tar, .tar.gz, .tar.bz2, .tar.xz, or .tar.Z. The file does not need to exist, but all parent directories must exist. ?Can we create a rpmgot.php hack?

Safe automatic pacman upgrades

safepac : This is an approach for automating pacman upgrades yet catching problematic updates before hand.

Building packages

requires: @base-devel, abs, fakeroot

makepkg -s

makeworld ?

Working with the serial console

Configure your Arch Linux machine so you can connect to it via the serial console port (com port). This will enable you to administer the machine even if it has no keyboard, mouse, monitor, or network attached to it (a headless server).

Configuration

Add this to the bootloader kernel line:

console=tty0 console=ttyS0,9600

From systemd:

systemctl enable getty@ttyS0.service

Installing Arch Linux using the serial console

Boot the target machine using the Arch Linux installation CD.
When the bootloader appears, select "Boot Arch Linux ()" and press tab to edit
Append console=ttyS0 and press enter
Systemd should now detect ttyS0 and spawn a serial getty on it, allowing you to proceed as usual

Note: After setup is complete, the console settings will not be saved on the target machine; in order to avoid having to connect a keyboard and monitor, configure console access on the target machine before rebooting.

Identifying files not owned by any package

pacman-disowned

#!/bin/sh

tmp=${TMPDIR-/tmp}/pacman-disowned-$UID-$$
db=$tmp/db
fs=$tmp/fs

mkdir "$tmp"
trap 'rm -rf "$tmp"' EXIT

pacman -Qlq | sort -u > "$db"

find /bin /etc /sbin /usr 
  ! -name lost+found 
  ( -type d -printf '%p/n' -o -print ) | sort > "$fs"

comm -23 "$fs" "$db"

Pacman one liners

Remove packages and its dependancies. pacman -Rs ...
List explicitly installed packages pacman -Qeq
List orphans pacman -Qtdq
Remove everything but base group pacman -Rs $(comm -23 <(pacman -Qeq|sort) <((for i in $(pacman -Qqg base); do pactree -ul $i; done)|sort -u|cut -d ' ' -f 1))
Listing changed configuraiton files pacman -Qii | awk '/^MODIFIED/ {print $2}'
Download a package without installing it pacman -Sw package_name
Manage pacman cache

paccache -h

PHP notes

2021-12-25T00:00:00+01:00

Notes on doing different things within the PHP language.

Object oriented introspection

property_exists(obj,prop_name)
method_exists(obj,method_name)
is_a(obj,'clas_name') or ($obj instanceof ClassName)

Dynamic coding

Call a method: call_user_func(array($obj,'method',...args...)
You can simply $obj->prop = value to add properties.
or you can use __set and __get. See http://php.net/manual/en/language.oop5.overloading.php

varargs

Git recipes

2025-01-01T00:00:00+01:00

A collection of small useful recipes for using with Git.

Use the command:

git init --bare --shared /srv/git/myrepo.git

Creates a repo with the following config:

core.bare = true: make it a bare repo
core.sharedrepository = 1 (same as core.sharedrepository = group): the repo directory and all directories later created in it will be managed by git to allow group read, write, and execute permissions (with the sgid bit set as well -- so as to work with users different primary group)
receive.denyNonFastforwards = 1: deny non fast-forward pushes to the repo

Make sure that the user and group ownership of the files is correct. You may need to run chown -R or chgrp -R to correct this.

If you want to fine-tune the user, group, or other users' permissions, use --shared=0NNN, where NNN are the standard user, group, and other bits for files (the execute and sgid bits on directories will be managed appropriately by git). For example, this allows read and write access to the user, and read-only access to the group (and no access to other):

git init --bare --shared=0640 /srv/git/myrepo.git

This allows read and write access to the user and group (and no access to other):

git init --bare --shared=0660 /srv/git/myrepo.git

This allows read and write access to the user and group, and read-only access to other:

git init --bare --shared=0664 /srv/git/myrepo.git

This is the default.

Rewriting history

Rolling back the last commit

if nobody has pulled your remote repo yet, you can change your branch HEAD and force push it to said remote repo:

git reset --hard HEAD^
git push -f

Restoring changes

So in the event that you want to go back to a previous version of a file. First you must identify the version using:

git log $file

Once you know which commit to go to, do:

git checkout $hash $file

Then

git commit $file

User friendly version ids

Creating version ids Use:

 git describe

Gives:

 $tag-$commit_count-$hash

However for this to work, you need to have a good tag set and a good tag naming convention.

Branches

Main branch names:

master - The main branch. Source code of HEAD always reflects production-ready status.
develop or dev - Main dev branch. HEAD always reflects state with the latest development changes for the next release. This can sometimes be called the "integration branch" and used to generate automatic nightly builds.

Also a variety of supporting branches to aid parallel development between team members, ease tracking of features, prepare for production releases and to assist in quickly fixing live production problems. Unlike the main branches, these branches always have a limited life time, since they will be removed eventually. Creating a new branch:

 git checkout -b new_branch develop
 # Creates a branch called "new_branch" from "develop" and switches to it
  git push -u origin new_branch
  # Pushes "new_branch" to the remote repo

Listing branches

 git branch      # List all local branches
 git branch -a  # List local and remote branches

Merging branches

 git checkout dev
 # Switches to branch that will receive the commits...
 git merge --no-ff "feature_branch"
 # makes the a single commit (instead of replaying all the commits from the feature branch)

Deleting branches

git branch -d branch_name    # Only local branches
git push origin --delete branch_name # Remote branch
git push origin :branch_name # Old format for deleting... prefix with ":"

Clean-up delete branches in remote repo from local repo...

git branch --delete branch
git remote prune origin

Tagging

Creating tags

Tag releases with

git tag -a $tagname -m "$descr"

This creates an annotated tag that has full meta data content and it is favored by Git describe.

Temporary snapshots

git tag $tagname

These are lightweight tag that are associated to a specific commit.

By default are not pushed. They need to be exported with:

git push origin $tagname

git push origin --tags

To pull tags (if there aren't any)

git fetch --tags

Deleting tags

git tag -d $tagname    # Local tags
git push --delete origin $tagname # Remote tags
git push origin :refs/tags/$tagname   # Remote tags (OLD VERSION)

Rename a tag:

git tag new old
git tag -d old
git push origin :refs/tags/old

Setting up GIT

git config --global user.name "user"
git config --global user.email "email"

Other settings:

[http]
  sslVerify = false
  proxy = http://10.47.142.30:8080/
[user]
  email = alejandro_liu@hotmail.com
  name = alex

Using ~/.netrc for persistent authentication

Create a file called .netrc in your home directory. Make sure you sets permissions 600 so that it is only readable by user. With Windows, create a file _netrc in your home directory. You may need to define a %HOME% environment variable. In Windows 7 you can use:

setx HOME %USERPROFILE%

set HOME=%HOMEDRIVE%%HOMEPATH%

The contents of .netrc (or _netrc) are as follows:

|machine $system
|   login $user
|   password $pwd
|machine $system
|   login $user
|   password $pwd

Creating new repositories

mkdir ~/hello-world
cd ~/hello-world
git init
# Creates an empty repository in ~/hello-world
touch file
git add file
git commit -m 'first commit'
# Creates a new file and commits locally
git remote add origin 'https://$user:$passwd@github.com/$user/hello-world.git
# Creates a remote name for push/pull
git push origin master
# Send commits to remote

Creating a bare repo:

mkdir templ
cd templ
echo "Initial commit" &gt; README.md
git add README.md
git commit -m"Initial commit"
git clone --bare .

Vendor Branches

Set-up

unzip wordpress-2.3.zip
cd wordpress
# Note, unzip creates this directory...
git init
git add .
git commit -m 'Import wordpress 2.3'
git tag v2.3
git branch upstream
# Create the upstream branch used to track new vendor releases

When a new release comes out:

cd wordpress
git checkout upstream
rm -r \*
# Delete all files in the main directory but doesn't touch dot files (like .git)
(cd .. &amp;&amp; unzip wordpress-2.3.1.zip)
git add .
git commit -a -m 'Import wordpress 2.3.1'
git tag v2.3.1
git checkout master
git merge upstream

A variation of vendor branches is to sync with an upstream fork in github. Read this guide on how to do that: Syncing a fork on github

GIT through patches

Creating a patch:

 ... prepare a new branch to keep work separate ...
 git checkout -b mybranch
 ... do work ...
 git commit -a
 .. create the patch from branch "master"...
 git format-patch master --stdout &gt; file.patch

To apply patch..

 ... show what the patch file will do ...
 git apply --stat file.patch
 .. displays issues the patch might cause...
 git apply --check file.patch
 .. apply with am (so you can sign-off)
 git am --signoff &lt; file.patch

Maintenance

git fsck
git gc --prune=now     # Clean-up
git remote prune origin # Clean-up stale references to deleted remote objects

Submodules

Add submodules to a project:

git submodule add $repo_url $dir

Clone a project with submodules:

git clone $repo_url
cd $repo
git submodule init
git submodule update

Or in a single command (Git >1.6.5):

git clone --recursive $repo_url

For already cloned (Git >1.6.5):

git clone $repo_url
cd $repo
git submodule update --init --recursive

To keep a submodule up-to-date:

git pull
git submodule update

Remove sub-modules:

git submodule deinit $submodule
git rm $submodule # No trailing slash!

setting git email per repository

Navigate to the work repository, then at the root folder run the following command to change the email.

git config --local user.email name@work.com

Note: this command only affects the current repository. Any other repositories will still use the default email specified in ~/.gitconfig.

Alternatively, you can have different configurations based on a directory path by using:

Contents of $HOME/.gitconfig

[includeIf "gitdir:~/work/"]
    path = .gitconfig-work
[includeIf "gitdir:~/personal/"]
    path = .gitconfig-personal

Getting rid of DRM on e-books and videos

2021-12-25T00:00:00+01:00

Instructions on how to remove DRM from E-Books and videos.

How to Remove DRM from Ebooks (and Back Up Your Library Permanently)

The easiest way to strip DRM from Kindle books (and Barnes and Noble, Adobe Digital Content, etc) is with the free ebook software Calibre, DRM removal plugins, and a copy of the Kindle desktop software (PC/Mac). These directions are for Kindle, but will work with Barnes and Noble, Adobe Digital Editions, and older formats. Here's what you need to do:

Download Calibre, the the plugins, and the Kindle Desktop software.
Unzip the contents of the plugin directory.
Open up Calibre and click on "Preferences."
Navigate to "Plugins" under the "Advanced" section.
Click "Load Plugin from file," and select K3MobiDeDRM_v04.5_plugin.zip from the directory you just unzipped.
Load up the Kindle app on your Mac or Windows computer and download all your books from Amazon.
Navigate to either C:Users[your username]DocumentsMy Kindle Content on Windows or [your username]My DocumentsMy Kindle Content on Mac.
Your books aren't named in any meaningful way, so just drag all the *.azw files into Calibre.
After a short wait (depending on the size of your library), Calibre will finish importing the books. Now you have a DRM-free backup of all your books on your computer.

It's a little convoluted, but once you get the hang of it, Calibre is a solid way to backup all your purchased ebooks.

How to Remove DRM from Movies and TV Shows

You can record directly from your computer using a screen recording tool (any of these five will do). You will, of course, have to wait for the entire movie since it operates essentially like dubbing, but if you already use screen recording tools it's a free option for backing up your movies.

Wordpress links

2021-12-25T00:00:00+01:00

This article describes how you creeate hyperlinks within Wordpress. There are a number of ways to do this, depending on the configuration and the types of data we are linking to.

Linking Without Using Permalinks

This actually works whether or not Permalinks are active. Using the numeric values found in the ID column of the Posts, Categories, and Pages Administration, you can create links as follows.

Posts

To link to a Post, find the ID of the target post on the Posts administration panel, and insert it in place of the '123' in this link:

<a href="index.php?p=123">Post Title</a>

Pages

To link to a Page, find the ID of the target Page on the Pages administration panel, and insert it in place of the '42' in this link:

<a href="index.php?page_id=42">Page title</a>

Date-based Archives

Year: <a href="index.php?m=2006">2006</a>
Month: <a href="index.php?m=200601">Jan 2006</a>
Day: <a href="index.php?m=20060101">Jan 1, 2006</a>

Linking Using Permalinks

If you have enabled permalinks, you have a few additional options for providing links that readers of your site will find a bit more user-friendly than the cryptic numbers. For posts, replace each Structure Tag in your permalink structure with the data appropriate to a post to construct a URL for that post. For example, if the permalink structure is:

/index.php/archives/%year%/%monthnum%/%day%/%postname%/

Replacing the Structure Tags with appropriate values may produce a URL that looks like this:

<a href="/index.php/archives/2005/04/22/my-sample-post/">My Sample Post</a>

To obtain an accurate URL for a post it may be easier to navigate to the post within the WordPress blog and then copy the URL from one of the blog links that WordPress generates. Review the information at Using Permalinks for more details on constructing URLs for individual posts.

Pages

Pages have a hierarchy like Categories, and can have parents. If a Page is at the root level of the hierarchy, you can specify just the Page's "page slug" after the static part of your permalink structure:

<a href="/index.php/a-test-page">a test page</a>

Once again, the best way to verify that this is the correct URL is to navigate to the target Page on the blog and compare the URL to the one you want to use in the link.

Date-based Archives

Year: <a href="/index.php/archives/2006">2006</a>
Month: <a href="/index.php/archives/2006/01/">Jan 2006</a>
Day: <a href="/index.php/archives/2006/01/01/">Jan 1, 2006</a>

International Phonetic Alphabet

2021-12-25T00:00:00+01:00

Phonetic Alphabet

1	2	3	4
Alpha	Kilo	Uniform	0 - Zero
Bravo	Lima	Victor	1 - Wun
Charlie	Mike	Whiskey	2 - Two
Delta	November	X-Ray	3 - Tree
Echo	Oscar	Yankee	4 - Fower
Foxtrot	Papa	Zulu	5 - Fife
Golf	Quebec	. decimal	6 - Six
Hotel	Romeo	(point)	7 -Seven
India	Sierra	. (full)	8 - Ait
Juliet	Tango	stop	9 - Niner

Makefiles

2021-12-25T00:00:00+01:00

Some notes on GNU Make. I always have to look-up these in the manual. Here now for my own convenience.

GNU Make automatic variables:

From http://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html.

$@
The file name of the target of the rule.
$%
The target member name
$<
The name of the first prerequisite.
$?
The names of all the prerequisites that are newer.
$^
The names of all the prerequisites.

To include files in Makefile only if they exist:

ifneq ($(wildcard _incfile_),)
  include _incfile_
endif

Issue Tracker

2021-12-25T00:00:00+01:00

Use DVCS as backend (GIT)
Output html
markdown
Prefer perl/python
Mostly RO so to avoid merge conflicts.

DITZ + git integration

Adding Markdown

lib/html.rb contains the functions that generate HTML
*.rhtml contain templates and call functions in lib/html.rb to generate (and format) output.

Note, if working with github, refer to http://github.github.com/github-flavored-markdown/ Using a markdown library:

http://maruku.rubyforge.org/usage.html
Pure Ruby
http://kramdown.rubyforge.org/
Pure Ruby (fast?)
https://github.com/rtomayko/rdiscount
C library
http://ruby.morphball.net/bluefeather/index_en.html
Pure Ruby?

Arch has: redcarpet, rdiscount, maruku, ruby-markdown, github-markdown

GIT INTEGRATION

Simple hooks

~/.ditz/hooks/after_add.rb:

Ditz::HookManager.on :after_add do |project, config, issues|
  issues.each do |issue|
    `git add #{issue.pathname}`
  end
end

~/.ditz/hooks/after_delete.rb:

Ditz::HookManager.on :after_delete do |project, config, issues|
  issues.each do |issue|
    `git rm #{issue.pathname}`
  end
end

GIT Extensions:

https://github.com/ihrke/git-ditz - Adds a "ditz" subcommand to git. See README on how it installs.

DITZ PLUGINS:

git-sync

This plugin is useful for when you want synchronized, non-distributed issue
coordination with other developers, and you're using git. It allows you to
synchronize issue updates with other developers by using the 'ditz sync`
command, which does all the git work of sending and receiving issue change
for you. However, you have to set things up in a very specific way for this
to work:

Your ditz state must be on a separate branch. I recommend calling it
bugs. Create this branch, do a ditz init, and push it to the remote
repo. (This means you won't be able to mingle issue change and code
change in the same commits. If you care.)
Make a checkout of the bugs branch in a separate directory, but NOT in
your code checkout. If you're developing in a directory called "project",
I recommend making a ../project-bugs/ directory, cloning the repo there
as well, and keeping that directory checked out to the 'bugs' branch.
(There are various complicated things you can do to make that directory
share git objects with your code directory, but I wouldn't bother unless
you really care about disk space. Just make it an independent clone.)
Set that directory as your issue-dir in your .ditz-config file in your
code checkout directory. (This file should be in .gitignore, btw.)
Run 'ditz reconfigure' and fill in the local branch name, remote
branch name, and remote repo for the issue tracking branch.

Once that's set up, 'ditz sync' will change to the bugs checkout dir, bundle
up any changes you've made to issue status, push them to the remote repo,
and pull any new changes in too. All ditz commands will read from your bugs
directory, so you should be able to use ditz without caring about where
things are anymore. This complicated setup is necessary to avoid accidentally mingling code
change and issue change. With this setup, issue change is synchronized,
but how you synchronize code is still up to you. Usage:

read all the above text very carefully
add a line "- git-sync" to the .ditz-plugins file in the project
root
run 'ditz reconfigure' and answer its questions
run ditz sync with abandon

git ditz plugin

This plugin allows issues to be associated with git commits and git
branches. Git commits can be easily tagged with a ditz issue with the 'ditz
commit' command, and both 'ditz show' and the ditz HTML output will then
contain a list of associated commits for each issue. Issues can also be assigned a single git feature branch. In this case, all
commits on that branch will listed as commits for that issue. This
particular feature is fairly rudimentary, however---|it assumes the reference
point is the 'master' branch, and once the feature branch is merged back
into master, the list of commits disappears. Two configuration variables are added, which, when specified, are used to
construct HTML links for the git commit id and branch names in the generated
HTML output. Commands added:

ditz set-branch: set the git branch of an issue
ditz commit: run git-commit, and insert the issue id into the commit
message.

Usage:

add a line "- git" to the .ditz-plugins file in the project root
run ditz reconfigure, and enter the URL prefixes, if any, from
which to create commit and branch links.
use 'ditz commit' with abandon.

COLLABORATION PLUGINS

issue-claiming

This plugin allows people to claim issues. This is useful for avoiding
duplication of work---|you can check to see if someone's claimed an
issue before starting to work on it, and you can let people know what
you're working on. Commands added:

ditz claim: claim an issue for yourself or a dev specified in project.yaml
ditz unclaim: unclaim a claimed issue
ditz mine: show all issues claimed by you
ditz claimed: show all claimed issues, by developer
ditz unclaimed: show all unclaimed issues

Usage:

add a line "- issue-claiming" to the .ditz-plugins file in the project
root
(optional:) add a 'devs' key to project.yaml, e.g:

issue labeling

This plugin allows label issues. This can replace the issue component
and/or issue types (bug,feature,task), by providing a more flexible
to organize your issues. Commands added:

ditz new_label [label]: create a new label for the project
ditz label : label an issue with some labels
ditz unlabel [labels]: remove some label(s) of an issue
ditz labeled [release]: show all issues with these labels

Usage:

add a line "- issue-labeling" to the .ditz-plugins file in the project
root
use the above commands to abandon

TODO:

extend the HTML view to have per-labels listings
allow for more compact way to type them (completion, prefixes...)

issue priority

This plugin allows issues to have priorities. Priorities are numbers
P1-P5 where P1 is the highest priority and P5 is the lowest. Internally
the priorities are sorted lexicographically. Commands added:

ditz set-priority : Set the priority of an issue

Usage:

add a line "- issue-priority" to the .ditz-plugins file in the project
root
use the above commands to abandon

Cleaning up Google Calendar

2021-12-25T00:00:00+01:00

Recipe for cleaning a google calendar.

Sign in to Google Calendar
Click on Calendar Settings (current version has this just above the list of personal calendars, under an arrow).
Click on "Delete" of the main calendar.
A confirmation dialog box appears telling you that that "This deletes all events on primary Calendar".

Automatically adding systems to an AD domain

2021-12-25T00:00:00+01:00

When using virtualisation it is very common to create template VMs that can be cloned from. This makes deployment much easier than having to install a new VM from scratch. Unfortunately, the cloned VMs lack any Active Directory memberships and the VMs have to be manually added to the AD domain. For automated deployment scenarios this is less than desirable. This recipe intends to solve that issue in a Hypervisor independant manner. This recipe uses a Visual Basic script that will automatically join a system to a domain during Windows system preparation. In Lab Manager these steps can be performed on a VM Template so that virtual machines cloned from it will be joined to the domain when the system customization process runs. A specific Active Directory Organizational Unit can be specified. The Visual Basic script will contain credentials used for joining the system to the domain. So, as a security measure the Visual Basic script is setup to be deleted at the end of a successful execution.

Prerequisites

Active Directory User Account with permissions to add Computer Objects.
LDAP path syntax to Active Directory Organizational Unit to add the Computer to.

Steps on the VM Template

Create Scripts Folder

C:\Windows\Setup\Scripts

Create Batch File

C:\Windows\Setup\SetupComplete.cmd

Start /wait cscript %WINDIR%\Setup\Scripts\AddDomain.vbs
Del %WINDIR%\Setup\Scripts\AddDomain.vbs

Create VBS File

C:\Windows\Setup\Scripts\AddDomain.vbs

Const JOIN_DOMAIN             = 1
Const ACCT_CREATE             = 2
Const ACCT_DELETE             = 4
Const WIN9X_UPGRADE           = 16
Const DOMAIN_JOIN_IF_JOINED   = 32
Const JOIN_UNSECURE           = 64
Const MACHINE_PASSWORD_PASSED = 128
Const DEFERRED_SPN_SET        = 256
Const INSTALL_INVOCATION      = 262144

strDomain   = "DomainName"
strOU       = "LDAP\OU\PATH"
strUser     = "Domain\Username"
strPassword = "Password"

Set objNetwork = CreateObject("WScript.Network")
strComputer = objNetwork.ComputerName

Set objComputer = _
  GetObject("winmgmts:{impersonationLevel=Impersonate}!" & _
  strComputer & "rootcimv2:Win32_ComputerSystem.Name='" _
  & strComputer & "'")

ReturnValue = objComputer.JoinDomainOrWorkGroup(strDomain, _
   strPassword, _
   strDomain & "\" & strUser, _
   strOU, _
   JOIN_DOMAIN + ACCT_CREATE)

Tip: Start Notepad as administrator to have save access to the folder. Set the correct values for StrDomain, StrOU, StrUser and StrPassword Example:

    strDomain = "best.adinternal.com" 
    strOU = "ou=Virtuals,ou=CRE R&D,ou=Beaverton,ou=Shared Management,dc=best,dc=adinternal,dc=com" 
    strUser& = "_adjoinuser" 
    strPassword = "$uperS3curePassw()rd!{13245}"

Deploy VM

Be sure Perform customization is checked and Microsoft Sysprep is selected on the VM Template properties. Clone the VM Template

Tip: Wait around 10 minutes before trying to login to the VM. During this time the VM is going through the sysprep process which will change the hostname to the name specified when cloning the VM to a configuration and join the domain. The process should be complete when the login screen displays [Ctrl]+[Alt]+[Delete] and prompts for a domain login.

Additional notes

To improve security we could for example not hardcode login credentials in the VB script. Instead, we could retrieve them from a web server (using SSL). This server could reset the Login password for the addDomain account and send that. Once this is completed, the password could be reset again. Also, the web server could check the IP address and referencing DNS/DHCP to see if this machine is indeed being authorised. Finally, we can place this in a different AD domain (with the appropriate trust relationships) so that you can apply additional security policies.

References

This recipe was originally published at: http://www.bonusbits.com/main/HowTo:Setup_a_VM_to_Automatically_Join_to_a_Domain
Unfortunately I am no longer able to reach this site.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007491
http://msdn.microsoft.com/en-us/library/windows/desktop/aa392154%28v=vs.85%29.aspx

Other examples:

SATA/IDE warm plug/unplug

2021-12-25T00:00:00+01:00

This is for SATA and IDE interfaces that do not automatically detect added/removed devices.

Scanning for newly added discs:

echo "- - -" > /sys/class/scsi_host/host0/scan

safely removing a disk

echo 1 > /sys/block/sda/device/delete

Other notes...

In the HP MicroServer, we can identify the host to scan by:

head -1 /sys/class/scsi_host/host*/proc_name

And look for the pata_atiixp. Should show ahci and usb-storage too.

Icons

2023-04-25T00:00:00+02:00

Finding icons:

iconfinder.

In 2023 I am using:

icons8

This allows to download icons in different formats, and also allows you to do tweaks such as:

changing colors
adding sub-icons
resizing
etc...

On-line Web Authoring Resources

2021-12-25T00:00:00+01:00

A collection of links for Web-Authoring. This focuses on using Web (HTML and CSS) technologies directly and not through a CMS like Wordpress.

http://www.webestools.com/
Free online tools, generators, services, scripts, tutorials.
http://www.google.com/webmasters/tools
Google Webmaster tools, gives you a peek of how your website
looks from google search engines perspective.
Free templates
http://www.freecsstemplates.org/
http://csscreme.com/
http://www.templatemo.com/ (Unrestricted)
http://www.free-css.com/ (GPL or CC)
http://www.free-css-templates.com/ (CC)

First steps...

2024-10-22T00:00:00+02:00

So finally took the time to re-launch the 0ink web site. This time used more off-the shelf software. So this site is just a another plain wordpress powered site. Actually I have to thank my son for introducing me to wordpress.

What happened is that my son, who is only seven wanted to have his own web site. (Due to peer pressure, kids these days...)

He has an Android tablet that he uses quite often. Since I knew that wordpress can be used to make decent looking web sites and there even was an Android app. Also knew that free wordpress hosting sites can easily be found... Make a story short, I set him up with a http://wordpress.com/ account and he was live on the 'Net in a matter of minutes. His website can be found here. This first foray got me intrigued, so I tested it on another free hosting site (here) and found it quite powerful so decided to use it for 0ink.net which seriously needed to move to a new host.

The old hosting service 110mb had been taken over by a new management team and the new free hosting service was not as appealing as before. Add in a little bit of bit-rot and that site quickly became an ugly mess. So now we are back again, and hopefully will be more maintainable.

We are back...

2021-12-25T00:00:00+01:00

After a long time in Limbo we are back with 0ink.net as a live site.

Local Perl packages

2021-12-25T00:00:00+01:00

Determine what is the local PERL5LIB configuration:

LIB=$(
  for d in `tr : ' ' <<<$PERL5LIB
  do
    if [ -w $d ] ; then
      echo $d
    break
  fi
  done)
PREFIX=`dirname $LIB`

Install sequence

#PREFIX=$HOME/cpan
#LIB=$HOME/cpan/lib
tar zxvf $perl-mod-tar
cd $unpacked-src-dir
perl Makefile.PL PREFIX=$PREFIX LIB=$LIB "$@"
make
make test
make install

Linux Keyboard Tips

2022-03-03T00:00:00+01:00

Miscellaneous hacks to use the keyboard under Linux.

Special Characters on X11

The compose key, when pressed in sequence with other keys, produces a Unicode character. E.g., in most configurations pressing Compose e ``` produces é. Compose keys appeared on some computer keyboards decades ago, especially those produced by Sun Microsystems. However, it can be enabled on any keyboard with setxkbmap. For example, compose can be set to right alt by running:

setxkbmap -option compose:ralt

Compose Sequences

    |   no-break space                      &brvbar;  broken bar              ||
    &shy;  soft hyphen             --           &micro;  micro sign              /U
    &iexcl;  inverted !              !!           &iquest;  inverted ?              ??
    &cent;  cent sign            C/ or C|        &pound;  pound sign           L- or L=
    &curren;  currency sign        XO or X0        &yen;  yen sign             Y- or Y=
    &sect;  section sign         SO or S! or S0  &para;  pilcrow sign            P!
    &uml;  diaeresis            &quot;&quot; or  &quot;        &macr;  macron               _^ or -^
    &acute;  acute accent            ''           &cedil;  cedilla                 ,,
    &copy;  copyright sign       CO or C0        &reg;  registered sign         RO
    &ordf;  feminine ordinal        A_           &ordm;  masculine ordinal       O_
    &laquo;  opening angle brackets  &amp;lt;&amp;lt;           &raquo;  closing angle brakets   &amp;gt;&amp;gt;
    &deg;  degree sign             0^           &sup1;  superscript 1           1^
    &sup2;  superscript 2           2^           &sup3;  superscript 3           3^
    &plusmn;  plus or minus sign      +-           &frac14;  fraction one-quarter    14
    &frac12;  fraction one-half       12           &frac34;  fraction three-quarter  34
    &middot;  middle dot           .^ or ..        &not;  not sign                -,
    &times;  multiplication sign     xx           &divide;  division sign           :-

    &Agrave;  A grave                 A`           &agrave;  a grave                 a`
    &Aacute;  A acute                 A'           &aacute;  a acute                 a'
       A circumflex            A^           &acirc;  a circumflex            a^
    &Atilde;  A tilde                 A~           &atilde;  a tilde                 a~
    &Auml;  A diaeresis             A&quot;           &auml;  a diaeresis             a&quot;
    &Aring;  A ring                  A*           &aring;  a ring                  a*
    &AElig;  AE ligature             AE           &aelig;  ae ligature             ae

    &Ccedil;  C cedilla               C,           &ccedil;  c cedilla               c,

    &Egrave;  E grave                 E`           &amp;egrave;  e grave                 e`
    &Eacute;  E acute                 E'           &amp;eacute;  e acute                 e'
    &Ecirc;  E circumflex            E^           &ecirc;  e circumflex            e^
    &Euml;  E diaeresis             E&quot;           &amp;euml;  e diaeresis             e&quot;

    &Igrave;  I grave                 I`           &igrave;  i grave                 i`
    &Iacute;  I acute                 I'           &iacute;  i acute                 i'
    &Icirc;  I circumflex            I^           &icirc;  i circumflex            i^
    &Iuml;  I diaeresis             I&quot;           &iuml;  i diaeresis             i&quot;

    &ETH;  capital eth             D-           &eth;  small eth               d-

    &Ntilde;  N tilde                 N~           &ntilde;  n tilde                 n~

    &Ograve;  O grave                 O`           &ograve;  o grave                 o`
    &Oacute;  O acute                 O'           &oacute;  o acute                 o'
    &Ocirc;  O circumflex            O^           &ocirc;  o circumflex            o^
    &Otilde;  O tilde                 O~           &otilde;  o tilde                 o~
    &Ouml;  O diaeresis             O&quot;           &ouml;  o diaeresis             o&quot;
    &Oslash;  O slash                 O/           &oslash;  o slash                 o/

    &Ugrave;  U grave                 U`           &ugrave;  u grave                 u`
    &Uacute;  U acute                 U'           &uacute;  u acute                 u'
    &Ucirc;  U circumflex            U^           &ucirc;  u circumflex            u^
    &Uuml;  U diaeresis             U&quot;           &uuml;  u diaeresis             u&quot;

    &Yacute;  Y acute                 Y'           &yacute;  y acute                 y'

    &THORN;  capital thorn           TH           &thorn;  small thorn             th

    &szlig;  German small sharp s    ss           &yuml;  y diaeresis             y&quot;

      Euro                    e=

Environment variables

Some unfriendly applications (including many GTK apps) will override the compose key and default to their own built-in combinations. You can typically fix this by setting environment variables; for instance, you can fix the behavior for GTK with:

export GTK_IM_MODULE=xim

Bash Tips

2023-12-09T00:00:00+01:00

Some bash one-liners:

echo ${!X*}

Will print all the names of variables whos name starts with X. To output the contents of a variable so it can be parsed by
bash

declare -p VARNAME

Pattern Matching

Operator: ${foo#t*is}

Function: deletes the shortest possible match from the left

Operator: ${foo##t*is}

Function: deletes the longest possible match from the left

Operator: ${foo%t*st}

Function: deletes the shortest possible match from the right

Operator: ${foo%%t*st}

Function: deletes the longest possible match from the right MNEMONIC: The # key is on the left side of the $ key and operates from the left. The % key is on the right of the $ key and operates from the right.

Substitution

Operator: ${foo:-bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, return bar.

Operator: ${foo:=bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, set $foo to bar and return bar.

Operator: ${foo:+bar}

Function: If $foo exists and is not null, return bar. If it doesn't exist or is null, return a null.

Operator: ${foo:?"error message"}

Function: If $foo exists and isn't null, return its value. If it doesn't exist or is null, print the error message. If no error message is given, it prints parameter null or not set. In a non-interactive shell, this aborts the current script. In an interactive shell, this simply prints the error message.

$$ for Subshell

When running a sub-shell in bash the $$ construct still returns the process id of the main shell. Use the following construct to determine the correct IP address:

mypid=$(sh -c 'echo $$PPID')

Yes, it looks nasty.

Retrieving an IP address.

Updated 2023-10-20

To get the IP address, the easiest way is to use:

ip -br a

Results in:

lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp1s0           DOWN           
eno1             UP             192.168.101.64/24 fd42:bf4e:715f:6ef5:95b0:ecc9:68fa:ac07/64 fe80::82db:4389:522c:332d/64 
wlp3s0           DOWN           
virbr0           DOWN           192.168.122.1/24 
docker0          UP             172.17.0.1/16 fe80::42:95ff:fe8e:29b0/64 
br-6f79780fe7d1  DOWN           172.18.0.1/16 
veth8758d36@if8  UP             fe80::a84f:c3ff:fe8f:45b/64

Which is easy to parse. Another option is:

ip -o a

Results:

1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host proto kernel_lo \       valid_lft forever preferred_lft forever
3: eno1    inet 192.168.101.64/24 brd 192.168.101.255 scope global dynamic noprefixroute eno1\       valid_lft 26700sec preferred_lft 26700sec
3: eno1    inet6 fd42:bf4e:715f:6ef5:95b0:ecc9:68fa:ac07/64 scope global dynamic noprefixroute \       valid_lft 1534sec preferred_lft 1534sec
3: eno1    inet6 fe80::82db:4389:522c:332d/64 scope link noprefixroute \       valid_lft forever preferred_lft forever
5: virbr0    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0\       valid_lft forever preferred_lft forever
6: docker0    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\       valid_lft forever preferred_lft forever
6: docker0    inet6 fe80::42:95ff:fe8e:29b0/64 scope link proto kernel_ll \       valid_lft forever preferred_lft forever
7: br-6f79780fe7d1    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-6f79780fe7d1\       valid_lft forever preferred_lft forever
9: veth8758d36    inet6 fe80::a84f:c3ff:fe8f:45b/64 scope link proto kernel_ll \       valid_lft forever preferred_lft forever

Has more information, but it is still fairly parsable.

Identifying virtual Network Interfaces

If you need to identify which network interfaces are virtua, use:

readlink /sys/class/net/virbr0

Results in:

../../devices/virtual/net/virbr0

The output contains /virtual/ in the path. Physical NICs would have something related to the bus that the NIC is connected to.

Keep e-mail private

2021-12-25T00:00:00+01:00

This is a handy tip.

If you don't want to give out your real email address to register for a site and don't want to go through the hassle of creating a spam email address, just point your browser to Guerrilla Mail upon loading the page you'll have an automatically assigned email address that is good for one hour.

Hopefully this helps reduce spam.

Git Workflows

2024-11-22T00:00:00+01:00

Start working on a Topic Branch
Keep Topic Branch current
Merge a Topic Branch
Start working on a HotFix
Keep HotFix Branch current
Merge a HotFix Branch
Finish working on a HotFix or Topic Branch
Create a New Release
Setup New Project
Setup to work on an existing project

This article describes my personal git Workflow.

Start working on a Topic Branch

This when we are implementing a new feature. Assumes that you have a working git repo.

git checkout -b "topic" dev
git push -u origin "topic"

From a different computer, you may want to work on an existing work branch.

git fetch origin
git checkout --track origin/topic

Keep Topic Branch current

While developing a topic we may want to bring any changes done to the dev/integration test...

git checkout topic
git merge dev

Merge a Topic Branch

Once all the development and testing for a topic is done...

git checkout dev
git pull
# switch to the dev (integration) branch
git merge --no-ff topic
# The --no-ff makes this a single commit.
# ... Update any changelogs and commit them...
git push

Start working on a HotFix

This when we want to fix a prod release bug. Assumes that you have a working git repo.

git checkout -b "topic" master
git push -u origin "topic"

From a different computer, you may want to work on an existing work branch.

git fetch origin
git checkout --track origin/topic

Keep HotFix Branch current

While developing a topic we may want to bring any changes done to the dev/integration test...

git checkout topic
git merge master

Merge a HotFix Branch

Once all the development and testing for a topic is done...

git checkout master
git pull
# switch to the dev (integration) branch
git merge --no-ff topic</p>
# The --no-ff makes this a single comit.
# ... update any changelogs and commit them ...
git push
git checkout dev
# We also want to add changes to dev...
git merge --no-ff topic

.. On another system....

git remote prune origin
git branch --delete topic

Finish working on a HotFix or Topic Branch

If really done, or if you want to abort this...

git branch -d topic
git push origin dev|master
# Use dev or master depending on being a topic branch or a hot
# fix branch respectively
git push origin :topic
# Delete the remote branch... Or ...
git push origin --delete topic

Create a New Release

We are ready for a new release...

git checkout dev
git push
git pull
# Make sure that dev is up-to-date in both directions...
git checkout master
git push ; git pull
# Make sure that master is up-to-date
git merge --no-ff dev
# ... fix version number ...
git commit -a -m"preparing release X.Y"
git tag -a X.Yrel -m"Release X.Y"
git push
git checkout dev
git merge --no-ff master
# ... bump version number ...
git commit -a -m"Bump version to X.Y+1"
git tag -a X.Y+1pre -m"New dev cycle for X.Y+1"
git push origin dev
git push origin --tags

Setup New Project

For setting up a new project.

mkdir project
cd project
# ... create files ...
git init
git add .
git commit -m"Initial commit"
git tag -a "0.0initial" -m "Initial commit"
git checkout -b "dev" "master"
git tag -a "0.0pre" -m "Development branch"
git push origin --tags

This sets up a local repo with two branches and some descriptive tags. The "master" branch for release code and the "dev" branch for development and integration. We now need to configure it on the remote repository.

git checkout master
git remote add origin "Remote repo URL"
git push origin master
git checkout dev
git push -u origin dev
git push origin --tags

Setup to work on an existing project

Setup clone:

git clone "Remote repo URL"
git push origin master

Nodes:
Single component availability (%):
Total Availability (%):

0ink.net

Ansible Custom Action Plugins

Introduction

Motivation

Where to place Action Plugins

Role or Collection scope

User or System Scoped

Configuring Plugin Paths

Basic layout

Error handling

Diagnostics and Verbose output

Run-time environment

Execution Context Flags

Path Resolution

Remote File Cleanup

Accessing task parameters

More Task Context via task_vars

Implementing check mode

Implementing diff mode

Template Rendering

Calling Modules from Action Plugins

Example: Delegating to a Custom Module

Key Concepts

Return Value

Best Practices

Return values

Common Keys

Custom Return Values

Conclusion

Ansible Custom Modules

Intro

What is a module

Where to store modules

Inside a Role

Inside a Collection

Custom Path via ansible.cfg or Environment Variable

Writing modules

Recommended: Python

Possible but Uncommon: Other Languages

Using Shell script

Sample Shell module

JSON input

Additional parameters

check_mode

ansible_diff

Generating JSON output

Final thoughts

Using sphinx for doc generation

Introduction

Basic functionality

Sample Usage

docs/Makefile and docs/make.bat

docs/Makefile

docs/make.bat

docs/conf.py

autodoc2

myst_parser

sphinx.ext.doctest

docs/index.md

docs/cli.md

Docstrings

Generating documentation

Publish to gh-pages

Conclusion

javascript snippets

Compute how an element is hidden

Iterate over LI elements

Adding/removing classes to/from elements

Get current date as YYYY-MM-DD

escape HTML characters

Copy to Clipboard

Additional OpenSSL tips

Self-signed certificates

Display cert extensions

Viewing certificate information

Checking server certificate

How to create a certificate chain ?

Converting a .pem certificate to pkcs12

New Apple requirements since 2023

Create a self-signed CA

More Task Context via `task_vars`

Custom Path via `ansible.cfg` or Environment Variable