MKVToolNix v74 released

Hey y’all!

MKVToolNix v74 is out. This is another one in a longer line of small-ish bug fix releases. Nothing too earth shattering. Anyway, have fun with it.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are still being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: SRT handling: added a hack called --engage keep_whitespaces_in_text_subtitles which disables stripping whitespaces from the start & end of each line of SRT entries during muxing. Part of the implementation of #3470.

Bug fixes

  • all: Windows: UNC paths of type \\?\C:\… are supported again by switching back to Boost’s file system library instead of using C++17’s file system library. Fixes #3058.
  • mkvmerge: AVC/H.264 & HEVC/H.265 packetizers: when appending tracks the pixel dimensions will be checked & muxing will be aborted if they don’t match. Fixes #3480.
  • mkvmerge: file type detection: file types that can be detected unambiguously by their content (e.g. Matroska, MP4, WAV…) will now preferred in the detection order over file types based on their extension. Prevents certain cases of mis-detection, e.g. DTS in WAV but with a file name extension of .dts being detected as the wrong type of DTS. Fixes #3462.
  • mkvmerge: SRT handling: whitespaces will now be stripped from the start & the end of each line of each entry, not just from the end of the last line. Part of the fix of #3470.
  • mkvextract: SRT extraction: whitespaces will not be stripped from the start & end of the whole entry anymore. Part of the fix of #3470.
  • MKVToolNix GUI: multiplexer: when adding multiple files that include certain file types (chapters, segment info and tag files) and when the choice where to add them is “all files to a single new tab”, these certain file types will now be added in the newly added tab as well instead of the current tab that had already been open. Fixes #3469.
  • build system: fixed compatibility with Ruby 3.2.0 by using FileTest.exist? instead of File.exists?, Dir.exists? & FileTest.exists? which were removed in that release.

Build system changes

  • MKVToolNix is now using & requiring Boost’s “file system” and “system” libraries again instead of C++17’s file system library. This is due to gcc’s library not supporting UNC paths of types \\?\C:\dir\file.ext.

Have fun 😁

MKVToolNix v73.0 released

Hey y’all!

Even though not much has happened with MKVToolNix over the last six or seven weeks, I’ve decided to finalize the first release of the year, v73. Nothing overly amazing in it.

One note for macOS users that isn’t stated in the NEWS below: this release’s DMG was built with FLAC support again. Sorry for the confusing in the previous version; that was due to build system setup SNAFUs on my end.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: greatly enhanced the speed of Base64 decoding which is used used e.g. when reading pictures embedded in Ogg or FLAC files or for binary data inside XML chapter & tag files. Even small blocks of data could take unreasonably long.
  • mkvmerge: added support for reading HEVC/H.265 video from FLV containers. Implements #3466.

Bug fixes

  • mkvmerge: splitting: fixed a typo in the string output in GUI mode regarding the timestamps before which splitting was done. See #3421.
  • MKVToolNix GUI: job output: after having run multiple jobs the “Clear output” functionality only reset the visible parts. However, using the “save output” function would also save the output generated before the user used “clear output”. Now only the output after the last use of “clear output” is saved. Fixes #3438.
  • build system: when probing for the FLAC library the Ogg libraries will be linked with the compiled test program as well. This fixes detection if the pkg-config settings for FLAC don’t list Ogg & the linker doesn’t find the library automatically (e.g. for my macOS builds). Fixes #3439.

Have fun 😁

Running Buildbot on IPv6

As I’ve struggled a bit here I thought I’d make this short write-up for others to find. The following writeup was done against Buildbot 3.6.1 on Ubuntu 22.04 & Arch Linux.

There are two well-known components to Buildbot: one master process (purposefully ignoring multi-master setup) & one or more workers. The master is the one providing the web interface, scheduling the work, keeping track of results & sending notifications. The workers do the work (well, duh!).

This means that we’re basically dealing with two different scenarios here:

  1. The master process must listen on IPv6 addresses for connections, both for the web interface & the workers which connect on a dedicated port.
  2. The workers must connect via IPv6 to the master.

Using the default configuration or the examples from the website neither of the two components use IPv6, for different reasons.

Both use the twistd Python framework for connection handling.

Making the master listen on IPv6

The configuration for the master is done solely via master.cfg, not in buildbot.tac.

Basically for every connection the master listens on a ConnectionString is used which contains both the port number & the address to listen on, e.g. "tcp:8010:interface=127.0.0.1" One might attempt to simply replace the IPv4 address 127.0.0.1 with an IPv6 address, but that doesn’t work as colons are used to separate key/value pairs within the ConnectionString. Therefore the colons must be escaped. Note that the syntax often used in other programs such as web browser to escape IPv6 addresses, placing them in square brackets such as [::], doesn’t work here either.

How are those colons escaped, though? With backslashes. However, these are Python strings, and Python uses backslashes for escaping in strings as well. Therefore in the source code we have to use two backslashes per colon. The resulting connection string might look like this: "tcp:8010:interface=\\:\\:" (which means “listen on all IPv6 & IPv4 addresses, regardless of the interface, on port 8010”).

Web interface handler

For the web interface listener, which is usually referred to as c["www"] (with c being the an instance of BuildmasterConfig), a dict is used. The port key can be a port number, or a full-blown connection string. Example:

c['www'] = {
    'port': "tcp:8010:interface=\\:\\:",
    'plugins': { 'waterfall_view': {}, 'console_view': {}, 'grid_view': {} },
    'auth': util.UserPasswordAuth({'user': 'supersecret'}),
    'authz': authz,
    'change_hook_auth': [strcred.makeChecker("file:changehook.passwd")],
    'change_hook_dialects': { 'gitlab': { 'secret': 'reallysecret'  } }
}

As hinted in the example the web interface handles both requests from us humans as well as change hooks from version control systems.

Worker handler

The worker handler is configured via c["protocols"]["pb"], and again a dict with a key called port is used which can be set to a connection string. For example:

c['protocols'] = {'pb': {'port': "tcp:interface=\\:\\::port=9989"}}

Making the workers connect via IPv6

The configuration is done in buildbot.tac.

The worker creates an instance of the Worker class. The first two parameters are the host address & the port number of the worker handler on the master’s side. Unlike the master, the parameters cannot seem to be ConnectionStrings.

Furthermore, the twistd client framework used here cannot resolve DNS AAAA records at all. If you provide a host name instead of an IPv6 or IPv4 address as the first argument, the library will only try to resolve it to an A record. For AAAA-only DNS records this means that name resolution fails, and the worker will not start. For DNS entries with both AAAA and A records, only the A record will be used, and the connection will be attempted via IPv4.

Luckily you can use an IPv6 address directly as the first argument. As there are no ConnectionStrings involved, colons can be left as-is. For example (all following arguments left out as they aren’t important):

s = Worker('2001:db8:1234::2', 9989, …)

Making the master connect via IPv6

We have to go back to the master once more as it can also make certain outbound connections that you might want to run via IPv6. These include but aren’t limited to:

  • email notifications
  • version control system status updates (Gitlab, Github etc.)

Email notifications

The email notifier also uses a pair of parameters specifying the SMTP server’s hostname & its port. Unfortunately it uses the same twistd client components that the worker uses, too, with the same DNS resolution issues. Sure, you can specify an IPv6 address directly instead of a host name, but that’ll be problematic if you want to enable TLS as certificate validation will now fail. I don’t have a good solution for IPv6-only setups in this case.

An example without authentication & without TLS:

mailer = reporters.MailNotifier(
    fromaddr="buildbot-master@example.com",
    sendToInterestedUsers=False,
    extraRecipients=["buildbot-failure-recipient@example.com"],
    relayhost='2001:db8:fefe::9876',
    generators=[generator_failing])
c["services"].append(mailer)

Version control status handlers

Handlers that signal the build status to version control systems such as Gitlab & Github don’t use the twistd framework for communication, it seems. They probably use something like urllib2. For their configuration they actually expect a base URL such as https://gitlab.com, and luckily for us, if that address resolves to an AAAA record then the status is indeed sent via IPv6. Nothing for us to do here.

Other considerations

Last but not least: the build instructions you create must also take IPv6 connectivity into account. If you configure git repositories on IPv4-only websites such as Github (shame!), you won’t be able to use them in an IPv6-only environment. Dual-stack setups will be fine, though.

On the other hand: external tools such as git that do not use the same Python libraries will resolve DNS entries to AAAA & A records & prefer AAAA = IPv6 over A = IPv4 — meaning configuring a git repo on sites such as Gitlab will indeed be retrieved via IPv6.

MKVToolNix v72.0 released

Heyo!

here’s MKVToolNix release v72. It fixes quite a lot of bugs, several one of them in the component driving the in-place file modification in mkvpropedit & the GUI’s chapter & header editors. Therefore I urge everyone to update.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: AV1 parser: the variable-width OBU size field will be re-written with minimal length if it’s encoded longer than necessary.
  • mkvmerge: when splitting is active the program will output the timestamps actually used for making the decision when to split. If GUI mode is active, a specially formatted line #GUI#splitting_before_timestamp <timestamp> is output as well. Lines prefixed with#GUI# are suitable for machine parsing, won’t be translated and are guaranteed not to change in format. Implements #3421.
  • MKVToolNix GUI: multiplexer: when dragging & dropping directories to the “attachments” tab, the files contained in those directories will be attached. Implements #3410.
  • MKVToolNix GUI: info tool: added information about the file (directory, size, modification timestamp) at the top of each tab. Implements #3407.

Bug fixes

  • mkvmerge: AV1 parser: fixed the parser completely aborting when parsing the OBU size field fails due to there not being enough data to parse. Instead the parser will remember the last known-good position & restart from there after more data is available. Fixes #3431.
  • mkvmerge: HDMV PGS subtitles: reverted the change that implemented a heuristic for detecting bogus timestamps & attempting to fix them. This was done to fix #3268. Unfortunately this affected valid subtitle files with intentional huge gaps in timestamps, e.g. forced subtitle tracks. The heuristic has simply been removed, fixing #3392.
  • mkvmerge: Matroska reader: fixed reading files with EBML Void elements before the Matroska Segment element.
  • mkvmerge: fixed reversed attachment selection: --attachments !4 would not copy any attachment instead of all attachments but the one with ID 4. Fixes #3427.
  • mkvextract: IETF BCP 47/RFC 5646 language tags: mkvextract will now use & prefer IETF BCP 47 track language elements if they’re present. Only affects the VobSub & USF subtitle extraction.
  • mkvpropedit, MKVToolNix GUI’s chapter & header editors: updated the list of deprecated Matroska elements. The applications will no longer try to write those elements, even if they’re found in the file to be modified. The programs will no longer abort with error messages such as assertion "false" failed. Fixes #3416.
  • mkvpropedit, MKVToolNix GUI’s chapter & header editors: when the Matroska version numbers stored in the EBML Head element are updated, the updated EBML Head element might be smaller than the existing one. In that case the programs used to shrink the EBML Head & write a small EBML Void element between the updated EBML Head & the following element, usually a Matroska Segment element. This isn’t widely supported by programs including MKVToolNix itself, causing them to declare such files as invalid. The programs will now create the EBML Void element inside the EBML Head element, making them a level 1 element instead of a level 0 element. Fixes #3355.
  • mkvpropedit, MKVToolNix GUI’s chapter & header editors: often the programs have to relocate the Master elements in which the modifications were done. In that case the Seek Head elements must also be updated to reflect to the Master elements’ new positions. If a file contained a Seek Head element at the start already and if that Seek Head was too small to contain the updated positions, the programs would end up in an endless loop trying to write data to the end, creating ever-growing files. This is now handled properly by voiding this too-small Seek Head & finding a proper space for a new one instead. Fixes #3338.
  • MKVToolNix GUI: header editor: fixed pixelated icons on higher display scaling values. Fixes #3420.

Have fun 😁