Converting Ultimate++ framework to Meson build system to make it externally consumable by others

Hello everyone,

For a long time I have been taking a look into Ultimate++ framework.

Ultimate++ is a framework that includes a GUI and its own IDE, called TheIde and allows you to create full multiplatform applications for the major operating systems (Linux, Windows, MacOS). It is, for me, one of the most underrated frameworks popularity-wise.

TheIdE includes a lot of things to be productive with it, from Code Assistance to Help, debugger and a GUI RAD tool.

It can make apps as sophisticated as the one shown below, has database connection APIs and includes OpenGL embedding and all kind of fancy stuff available.

What makes Ultimate++ stand out, in my opinion, is that it uses C++ in a way that makes the coding style very efficient: the GUI RAD tool emits regular C++ that can be embedded, widgets are just normal objects with their normal scopes and it makes for a very intuitive user experience. SQL and other image embedding uses exactly the same strategy. You end up with everything in your code embedded directly from the RAD tools.

TheIde RAD tool in Ultimate++

You can get an idea of the level of sophistication that this framework allows by looking at its applications page here.

What is the problem with Ultimate++? IMHO…

I was eager to use Ultimate++ for some time, I stumbled all the time with a problem: you have to use their IDE and all things get packed their own way. This creates a lot of friction with my workflow, and I am guessing it could be the same for other people.

All the work you do falls into their own environment. I do not have anything against their environment, but the package generates a custom package system of their own that makes it its own realm of how to work.

This does not play nice with the rest of build systems nowadays, such as CMake/Meson/PreMake and other build systems in use such as MsBuild, XCode, etc.

Fixing the libraries consumption problem (and other free stuff on the way)

I wondered how I could solve the problem of package consumption. I could have added some .pc files to it and be done with it. But I do not think that is enough.

Since I am quite familiar with Meson (and CMake, actually), I gave a try at how much it would take to convert Ultimate++ to use Meson.

As I convert it, I could get some nice extras:

• Compile in many modes: debug, release, ASAN, static/dynamic linking etc.
• Build system unification among Windows, Linux and Mac
• Cross-compile the environment from other OSes
• Emit pkg-config and .cmake files for its libraries to be consumed
• Make collaboration easier from outside of the project
• Some of the embedded custom dependencies can be extracted to use Meson wraps/Conan dependencies or system dependencies transparently. Namely, correct external dependency use dictated by the user, not by the build system author

Converting Ultimate++ to Meson

I needed a strategy to start. I decided to convert on top of the stable 2022.2 tag.

I took a look at the single, monolithic Makefile that is there to compile all the needed stuff. This Makefile seems to compile enough of it to build TheIDE. The first thing I did is to take out my Linux laptop and start from there.

I compiled with the original build system to be able figure out the outputs and inspect the generated libraries. I did some grepping to extract the source files and I started a top-level Meson file for the project.

Since I want that the Meson conversion can also compile in Windows and Mac, dependency handling was critical. I decided I would extract the internally-packaged dependencies to point them to the build system and that the user of the build system chooses what to use transparently, whether those are Ubuntu repositories, Meson wraps or Conan downloaded packages. This is important for being able to compile things in Windows down the road when I make the conversion and also improves modularity. It also makes possible to point to the same dependency. For example, if some package uses zlib and another entirely different dependency uses zlib, you can coordinate it and make them point to the same version.

The coversion to Meson is still work in progress but it is already working in Linux under a static library build setup. It is compiling a good deal of libraries + TheIDE successfully and it runs correctly under Ubuntu Linux 19.04.

What took to convert it to Meson

Basically, the conversion was relatively straightforward and I am quite impressed at how well-thought the dependency handling is done and how well it combines with the command-line. You can use system dependencies, avoid to download them, fallback to subprojects dependencies even if the system ones are installed, overriding them one by one.

Except for some struggles with the way some dependencies were included in Ultimate++ framework, it was quite ok.

What I did:

• compile statically TheIde and have all dependencies (libraries) exposable via pkg-config. Need still to add the call to pkg config generate from Meson, but that is one-liners, same as adding .cmake packages for consumption
• I extracted and used package management dependencies that were originally embedded in the source code for: liblzma, lzma sdk, zlib, xxhash, libpng, zstd and pcre. These ones are now detected by Meson logic in the build system and can be overriden as per the user preferences

The project declaration and handling of configuration file generation can be seen below:

This does a few things. First, it detects the date, hostname and whoami programs. After that, it runs some commands to gather the information. It creates a data configuration object and it emits the corresponding configuration file. Note that you do not need an input file necessarily (but you can do it and replace variables in your own .in file). A file will be emitted from the configuration directly. When I go to windows, I will figure out what programs to use for the same task, and if it gets bad, I can always fallback to use a python script as multi-platform program.

Libraries compilation

Basically all libraries were compiled following the following pattern and traversing the directory structure with subdir calls:

Some of the plugins had some external dependencies. And here Meson can help a lot. Look for example at this:

The zlib, liblzma and lzmasdk were embedded dependencies at first, with custom source code inside the repo compiled. The includes pointed to the internal directories.

Now they are just using the Meson logic to detect dependencies. This means they could come from the system, from another package manager such as Conan or from a Wrap and be compiled along with your code. This is extremely flexible and makes your life, especially in Windows, much better.

How does Meson work and what did I have to do for dependency handling?

Meson wraps to the rescue

Wraps are the package management system that Meson uses. The idea is that you can put yoursubproject.wrap files in your subprojects directory and Meson will take care of the rest. There are some wraps readily available for you here. The condition for a wrap to be usable is that the package supports the Meson build system. Since most packages do not, there are ways to provide the corresponding patches to add Meson support and point to those patch files from the wrap files, which will be applied after downloading/unpackaging your project source. All packages in subprojects will be compiled from source along with your project in the case of wraps.

In Meson you should flatten all your dependencies so that they are visible, since having dependencies nested with different versions in different subprojects can cause all kinds of trouble. In case you have some dependency for two subprojects, they should all point to the same version (dependency(‘somedep’) will do automatically point to the same dependency for all the process) . This avoids disgusting errors such as accidental ODR violation.

The files can come from archives, from git, svn, hg or can point to a local directory. There is support for patching files. This is handy because most projects do not provide a Meson build system. You can apply an overly or patches on top of them. The patch_* files are actually overlays. For real patch files use diff_files directive in your .wrap files.

The wraps will download if necessary, extract, apply patches/overlays and compile dependencies, though there are switches to avoid downloading and others.

You can for example force Meson to use selected subproject dependencies instead of the system ones by using --force-fallback=libpng,zstd if you already have those dependencies installed in your system.

The system is very customizable.

For liblzma, there is a wrap available:

The idea is that you put .wrap files in your subprojects directory. After that, those will be used by default if system dependencies are not detected.

The liblzmawrap file basically says where to download the original archive, the overlay files (patch_url) and the dependency it provides. This .wrap file provides the dependency('liblzma')if liblzma is not found in the system and proceeds to compile it along with your project. That is indicated by the [provide] section. The [provide] section in the wrap above lists the internal name for a dependency variable in the meson.build file and to what name it maps when you call dependency. You can also provide program_names if you have some executable you want to detect. Note that this is just a transitional mechanism and that the newer way to do it is by calling meson.override_dependency('depname', lib_dep) from inside your meson.build file.

You can also set some subproject options separately if needed with the syntax -Dmyproject:option=value or in Machine files if things get more complex.

For the lzmasdk I was not so lucky, since there was not one wrap file available: I downloaded the files from their website, extract them in a directory, added a couple of meson.buildinto the directory pointed by the wrap files and added the corresponding wrap file to my subprojects to mimic what already existed embedded in plugin/lib/ directory, which was holding the lzmasdk embedded directly in the build system.

Note the difference now in the [provide] section: I am using the name of the dependencies without naming any internal meson.build variable. This is because in the internal build system it was used meson.override_dependency('liblzmasdk', liblzmasdk_dep) and Meson will find it. You can do the same for program_names

These were basically the steps I followed for each directory to convert Ultimate++ to Meson build system.

There were some other caveats when compiling. For example, Meson includes the source directory and build directory for a target by default. This provoked some #include clashes, since there were includes in Ultimate++ that were named exactly as their libraries ,`#include <png.h>` for example, which pointed to their own internal png in the source directory where the compilation was happening. The solution is to use for the targets that clash implicit_include_directories: false removing the accidental inclusion.

Note that Meson can do way more than this, such as cross-compiling and it is very clear about when you are compiling a program to be used by the host or the build machines through the native:bool flag that many commands support. You can add flags only to one environment without getting confused where it will show up, the syntax is almost Python and very intuitive.

It also has things such as feature objects and disablers that reduce the conditionals boilerplate by a lot. For example you can shorcircuit building all dependencies that depend on a non-detected dependency by using a disabler without changing any other part of the build code.

Next steps

The next steps are:

• make Meson build system work in Windows
• make Meson build system work in Linux
• Add .pc and .cmake files for consumption from Meson directly
• create a Conan recipe that compiles from Meson build system so that way more generators can be used

I will also separate clearly public from private dependencies. Still figuring out if they are correctly arranged, for sure there are some errors there.

Final words

It took me some 8–9 hours to do the full conversion until being able to run TheIde and figure out all dependencies, etc.

The repository with the in-progress work where Linux already compiles can be found here. Please, note that this is experimental and it could fail in your machine, since it is not thoroughly tested.

Sorry for any typos/bad writing, I wrote this in a rush to just be able to expose it in case someone finds it fun/useful.

Thanks for reading!

NOTE: Please note that I do not have any affiliation with Ultimate++ project beyond an interest in contributing the improvements to the build system, all things shown here are out of my own interest in it and nothing else.