Following up on Building O3DE for macOS

This is a follow up to my original post about building O3DE on macOS. Ideally, you should read that first, to understand where things left off, and get an idea of how dumb I am before reading this. Tl;dr: I got it to build, but couldn't open anything in the editor, and to answer the second part, very dumb.

Further Troubleshooting Steps

At some point after I hit publish, it occurred to me that there were other things I could try, in an effort to salvage something from the work I'd already put into building this engine.

Registering the Engine

At the end of the build process, I failed to register the engine. I don't remember seeing this step mentioned in the GitHub thread about building for macOS, but the people posting on there probably assumed anyone attempting this would have some familiarity with the O3DE build process.

As far as I can tell, this step essentially adds a line to a JSON configuration file, which tells O3DE where to find the engine. Failing to do this didn't exactly prevent the Project Manager from attempting to launch the Editor. Remembering to do it didn't seem to help the Editor get any further in its startup process. Either way, the documentation calls it out, and I did register my build prior to attempting any other troubleshooting, so it may be important.

Actually Building my Project

In my first post, I ran into a glitch because O3DE couldn't see my CMake installation. Maybe this has something to do with the fact that O3DE is running in Rosetta 2 while CMake is running natively on ARM. Maybe O3DE just doesn't understand how to find external tools like that on a Mac. I don't have the time, patience, or resources to test these theories, so you should probably take that as the incoherent ramblings of someone who doesn't know what they're talking about.

But in the heat of the moment, it didn't occur to me that I could just do the exact same thing I'd been doing all along, which is simply to build the project using CMake in the terminal. So I attempted to run CMake from the project folder, using arguments I pulled out of my butt:

$ cmake -B ./build/mac_xcode -S . -G Xcode
$ cmake --build build/mac_xcode --target Editor --config profile

Was this the right set of arguments for what I was trying to do? Should I try looking at the documentation instead of trying to wing it? It definitely chugged away for a while before I got that answer:

Just for funsies, I tried opening the project in the Editor anyway. Nothing interesting happened.

Going Minimal

At this point, I decided to go the MVP route. O3DE has a 'Minimal' project template which includes only the Atom renderer, ImGui, and the Camera Framework. I figured this would give me the least amount of surface area for something to go wrong, and it would take less time to figure out if the build failed. I also decided to glance at the documentation for building projects from the command line on Linux. It looks like they fixed the 404 error messages I got last time I browsed the documentation.

The instructions at least confirmed that I mostly had the right idea earlier. What I actually wanted to do was build the <ProjectName>.GameLauncher and Editor targets. That command looks like:

$ cmake -B build/mac_xcode -S . -G Xcode
$ cmake --build build/mac_xcode --target Minimal.GameLauncher Editor --config profile

Spoiler alert: that failed, too. Both builds seemed to get stuck when attempting to build the Asset Processor. It doesn't seem to give any indication of what went wrong, but there's probably a log file somewhere that I'm too dumb to find and way too dumb to be able to read.

The documentation also clarifies that users who are building their projects this way should use the Editor app that gets generated in the project's build directory, instead of the prebuilt one. At this point, I feel like I took a wrong turn in the O3DE rabbit hole, and ended up in the House of Leaves.

The Philosophy of Time Travel

Last time, I did point out an unexplored path – commit 4ac8adf. Allegedly, this is the most recent commit that produces a functioning macOS build, depending on how one defines 'functioning'. I acknowledged that this might work, but felt like it would be a long download, and a lot of mucking around in CMake just to produce an engine that supposedly crashes a lot, and doesn't import assets properly.

But if this is the most recent commit that works, I figured I could go back to the most recent release build at the time when 4ac8adf was committed, which is O3DE 22.10. At least that build is stable on the supported platforms, so it shouldn't crash as much, assuming I can get it running. Of course, you know what happens when you assume. The build kept failing. I decided to fetch 4ac8adf. Instead, I got distracted.

Commit b39157b

I'm not sure why I didn't notice this before, but the O3DE maintainers pushed commit b39157b to the Development branch of the engine about a week before I first tried to build the engine. It caught my eye because the little commit message says:

Fixed Archive Gem build on MacOS (o3de#16161)

That commit literally touched the first item on the page, so I guess that's payback for all the times I made fun this one specific coworker I have, who is terrifyingly oblivious. At the time, I felt more comfortable trying to build a release version of the engine, but this seemed promising. Someone else was specifically working on getting this to work on macOS, who actually knows what they're doing, unlike me.

Unsurprisingly, the build failed. But for whatever reason, I decided to dig a little deeper. I had to scroll past a wall of useless information before I could get to the actual error, but I managed to find it.

As it turns out, O3DE uses a convention that will treat compiler warnings as if they were actual errors, and refuse to do the build unless you fix it. Generally speaking, this is a good practice, as it forces developers to address the kind of minor issues that normally won't prevent a project from building, but which are incorrect in one way or another. Specifically speaking about what I'm working on right now, it's mostly just annoying, since it's catching things that I didn't write, and don't understand. I turned that off, and the build succeeded.

But this was just the beginning, since I still needed to create a minimal project, and build it in the terminal, because O3DE can't find the CMake installation that literally birthed it onto my machine. This took about the same amount of time as the full engine build, because I'm pretty sure it was a full engine build. The build directory for my project seems to have a full copy of the Editor, Asset Processor, and other O3DE applications. Was I not supposed to build the engine by itself, first? Do I need to register this new instance? The documentation is unclear.

So Close, Yet So Far

Once the build process finally succeeded, I launched the editor. At first, it didn't look like it actually did anything at all, but after I gave up and started mucking around with something else, I got a little popup asking me to grant the Editor access to my Documents folder. I clicked Hell, yeah!

Unlike my previous attempt, it wasn't long before the Asset Processor popped up. I've only used O3DE on Windows, so I wasn't expecting the whole window to show up, but the editor seemed to connect to it, and it reported that there were a bunch of assets that it needed to process. The number went up, and then it gradually went down until it hit about 333, and the splash screen disappeared. The editor crashed. The Asset Processor did not. I tried reopening it, but it kept getting to 333, and crashing.

I let the Asset Processor keep running for a while, and it did eventually get down to zero on its own. After that, I tried running the Editor again, and it continued to crash. When that happens, macOS seems to catch it, and it provides a popup with crash details. I saw a lot of references to different Qt libraries, so maybe it choked trying to draw the Editor UI, but I'm not qualified to make that diagnosis.

Minimal.GameLauncher

I'd already written an initial version of this post, and published it before it occurred to me that I had built the Game Launcher target for the project, as well. It shouldn't be using anything Qt related, so I wondered if I could get any further by trying to run it directly. After a bit of a delay, I got this screen:

This didn't feel like a huge victory at first, since the screen was almost completely blank. It was also rendering at 40 fps at 1080p resolution (weird) with 2x MSAA. It also crashed, eventually. But then I remembered that this was just a blank project, and I never had a chance to add a level to it, much less configure what level it was supposed to display on launch. I also started to wonder what performance would look like if I did an iOS build of the project, as that should run natively on Apple Silicon, using the Low End rendering pipeline (for a less brutal FPS average).

In other words, on some level, O3DE actually does run on macOS. I still need to figure out how to make that useful to anything I'm actually working on, but at the very least, it's still somewhat encouraging.

The Case for Optimism

While I'm certainly disappointed that I wasn't able to see how many (or how few) FPS the M1 chip can pump out trying to render a doofy looking shaderball, I have a little more hope than I did the last time I wrote about this:

• Someone is actively working on getting O3DE to run on macOS
• I got better results from the latest development branch than trying to build the last two stable releases
• Against all odds, I'm actually learning a surprising amount about how O3DE works, even when it's not actually working

At this point, it's probably too soon to start hoping for an official macOS build of O3DE 23.10, or even official build documentation. But hopefully things will continue to improve at a steady pace. Maybe I'll bite the bullet and finally sign up for Discord so I can talk to someone from sig-platform or sig-build, as someone else suggested. That's not going to happen, but at least I feel better about the other possibilities.