On Building O3DE for macOS

So, I did a thing. I got an Apple Silicon MacBook, and tried to get O3DE working on it.

Tried.

For the record, I'm not a real developer by any stretch of the imagination. I once had a job writing mostly JavaScript for, like, a year. I'd say I know enough to cringe at that hacking scene from NCIS, but anyone who can actually write a CMakeLists.txt file will probably think a lot of what I'm about to say is equally as stupid (or 'misguided', if you want to be diplomatic about it – I can read between the lines).

Expedition 1: 17 June, 2023

As of the time of writing (i.e. O3DE 23.05), official documentation calls macOS support experimental. We have system requirements for the engine on Windows with Visual Studio, or Linux with LLVM/Clang, and a tiny blurb about macOS:

Undaunted, I went to the O3DE GitHub page to obtain the source. The README.md calls out that the project uses Git LFS. I tried to enable that with the Git version that macOS offers to install when you type git into a fresh terminal before XCode finishes downloading, but it kept complaining that lfs wasn't a recognized command. I ended up using GitHub desktop, which cloned the repository with no issues. It even offered to enable LFS for me once it finished. Neato!

Since I (severely) lack expertise in macOS development, Google pointed me to this document from the NYU Tandon School of Engineering indicating how I might proceed to build an OpenGL project using CMake and XCode:

• Install Homebrew
• Use Homebrew to install CMake (i.e. type brew install cmake in the terminal)
• Use CMake to generate an XCode project

mkdir build
cd build
cmake .. -G Xcode

Once I got the project open in XCode, I pressed the little play button on the left panel. It popped up a little dialog warning me that the project was configured for a custom architecture, and asking if I wanted XCode to switch it to one with Apple Silicon support. Due to naivety and hubris, I pressed Update and Build, and let it do its thing while I tried to find the specific moment in an hour and six minute video where the dude said something that I thought would be funny to quote in my writeup.

When I finally remembered that I had left XCode running, I went back to check on it, and (predictably) there were errors:

This header is only meant to be used on x86 and x64 architecture

Cool. I guess they weren't kidding about needing x86_64. The header in question is part of LLVM, and apparently handles "MMX intrinsics", which sounds like it's way closer to bare metal than I'm comfortable being. I didn't want to try to figure out where O3DE was trying to do MMX things, or how to work around it, since we've already established that I'm a JavaScript dude. What's an architecture? What's performance? I don't know. I write JavaScript.

At this point, it was time to touch grass.

Expedition 2: 18 June, 2023

Today's theme is:

** BUILD FAILED **

When I woke up, I wondered what would have happened if I hadn't told XCode to 'Update and Build', and just decided to do an Intel build like God intended, and run it through Rosetta 2. I felt good about it. This certainly isn't the worst regret to ever wake me up in the night.

I opened GitHub Desktop, reverted all the changes that the build process had made, and deleted the contents of the build directory (guess that folder's in the .gitignore?). I used CMake to generate the XCode project like before, and pressed Play like before. But this time, I pressed "Build" instead of "Update and Build", because I'm smarter than I was yesterday. I went back to YouTube.

This time, when I came back to XCode, I saw a different message:

But that's not today's theme. You see, when I navigated to the build directory and tried to run the O3DE Editor, it wouldn't let me. I opened the directory in the Finder, and it listed the Editor with a size of 3 KB.

O3DE is a lot of things, but 3 KB is not one of them.

I went back to Google. Maybe someone has already written an O3DE macOS build tutorial like I'm trying to write. Or maybe someone published their own unofficial build for macOS like I'm trying to do. If that's the case, Google either doesn't know about it, or doesn't realize that's what I'm asking for. At one point, the search results pulled up something I wrote:

I went back to the build instructions for Linux, and found the CMake arguments used there. I adapted that to the following command:

cmake --build . --target Editor --config release -j 8

CMake did its thing for a while, but it hit me with a

** BUILD FAILED **

No worries. Right under that, there's a little note (emphasis mine):

CMake unity builds are on by default. This is a CMake feature that can greatly improve build times by merging source files into single compilation units. If you encounter a build error, disabling unity builds might help debug the problem. To disable unity builds, run the cmake project generation command with the -DLY_UNITY_BUILD=OFF argument to regenerate your project files.

So I ran these commands in the terminal:

cmake .. -G Xcode -DLY_UNITY_BUILD=OFF
cmake --build . --target Editor --config release -j 8

It ran for a while.

** BUILD FAILED **

Christ.

Expedition 3: The GitHub Discussion

At this point, I vaguely remembered seeing some kind of a forum post where someone from the O3DE team said something to the effect that there was like one guy on the team who was using the engine on a Mac – what a weirdo. Maybe that was on Reddit. Maybe I hallucinated it. I tried to find it again as a form of procrastination, but instead I found a GitHub discussion where an alleged game development professor was asking about the availability of O3DE for students who only had access to macOS. The original post was from August, 2022, about two months before 22.10 came out.

This discussion yields a few important nuggets of information. An O3DE maintainer by the handle rgba16f (and the face of Grumpy Cat) writes:

There isn't an official doc that lists what it would take to promote Mac to officially supported.

From my personal local tests the main thing that is broken on Mac (vs PC) is user mouse input, trying to control the camera in the editor will quickly show unexpected behavior. As far as I know this is an error in our QT layer. The main impediment to that getting fixed is available engineering time.

There are also other polish issues to do with the AssetProcessor, packaging the build into an App bundle, etc. These are not blockers to working on the platform but they do reduce the quality of the experience.

I would recommend taking your question about nightly Mac builds to sig-platform and/or sig-build on discord.

I'm way too socially anxious to talk to strangers on Discord about something I already feel dumb trying to figure out, even though everyone I've seen on the O3DE YouTube channel seems friendly, passionate, and helpful. The mouse control thing is concerning, but not as concerning as what shdwshard adds:

There are no prebuilds for MacOS, so you'll need to check out the source and build it yourself. Adding to the trouble is that both main and development produce unusable mac binaries at the moment.

The unusable binaries comment was from January of this year, so yikes. He also provided a follow up comment:

Through bisecting, I found 4ac8adf is the most recent which will function on my M1 Mac. You might try that as a starting point, along with the following build commands:

# Configure the editor's build
cmake -B ./build/mac_xcode -S . -G Xcode -DLY_ASSET_DEPLOY_MODE=LOOSE -DLY_ASSET_DEPLOY_ASSET_TYPE=mac
# Build the editor
cmake --build build/mac_xcode --target Editor --config profile
# Run the editor
./build/mac_xcode/bin/profile/Editor.app/Contents/MacOS/Editor

Anthony-Gaudino adds,

Also, for anyone trying this, you should install Rosetta by running softwareupdate --install-rosetta and also install Cmake: https://formulae.brew.sh/formula/cmake#default

4ac8adf worked for me on M1 Mac, but a very recent version failed while building the code.

With 4ac8adf I created a small project and none of the FBX assets build, so I had no 3D assets in editor.

The editor also crashes very often.

Well, I guess that means it's time for another build.

I tried building the project as is, and attempted a release configuration instead of profile, but that failed. I tried again using the 'profile' config, and it ran a lot longer. The bottom of my MacBook got a lot hotter. Then, finally, it happened:

** BUILD SUCCEEDED **

Test Driving The First Build

Unlike my XCode build, this version of Editor.app weighs in at a more plausible (if worryingly large) 1.33 GB. I clicked on it. It wanted to access my Documents folder, so granted permission. It threw the following error:

I ignored it. For a while, nothing happened. But suddenly:

Unfortunately, it didn't go any further, which would make sense, since I didn't tell it to load a project. I tried running o3de.app, instead. That gave me the normal Project Manager, and an Automated Testing project that was already there. It had the little warning icon that shows when a project needs to be built, so I hit Build Now, which told me I needed to install CMake.

What have I been fighting with for the last two days?

I tried installing the GUI version in case that helps. At first, O3DE didn't see it, but that might have been because I hadn't opened it yet, since it seemed to work later. I did have to open my project in XCode and build it that way (again, use "Build", not "Update & Build"). But even after that, the editor wouldn't open it. I left it open on that splash screen overnight, but nothing happened. I could probably try it out with that last known working build, but that seems like a lot of work for something that "crashes very often", where "trying to control the camera in the editor will quickly show unexpected behavior". I have it running well enough on a Windows PC (with a dedicated GPU!), so I can wait for the macOS port to be a little more fully baked.

Conclusions

At the end of the day, the O3DE developers weren't kidding about the state of the engine on macOS. And while this journey has certainly been frustrating, and ultimately unfulfilling, I don't want to come across as overly harsh or negative. I like using this engine on Windows. I like the way the Atom renderer looks. The people who build this thing seem great. I want to see it succeed.

I'd still like to do some more experimentation. Maybe things will work a little better if I use macOS as the build target (i.e. for playing the game), instead of trying to set it up as the host platform (i.e. the editor). But macOS and iOS targets also seem to be labeled as experimental, so I have reservations.

Ultimately, I'd like to see O3DE move towards being more self contained, and easier to get running. Unity, Unreal Engine, and Godot don't necessarily need Visual Studio, XCode, or Clang to get started. Similarly, I could see a lot of value in a version of O3DE where users can build simple games using Lua or Script Canvas, without even having a full suite of development tools installed. You might even be able to get away with making that an iPad app (for the iPad Pro, anyway).

Obviously, that's not much of a concern for serious AAA game developers, but O3DE is trying to position itself as "The open source, real-time 3D engine". Unfortunately, no matter how great the Atom renderer looks, or how well it can run on a variety of devices, it won't attract anyone from fields like architectural visualization, virtual film production, or general 3D simulations as long as it's asking artists to touch CMake.