Background on applehv vs. qemu

Podman with QEMU virtualization on Mac comes with a big performance-penalty for bind-mounts, see containers/podman#16994 .

As of 25.1.24, virtualization with Apple Hypervisor is available with the Podman Desktop pre-release 1.7.3

This Gist is supposed to keep track of how to install it and test it with Docker Compose and Testcontainers.

Known problems of 4.9.0

Beware that as of 26.1.24, containers/podman#21288 means that a reboot of the host (e.g. laptop) will invalidate the applehv machine, and the only known workaround is to delete and recreate it! The fix is already merged into main, but hasn't made it into 4.9.0

Installing Podman with applehv virtualization

As recommended by @rhatdan ¹ , for testing purposes it's better to use the Podman Desktop .img installer or Podman .pkg installer to rule out issues with brew packaging.

Beware that if you install via .dmg, and when you don't have podman installed via brew, that the .dmg will install a .pkg with Podman CLI. The latter then can only be uninstalled manually, podman-desktop/podman-desktop#5735

Installation steps²:

Uninstall anything podman-related with brew uninstall
- brew uninstall podman podman-desktop
Remove any traces of previous Podman installations, including VMs (i.e. this will delete your existing VMs!)

rm -rf ~/.config/containers/
rm -rf ~/.local/share/containers
rm ~/.ssh/podman*

Currently must manually install vfkit, because .dmg installer currently misses it³:

brew tap cfergeau/crc
brew install vfkit

see https://github.com/crc-org/vfkit/releases for current vfkit release

From here you might either want to go with an official release installer, or, if you are adventurous, try a Podman 5 pre-release built from HEAD of the Podman main branch. The latter was surprisingly stabe for me, and you might evade bugs that are already fixed on the main branch.

Use Podman 4.9.x from installer

Download Podman Desktop .dmg from Podman Desktop Releases , preferrably 1.7+
- alternatively for CLI only, download .pkg installer from Podman releases (not Desktop), preferrably 4.9.0+
install it

Use Podman 5.0.0-dev via brew HEAD

This will build podman from its Github main branch, and that seems to be the only way to get a current build of Podman on Mac. The result was surprisingly stable for me!

brew install podman --HEAD

This will ony install Podman CLI. You can combine this with a recent Podman Desktop build from https://github.com/containers/podman-desktop/releases , as that will detect Podman CLI being installed already (via brew) and leave it untouched.

Create the applehv machine

create applehv machine and start it:

export CONTAINERS_MACHINE_PROVIDER=applehv
podman machine init
podman machine start

Add applehv to Podman config Add applehv to Podman config

# Add the 2 lines below below to ~/.config/containers/containers.conf
[machine]
Add applehv to Podman config
provider="applehv"

Make applehv machine rootful

Around using Docker Compose + Testcontainers, it turned out to be necessary at some point to run Podman rootful

podman machine stop
podman machine set --rootful
podman system connection default podman-machine-default-root
podman machine start

Using Docker Compose

You can use docker-compose installed via brew with podman as a substitute for docker.

Preconditions

Follow Using podman-mac-helper on macOS | Podman Desktop (podman-desktop.io)
must set DOCKER_HOST,
- but in contrast to what Using the DOCKER_HOST environment variable | Podman Desktop (podman-desktop.io) says, must set it to the rootful connection reported by podman system connection list , e.g. ssh://[email protected]:54328/run/podman/podman.sock !⁴
must symlink docker executable, do not simply alias it or e.g. exec-maven-plugin won’t find docker command
- sudo ln -s /opt/podman/bin/podman /usr/local/bin/docker

Docker Compose Troubleshooting

Problem: bind mount volume via Docker Compose cannot be accessed by container:

e.g. Java container shows Caused by: java.nio.file.AccessDeniedException: /folder-mount

2) Can't use volume mount, get permission denied in podman/troubleshooting.md says

This is sometimes caused by SELinux, and sometimes by user namespaces⁵.

SELinux can be disabled in the docker-compose.yaml as described in How to disable SE Linux from docker compose : r/docker (reddit.com):

In docker-compose.yaml, within services/<service-name>/:

security_opt:
 - "label=disable"

If you don't want or aren't able to amend your docker-compose.yaml, then SELinux can also be disabled completely in the machine, see How to disable SELinux in Fedora CoreOS - TechOverflow

Unfortunately, turning SELinux off completely is necessary for Testcontainers w/ Docker Compose, see below.

Problem: Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

When issuing docker-compose --file ... up:

error during connect: Get "http://docker.example.com/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dws-magnolia-webapp%22%3Atrue%7D%7D": command [ssh -o ConnectTimeout=30 -l root -p 49671 -- 127.0.0.1 docker --host unix:///run/podman/podman.sock system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: [email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Pruning all unused containers surprisingly helped me once here.

Using Testcontainers

Testcontainers troubleshooting

This is with a simple org.testcontainers.containers.ComposeContainer to run a custom docker-compose.yaml:

    private static ComposeContainer compose;
    
    @BeforeAll
    public static void startContainers() {
        compose = new ComposeContainer(
                new File("docker-compose.yaml"));
        compose.start();
    }

Status 500: make cli opts(): making volume mountpoint for volume .. podman.sock: operation not supported

When run from Maven and debugged:

{"cause":"operation not supported","message":"make cli opts(): making volume mountpoint for volume /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: mkdir /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: operation not supported","response":500}

{"cause":"changing to intended-new-root directory \"/private/tmp/.testcontainers-tmp-5348608840095356915\": chdir /private/tmp/.testcontainers-tmp-5348608840095356915: not a directory","message":"error in copier subprocess: changing to intended-new-root directory \"/private/tmp/.testcontainers-tmp-5348608840095356915\": chdir /private/tmp/.testcontainers-tmp-5348608840095356915: not a directory","response":500}

	Caused by: com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.NoHttpResponseException: localhost:2375 failed to respond
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.DefaultHttpResponseParser.createConnectionClosedException(DefaultHttpResponseParser.java:87)

When Junit Test is run inside Eclipse Debugger:

14:24:45.368 [main] ERROR tc.docker:24.0.2 - Could not start container
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"operation not permitted","message":"lsetxattr /var/run/docker.sock: operation not permitted","response":500}

When disabling SELinux entirely in the machine, running from Maven still yields:

com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"operation not supported","message":"make cli opts(): making volume mountpoint for volume /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: mkdir /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: operation not supported","response":500}

Solution

Must set DOCKER_HOST to the rootful connection URI reported by podman system connection list, in contrast to what the documentation on DOCKER_HOST says

Status 500: lsetxattr /var/run/docker.sock: operation not permitted

When run in Maven:

15:38:22.127 [main] ERROR tc.docker:24.0.2 - Could not start container
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"operation not permitted","message":"lsetxattr /var/run/docker.sock: operation not permitted","response":500}

Solution

The solution is to disable SELinux in the entire Podman machine, see Docker Compose Troubleshooting above.

in https://github.com/containers/podman/issues/20776#issuecomment-1867633199 ↩
mostly copied from @GabrielDillenburg's comment https://github.com/containers/podman/issues/20776#issuecomment-1887085665 ) ↩
see Unable to test podman on mac with provider applehv due to vfkit missing · Issue #21064 · containers/podman (github.com) ↩
should file issue for https://github.com/containers/podman-desktop/blob/8f29800fb18e2adf6445dba1e3ee703d0b6b2d6e/website/docs/migrating-from-docker/using-the-docker_host-environment-variable.md?plain=1#L77 ↩
this is either same as 'Permission denied' on volumes in Mac OS with podman machine · Issue #17560, or SELinux has to be disabled ↩

exoego/testing-podman-applehv.md

Background on applehv vs. qemu

Known problems of 4.9.0

Installing Podman with applehv virtualization

Use Podman 4.9.x from installer

Use Podman 5.0.0-dev via brew HEAD

Create the applehv machine

Make applehv machine rootful

Using Docker Compose

Preconditions

Docker Compose Troubleshooting

Problem: bind mount volume via Docker Compose cannot be accessed by container:

Problem: Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

Using Testcontainers

Testcontainers troubleshooting

Status 500: make cli opts(): making volume mountpoint for volume .. podman.sock: operation not supported

Solution

Status 500: lsetxattr /var/run/docker.sock: operation not permitted

Solution

Footnotes