Is it possible to run pre-built MacOS images from Packer?

[alexellis]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Is it possible to run pre-built MacOS images from Packer?

Describe the solution you'd like
A clear and concise description of what you want to happen.

I saw that Tart CI (licensed under a non-OSS license) shows how to build images using Packer, which can be saved and then loaded up on demand for things like CI runners.

Since I know of Lima which uses vz, I wanted to ask you whether this is possible with vz? The example only mentions being able to load a restore image, which doesn't sound like the same thing.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

It'd be interesting to be able to launch an Arm64 MacOS VM on Apple Silicon to run CI builds.

Seems like the main use-case here or the well-trodden path is Linux VMs on Apple hardware?

[Code-Hex]

@alexellis Thanks for creating this issue!
I'm sorry I don't have knowledge of Packer and Tart CI. I believe you mentioned this article: https://tart.run/integrations/vm-management/#building-with-packer

As you know, This library is a Go wrapper for Virtualization.framework, so what Virtualization.framework can't do, this library can't do either.

[alexellis]

Thanks for the response.

I suppose I can ask it a different way.

Could you see a way for us to create a disk image that could be launched as required for a CI runner using this library?

Of if that were your task, how would you go about it?

Unfortunately, the Tart CI stuff including their Packer integration has a license that isn't suitable for that kind of usage.

Do you always have to start from a restore image?

Is there a way to input things into an image at start-up time - parameters, secrets etc?

[alexellis]

By the way, I had an issue building the code, what would you suggest? This is Ventura with Apple Silicon and Go 1.20.

make all
go build -o virtualization .
# github.com/Code-Hex/vz/v3
virtualization_12_arm64.m:15:46: warning: incompatible pointer to integer conversion passing 'void *' to parameter of type 'uintptr_t' (aka 'unsigned long') [-Wint-conversion]
../../virtualization_12_arm64.h:41:53: note: passing argument to parameter 'cgoHandle' here
virtualization_12_arm64.m:432:47: warning: incompatible pointer to integer conversion passing 'void *const' to parameter of type 'uintptr_t' (aka 'unsigned long') [-Wint-conversion]
../../virtualization_12_arm64.h:40:46: note: passing argument to parameter 'cgoHandle' here
# github.com/Code-Hex/vz/v3
../../virtualization_arm64.go:554:4: cannot use _cgo3 (variable of type _Ctype_ulong) as unsafe.Pointer value in argument to _Cfunc_installByVZMacOSInstaller
../../virtualization_arm64.go:554:4: cannot use _cgo4 (variable of type _Ctype_ulong) as unsafe.Pointer value in argument to _Cfunc_installByVZMacOSInstaller
make: *** [build] Error 1

[alexellis]

I modified go.mod for the MacOS example to stop using the local source code, and to use the latest published release.

The VM boots now, but I want to send key strokes or commands to the VM. Is that something you've done? I see there's a keyboard configuration but there's no vm.SendKey or similar method available

[cfergeau]

I want to send key strokes or commands to the VM.

On a linux command line VM, you could use a serial device for this, or ssh connection.
With a GUI macOS VM, I'm not sure Apple's virtualization framework provides a way of sending keystrokes to the VM. Usually this is done with the OS accessibility APIs, might be possible to implement a vm.SendKey method with thes APIs to achieve what you need?

[cfergeau]

By the way, I had an issue building the code, what would you suggest? This is Ventura with Apple Silicon and Go 1.20.

@mpoindexter this compilation failure seems related to your PR.

[cfergeau]

I want to send key strokes or commands to the VM.

On a linux command line VM, you could use a serial device for this, or ssh connection. With a GUI macOS VM, I'm not sure Apple's virtualization framework provides a way of sending keystrokes to the VM. Usually this is done with the OS accessibility APIs, might be possible to implement a vm.SendKey method with thes APIs to achieve what you need?

A colleague of mine pointed me at https://github.com/adrianriobo/goax/blob/main/pkg/os/darwin/applescript/applescript.m

[alexellis]

What I'd like to get to - is a way to send key strokes to get from the initial "setup screen" after restoring the image, to a working machine where other operations could be performed.

I don't think SSH is enabled when you get the "Bonjour Hello!" screen and setup wizard? Perhaps it may not accept serial device input at tha ttime either?

It looked like the way the Tart creators get past this is by sending all the various keystrokes to tab around and click "confirm" to various pages until they land into a VM that's ready to be automated. Perhaps they also have keystrokes to open a shell and start installing packages.

If you wanted to get that restored image to a point where you can open a terminal or run commands against it, what would you do, knowing what you know now?

[cfergeau]

I never ran a macOS VM, so my knowledge stops there :(

[Code-Hex]

@alexellis

I'm not sure what are the requirements (i.e. disk format). but there is a way to create disk image to use this function.

Could you see a way for us to create a disk image that could be launched as required for a CI runner using this library?

I haven't looked into it properly, but I understand that utilizing it with CI probably requires installing some sort of command line or something. If you want to create a custom Disk image, this repository (CreateGuestImage.sh) may be helpful. (Not my work, so I haven't looked into it properly. Sorry if I'm wrong)

When the CI execution environment is the host and the macOS running on the host is the guest, a way to execute commands from the host to the guest is likely to be needed. For this purpose, I would pre-install an agent to run a command line using some kind of ssh protocol similar to Lima, using the above script.

Note
ssh is selected because vsock server cannot be started on macOS.

Of if that were your task, how would you go about it?

---

This is probably as far as I can show 🙏

If you have acquired knowledge in this area, it would be helpful to future users if you could add it to the wiki below.

https://github.com/Code-Hex/vz/wiki

[alexellis]

The problem as I see it is a bit of a chicken and egg:

The system is always in the restore state at the "hello, bonjour" page.

Even if we mounted a second disk with extra software on it, we couldn't install it or start SSH, because we're not yet at that stage.

We need a way to send key presses into the MacOS VM via the keyboard or another mechanism to advance through the wizard, then once it's fully completed, we could install an agent and communicate that way.

I have had a look through the way TartCI does this with Packer, but I haven't been able to find the code or mechanism yet that they use to get the key presses for the installer - https://github.com/cirruslabs/macos-image-templates/blob/master/templates/vanilla-ventura.pkr.hcl#L21C19-L21C19

Even after they've done that installation, it's not clear how they proceed to enter CLI commands like setting up sudoless login - https://github.com/cirruslabs/macos-image-templates/blob/master/templates/vanilla-ventura.pkr.hcl#L88C3-L88C22

There's some mention of VNC in the codebase, https://github.com/cirruslabs/tart/blob/f68297097e5f043191b52302fed17f7ff87ce972/Sources/tart/Commands/Run.swift - but I didn't think VNC was running when a machine boots up - before the setup is fully completed.

The alternative (which is not a great solution) is to do all of this initial installation manually, and install and agent, then save the system as a disk image, which is cloned and (if possible, wiped) upon boot-up by that custom agent, before it accepts remote procedure calls or SSH calls.

[cfergeau]

There's some mention of VNC in the codebase, https://github.com/cirruslabs/tart/blob/f68297097e5f043191b52302fed17f7ff87ce972/Sources/tart/Commands/Run.swift - but I didn't think VNC was running when a machine boots up - before the setup is fully completed.

Yes as far as I can tell they send these boot commands over VNC https://github.com/cirruslabs/packer-plugin-tart/blob/b25da725f429c5cddcb146f37e381596929efd2f/builder/tart/step_run.go#L210-L239

[Code-Hex]

@alexellis Ah, maybe I understand what do you want to do.

Virtualization framework supports USB keyboard. This is enabled to send key events from keyboard you are used.

I believe the code in tart is here:
https://github.com/cirruslabs/tart/blob/f68297097e5f043191b52302fed17f7ff87ce972/Sources/tart/Platform/Darwin.swift#L106

There is support in also vz.

Perhaps Packer's boot_command is prepared to set up the OS on the first boot. If this has the role of simulating the keyboard, I feel that these combinations are possible.

Note that, Some special keys that are exclusive to macOS don't seem to be available yet.
See: https://developer.apple.com/documentation/virtualization/vzmackeyboardconfiguration

However, for first-time setups, I felt that the features mentioned at the beginning were sufficient. It is also used in the macOS example.

[alexellis]

I think I found out how they do the bootstrap..

Seems like there is a hidden private API that enables a built-in VNC server within the installer.

https://github.com/cirruslabs/tart/blob/f45551cbf0e6cdff0d488a92c16b6d6b09faae22/Sources/tart/VNC/FullFledgedVNC.swift#L9

https://github.com/cirruslabs/tart/blob/main/Sources/tart/Commands/Run.swift#L48

Do you think that's possible over CGO?

Virtualization framework supports USB keyboard. This is enabled to send key events from keyboard you are used.

Do you have a code example of how to send keys using it from the controlling example program?