I got my hands on a mac install DMG file too, and i've had one hell of a time trying to convert it. I've tried both dmg2iso and dmg2img and had errors similar to yours. UltraISO was able to convert the image ok, but the image wouldn't boot in virtualbox O_o
Give me some time, when i find a solution i'll let you know ^_^
burning the .dmg file format
Well shit. I've googled my ass off and every solution i've found has failed.
I did manage to convert it to an iso, but even though the dmg showed the mac installation files in 7zip, the iso just had the installer for bootcamp O_o
The only possible solution i can see is opening the .dmg file in 7zip, extracting the .hfs file and then extracting everything in the .hfs file. These seem to be the actual mac os installer files. However, these can be burned to an ISO, but i can't find a boot image file anywhere within the installer files...
This is irritating...
I did manage to convert it to an iso, but even though the dmg showed the mac installation files in 7zip, the iso just had the installer for bootcamp O_o
The only possible solution i can see is opening the .dmg file in 7zip, extracting the .hfs file and then extracting everything in the .hfs file. These seem to be the actual mac os installer files. However, these can be burned to an ISO, but i can't find a boot image file anywhere within the installer files...
This is irritating...
¯\_(ツ)_/¯ It works on my machine...