I will show you how to do that without any third party software, just by using the OS X Disk Utility app.
1) First of all, open up the Disk Utility app on your mac, which is located at /Applications/Utilities path.
2) Now from Images menu select Convert navigate and choose the DMG format file.
3) Now you need to enter the filename you want to convert, set the value of "Image Format" to DVD/CD master and press on "Save" button.
It will save the file as CDR if you want to convert it to ISO then do next.
From the OS X desktop open the "Finder" menu and select "Preferences".
In opened window go to "Advanced" tab and make sure that the "Show all filename extensions" line are checked.
Simply right click on that CDR format file and choose from context menu "Rename".
Now change the "cdr" format to "iso", in warning window click on "Use .iso" button.