You can also enter "open -a firefox" in amule where lionel77 mentions, which opens the app called firefox (hear hear...) in the /Applications dir just like you would have doubleclicked it with the mouse.
For firefox, there is this nice Option in Preferences under Tabed Browsing, saying "open sites opened by other applications in a new tab in the active wndow / in a new window / in the last active Tab or window" (translated). The first two options do the trick for me.
So at least for Firefox, "open -a firefox" should be the command to be executed by amule by default when you choose "Firefox" in the dropdown. The way its implemented right now expects firefox to be in your PATH, which will for 99.99% of all mac users not the case. Sadly, I haven't found out how to pass options to the app started with the "open" command, I think its not possible (stupid implementation).
You can even start your defaultbroser calling "open
http://www.google.ch" (or in the case of firefox open the site in another tab / window, if you have done the settings explained above). Setting firefox as defaultbrowser has to be done over safari's setting (pervert hehe).
Cheers
sssnake