The prefix doesn't really matter (although it may cause problems), you can of course use it, depends on where you want the files to be installed.
But you are right, it doesn't work with my line. For instance, I should have used --without-expat and even then there's still the xml.cpp problem. I'll try to have a look at it later again, please use the wxall-source in the meanwhile. As long as you use --disable-gui, you get the same as with wxbase, i.e. no GUI libraries.