Jon,
First check that you have a reasonable squish gap setting this should be 1.4 – 1.6mm. Measure it with a piece of solder through the plug hole and make sure it goes right to the edge of the bore as the squish band is quite narrow. Add more shims until its correct. If its way too low the noise may be pinking from too high a cr. Certainly worth checking anyway.
Your dead short is very unlikely to be the bulb unless its fitted incorrectly. The lights (and everything else) are powered by the battery and the regulator will adjust dynamo output to cope with load. If it can cope with dip bean then most unlikely that it could not cope with main beam. My guess is you have a short to earth. All you can do is chase the circuit from the ignition/lighting switch via the dip switch and back to the bulb. As it works on dip beam, the power is clearly getting to the dipswitch ok so most likely the fault is in the dipswitch itself, or in the mainbeam wire. Try just removing the dipswitch from the handlebar first. If that cures the problem then the fault is a short from the switch to earth and should be easy enough to spot.
Peter