Claire Blackshaw gives brief intro to using spreadsheets in Opinion: Stop Being The Useless Designer – Excel and Formulas.
Excel (and other spreadsheet programs such as Numbers and Open Office) has excellent tools to quick prototype game system behaviors, as Blackshaw points outs.
For prototyping complex behaviors, writing simple computer programs in unavoidable, I think. I have found Python extremely useful for quick prototyping. There are handy libraries for Python to create and analyze simulations; for example, SimPy for creating sumulations and rpy for statistical calculations (rpy requires R). Pygame can be used for creating playabe prototypes.
Petri Lankoski 