Don't have the effort/time to do a huge writeup tonight. Here are a few things you can look into that they use. Note: this is for MBG, as MBP's custom renderer uses entirely different methods.
OpenGL
for rendering, it does all the 3d graphics and stuff.
Immediate mode rendering
because this version of Torque was released before shaders really caught on.
Stencil shadow volumes
for shadows. Compare to
shadow mapping
where the shadow values are stored in textures instead of geometry.
Gouraud shading
smooths the light values between the different faces so you don't get choppy spheres or loops when there are a low number of polygons.
Binary Space Partitioning
for optimizing the renderer. This speeds up time to draw the scene, but takes extra time because you need to build the BSP tree (map2dif does this for difs, and the game also does this when loading).
Jeff could go into more detail on most of these, as well as the wiki pages that I linked. It gets into advanced mathematics (linear algebra and calculus, hooray!) really fast, so be warned.