Do I need to normalize the vector before the culling?
What transforms do I apply to the normal?
What 3D space(Object, World, Eye, etc..) should my polygon be in before attempting backface removal?
I've tried my hand at it, but I'm having issues, currently, here's my barely working code. It does the culling, but only does it right when my object is at a certain angle.
void drawPolygon(Polygon3D &p, bool statik=false){
int verts = p.numVertices();
POINT points[verts];
Vector3 tmp, normal, Lnormal; //Normals for backface removal and lighting respectively
normal = R*getNormalOfPolygon(p);
normal.normalize();
for(int i=0; i<verts; i++){
if(statik) tmp = C*p.getVertex(i);
else tmp = A*p.getVertex(i);
/////// Clipping ///////
if(tmp.z() > -2 || tmp.z() < -55){ return; }
/////// Backface Removal ///////
if(dotproduct(normal, Cam.getDirectionVector()-tmp) < 0) return;// If the angle is less than 90, it can't be seen, so we skip it.
...
What you need to do is to project* the polygon's normal vector onto the eye's direction vector. They can be in any coordinate space, as long as they're both in the same one.
If the result is zero, then the normal vector is perpendicular to the eye vector, meaning that you're viewing the polygon edge-on. So you probably want to cull it in that case. If the result is positive, then the normal vector is pointing more-or-less in the same direction as the eye vector (well, they're in the same hemisphere at least). In other words, it's facing away from the camera, so cull it. But if the result is negative, then the polygon is facing towards the camera so you should display it.
*Instead of doing projection you can just use the dot product, since it will still have the same sign (plus it's faster). This is because to get from the dot product to the scalar projection, you just divide by the length of one the the vectors, and length is always positive (by definition).
For a similar reason, you do not need to normalise either of the vectors beforehand; the divisions involved in normalising are positive and so do not affect the sign.
BTW, if you use OpenGL or DirectX they do backface culling for you. I think it might be faster, since the GPU does the culling for you - on the other hand, it wastes pipeline bandwidth sending triangles that will get culled, which could slow things down. So I don't know; I'm not exactly an expert. I do know that it is less effort though. =) Of course, if you're writing your own software renderer then this doesn't apply.