Well, you came close. :P Enter() and Exit()'s job is to determine whether or not to allow the movement by returning true or false; the default action of them doesn't actually affect anything, but only decides whether or not the movement should be allowed according to density and returns 1 or 0 accordingly. So if you don't pass on the return value their parent proc returns, calling it has no effect whatsoever, same thing as if you put a line of just rand(1,2) - it calculates a value and returns it, but it isn't used and is lost. The overrides you've posted would always return null, which is false, so all applicable movement attempts are denied. The way to work with these should be to return 0/false if you want the movement to be denied, and otherwise return the parent proc so any other parent/default movement rules are 'applied' and will determine if the movement succeeds or not.
Misc note: I wouldn't go in favor of using vars like mob.NPC to determine if a mob is an NPC one; that seems a fairly extraneous var. I usually have it set up so I can do something like if(istype(Mob,/mob/NPC)), but otherwise you can still check the key var to see if the mob is a player mob (or of course the client to see if a player is currently playing as that mob).
Misc note #2: As one may want to have a specific turf/area (instead of the NPC-blocking one) in that spot and you can't have 2 in the same spot, one may wish to use a different method of flagging a turf/area as NPC-blocking, such as with a simple variable on that atom.

Also, while this method would work to keep the NPCs in their designated area, the best way to go about such a thing is to remove the "problem" at the source: modify the NPCs AI code to never even attempt to move if the destination is a disallowed one, and that way Move() calls are spared and the NPC can decide to do something else instead, which is probably more productive than wasting time attempting a movement that will never succeed. For example, if the NPC's current target has moved beyond the NPC's allowed perimeter, switch to attacking it with ranged attacks (if applicable) instead of chasing, otherwise give up chasing (and try to find and attack a different target in the allowed area, when applicable).
This way of thinking can apply to mostly everything the AI can be denied from doing; if you don't root it out at the source, the result will be that instead of doing more desirable things (like in the example above), the NPC will try (usually over and over) to do the forbidden action and fail (unlike say a smart person or player that would realize he can't do that and try doing something else). This can lead to weaker and exploitable AI, so it should be avoided, as it is still easy to do anyway.

EDIT: I'll include a small example for the sake of the OP, to make this wall of text somewhat easier to grasp. Let's say that 'before', your AI code looked something like this (psuedo-code, illustration-only):
'After' [you change it so the NPC doesn't try to go into a forbidden area], it'd look like this:

http://www.byond.com/developer/FlameSage/FS_Walkback

i think you might be able to use this in some smart way.. making it so if it doesnt have a target it walks back to its original position(created position)

8BitGeek wrote:

I want to be able to set up an area that makes it so any NPCs in this area cannot leave.

Although Bakasensei's code had an error in it, it was still (aside from mentioned error) exactly what you asked for.

Taking what you said strictly and exactly as you put it, it is as simple as that code snippet. Any mob without an attached client controlling it will be denied exit, and everything else will be treated normally.

If you want it to be smarter, then what Kaioken said will benefit you, as it was basically just a suggestion to do the same thing but handled from the mob's point of view so that the mob can take its own special action on entry-failure.