Wired News is running an article on just how fast the X-Box is expected to be. They're giving credit for its performance to two factors: the nVidia graphics chipset, and the memory architecture of the X-Box, where the main and graphics processors share main memory.
From the article:
The XGPU will be able to animate 125 million polygons per second. NVidia's current top chip, the GeForce 2 Ultra, maxes out at 31 million polygons per second, and Xbox-like performance isn't likely to happen on a PC any time soon. There's absolutely no way you could render 125 million polygons a second across an AGP bus, said Tony Tomasi, director of product marketing at nVidia. The entire architecture of the XGPU is built around unified architecture where it shares memory with the processor, as opposed to shuttling it across a bus. So that changes the way the whole thing works.
Of course, why one couldn't patch a major consumer operating system to use a similar architecture is a separate question...