What's wrong with prioritization of packets?

For individual routers: It adds additional cpu and memory requirements, along with additional code complexity. This increases router cost and reduces reliability. Further, if the router is the bottleneck, rather than the wire, it reduces traffic. For the routing fabric: It increase routing protocol complexity, increasing the probability of misconfiguration. And adding delay to selected traffic classes may have difficult-to-predict effects, including lowering total goodput, and perhaps decreasing statistical multiplexing.