You will lose both torque and HP and fuel efficiency. There is no benefit for decreasing the size of the throttle body.
With smaller throttle body, all you do is create the bottle beck for the air entering the engine. Less air- less power, less efficiency.
In theory, smaller throttle body can increase air speed but it will decrease volume. On our engines, I do not see this benefiting the engine at all.