As Clements already mentioned, I had actually experimented with TruForm (implemented in software) waaaaay back in a D3D plugin I was writing. It worked pretty well for some things (Mario's head has never looked so smooth), but it made things sorta unplayable in-game, in SM64 anyway (the only game that plugin supported).
The problem was that it smoothed the terrain as well as the objects, so you ended up walking through hills alot because the game engine was still doing collision detection with the non-smoothed polys.
Since then I have given it some thought, and I've come up with a couple solutions that might make it feasible. First, make it ignore polys with a matrix-stack depth below a specified number. This solution would depend alot on how the game handles transformation, but the idea is that the terrain will usually only be transformed for view (matrix stack index of 0), while characters and objects would be transformed into world coordinates, and then to view (matrix stack index of at least 1). I haven't actually tried it to see if it would work, though...
The other solution I came up with is to give the ability to scale the amount of smoothing done to the polys. This would allow the user to find a balance between improved quality, and clipping errors.
Another problem is that I can't use the TruForm implemented into ATI's drivers because I have to do transformation in software to properly emulate the N64's transformation. This means I have to implement it in software, which can get kinda slow.
Anyway, that's as far as I've gotten into it. I may try to introduce poly-smoothing in future versions of glN64, but no promises.
Orkin