I've been thrashing away on oldlipstikred, my custom guitar, and Thinking about the state of visibility systems as a whole. It seems more and more people are interested in such things, but have to ask other people about their systems, or use the two well documented ones, portals and PVS + BSP. Personally, I think PVS was a good idea (iD-ugh), but if it had been a better set in the first place, the whole 3d research world would not be in this state of perpetual borrowing. Setting a personal initutive and developing your own Visibility set is a damn good idea, as you can custom suit your engine. PVS suited Quake, Portals suited Descent, and but I know descent had too many limitations, and Quake, although a leap forward, didn't quite get to the optimal way for its environment.
I personally have a few different equations of my own that I have developed, and I have let others in on them. The Phantom for instance, designed his new realtime based visibility equation on one of the ones I developed. Personally we discussed it and have both worked hard on the algo, and we have gone down different paths which suit our own style of engine.
One of the equations I have developed, not the one mentioned above, is documented, although with a rather poor description, on my page www.claustrophobe.8m.com. It is not really that realtime modifiable, but it does allow for a large percent of realtime modification. Not many people payed attention to this DOC, and I personally am using an OCTREE based system, called "The Realtime modifiable perfect 8 set" or REMPES, which I have mentioned, and many people know how works, but only a few know some of the optimisation details that make mine much faster than the regular system. I have two new optimisations (ones I just worked out) called virtual nodes and the 7 point system which should give me a rather nifty speed system, considering the 7 point system cuts down 64 plane side tests to 7 and the virtual node system can cut tests by a large amount aswell. By using the virtual node system, which happens to use the 7 point test for optimal efficiency, I can actually reduce many plane tests to simple > and < tests.
But the main thing to note is, that both these, although I haven't explained are actually rather simple tests, which can gain a fair bit of efficiency. Simply reordering code into the most efficient order, where the most things can be cached is also efficient. Simplicity is often a great improvement. That is not to say that all speed ups are simple. One of the speed ups I am studying at the moment is using parts of the other visibility system documented on my page, to significantly speed up testing, by calculating which polygons never have to be tested to other occluders, by splitting the world into 3d convexities. I could still dynamically modify parts of the world, either inserting convexities, or normal models, that wouldn't use the test, but we are talking a rather quick system here.
The trick to designing a quick visibility system, is often to invent a reasonable one, and then work out generic optimisations, by either taking from other methods, or realising simplistic optimisations, or working out case optimisations, which can avoid large amounts of pre-processing.
PS Visit my site, and check out the two lame docs there, you might be able to see something that will help you with your own idea. Jaap Suter saw something in the shadow doc that helped him get a grip on some vis stuff. so why not?