Not logged in, Join Here! or Log In Below:  
 
News Articles Search    
 

 Home / 3D Theory & Graphics / bilinear filtering precision Account Manager
 
Archive Notice: This thread is old and no longer active. It is here for reference purposes. This thread was created on an older version of the flipcode forums, before the site closed in 2005. Please keep that in mind as you view this thread, as many of the topics and opinions may be outdated.
 
ganchmaster

March 17, 2005, 04:16 PM

I have a shader which renders an environment map with a reflection vector computed from the eye using a normal looked up from a normal map. I'm getting some odd stepping artifacts in the colors resulting from this operation, and I'm wondering if I am being affected by some calculations with limited internal precision. Unfortunately the bump-reflection calculation seems like one that could be sensitive to precision issues.

I'm using HLSL with 2.0a profile on a Nvidia 6800GT.

Does anyone know what kind of precision results from the bilinear filtering of the normal map? My pixel shader uses floats, but the values from the normal map are 8-bit integers (for testing I'm using R8G8B8 textures). So when I say float3 bumpNormal = 2 * tex2D(BumpSampler, uv) - 1, what level of precision am I getting? Are the 8-bit values (mapped into 0-1 space) interpolated with full floating point precision, or just with their own 8-bit precision (ie steps of 1/255), or something else?

Thanks for any info...

 
Reedbeta

March 18, 2005, 02:03 AM

You should renormalize the normal after filtering. I had the same issue; it happens because bilinear filtering gets the normal far enough away from unit length that it produces a visible stepping effect, like you said. Precision is not the issue; it's just the way bilinear filtering works. Technically, we should use spherical interpolation for the normals, but bilinear + renormalize seems to work fine.

 
ganchmaster

March 18, 2005, 11:07 AM

That's not it. I'm renormalizing the normals already; I find the highlights look much better.

I did some experiments and it seems to me that the incoming precision from the texture lookup is only 8-bit precision (steps of 1/255 between 0 and 1). So if I have a texture with a gradient that steps in increments of 1 per channel, there's no internal interpolation between those values; it still steps in discrete steps of 1/255.

I figured this out by, more or less, making a texture with a gradient, multiplying the values I get from it by 10, and blowing it up real big on the screen so I could so I could get lots of samples between texels. On the screen the values increased in steps of 10. If there were higher internal precision in the bilinear interpolated value that was passed to my pixel shader, I would have seen a smooth gradient, not steps of 10.

This is disappointing because it produces artifacts in an effect I was trying to do. I'm still hoping there's something I can do to mitigate this. I know I could "manually" do the bilinear by sampling adjacent texels but that's a pretty ugly and expensive solution. (BTW, yes I am using "LINEAR" filtering in my samplers). Comments? Ideas?




 
Axel

March 18, 2005, 11:17 AM

Are you using an ATi card? They only use 5 bits of precision for bilinear filtering :(

 
ganchmaster

March 18, 2005, 11:43 AM

As the original post states, I'm using a Nvidia 6800GT.

 
SimmerD

March 18, 2005, 11:58 AM

Another thing to keep in mind is that the 2.x profiles only require 8-bit color interpolation. If you want fp16 or fp32 color interpoaltion, you need to use the 3.0 profile instead.

 
ganchmaster

March 18, 2005, 03:21 PM

SimmerD wrote: Another thing to keep in mind is that the 2.x profiles only require 8-bit color interpolation. If you want fp16 or fp32 color interpoaltion, you need to use the 3.0 profile instead.


I tried switching to vs_3_0 and ps_3_0, and it made no difference to my test. Do I need to do something else to get fp16 or fp32 color interpolation?

Is this something that only works with fp16 or fp32 textures? I'd rather stick with 8-bit but still have the high-precision interpolation.

Also, are we referring to the same thing? I am talking about precision of the bilinear interpolation from texture that the pixel shader reads; are you possibly referring to interpolation of COLOR values in the pixel shader input registers?

 
David Olsson

April 19, 2005, 10:23 AM

I've been having the same problem, have you found any solution yet ?
I need better filtering than 8bit otherwise my results look very ugly.

 
ganchmaster

April 19, 2005, 02:47 PM



David Olsson wrote: I've been having the same problem, have you found any solution yet ? I need better filtering than 8bit otherwise my results look very ugly.


I never did find a solution to the filtering precision. I have emailed NVidia about it and never got a response. I have also asked on a couple of forums and no-one seems to really know how this is supposed to work, or if they know, they aren't saying :). I guess that sampling from a floating point texture format might work better, but I haven't tested that, as it wasn't a good option in my case.

I did eventually figure out that part of the ugliness I was getting was due to use of the texCUBEbias instruction. It introduced weird line artifacts for some reason. I was able to reproduce this in the NVidia FX composer, and I sent them a sample, but never heard back. When I switched to texCUBElod, the artifacts went away. So I guess I'd say to anyone reading this, watch out for texCUBEbias, at least on NVidia chips!

Are you having a problem with a reflection mapping operation, or with some other precision sensitive situation?

 
David Olsson

April 19, 2005, 03:12 PM

I'm doing sphere tracing in a pixelshader while encoding the distance function in 3d texture. 8bit filtering for each channel show up as really ugly artifacts, 16bit would look better but 32bit floating point would be preferably. So what I'm looking at is low precision in the filtering will result in big artifacts in the geometry itself.

 
This thread contains 10 messages.
 
 
Hosting by Solid Eight Studios, maker of PhotoTangler Collage Maker.