Presented to the UCI/OSA Color/Vision Meeting, October 2001, but still work in progress

Classification Weights and Internal Noise Level Estimation

Albert J. Ahumada, Jr.

Abstract

For the case of linear discrimination of two stimuli in white Gaussian noise in the presence of internal noise, a method is described for estimating linear classification weights from the sum of noise images segregated by stimulus and response. The recommended method for combining the two response image for the same stimulus is to difference the average images. Weights are derived for combining images over stimuli and observers. Methods for estimating the level of internal noise are described, especially for the case of repeated presentations of the same noise sample. Simple tests for particular hypotheses about the weights are shown based on observer agreement with a noiseless version of the hypothesis.

List of symbols in order of appearance

m the number of image components

S _s s = 0 to 1, 1 by m signal vector

p _s s = 0 to 1, probability of signal S _s

N 1 by m noise vector with components N _i , i = 1, m

T 1 by m trial stimulus vector with components T _i , i = 1, m

E[.] averaging or expectation operator

Var[.] variance computing operator

s² variance of N _i

L 1 by m classification vector

b bias of linear classifier

R _r r = 0 to 1, the event that the observer made response r

||.|| vector length, ||L|| = (LL ^T) ^½

Pr{} probability of enclosed event

p_sr probability of response R _r given signal S _s

F _Z(.) cumulative standard normal distribution function

d ₀' sensitivity of linear classifier

b ₀ shifted bias of linear classifier, b – L S ₀ ^T

Z(.) functional inverse of the cumulative standard normal distribution function, F _Z ^-1(.)

L _I classification vector L of the ideal observer

d _I' sensitivity of the ideal observer

r the sampling efficiency of L, L L _I ^T

b _{0, H} random shifted bias of the human observer model

g² variance of b _H

a² proportion of external noise in the classification variable, 1/ (1+ g² )

d _H' sensitivity of the human observer model

b _H performance bias of the human observer model

f _Z(.) standard normal distribution density function

A _SR the averages of the noises N for the trials segregated by signal S _S and detection response R _R

N _SR a random noise N conditional on signal S _S and detection response R _R

v _SR the expected value of N _SR when m = 1.

U an orthonormal m by m transformation

I the m by m identity transformation

z _i a standard normal variable

M _r r = 0 to 1, the event that the error free model made response r

p_MS0 the probability of event M ₀ given that the signal was S _s

b _Ms the signal dependent error free model criterion, b ₀ if S ₀ , or b ₀ - d ₀' if S ₁

Linear classification of two signals in additive white Gaussian noise

The signals and noise

S ₀ and S ₁ are 1 by m signal vectors, presented with probabilities p ₀ and p ₁ = (1 - p ₀) for n trials. On each trial a random noise sample vector N is added to the signal, so the trial stimulus

T = S ₀ + N

or

T = S ₁ + N.

N is a 1 by m vector of independent samples of identically distributed Gaussian variables N _i with

E[N _i] = 0

and

Var[N _i] = E[(N _i - E[N _i])²] = s²,

where E[.] is the averaging or expectation operator and Var[.] computes the variance. Without loss of generality we can assume that the noise has been normalized by its standard deviation so that

s² = s = 1.

The linear observer model

The linear observer classifying the noisy signals as R ₀ or R ₁ would use a vector L

and respond R ₁ if and only if

L T ^T > b,

where b is a response criterion and ^T indicates the matrix transpose operator, so that

             m

L T ^T = å L _i T _i .

             i

Also without lack of generality we will assume that L has unit length (L and b have already been divided by the length of L) so that

||L|| = (LL ^T) ^½ = 1.

The performance of an observer is characterized by the error rates

p₀₁ = Pr{R ₁ | S ₀},

the probability of signal S ₀ being followed by response R ₁ and

p ₁₀ = Pr{R ₀ | S ₁},

the probability of signal S ₁ being followed by response R ₀.

For the linear classifier with vector L and criterion b,

L N ^T

is Gaussian with mean zero and unit variance. Hence

p ₀₁ = Pr{L (S ₀ + N) ^T > b } = 1- F _Z( b – L S ₀ ^T )

and

p ₁₀ = Pr{L (S ₁ + N) ^T < b } = F _Z( b – L S ₁ ^T ),

where F _Z(.) is the cumulative standard Gaussian distribution function.

If we define sensitivity and bias parameters

d ₀' = L (S ₁ - S ₀) ^T

and

b ₀ = b – L S ₀ ^T,

then the error rates are

p ₀₁ = 1 - F _Z (b ₀)

and

p ₁₀ = F _Z (b ₀ – d ₀').

These parameters can be found from the error rates as

b ₀ = Z(1 - p ₀₁)

and

d ₀' = b ₀ - Z(p ₁₀),

where

Z(p) =F _Z ^-1(p)

is the functional inverse of the cumulative standard normal distribution function F _Z(.).

The ideal observer

The ideal observer classifying the noisy signals as R ₀ or R ₁ would use the linear classifier

L _I = (S ₁ - S ₀) / ||S ₁ - S ₀||.

For the ideal observer,

d _I' = L _I (S ₁ - S ₀) ^T = ||S ₁ - S ₀|| .

The efficiency of a non-ideal linear classifier is given by

(d ₀'/d _I') ² = (L L _I ^T ) ² = r ²,

the square of the correlation between the actual and the ideal classifier coefficients, sometimes called the sampling efficiency.

A noisy human observer model

Human observers classify the same images different ways on different presentations. This is modeled here by assuming that the observer's criterion b _{0, H} (corresponding to b ₀) is a normally distributed random variable with

E[b _{0, H}] = b ₀

and

Var[b _{0, H}] = g²

independent of the noise N. It does not matter whether the variability is added to the criterion or the classification function value. Since the noiseless criterion b ₀ was defined as the criterion for a variable with unit variance, the parameter 1+ g² can be interpreted as the total variance of the classification variable and

a² = 1/ (1+ g² )

as the proportion of variance in the classification variable that is from the external noise N. The error probabilities are now

p ₀₁ = Pr{L N ^T > b _{0, H} }

= Pr{(L N ^T - (b _{0, H} - b ₀)) / (1+ g²) ^1/2 > a b ₀ }

= 1- F _Z(a b ₀)

and

p ₁₀ = F _Z(a (b ₀ - d ₀')).

If we define the observer's sensitivity and biases as

d _H' = a d ₀'

and

b _H = a b ₀,

then we can compute these parameters from the human model observer error rates as

b _H = Z(1 - p ₀₁)

and

d _H' = b _H - Z(p ₁₀).

The efficiency of the human observer model is

(d _H'/d _I') ² = a² r ².

Since r ² £ 1, a lower bound for alpha is given by

a ³ d _H'/d _I',

and an upper bound for g² is given by

g² £ (d _I'/d _H') ² - 1.

These bounds are reached when L is L _I and the inefficiency is only the result of the internal or criterion noise.

The classification images

The classification image components are the four average noises A _SR, the averages of the noises N for the trials segregated by signal S _S and detection response R _R . We would like to find the mean and the variance of the pixels of A _SR as a function of the parameters (S ₁, S ₀, L, b ₀, and g or a).

The single pixel case

In the single pixel (m=1) case we are trying to find the mean of a single Gaussian variable N that has been truncated by a random criterion b _H. Let N _SR be the truncated variable when s was the stimulus and r was the response and

v _SR = E[N _SR].

Then since ||L|| = 1, L = 1 so

L N ^T < b _{0, H}

if and only if

N < b _{0, H}.

So in the case that s = r = 0,

v ₀₀ = E[N ₀₀] = E[N | N < b _H]

and for the other cases

v ₀₁ = E[N ₀₁] = E[N | N > b _H]

v ₁₀ = E[N ₁₀] = E[N | N < b _H - d ₀']

v ₁₁ = E[N ₁₁] = E[N | N > b _H - d ₀'] .

Single pixel, no noise

Consider now the single pixel case when there is no noise in the criterion (b _{0, H} = b ₀).

b ₀

E[N ₀₀] = E[N | N < b ₀] = ( ò z f _Z(z) dz )/ F _Z(b ₀) = - f _Z(b ₀)/ F _Z(b ₀),

                                           -¥

where the integration of z exp(-z ² /2) is enabled by the variable substitution

t = -z ²/2.

If b ₀ is zero,

E[N ₀₀] = -(2/p) ^1/2 = -0.7979,

the expected value of the absolute value of a standard Gaussian.

Similarly,

¥

E[N ₀₁] = E[N | N > b ₀] = ( ò z f _Z(z) dz ) / (1-F _Z (b ₀)) = f _Z (b ₀)/ (1-F _Z (b ₀)).

b ₀

Single pixel, noisy criterion

The Gaussian criterion case can be reduced to the fixed criterion case by a change of variables. Let z ₀ be the standard Gaussian used to form the criterion b _H, so that

b _{0, H} = g z ₀ + b ₀ .

Then

v ₀₁ = E[N ₀₁] = E[N | N > b] = E[ N | N < g z ₀ + b ₀].

If we let

z ₁ = a (N – g z ₀)

and

z₂ = a (g N + z₀)

The new variables z ₁ and z ₂ are independent (E[z ₁ z ₂] = 0), standard (E[z ₁] = E[z ₂] = 0, Var[z ₁] = Var[z ₂] = 1) Gaussian variables, such that

N = a (z ₁ + g z ₂)

and

N < g z ₀ + b ₀

if and only if

z₁ < ab ₀.

So

v ₀₀ = E[N ₀₀] = E[N | N < g z ₀ + b ₀] = E[a (z ₁ + g z ₂) | z ₁ < a b ₀]

= a E[ z ₁ | z ₁ < ab ₀]

= - a f _Z (a b ₀) / F _Z (a b ₀)

= - a f _Z (b _H) / F _Z (b _H)

= - a f _Z (Z(p ₀₀))/p ₀₀ .

Expressed in terms of p ₀₀ the effect of the criterion noise on the mean v ₀₀ is just to reduce v ₀₀ by the factor a.

Similarly,

v ₀₁ = E[N ₀₁] = E[N | N > b _{0, H}] = E[a (z ₁ + g z ₂) | z ₁ > a b ₀]

= a E[ z ₁ | z ₁ > a b ₀]

= a f _Z (a b ₀) / (1 - F _Z (a b ₀))

= a f _Z (b _H) / (1 - F _Z (b _H))

= a f _Z (Z(p ₀₀)) / (1-p ₀₀)

= a f _Z(Z(p ₀₁)) / p ₀₁,

since p ₀₁ = 1 - p ₀₀ and

Z(p) = -Z(1 - p).

If false alarms are less frequent than correct rejections (p ₀₁ < p ₀₀ ), then

|v ₀₁| > |v ₀₀|,

a larger absolute expected value on a false alarm than a correct rejection trial.

The signal case is the same with the criterion changed to b _{0, H} – d ₀' so that

v ₁₀ = E[N ₁₀] = E[N | N < b _{0, H} – d ₀'] = -a f _Z (Z(p ₁₀))/p ₁₀,

and

v ₁₁ = E[N ₁₁] = E[N | N > b _{0, H} – d ₀'] = a f _Z (Z(p ₁₁))/p ₁₁ .

Again, if misses are less frequent than hits (p ₁₀ < p ₁₁ ), then

|v ₁₀| > |v ₁₁|,

a larger absolute expected value on a miss than a hit trial. Regardless of the signal, the expected value only depends on the response proportion and the criterion variability.

The multiple pixel case

Another independent variable transformation allows the single pixel case result to solve the multiple pixel case. Let us first examine the means and variances of the pixels of N ₀₀. For any vector L of unit length, it is possible to construct an orthonormal transformation U whose first row is L, that is

U = (L ^T L ₁ ^T … L _m-1 ^T) ^T ,

such that

U ^T U = I,

where I is the identity transformation (the transpose of U is its inverse).

When this transformation is applied to N ^T we get a new noise U N ^T whose distribution is the same as that of N ^T, but whose first pixel is L N ^T. On an S₀ trial, a noise vector N will be classified as N ₀₀ if and only if the first pixel of U N ^T,

z ₁ = L N ^T < b _{0, H}.

The rest of the pixels (z ₂, ..., z _m) of U N ^T are independent standard Gaussian variables (with mean zero and variance 1).

E[N ₀₀ ^T] = E[ N ^T | L N ^T < b _{0, H}]

= E[U ^T U N ^T | L N ^T < b _{0, H}]

= E[U ^T (z ₁, z ₂, ... , z _m )^T | z ₁ < b _{0, H}]

= U ^T E[(z ₁, z ₂, ... , z _m )^T | z ₁ < b _{0, H}]

= U ^T (v ₀₀, 0, ... , 0 )^T

= v ₀₀ L ^T.

Using similar argument for the other cases, leads to the general result that

E[N _SR ] = v _SR L .

The mean of a classified noise is proportional to the classifying vector L. The variance of individual elements of N _SR ^,

Var[N _SR, i ] = L _i² Var[z ₁ | S _S "S _R "]+ (1-L _i²) ,

which is bounded by Var[z ₁ | S _S "S _R "] and one. Truncation of a Gaussian can only decrease the variance, so

Var[N _SR, i ] < 1.

Since ||L|| = 1, if there are very many significant weights in L they will have to be small, so that

Var[N _SR, i ] ~ 1.

Let A _SR be the average value of a number n _SR of N _SR. Any combination of the form

L _est = w ₀₁ A ₀₁ – w₀₀ A ₀₀ + w ₁₁ A ₁₁ – w ₁₀ A ₁₀,

with positive weights w _SR will be an estimate of L times a positive constant. To maximize the signal to noise ratio of an estimate of L from a sample with n _SR approximately independent samples of each type, the individual estimates N _SR /v _SR should be weighted inversely by the square of their variances, which are approximately 1/v _SR², so we should weight each N _SR by v _SR .

A good un-normalized estimate of the classifier L is thus given by the v _SR weighted sums n _SR A _SR .

L _G = v ₀₁ n ₀₁ A ₀₁ - v₀₀ n₀₀ A ₀₀ + v ₁₁ n ₁₁ A ₁₁ - v ₁₀ n ₁₀ A ₁₀ .

If we replace p _SR in v _SR by p _SR = n _SR / n _S, taking advantage of the fact that

f _Z (Z(p))= f _Z (Z(1-p)),

we obtain

L _G = a ( n ₀ f _Z (Z( p ₀₁)) (A ₀₁ - A ₀₀ ) + n ₁ f _Z (Z( p ₁₀)) (A ₁₁ - A ₁₀) ).

The more frequent stimulus should be given more weight and the more error prone stimulus should be given more weight. If both stimuli are equally frequent and the error rates are equal, the formula is proportional to

L _ave = A ₀₁ – A ₀₀ + A ₁₁ – A ₁₀ ,

the combination rule originally used by Ahumada and Beard (). Another way of expressing the good weighting scheme for combining average classification images A _{SR, O} over responses, stimuli, and observers, in that order that to combine over responses, just take the difference,

L _{G, S, O} = A _{S1, O} - A _{S0, O}.

When combining images for different stimuli, they need to be weighted by factors involving the relative frequencies of the stimuli and the extremeness of the error proportions for the stimuli.

L _{G, O} = w _{1, O} L _{G, 1, O} + w _{0, O} L _{G, 0, O}.

where

w _{S, O} = n _{S, O} exp( - Z( p _{SS, O})²/2) .

If estimates are to be combined over M observers, they need to be weighted by the observer's proportion of variance due to the external noise variance, a² = 1/(1+ g²), and the number of trials run by the observer (which is included here in the n _{S, O}).

L _G = a ₁² L _{G, 1} + a ₂² L _{G, 2} + ... + a _M ² L _{G, M} .

Measuring the internal noise

Response agreement with the same external noise sample

Ahumada and Beard (1998 ARVO) presented a method for estimating the amount of internal noise relative to the external noise (alpha or gamma) using the agreement between observer responses on pairs of trials where the same external noise and the same stimulus were presented. They did not specify a linear observer model, but they effectively assume that the decision component based on the noise corresponding to our

z ₁ = L N ^T

is a standard Gaussian and the rest of their assumptions correspond to our model with criterion variability.

The subscripts i and j are added to indicate two separate trials. The agreement probability for a particular signal and response condition with the same noise is denoted by

p _ASR = Pr{R _{R, i} , R _{R, j} | S _{S, i} , S _{S, j} , N _i = N _j }.

To obtain this probability we will compute it conditional on the value of z ₁ and then average over the possible values of z ₁.

Conditional on the value of z ₁, for S = 0, the probability of a response R = 0 is given by

Pr{R _{0, i} | S _{0, i} , z ₁ } = Pr{ z ₁ < b _{0, H} | z ₁}

= Pr{ z ₁ < g z ₀ + b ₀ | z ₁}

= Pr{ (z ₁ - b ₀) / g < z ₀ | z ₁}

= F _Z ((b ₀ - z ₁)/ g).

For another response to the same signal and the same noise the criterion variability would be independent and the probability of two responses of R = 0 would be given by

Pr{R _{0, i} , R _{0, j} | S _{0, i} , z ₁ } = Pr{ z ₁ < b _{0, H} | z ₁}²

= F _Z((b ₀ - z ₁)/ g) ².

= F _Z((b _H / a - z ₁) / g) ².

= F _Z((b _H (1 + g² ) ^0.5 - z ₁) / g) ².

The probability of two correct R _{0, i} responses to the same noise is then

            +¥

p_A00 = ò F _Z((b _H (1 + g² ) ^0.5 - z ₁) / g) ² f _Z(z ₁) dz ₁,

           -¥

and

            +¥

p_A01 = ò (1- F _Z((b _H (1 + g² )) ^0.5 - z ₁) / g) ² f _Z(z ₁) dz ₁.

           -¥

The equations for arbitrary S can be written

            +¥

p_AS0 = ò F _Z((Z(p _S0)(1 + g² ) ^0.5 - z ₁) / g) ² f _Z (z ₁) dz ₁,

           -¥

and

            +¥

p_AS1 = ò (1 - F _Z((Z(p _S0)(1 + g² ) ^0.5 - z ₁) / g)) ² f _Z (z ₁) dz ₁.

           -¥

Ahumada and Beard (1998) used a least squares search method to solve for g as a function of the probability of agreement for both responses ( p_AS0 + p_AS1 ) and p _S0, but it is sufficient to solve for the probability of agreement for either one of them.

Estimating observer noise using a model classifier

Ahumada and Beard (1998) also derived an estimate for g based on the assumption that z ₁ comes from a known model. This allows falsification of the model if the estimates of g are not consistent.

One estimate is based on the ratio of the performance of the model, d ₀', to that of the observer, d _H' . From above we have

d _H' = a d ₀' ,

so

a = d _H' / d ₀' .

and

g = ((d ₀' / d _H')) ² - 1) ^0.5.

Another estimate comes from the trial by trial agreement between the observer and the model.

For a given value of z₁ , the event M ₀ that the model will respond that the signal was S ₀ when it actually was S _S will occur if z ₁ is less than the criterion for the model,

b _MS = Z(p_MS0),

which is b ₀ if s = 0 or b ₀ - d ₀' if s = 1. The probability that the observer and the model agree on this response will be

b _MS

Pr{R ₀, M ₀} = ò F _Z((b _H (1 + g² ) ^0.5 - z ₁) / g) ² f _Z (z ₁) dz ₁.

                     -¥

Note that the other possibilities can be computed from this and simpler ones, from the relationships

Pr{R ₀, M ₀}+ Pr{R ₀, M ₁} = Pr{R ₀} ,

Pr{R ₀, M ₀}+ Pr{R ₁, M ₀} = Pr{M ₀} ,

and

Pr{R ₁, M ₁}+ Pr{R ₀, M ₁} = Pr{M ₁} .

Examples of Model Testing based on Noise Estimates

Now we show two tests of the hypothesis that the observer is a noisy version of a particular model. The particular model we will test is a linear classifier, whose classification image is similar to that of the observers. For the first test we compare predictions of a ² from observer self agreement with predictions of a ² from detection performance, where the proportion of observer variance from external noise a ² is then given by the square of the ratio of the observer d _H' to the model d ₀',

a ² = (d _H' / d ₀' ) ² .

Error bars for the self-agreement estimates were computed by generating 95% confidence limits for the proportion of self-agreement and then computing the corresponding noise proportion. These proportions were based on small samples compared to those used to compute the d's. The results for the observers with the highest d's, PW and DF indicate that if they were noisy versions of the model, they should agree more with themselves considering how well they do, so they must be using some other measure. Observer BLB performance and agreement are consistent with her following the model.

Observer-Model Agreement Example

For a given level of internal noise, the formulas above predict the probability of agreement of the noisy model (observer) and the noise free model. The same model as above was hypothesized and the level of the internal noise was based on the ratio of the human observer and the model's d's. An unbiased criterion was used for the model's responses,

b ₀ = 0.5 d ₀'.

The figure plots the ratio of the amount the observed agreement exceeds the amount of agreement expected for independent proportions to the amount the predicted agreement exceeds the independence prediction.

(

That is, the observer-model agreement proportions are converted to the fraction of the distance from the independence prediction to the model prediction. The error bars are 95% confidence limits. The intervals for PW and DF are bounded away from 1, so they are not consistent with the model by this measure either, while BLB remains consistent.

References

C.K. Abbey, M.P. Eckstein, F.O. Bochud (1999) Estimation of human-observer templates for 2 alternative forced choice tasks, Proc. SPIE (E.A. Krupinski, Ed.) 3663:284-295.

C.K. Abbey, M.P. Eckstein (2000) Estimates of human-observer templates for simple detection tasks in correlated noise, Proc. SPIE (E.A. Krupinski, Ed.) 3981, 70-77.

C.K. Abbey, M.P. Eckstein (2001a) Maximum-likelihood and maximum a-posteriori estimates of human-observer templates, Proc. SPIE (E.A. Krupinski and D.P. Chakraborty, Eds.) 4325.

C.K. Abbey, M.P. Eckstein (2001b) Theory for estimating human-observer templates in two-alternative forced-choice experiments, Proc. 17th International Conference on Inferential Processes in Medical Imaging (M.F. Insana and R. Leahy, Eds.) Springer-Verlag, Berlin.

A.J. Ahumada, Jr. (1996) Perceptual classification images from vernier acuity masked by noise, Perception 26 (ECVP Suppl.), 18 (abstract). HTML

A.J. Ahumada, Jr., B.L. Beard (1998) Response classification images in vernier acuity, Investigative Ophthalmology and Visual Science 39 (4, ARVO Suppl.) S1109 (abstract). HTML

A.J. Ahumada, Jr., B.L. Beard (1999) Response Correlation Functions from Multiple Sensor Models, Psychonomics Society Annual Meeting, Los Angeles, CA. HTML

Albert J. Ahumada Jr., Bettina L. Beard (1999) Classification Images for Detection, Investigative Ophthalmology and Visual Science 40 (4, ARVO Suppl.), S572 (abstract). HTML

A.J. Ahumada, Jr., J. Lovell (1971) Stimulus features in signal detection, Journal of the Acoustical Society of America 49, 1751-1756.

A.J. Ahumada, Jr., R. Marken, A. Sandusky (1975) Time and frequency analyses of auditory signal detection, Journal of the Acoustical Society of America 57, 385-390.

E. Barth, B.L. Beard, A.J. Ahumada, Jr. (1999) Nonlinear features in vernier acuity, in Human Vision and Electronic Imaging III, ed. B.E. Rogowitz and T.N. Pappas, Proc. Vol. 3644, pp.88-96, SPIE, San Jose, CA. HTML

B.L. Beard, A.J. Ahumada, Jr. (1997) Relevant image features for vernier acuity, Perception 26, 38 (abstract).

B.L. Beard, A.J. Ahumada, Jr. (1998) Technique to extract relevant image features for visual tasks, in Human Vision and Electronic Imaging III, ed. B.E. Rogowitz and T.N. Pappas, Proc. Vol., 3299, pp.79-85, SPIE, San Jose, CA. HTML

B.L. Beard, A.J. Ahumada (2000) Response Classification Images for Parafoveal Vernier Acuity, Investigative Ophthalmology and Visual Science 41 (4, ARVO Suppl.), S804 (abstract). HTML

D.C. Edwards, M.A. Kupinski, R.M. Nishikawa, C.E. Metz (2000) Estimation of linear observer templates in the presence of multi-peaked gaussian noise through 2AFC experiments, Proc. SPIE (E.A. Krupinski, Ed.) 2981, 86-96.

J. Gold, P.J. Bennett, and A.B. Sekuler (1999) Identification of band-pass filtered letters and faces by human and ideal observers, Vision Research 39(21), 3537-3560.

J.M. Gold, R.F. Murray, P.J. Bennett, and A.B. Sekuler (2000) Deriving behavioral receptive fields for visually completed contours, Current Biology 10(11), 663-666.

J.M. Gold, R.F. Murray, A.B. Sekuler, P.J. Bennett (2000) Snapshots of visual completion, Investigative Ophthalmology and Visual Science 41(4), S317.

K. Knoblauch, J.P. Thomas, M. D'Zmura (1999) Feedback temporal frequency and stimulus classification, Investigative Ophthalmology and Visual Science 40(4), 4171.

K. Knoblauch, R. Yssaad-Fesselier (2000) Stimulus features used during learning of an unknown visual signal, Investigative Ophthalmology and Visual Science 41(4), S46.

D.G. Pelli, B. Farell (1999) Why use noise? Journal of the Optical Society of America A 16(3), 647-653.

D.L. Ringach (1998) Tuning of orientation detectors in human vision, Vision Research 38(7), 963-972.

J.A. Solomon (2000) A picture of orientation discrimination, Investigative Ophthalmology and Visual Science 41(4 ARVO Suppl.), 4241 B188 S1 (abstract).

J.A. Solomon (2000) Channel selection with non-white-noise masks, Journal of the Optical Society of America A, 17(6), 986-993.

J.A. Solomon, M.J. Morgan (1999) Reverse correlation reveals psychophysical receptive fields, Investigative Ophthalmology and Visual Science 40(4 ARVO Suppl.), S572 (abstract).

A.B. Watson, R. Rosenholtz (1997) A Rorschach test for visual classification strategies, Investigative Ophthalmology and Visual Science 38(4 ARVO Suppl.), S1 (abstract).