## Evaluation And Use Of State Estimators

James R. Werti

14.1 Prelaunch Evaluation of State Estimators

14J Operational Bias Determination

### 14.3 Limitations on State Vector Observability

Chapters 10 through 13 have provided the basic analytic foundations of attitude determination and attitude state vector estimation or data filtering. This chapter discusses practical aspects of state estimation and makes specific suggestions for the evaluation and operational use of estimators. There is no optimum procedure for analyzing attitude data, and much of the discussion is necessarily subjective. This chapter is intended as a practical guide, to' be modified as appropriate for the problem under consideration. Section 14.1 describes specific procedures for the prelaunch evaluation and testing of estimators, and Section 14.2 describes the operational use of estimators. Finally, Section 14.3 discusses the limitations on state vector estimation and procedures for determining which state vector elements should be solved for in a particular situation. Although the discussion is as spacecraft independent as possible, the examples used are drawn primarily from systems using horizon sensors and Sun sensors on a spinning spacecraft, such as the CTS spacecraft described in Section 1.1.

### 14.1 Prelaunch Evaluation of State Estimators

Given a functioning estimator or data filter, how do we determine whether it behaves correctly? The two basic requirements for testing estimators are a data simulator and a detailed test plan. Normally, real data is not sufficient for testing because the correct solution is unknown. However, it may be possible to use real data when the attitude has been determined by more accurate, redundant sensors. For example, attitudes based on SAS-3 star tracker data were used to evaluate infrared horizon sensor data [Hotovy, 1976] and a similar procedure is anticipated for MAGSAT [Levitas, el al., 1978]. When a data simulator is used it must be at least as sophisticated (i.e., incorporate as many effects) as the state estimator and preferably more sophisticated.

A test plan for the evaluation of an estimator should be designed by an individual or group other than those who designed the estimator itself; otherwise, that which was overlooked in the design will also be overlooked in the testing. The test should be designed to test each part of the estimator independently and should also test that all of the parts of the estimator work together. Unfortunately, with a complex state estimation system, it is effectively impossible to test all possible combinations of processing options. Therefore, it is important to attempt to identify both combinations that may present problems (e.g., a change in sensors in the middle of a data pass) and those that will normally be used in practice.

There are three levels of testing that may be performed on any state estimator: (1) testing under nominal and contingency mission conditions, (2) testing under conditions specifically designed to identify and isolate problems, and (3) testing under conditions specifically designed to identify system limits. The first type of test is the least severe and is the minimum requirement for any test procedure. There are two major problems with type (1) tests—it is unknown how the estimator will behave under unanticipated (but possible) mission conditions and it is not really known what to expect from nominal mission conditions. The nominal situation is usually sufficiently complex that analytic solutions for the various parameters are not available. Therefore, the only conclusions that can be drawn from tests under nominal or contingency conditions are that the estimator behaves approximately as one would expect or that it obtains the correct answer on "perfect" data within the limits required by the design and the conditions anticipated. Both of these conclusions are weak, although they may be adequate for some purposes.

Type (2) tests eliminate much of the ambiguity by choosing tests specifically designed to identify problems or evaluate performance;al though the conditions chosen may be unrealistic. The usual procedure here is to choose conditions unrealistically simple such that analytic solutions for the observables are available or at least such that the analyst has an intuitive "feel" for the results. For example, horizon sensor modeling might be tested using a spherical Earth model and a circular, equatorial spacecraft orbit; star sensor calibration algorithms might be tested using an evenly spaced, rectangular grid of stars.

Type (3) tests are the most stringent and are the analog of destructive testing in civil engineering because the intent of the test is to determine the limits of system performance. For example, the attitude might be chosen at the celestial pole to determine how the estimator handles coordinate singularities; or unmodeled biases, such as a deliberate orbit error, might be included in the data to determine how well the estimator behaves in the presence of unmodeled systematic variations. The latter test is particularly valuable because unmodeled systematic errors are the practical accuracy limit for most state estimators.

Finally, we describe three specific tests to examine the operational characteristics of state estimators. These are designed to test the statistical computations and resulting uncertainties, the accuracy of observation models, and the accuracy of computed partial derivatives.

Statistical and Uncertainty Tests. For any state estimator, we want to determine whether uncertainties calculated by the estimator truly reflect the variations in the solutions due to the noise on the data. This cao be conveniently done with a data simulator which adds pseudorandom noise to the data based on an álgebraic random number generator. (For a discussion of the characteristics of algebraic pseudorandom number generators, see, for example, Carnahan, el al.y [1969].) A series of 10 to 20 test runs are made under identical conditions including the addition of Gaussian noise to the data; the only difference between the runs should be the seed or starting value used for the algebraic random number generator. The standard deviation of the resulting attitude and bias parameter solutions is then a real measure of the spread in the solutions due to the noise on the data; further, it is fully independent of the statistical computations within the estimator (i.e., the spread is dependent only on the estimator solution and not on its statistics). Therefore, this solution spread can be compared with the uncertainties computed by the estimator to determine the accuracy of these computations. Note that the computed uncertainties should be nearly the same for each test run. This test is particularly effective when the underlying statistical analysis is uncertain, as in the case in which nonlinearities may become important or those in which quantized measurements are assumed continuous.

To evaluate this test we need to know how well the standard deviation, st, of the resulting attitude or bias solutions, x„ measures the actual variance, of, in the state vector solution; that.is, how closely should the observed agree with the value of o, computed by the state estimator. Given n random samples from a normally distributed population, the sf should have a chi-square distribution (see, for example, Freund [1962]) such that where v = n- 1 and (1—2a) is the confidence interval for the results. For example, if we make 10 runs on a simulated data set for which the resulting standard deviation in the 10 values of element i is 0.015 deg, then for a 90% confidence level, we obtain from standard statistical tables Xo.os,9= 16.92 and X0.9j.9= 3.325. Therefore, with 90% confidence, o, lies in the interval (9 x 0.0152/16.92)1/2 = 0.011 deg to 0.025 deg; that is, the computed uncertainty from the estimator should lie in this range 90% of the time.

Observation Model Tests. The best procedure for testing the accuracy of individual models is to execute the state estimator on simulated, noisefree data and closely examine the solutions or solution residuals. The noise in the system is then the unavoidable noise due to machine round off. For the estimator to be functional, this must be below that anticipated for the actual data; nevertheless, it clearly demonstrates any existing differences between simulator models and models incorporated into the estimator, such as slightly different ephemerides or sensor models.

Partial Derivative Tests. The partial derivatives in an estimator may be tested for accuracy by the following method. Set all of the state vector elements except one to their correct, known value in a simulated data set. Set the one element being tested off of the correct value by a small amount, such as 0.1 to 0.01 deg. Assuming that the problem is approximately linear over this small range, the estimator should converge to the correct solution in a single iteration. If the first iteration gives an answer which is significantly in error, then the partial derivatives are probably computed incorrectly. However, even if the partial derivatives are incorrect, the estimator may still converge slowly provided that the sign of the partials is correct. Of course, the estimator will generally operate more efficiently if the values of the partial derivatives are correct. It is also possible that nonlinearities may be important even in a very small region around the correct solution. If this is suspected, then an independent test of the partial derivatives should be sought.

142 Operational Bias Determination

This section is concerned primarily with estimators used for bias determination, a process of state estimation or data filtering that is performed only infrequently during the life of a space mission, to determine various calibration parameters or biases; these will then be fixed at their estimated values and subsequently used for routine attitude determination. Therefore, we are concerned here with how to get the most possible information about biases from a given set of data. In practice, most of the time spent in bias determination is devoted to selecting the data to be processed (ie*. selecting the data base to be used and eliminating individual samples) and verifying the solution in as many ways as possible.

Operational bias determination is governed by both mission requirements and time constraints. State estimators are used to solve for attitude and biases simultaneously either to find the best estimate of bias parameters or to find the best attitude estimate to be used in conducting maneuvers. The estimation process may be divided into the following two or three steps: (1) a rapid preliminary state vector estimate to provide backup in case of software failure or unanticipated timeline changes (this step is necessary only in real-time analysis); (2) determination of the best possible attitude and bias parameters; and (3) validation of the results, possible revision of the answer, and evaluation of the uncertainties.

To carry out the above program requires a systematic and predesigned procedure to accomplish each step and to record mission parameters and the results of the various tests performed. Recordkeeping is particularly important in areas such as the spacecraft control environment, in which large quantities of data are ordinarily processed and reaccessing data may be difficult or time consuming.

Selecting Data Sets for Processing. To obtain the best possible estimate of bias parameters, we would like to obtain data bases* with the greatest possible information content. To do this, we use the procedures described in Section 14.3 and Chapter 11 to evaluate the information content and correlations for the various data sets that will be available or could become available. Conclusions are then tested on simulated data to establish a formal operational procedure.

An advantage of the geometrical analyses of Chapter 11 and Section 14.3 is that many potential data sets may be evaluated quickly with minimal computer support Consequently, it may be possible to evaluate many possible geometrical conditions for bias determination or sensor calibration. With this information, data collection can be planned specifically for sensor evaluation. For example, after firing the apogee boost motor for SMS-2, GOES-1, and CTS to put the spacecraft into a near synchronous orbit, an attitude maneuver was required to bring the spin axis to orbit normal. This large maneuver was accomplished by a series of smaller maneuvers with intermediate attitudes, or stops, used to obtain one orbit of data for sensor evaluation [Chen and Wertz, 1975; Wertz and Chen, 1976; Tandon, et al., 1976]. The steps were chosen by evaluating the different available geometries along the maneuver path.

Although making the best use of existing maneuvers or normal geometry is the most efficient procedure for sensor evaluation, it may be appropriate to use the geometrical analyses to plan specific calibration maneuvers or maneuvers designed explicitly to calibrate or evaluate hardware performance. Small attitude maneuvers are frequently used with gas jet systems to evaluate the approximate performance

*Data base in this context is any collection of data that can be processed together through a state estimator.

of the jets before using them for long maneuvers. (See, for example, Werking, et al., [1974].) Attitude sensor calibration maneuvers were used on AE-4 and -5 to measure the sensor biases early in each mission to provide accurate deterministic attitudes and are planned for SMM for the calibration of gyros and Sun and star sensors [Branchflower, et al., 1974],

Sample Operating Procedure. Having obtained data suitable for state vector estimation, we would like to process this data to obtain the most information possible. We describe here a sample procedure for spinning spacecraft used with some variations for launch support for CTS; for spacecraft in the AE, SMS, and GOES series; and anticipated for upcoming missions [Chen and Wertz, 1975; Tandon, et al., 1976; Wertz, et al., 1975]. This procedure is based on a processing system similar to the CTS system described in Section 21.2, incorporating a deterministic processor, a bias determination subsystem (normally, a state estimator), and a means for comparing predicted and observed data. The operating procedure which has proved successful in these cases is as follows:

Obtain a Preliminary Solution:

1. Manually select data or check the automatic data selection process.

2. Use the deterministic processor to provide an unbiased initial estimate for the bias determination subsystem.

3. Use the bias determination subsystem to obtain a preliminary state vector estimate based on prelaunch selection of the parameters to be solved for. (See Section 14.3.)

Obtain the Best A vailable Solution:

4. Use an iterative procedure to eliminate data anomalies, as described in Section 9.4.

5. Again, use the deterministic processor on the final set of selected data to provide an unbiased estimate for the bias subsystem.

6. Obtain a converged solution with the state estimator. Convergence should be based on the convergence of the state vector elements themselves, rather than on statistical tests which may give unrealistic results if the problem is nonlinear, if some of the measurements are quantized, or if the estimator includes the option of data rejection. To test for convergence, the change in each of the state vector elements may be required to be below a predetermined level (e.g., 0.001 deg for angular parameters) or the change may be required to be some fraction (e.g., 0.1 or less) of the change on the preceding iteration. As discussed in Section 14.3, a solution should be obtained which provides the maximum number of state vector elements which can be solved for.

Test the Solution. Four classes of tests are available:

7. Use the bias parameters in the deterministic processor to verify that the systematic errors in the deterministic solutions have been reduced. (See Figs. 11-29, 11-30, and 9-18.)

8. Determine the consistency of the results by comparing the predicted and the observed data. Note any systematic behavior of the residuals. (See Figs. 9-17 through 9-19.)

9. Add known biases to the real data and see if they can be recovered by the same procedure which was used to obtain the final answer.

10. To ensure that the estimator is operating in a "linear" region, try different initial estimates for the state vector parameters and check to see that the estimator converges to the same answer.

Discussion. Typically, state estimation accuracies are limited by systematic rather than statistical errors because when statistical errors dominate, more data is normally available that could be processed to further reduce the statistical error. Therefore, it is important to observe and record the level of systematic variations and to attempt to estimate the systematic uncertainties. The uncertainties computed by the state estimator will be unrealistically low because they will account only for the statistical variation.

One procedure for estimating the systematic uncertainties in a given set of bias parameters is to examine the scatter among independent determinations. Here, the estimator is used to analyze data taken with different orbits and different geometries, but from the same sensors. The results from the different data sets may then be statistically combined to obtain improved estimates, of attitude parameters and their uncertainties. (See, for example, Chen and Wertz [1975].)

Although the above procedure provides the most unambiguous quantitative results, truly "independent" data sets are rarely, if ever, available. Therefore, the resulting uncertainties may still be unrealistically low. We may test further for systematic uncertainties by using an a priori knowledge of unmodeled biases such as errors in the orbit, the Earth model, or unobservable sensor biases. The unmodeled biases are applied to simulated data, the estimator is used to solve for the state vector parameters, and the uncertainties are estimated by the amount that the computed state differs from the known values.

A final procedure for testing the level of systematic errors directly is to apply an unmodeled bias to the data and to adjust the magnitude of the bias to give systematic residuals of the same magnitude as those observed in the real solutions. The amount by which the state vector elements are changed by this systematic error is then an estimate of the parameter uncertainty, due to the unknown systematic error. The "unmodeled" bias here may be conveniently chosen as one of the state vector elements which is not solved for during the test.

None of the above methods for determining a realistic uncertainty estimate is completely satisfactory, and the procedures for obtaining the best attitude state vector estimate cannot be quantised. Thus, there is still a need for sophisticated judgment on the part of the operator of a state estimator. The most effective operational procedure for attitude determination is the use of a deterministic processor for continuous, routine attitude determination with the occasional use of a state estimator to determine biases and evaluate possible systematic errors.

### 143 Limitations on State Vector Observability

In general, a good state estimator should provide more state vector elements than can be solved for simultaneously with any real data pass; if not, there is insufficient flexibility in the estimator. Thus, the first requirement for analysis with any estimator—whether in prelaunch testing or operational use—is to correctly choose the state vector elements to be solved for. In principle, this is a straightforward matter based x>n the estimator covariance matrix; any set of elements for which the off-diagonal terms in the covariance matrix remain small can be resolved by the estimator. In practice, model inadequacies, nonlinearities, and data anomalies frequently make the covariance matrix difficult to interpret and, therefore, of very limited use in choosing the solved-for parameters.

Before discussing practical procedures for choosing state vector elements, we distinguish three types of state vector elements: (I) the attitude itself, which is the basic parameter to be determined; (II) biases which may be represented by a deviation in one particular measurement type—for example, a bias in the Sun angle, an azimuth bias of the Sun sensor relative to a horizon sensor, or a magnetometer bias; and (III) biases which are not associated with any one particular measurement, for example, a bias on the angular radius of the Earth or any orbit parameter such as an orbital in-track error.

If the state vector element being considered consists only of the attitude and constant type 11 biases, it is sufficient to determine an accurate attitude one time because if the attitude is known, the biases are determined by the various sensor measurements. In addition, correlations among type II biases are the same as those among their respective measurements. Therefore, the analysis of the information content of measurements also indicates a correlation among biases. For example, a correlation between the Sun angle and the Sun-to-Earth rotation angle measurement, as described in Section 11.4, implies a correlation between the Sun angle bias and azimuth bias. In contrast, type III biases cannot be analyzed by the general procedure for type II biases and must be treated individually.

In general, we would like to solve for as many parameters as possible, so long as a meaningful, converged solution can still be obtained. We would like to know beforehand which combinations of parameters are uncorrelated and therefore can be resolved. As a general hypothesis, we suggest that the greater the variation in the geometry, the larger the number of parameters which can be solved for. Repeated measurements under the same geometrical conditions, for example, measuring the Earth width by a spacecraft in a circular orbit with its spin axis at orbit normal, do not provide new information with which to distinguish various sensor biases; such as deviations in the mounting angle of the sensor, the angular radius of the Earth, or the semimajor axis of the spacecraft orbit.* Repeated measurements under the same geometrical conditions serve only to reduce the statistical noise, but this is of limited practical value because the uncertainties are normally dominated by systematic rather than statistical errors. However, if the geometrical conditions change (for example, if the orbit is noncircular or the attitude is not at orbit normal), each measurement provides new information and more parameters may be solved for. Thus, state vector estimation is best done with data which incorporate the widest possible variety of geometrical relationships among the attitude reference vectors.

To determine explicitly which state vector elements are observable or distinguishable, four procedures have been found to be useful and have been successfully applied to the analysis of real spacecraft data: (1) trial and error with

•Although biases may be indistinguishable in the instance cited, a full orbit of data may determine the attitude very precisely. The importance of determining the biases depends on whether it is the attitude at-the time of die data pass which is important, or whether the biases themselves are needed so that accurate attitudes may be determined at a later time when less or different data are available.

simulated data, (2) analysis of correlations among different measurement types (for type II state vector elements which are correlated with specific observations), (3) analysis of the information content of a single measurement type, and (4) analysis of geometrical procedures which allow a particular parameter to be resolved.

1. Trial-and-Error Procedures. In a trial-and-error procedure, we use the state estimator to process real or simulated data, trying various combinations of state vector elements to determine which combinations give converged solutions. In the case of simulated data, we may also determine which combinations return approximately the correct answer. This procedure is practically useless as a general analytic technique for more than 5 to 10 parameters because of the many possible combinations of state vector elements and processing options. Also, it is difficult to obtain any general insight from trial-and-error analysis which can be applied to conditions different from those tested. Nevertheless, trial and error provides a procedure for testing analytic results obtained by other methods, and may be the most economical procedure for single applications. It serves as the basic test for other techniques and all analytic conclusions should be subsequently tested, so far as possible, first on simulated data and then on real data. Each of the procedures presented below has been tested on both real and simulated data.

2. Correlations Among Different Measurement Types. If two measurements are correlated over some region, the associated type II biases are also correlated over that region and are difficult to distinguish. Therefore, the analysis of Chapter 11 concerning correlations between measurement types may be applied directly to correlations between type II biases. For example, Fig. ll-26(a) indicates that for the SMS-2 spacecraft in its transfer orbit to geosynchronous altitude, the Sun angle/nadir angle correlation angle was near zero during the entire coverage of the Earth by horizon sensor 2. Therefore, the Sun angle and nadir angle measurements are providing nearly the same information about the attitude, and their associated biases are strongly correlated. Thus, we expect that a Sun angle bias is difficult to distinguish from a sensor mounting angle bias* with these data. This conclusion, and similar conclusions with data from other sensors, was confirmed through the analysis of both real and simulated SMS-2 data [Chen and Wertz, 1975].

Although this procedure can provide information on the correlation of biases quickly and easily, it is limited to type II biases and is also limited to data spans over which the geometry does not change greatly. As the data span becomes longer, such that the correlation is changing, the interpretation in terms of correlated and uncorrelated biases is less clear. Procedure 3 is concerned specifically with the changing correlation over long data spans.

3. Information Content of a Single Measurement Type. Procedure 2 is only applicable to correlations at one point or over a small region; however, large data passes are normally needed for bias determination. Therefore, we would like to examine the information content of any single type of attitude measurement by determining the correlation angle for measurements of the same type at the

*Note that although a sensor mounting angle bia3 and a true nadir angle bias are not identical, the distinction is not critical in this case. Both biases shift the computed attitude in the same direction, but the relative amount of the shift is not linearly related for the two biases.

beginning and the end of (or throughout) a long data pass. If this correlation angle is small and if the possibility of a bias in the measurement exists, there is little information content in the measurement Conversely, if the correlation angle is large, even in the presence of a constant bias on. the measurement, it may be possible to use that measurement to determine both the attitude and the corresponding type II bias.

Figure 14-1 illustrates the qualitative physical basis for this interpretation for the case of a cone angle measurement for which the possible attitude loci are small circles on the celestial sphere. For concreteness, we assume that the measurement is a direct measure of the nadir angle and that the +'s are the orientations of the Earth (i.e., the centers of the various nadir cones) at the beginning and end of a data pass. The solid line corresponds to the possible attitudes assuming that there is no bias in the measurement, Tlie dashed line corresponds to the possible attitudes if there is a bias Aij in the nadir angle measurement. The correlation angle, between the nadir angle measurement at the beginning (time f,) and end (time tj of the data pass is just the angle of intersection of the two nadir cones. In Fig. 14-1(a), this correlation angle is small. Therefore, the horizontal component of the attitude is poorly defined. However, if there is the possibility of a nadir angle bias, then the radius of the cone is unknown and the vertical component of the attitude is also poorly defined. In this example we have, in effect only one measurement and one potential bias and, therefore, this measurement provides no information about the attitude. In practice, the bias is not totally unknown and may normally be expected to fall within some assumed limits. However, in the logic of the state

NADIR CONB3 NADIR CONE,

EARTH, EARTHj

4 ♦ Ja) ATTITUDE UNDETERMINED

(b> ATTITUDE LIES ALONG DOTTED LINE

## Post a comment