---
title: "Counts to Effort: How the Estimate and its Error Are Built"
output:
  rmarkdown::html_vignette:
    highlight: null
vignette: >
  %\VignetteIndexEntry{Counts to Effort: How the Estimate and its Error Are Built}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

This page explains what `estimate_effort()` computes from angler counts and
why its standard error has the parts it has. It is for readers who know their
survey and want to see the arithmetic behind the numbers. For the steps of an
analysis, see `vignette("tidycreel")`.

The chain has three steps, and each section below takes one:

```
one count  ->  one day's effort  ->  stratum total  ->  standard error
              (count x hours)       (N_h / n_h x sum)   (between days, within days)
```

```{r load, message = FALSE}
library(tidycreel)
```

## The sampling unit is the day

A creel season is divided into **strata**, groups of days expected to be
alike, such as weekdays and weekend days. Within a stratum, some days are
sampled and the rest are not. Each day belongs to exactly one stratum, and
the day is the **primary sampling unit** (PSU): it is what was drawn at random,
so it is what the standard error is computed over.

Write $N_h$ for the number of days in stratum $h$ and $n_h$ for the number
sampled. Everything below happens within one stratum. Strata are estimated
separately and added.

## From a count to a day's effort

An **instantaneous count** is the number of anglers fishing at one moment. It
is a number of anglers, not an amount of effort. If the count was taken at a
random moment in a fishing period of length $T$ hours, the count times $T$ is
an unbiased estimate of the angler-hours in that period (Hoenig et al. 1993):

$$\hat{e}_i = c_i \times T_i$$

tidycreel makes this multiplication when you name the column holding $T$ with
`period_length_col`, and reports the result in angler-hours. Without it, the
estimate is the counts expanded but not multiplied, its unit is `NA`, and
`estimate_effort()` warns that no period length was supplied. tidycreel never
assumes a unit it was not given.

## From sampled days to the stratum total

The stratum total is the number of days times the mean daily effort on the
sampled days:

$$\hat{E}_h = N_h \, \bar{e}_h = \frac{N_h}{n_h} \sum_{i=1}^{n_h} \hat{e}_i$$

$N_h / n_h$ is the expansion factor: when one day in two was sampled, each
sampled day stands for two.

Here is a stratum of 8 weekdays, 4 of them sampled, with one count per day on
a 10-hour fishing day:

```{r one-count}
calendar <- data.frame(
  date     = as.Date("2024-06-03") + 0:7,
  day_type = "weekday"
)
counts <- data.frame(
  date      = as.Date("2024-06-03") + 0:3,
  day_type  = "weekday",
  n_anglers = c(15L, 25L, 35L, 45L),
  hours     = 10
)

design <- creel_design(calendar, date = date, strata = day_type)
design <- add_counts(design, counts, period_length_col = hours)
estimate_effort(design, target = "period_total")
```

By hand: daily effort is 150, 250, 350 and 450 angler-hours, a mean of 300,
and $\hat{E} = 8 \times 300 = 2{,}400$ angler-hours.

`target = "period_total"` asks for the total over all 8 days. The default,
`target = "sampled_days"`, is the total over the 4 days counted (1,200 here);
`vignette("tidycreel")` shows the two side by side.

## The standard error: variation between days

The estimate is uncertain because a different draw of days would have given a
different mean. That uncertainty is estimated from how much the sampled days
differ from each other. $s_b^2$, the sample variance of the daily efforts
$\hat{e}_i$, measures it, and with one count per day the variance of the
stratum total is

$$\widehat{\text{Var}}_{\text{between}}(\hat{E}_h) = N_h^2 \, \frac{s_b^2}{n_h}$$

For the stratum above, $s_b^2 = 16{,}667$, so

$$\widehat{\text{Var}} = 64 \times \frac{16{,}667}{4} = 266{,}667,
\qquad \text{SE} = 516.4$$

which is the `se` and `se_between` printed above.

### Why there is no finite-population correction here

Days are drawn without replacement, and survey texts usually multiply this
variance by the **finite-population correction** $1 - n_h/N_h$: sample half
the stratum and half the day-to-day uncertainty is gone; sample all of it and
none is left. That reasoning covers only one of the two reasons a day's
estimate is uncertain. The other is **when in the day the count fell**: a
count at another moment would have given a different number, and sampling more
days does nothing about that. $s_b^2$ carries both. Multiplying it by
$1 - n_h/N_h$ shrinks both, and with every day sampled it would report an SE
of 0 although each count still fell at a random moment.

The two-stage variance below puts the within-day part back, but it needs at
least two counts on a day to estimate it. So tidycreel applies the correction
to a stratum only when every sampled day in it has two or more counts. With
one count per day it leaves the correction out, which is conservative: the
standard error is somewhat too large rather than too small (Cochran 1977,
equation 10.42). Up to tidycreel 8.0.0 the correction was applied in every
case, and single-count standard errors were too small (#505).

### One count per day: `se_within` is 0, and what that means

With one count per day, `se_within` is reported as 0. That does **not** mean
the number of anglers was steady through the day: $s_b^2$ contains the
within-day variation, but with one count per day it cannot be separated out.
The 0 is a placeholder for "not estimated"; #504 proposes reporting `NA`
there instead.

## Several counts per day: two-stage variance

When a day has $k$ counts at random times, the day's effort is estimated from
their mean, and the day is now itself a sample: of the moments in the day, $k$
were observed. That makes the design **two-stage** (days, then moments within
days), and the variance has a term for each stage (Cochran 1977, chapter 10):

$$\widehat{\text{Var}}(\hat{E}_h) =
\underbrace{N_h^2 \left(1 - \frac{n_h}{N_h}\right) \frac{s_b^2}{n_h}}_{\text{between days}}
\; + \;
\underbrace{N_h \, \frac{s_w^2}{k}}_{\text{within days}}$$

- $s_b^2$ is the sample variance of the daily mean efforts.
- $s_w^2 = \dfrac{\sum_i \sum_j (\hat{e}_{ij} - \bar{e}_i)^2}{n_h (k - 1)}$ is
  the within-day variance, pooled over the sampled days.

The within-day term is the second-stage term of Cochran's two-stage variance,
$f_1 (1 - f_2) s_2^2 / (nm)$ scaled to a total, with the second-stage sampling
fraction $f_2$ taken as zero because a day contains effectively unlimited
moments. Pope et al. (Chapter 17 draft) give the same two terms for creel
counts. `se_between` and `se_within` are the square roots of the two terms, and
`se` the square root of their sum.

**What within-day variance measures.** It is how much the number of anglers
present changes over the day: a morning bite and an empty afternoon give a
large $s_w^2$. It is not observer error; an exact count at a different moment
would still differ.

The same stratum, now with a morning and an afternoon count on each sampled
day. The daily means are the same as before (150, 250, 350, 450
angler-hours), so the between-day term does not change:

```{r two-counts}
counts_2 <- data.frame(
  date       = rep(as.Date("2024-06-03") + 0:3, each = 2),
  day_type   = "weekday",
  count_time = rep(c("am", "pm"), 4),
  n_anglers  = c(10L, 20L, 20L, 30L, 30L, 40L, 40L, 50L),
  hours      = 10
)

design_2 <- creel_design(calendar, date = date, strata = day_type)
design_2 <- add_counts(design_2, counts_2,
  count_time_col = count_time, period_length_col = hours
)
estimate_effort(design_2, target = "period_total")
```

By hand: each day's two efforts differ by 100 angler-hours, so each day's sum
of squares is $2 \times 50^2 = 5{,}000$ and
$s_w^2 = 4 \times 5{,}000 / (4 \times 1) = 5{,}000$. The within-day variance is
$8 \times 5{,}000 / 2 = 20{,}000$, an SE of 141.4. The total SE is
$\sqrt{133{,}333 + 20{,}000} = 391.6$.

The formula assumes every sampled day in a stratum has the same number of
counts. Days with different numbers of counts are open as #405.

### More days or more counts per day?

The two terms show where the uncertainty comes from, but not, on their own,
what to change. Adding sampled days reduces all of the variance, the
day-to-day part and the within-day part; adding counts per day reduces only
the within-day part, and is usually cheaper than adding a day. The best
split depends on both the variance components and the cost of a day versus
the cost of a count; Cochran (1977, section 10.6, equation 10.26) gives the
optimum number of counts per day. There is
no fixed ratio of `se_within` to `se` beyond which one choice wins.

## Progressive counts

A **progressive count** is a different field method. The observer travels the
whole section once (a **circuit**, taking $\tau$ hours) and counts every angler
passed. The day's effort is

$$\hat{e}_d = C_d \times T_d$$

where $C_d$ is the circuit's count and $T_d$ the hours the fishery is open that
day.

**Why $\tau$ does not appear.** Follow Hoenig et al. (1993, p. 725). The circuit
samples one $\tau$-hour block of the day, so the count expands over its own
duration first: $C_d \times \tau$. The day is made of $K = T_d / \tau$ such
blocks, and the circuit's block was drawn at random from them, so the day's
effort is

$$\hat{e}_d = C_d \times \tau \times \frac{T_d}{\tau} = C_d \times T_d$$

$\tau$ enters twice, once expanding the block and once counting the blocks, and
cancels. It has already done its work in **defining the blocks the circuit was
scheduled within**, which is why `generate_progressive_start()` needs it and
the final arithmetic does not. `add_counts()` still requires it: a fishing
period shorter than one circuit ($K < 1$) is an error, since no circuit could
finish, and $\tau = T_d$ ($K = 1$) is the legitimate all-day circuit of Robson
(1961).

Six weekdays, three sampled, a 2-hour circuit on an 8-hour day:

```{r progressive}
calendar_p <- data.frame(
  date     = as.Date("2024-06-03") + 0:5,
  day_type = "weekday"
)
prog_counts <- data.frame(
  date        = as.Date("2024-06-03") + 0:2,
  day_type    = "weekday",
  n_anglers   = c(15L, 20L, 25L),
  shift_hours = 8
)

design_p <- creel_design(calendar_p, date = date, strata = day_type)
design_p <- add_counts(design_p, prog_counts,
  count_type = "progressive",
  circuit_time = 2,
  period_length_col = shift_hours
)

design_p$counts
estimate_effort(design_p, target = "period_total")
```

`add_counts()` stores each day's $C_d \times T_d$: 120, 160 and 200
angler-hours. The total is $6 \times 160 = 960$ angler-hours. One circuit a day
gives one estimate a day, so the variance is the between-day term alone, as for
one instantaneous count a day, and for the same reason it carries no
finite-population correction.

The estimate is unbiased only when the circuit's start time, start point and
direction were randomised and the observer moved faster than the anglers.
None of that can be checked from the counts table;
`vignette("progressive-count-surveys")` gives the field conditions.

## Summary

| Data | Day's effort | Variance terms | `se_within` |
|---|---|---|---|
| One instantaneous count per day | $c_i T_i$ | between days, no FPC | 0: not estimated (#504) |
| $k$ instantaneous counts per day | mean of $c_{ij} T_i$ | between days + within days | estimated |
| One progressive circuit per day | $C_d T_d$ | between days, no FPC | 0: not estimated (#504) |

In every case the stratum total is $N_h$ times the mean daily effort and
strata are added. The between-day term carries the finite-population
correction only when every sampled day in the stratum has two or more counts,
because only then is the within-day variation it would remove put back.

## References

- Cochran, W. G. (1977). *Sampling Techniques*, 3rd edition. Wiley, New York.
  Chapter 10, subsampling with units of equal size.
- Hoenig, J. M., Robson, D. S., Jones, C. M., and Pollock, K. H. (1993).
  Scheduling counts in the instantaneous and progressive count methods for
  estimating sportfishing effort. *North American Journal of Fisheries
  Management*, 13, 723–736.
- Pope, K. L., et al. Creel surveys. Chapter 17 (unpublished draft).
- Robson, D. S. (1961). On the statistical theory of a roving creel census of
  fishermen. *Biometrics*, 17, 415–437.