Probability generating functions

Probability generating functions are a useful tool for studying discrete random variables, taking values in \(n = 0, 1, 2 ...\). Each probability mass function has a unique probability generating function and vice versa. The moments of a random variable can be obtained straightforwardly from its probability generating function. Probability generating functions are useful when dealing with sums and random sums of independent random variables.

Definition

Definition 30 (Generating function)

Given a sequence \(u_0, u_1, ...\), its generating function is

\[\begin{align} \sum^\infty_{n = 0} u_n s^n = u_0 + u_1 s + u_2 s^2 + ... \end{align}\]

for all values of \(s\) for which the converges absolutely.

The generating function is a general definition, that is not specific to probability theory. When the terms of the sequence \(u_0, u_1, ...\) are the values of a probability mass function, then the generating function is called a probability generating function.

Definition 31 (Probability generating function)

Let \(X\) be a random variable on \((\Omega, \mathcal{F}, \mathbb{P})\), which takes values on the non-negative integers and let \(p_n = \mathbb{P}(X = n)\). Then the probability generating function (or pgf) of \(X\) is defined as

\[\begin{align} G_X(s) = \sum^\infty_{n = 0} p_n s^n = p_0 + p_1 s + p_2 s^2 + ... \end{align}\]

for all values of \(s\) for which the sum converges absolutely.

Uniqueness of PGFs and examples

One very useful result is that if two random variables have the same pgf, then they have the same pmf - and vice versa.

Theorem 19 (Uniqueness of pgfs)

Let \(X\) and \(Y\) be discrete random variables with probability generating functions \(G_X\) and \(G_Y\). Then

\[\begin{align} G_X(s) = G_Y(s) \text{ for all } s \iff \mathbb{P}(X = k) = \mathbb{P}(Y = k ), \text{ for } k = 0, 1, 2, ... \end{align}\]

One direction of this result follows by the definition of the pgf, whereas the other can be shown by taking the Taylor expansion of \(G_X\) and \(G_Y\), and observing that all coefficients are equal, which shows that the pmfs of \(X\) and \(Y\) are identical.

Bernoulli

If \(X\) has the Bernoulli distribution with parameter \(p\), then

\[\begin{align} G_X(s) = q + ps, \text{ where } q = 1 - p. \end{align}\]

Binomial distribution

If \(X\) has the binomial distribution with parameters \(p\) and \(n\), then

\[\begin{align} G_X(s) = (q + ps)^n, \text{ where } q = 1 - p. \end{align}\]

Poisson distribution

If \(X\) has the binomial distribution with parameter \(\lambda\), then

\[\begin{align} G_X(s) = e^{\lambda(s - 1)}. \end{align}\]

Geometric distribution

If \(X\) has the geometric distribution with parameter \(p\), then

\[\begin{align} G_X(s) = \frac{ps}{1 - qs}, \text{ where } q = 1 - p. \end{align}\]

Negative binomial distribution

If \(X\) has the negative binomial distribution with parameters \(p\) and \(n\), then

\[\begin{align} G_X(s) = \left(\frac{ps}{1 - qs}\right)^n, \text{ where } q = 1 - p. \end{align}\]

Moments

We are often interested in the moments, such as the mean, of a random variable as these summarise certain aspects of its pmf.

Definition 32 (Moment)

The \(k \geq 1\) moment of a random variable \(X\) is the quantity \(\mathbb{E}(X^k)\).

We can easily obtain all moments of a random variable from its pgf, as stated by the following result.

Theorem 20 (Moments from pgf derivatives)

Let \(X\) be a random variable with pgf \(G_X\). Then

\[\begin{align} G_X^{(k)}(1) = \mathbb{E}\left(X(X - 1)...(X - k + 1)\right), \end{align}\]

where the \(G_X^{(k)}\) notation denotes the \(k^{th}\) derivative of \(G_X\).

The above result is useful in computing higher order moments of random variables, by first finding the pgf and taking derivatives.

Sums of independent variables

Theorem 21 (Independence \(\implies\) \(G\) factorises)

If \(X\) and \(Y\) are independent random variables, each taking values on the non-negative integers, then

\[\begin{align} G_{X + Y}(s) = G_X(s) G_Y(s). \end{align}\]

By extension, if \(X_1, X_2, ..., X_n\) are independent random variables, then their sum \(X = X_1 + X_2 + ... + X_n\) has pgf \(G_X(s) = G_1(s)G_2(s)...G_n(s)\). One very useful consequence of the above result is that we can easily find the pmf of a sum of random variables by simply taking the product of (known) pgfs and matching them against other (known) pgfs. For example, by inspecting the example pgfs above, we see that:

• If \(X\) and \(Y\) are independent and binomially distributed with parameters \(p\) and \(n\) and \(p\) and \(m\), then \(X + Y\) is also binomially distributed with parameters \(p\) and \(n + m\).

• If \(X\) and \(Y\) are Poisson distributed with parameters \(\lambda\) and \(\mu\), then \(X + Y\) is also Poisson distributed with parameter \(\lambda + \mu\).

• If \(X\) and \(Y\) are negative binomially distributed with parameters \(p\) and \(n\) and \(p\) and \(m\) respectively, then \(X + Y\) is also negative binomially distributed with parameters \(p\) and \(n + m\).

• If \(X_1, X_2, ..., X_n\) are independent and geometrically distributed, then \(X_1 + X_2 + ... + X_n\) is negative binomially distributed with parameters \(p\) and \(n\).

In some problems of interest, we may have a sum of \(N\) i.i.d. random variables, where \(N\) is itself a random variable. In this case, the following result, called the random sum formula, is very useful.

Theorem 22 (Random sum formula)

Let \(N\) and \(X_1, X_2, ...\) be random variables taking values on the non-negative integers. If \(N\) has pgf \(G_N\) and the \(X_n\) are independent and identically distributed, with pgf \(G_X\), then the pgf of the sum

\[\begin{align} S = X_1 + X_2 + ... + X_N \end{align}\]

has pgf

\[\begin{align} G_S(s) = G_N(G_X(s)). \end{align}\]

Using \(G_S\) we can easily determine the moments of a random sum. The random sum formula is especially useful when studying branching processes.