All figures and data are obtained from the California Lottery.

____________________________________________________
Fantasy 5

The following figures show a playslip and a sample ticket for the game of Fantasy 5.

Figure 1

Figure 2

In the game of Fantasy 5, the player chooses 5 numbers from 1 to 39. If all 5 chosen numbers match the 5 winning numbers, the player wins the top prize which starts at $50,000 and can go up to $500,000 or more. The odds of winning the top prize are 1 in 575,757. There are lower tier prizes that are easier to win but with much lower winning amounts. The following figure shows the prize categories and the winning odds of Fantasy 5.

Figure 3

All 5 of 5
In matching the player’s chosen numbers with the winning numbers, the order of the numbers do not matter. Thus in the calculation of odds, we use combination rather than permutation. Thus we have:

Based on , the odds of matching all 5 winning numbers is 1 in 575,757 (the odds of winning the top prize).

Any 4 of 5
To match 4 out of 5 winning numbers, 4 of the player’s chosen numbers are winning numbers and 1 of the player’s chosen numbers is from the non-winning numbers (34 of them). Thus the probability of matching 4 out of 5 winning numbers is:

Any 3 of 5
To find the odds for matching 3 out of 5 winning numbers, we need to find the probability that 3 of the player’s chosen numbers are from the 5 winning numbers and 2 of the selected numbers are from the 34 non-winning numbers. Thus we have:

Any 2 of 5
Similarly, the following shows how to calculate the odds of matching 2 out of 5 winning numbers:

____________________________________________________
SuperLOTTO Plus

Here are the pictures of a playslip and a sample ticket of the game of SuperLOTTO Plus.

Figure 4

Figure 5

Based on the playslip (Figure 4), the player chooses 5 numbers from 1 to 47. The player also chooses an additional number called a Mega number from 1 to 27. To win the top prize, there must be a match between the player’s 5 selections and the 5 winning numbers as well as a match between the player’s Mega number and the winning Mega number (All 5 of 5 and Mega in Figure 6 below).

Figure 6

All 5 of 5 and Mega
To find the odds of the match of “All 5 of 5 and Mega”, the total number of possibilities is obtained by choosing 5 numbers from 27 numbers and choose 1 number from 27 numbers. We have:

Thus the odds of matching “All 5 of 5 and Mega” are 1 in 41,416,353.

Any 5 of 5
To find the odds of matching “All 5 of 5″ (i.e. the player’s 5 selections match the 5 winning numbers but no match with the Mega winning number), we need to choose 5 numbers from the 5 winning numbers, choose 0 numbers from the 42 non-winning numbers, choose 0 numbers from the 1 Mega winning number and choose 1 number from the 26 non-Mega winning numbers. This may seem overly precise, but will make it easier to the subsequent derivations. We have:

Any 4 of 5 and Mega
To calculate the odds for matching “any 4 of 5 and Mega”, we need to choose 4 out of 5 winning numbers, choose 1 out of 42 non-winning numbers, choose 1 out of 1 Mega winning number, and choose 0 out of 26 non-winning Mega numbers. We have:

Any 4 of 5
To calculate the odds for matching “any 4 of 5″ (no match for Mega number), we need to choose 4 out of 5 winning numbers, choose 1 out of 42 non-winning numbers, choose 0 out of 1 Mega winning number, and choose 1 out of 26 non-winning Mega numbers. We have:

Any 3 of 5 and Mega
To calculate the odds for matching “any 3 of 5 and Mega”, we need to choose 3 out of 5 winning numbers, choose 2 out of 42 non-winning numbers, choose 1 out of 1 Mega winning number, and choose 0 out of 26 non-winning Mega numbers. We have:

The rest of the calculations for SuperLOTTO Plus should be routine. It is a matter to deciding how many to choose from the 5 winning numbers, how many to choose from the 42 non-winning numbers as well as how many to choose from the 1 winning Mega number and how many to choose from the 26 non-winning Mega numbers.

Any 3 of 5

Any 2 of 5 and Mega

Any 1 of 5 and Mega

None of 5 only Mega

____________________________________________________
Mega Millions

The following are a playslip, a sample ticket and the winning odds of the game of Mega Millions.

Figure 7

Figure 8

Figure 9

Based on the playslip (Figure 7), the player chooses 5 numbers from 1 to 56. The player also chooses an additional number called a Mega number from 1 to 46. To win the top prize, there must be a match between the player’s 5 selections and the 5 winning numbers as well as a match between the player’s Mega number and the winning Mega number. The calculation of the odds indicated in Figure 9 are left as exercises.

Example 1
Suppose that a student took two multiple choice quizzes in a course for probability and statistics. Each quiz has 5 questions. Each question has 4 choices and only one of the choices is correct. Suppose that the student answered all the questions by pure guessing. Furthermore, the two quizzes are independent (i.e. results of one quiz will not affect the results of the other quiz). Let be the number of correct answers in the first quiz and be the number of correct answers in the second quiz. Suppose the student was told by the instructor that she had a total of 4 correct answers in these two quizzes. What is the probability that she had 3 correct answers in the first quiz?

On the face of it, the example is all about binomial distribution. Both and are binomial distributions (both ). The sum is also a binomial distribution (). The question that is being asked is a conditional probability, i.e., . Surprisingly, this conditional probability can be computed using the hypergeometric distribution. One can always work this problem from first principle using binomial distributions. As discussed below, for a problem such as Example 1, it is always possible to replace the binomial distributions using a thought process involving the hypergeometric distribution.

Here’s how to think about the problem. This student took the two quizzes and was given the news by the instructor that she had 4 correct answers in total. She now wonders what the probability of having 3 correct answers in the first quiz is. The thought process is this. She is to pick 4 questions from 10 questions (5 of them are from Quiz 1 and 5 of them are from Quiz 2). So she is picking 4 objects from a group of two distinct types of objects. This is akin to reaching into a jar that has 5 red balls and 5 blue balls and pick 4 balls without replacement. What is the probability of picking 3 red balls and 1 blue ball? The probability just described is from a hypergeometric distribution. The following shows the calculation.

We will show below why this works. Before we do that, let’s describe the above thought process. Whenever you have two independent binomial distributions and with the same probability of success (the number of trials does not have to be the same), the conditional distribution is a hypergeometric distribution. Interestingly, the probability of success has no bearing on this observation. For Example 1, we have the following calculation.

Interestingly, the conditional mean , while the unconditional mean . The fact that the conditional mean is higher is not surprising. The student was lucky enough to have obtained 4 correct answers by guessing. Given this, she had a greater chance of doing better on the first quiz.

__________________________________________________
Why This Works

Suppose and and they are independent. The joint distribution of and has 36 points in the sample space. See the following diagram.

Figure 1

The probability attached to each point is

where and .

The conditional probability involves 5 points as indicated in the following diagram.

Figure 2

The conditional probability is simply the probability of one of the 5 sample points as a fraction of the sum total of the 5 sample points encircled in the above diagram. The following is the sum total of the probabilities of the 5 points indicated in Figure 2.

We can plug into and work out the calculation. But is actually equivalent to the following because .

As stated earlier, the conditional probability is simply the probability of one of the 5 sample points as a fraction of the sum total of the 5 sample points encircled in Figure 2. Thus we have:

With the terms involving and cancel out, we have:

__________________________________________________
Summary

Suppose and and they are independent. Then is also a binomial distribution, i.e., . Suppose that both binomial experiments and have been performed and it is known that there are successes in total. Then has a hypergeometric distribution.

where .

As discussed earlier, think of the trials in as red balls and think of the trials in as blue balls in a jar. Think of the successes as the number of balls you are about to draw from the jar. So you reach into the jar and select balls without replacement. The calculation in gives the probability that you select red balls and blue balls.

The probability of success in the two binomial distributions have no bearing on the result since it gets canceled out in the derivation. One can always work a problem like Example 1 using first principle. Once the thought process using hypergeometric distribution is understood, it is a great way to solve this problem, that is, you can by pass the binomial distributions and go straight to the hypergeometric distribution.

__________________________________________________
Additional Practice
Practice problems are found in the following blog post.

How to pick binomial trials

Consider the following problem. Suppose that a certain professor is lazy and lets his students grade their own quizzes. After he collects the quizzess from his students, he randomly assigns the quizzes back to these students for grading. If a student is assigned his or her own quiz, we say that it is a match. We have the following questions:

• What is the probability that each of the students is a match?
• What is the probability that none of the students is a match?
• What is the probability that exactly of the students are matches?
• What is the probability that at least one of the students is a match?

The above problem is called the matching problem, which is a classic problem in probability. In this post we solve the last question indicated above. Though the answer is in terms of the total number of quizzes , it turns out that the answer is independent of and is approximately . Thus if the professor assigns the quizzes randomly, it will be very unusual that there is no match.

The last question above is usually stated in other matching situations. One is that there are couples (say, each couple consists of a husband and a wife) in a class for ballroom dancing. Suppose that the dance instructor randomly matches the men to the ladies. When a husband is assigned his own wife, we say that it is a match. What is the probability that there is at least one couple that is a match?

The key to answering this question is the theorem stated in Feller (page 99 in chapter 4 of [1]). We state the theorem and make use of it in the solution of the last question above. A sketch of the proof will be given at the end. For ideas on the solutions to the first three questions above, see this previous post.

The union of events
For any events that are defined on the same sample space, we have the following formula:

where

,

,

Note that in the general term , the sum is taken over all increasing sequence , i.e. . For , we have the following familiar formulas:

The Matching Problem

Suppose that the students are labeled . Let be the even that the student is assigned his or her own quiz by the professor. Then is the probability that there is at least one correct match.

Note that . This is the case since we let the student be fixed and we permute the other students. Likewise, , since we fix the and students and let the other students permute. In general, whenever are distinct integers and are increasing, we have:

We now apply the formula (1). First we show that for each where , . Since there are many ways to have matches out of students, we have:

Applying the formula for the union of events, we have:

Note that the left-hand side of the above equality is the first terms in the expansion of . Thus we have:

The above limit converges quite rapidly. Let . The following table lists out the first several terms of this limit.

Sketch of Proof for the Formula
The key idea is that any sample point in the union is counted in exactly one time in the right hand side of (1). Suppose that a sample point is in exactly of the events . Then the following shows the number of times the sample point is counted in each expression:

Thus the sample point in question will be counted exactly times in the right hand side of the formula.

The following is the derivation that .

Reference

1. Feller, W., An Introduction to Probability Theory and its Applications, Vol. I, 3rd ed., John Wiley & Sons, Inc., New York, 1968

Previous Posts on The Matching Problem
The Matching Problem

More About the Matching Problem

Tis the Season for Gift Exchange

The above example is a description of a classic problem in probability called the matching problem. There are many colorful descriptions of the problem. One such description is that there are married couples in a ballroom dancing class. The instructor pairs the ladies with the gentlemen at random. A match occurs if a married couple is paired as dancing partners.

Whatever the description, the matching problem involves two lists of items that are paired according to a particular order (the original order). When the items in the first list are paired with the items in the second list according to another random ordering, a match occurs if an item in the first list and an item in the second list are paired both in the original order and in the new order. The matching problem discussed here is: what is the probability that there is at least one match? Seen in this light, both examples described above are equivalent mathematically.

We now continue with the discussion of the random gift selection example. Suppose that there are people in the party. Let be the event that the person selects his or her own gift. The event is the event that at least one person selects his or her own gift (i.e. there is at least one match). The probability is the solution of the matching problem as described in the beginning. The following is the probability .

The answer in is obtained by using an idea called the inclusion-exclusion principle. We will get to that in just a minute. First let’s look at the results of for a few iterations.

In a party with random gift exchange, it appears that regardless of the size of the party, there is a very good chance that someone will end up picking his or her own gift! A match will happen about 63% of the time.

It turns out that the answers in are related to the Taylor’s series expansion of . We show that the probability in will always converge to . Note that the Taylor’s series expansion of is:

Consequently, the Taylor’s series expansion of is:

Note that the sum of the first terms of the series in is the probability in . As the number of partygoers increases, the probability will be closer and closer to . We have:

The equation says that it does not matter how many people are in the random gift exchange, the answer to the matching problem is always in the limit. The equation says that the probability of having no matches approaches . There is a better than even chance ( to be more precise) that there is at least one match. So in a random gift exchange as described at the beginning, it is much easier to see a match than not to see one.

The Inclusion-Exclusion Principle

We now want to give some indication why provides the answers. The inclusion-exclusion principle is a formula for finding the probability of the union of events. For the union of two events and the union of three events, we have:

To find the probability of the union of events, we first add up the probabilities of the individual events . Because the resulting sum overshoots, we need to subtract the probabilities of the intersections . Because the subtractions remove too much, we need to add back the probabilities of the intersections of three individual events . The process of inclusion and exclusion continues until we reach the step of adding/removing of the intersection . The following is the statement of the inclusion-exclusion principle.

In , is the sum of all probabilities , is the sum of all possible probabilities of the intersection of two events and and is the sum of all possible probabilities of the intersection of three events and so on.

We now apply the inclusion-exclusion principle to derive equation . The event is the event that the person gets his or her own gift while the other people are free to select gifts. The probability of this event is . There are many ways of fixing 1 gift. So .

Now consider the intersection of two events. The event is the event that the person and the person get their own gifts while the other people are free to select gifts. The probability of this event is . There are many ways of fixing 2 gifts. So .

By the same reasoning, we derive that and and so on. Then plugging into , we obtain the answers in .

The matching problem had been discussed previously in the following posts.

The matching problem

Mote about the matching problem

Consider a counting process that describes the occurrences of a certain type of events of interest in a unit time interval subject to three simplifying assumptions (discussed below). We are interested in counting the number of occurrences of the event of interest in a unit time interval. As a concrete example, consider the number of cars arriving at an observation point in a certain highway in a period of time, say one hour. We wish to model the probability distribution of how many cars that will arrive at the observation point in this particular highway in one hour. Let be the random variable described by this probability distribution. We wish to konw the probability that there are cars arriving in one hour. We start with using a binomial distribution as an approximation to the probability . We will see that upon letting , the is a Poisson probability.

Suppose that we know , perhaps an average obtained after observing cars at the observation points for many hours. The simplifying assumptions alluded to earlier are the following:

1. The numbers of cars arriving in nonoverlapping time intervals are independent.
2. The probability of one car arriving in a very short time interval of length is .
3. The probability of having more than one car arriving in a very short time interval is esstentially zero.

Assumption 1 means that a large number of cars arriving in one period does not imply fewer cars will arrival in the next period and vice versa. In other words, the number of cars that arrive in any one given moment does affect the number of cars that will arrive subsequently. Knowing how many cars arriving in one minute will not help predict the number of cars arriving at the next minute.

Assumption 2 means that the rate of cars arriving is dependent only on the length of the time interval and not on when the time interval occurs (e.g. not on whether it is at the beginning of the hour or toward the end of the hour).

Assumptions 3 allows us to think of a very short period of time as a Bernoulli trial. Thinking of the arrival of a car as a success, each short time interval will result in only one success or one failure.

Assumption 2 tells us that non-overlapping short time intervals of the same length have identical probability of success. Overall, all three assumptions imply that non-overlapping short intervals of the same length are inpdendent Bernoulli trials with identical probability of success, giving us the basis for applying binomial distribution.

To start, we can break up the hour into 60 minutes (into 60 Bernoulli trials). We then consider the binomial distribution with and . So the following is an approximation to our desired probability distribution.

Conceivably, there can be more than 1 car arriving in a minute and observing cars in a one-minute interval may not be a Bernoulli trial. For a one-minute interval to qualify as a Bernoulli trial, there is either no car arriving or 1 car arriving in that one minute. So we can break up an hour into 3,600 seconds (into 3,600 Bernoulli trials). We now consider the binomial distribution with and .

It is also conceivable that more than 1 car can arrive in one second and observing cars in one-second interval may still not qualify as a Bernoulli trial. So we need to get more granular. We can divide up the hour into equal subintervals, each of length . Assumption 3 ensures that each subinterval is a Bernoulli trial (either it is a success or a failure; one car arriving or no car arriving). Assumption 2 ensures that the non-overlapping subintervals all have the same probability of success. Assumption 1 tells us that all the subintervals are independent. So breaking up the hour into moments and counting the number of moments that are successes will result in a binomial distribution with parameters and . So we are ready to proceed with the following approximation to our probability distribution .

As we get more granular, . We show that the limit of the binomial probability in is the Poisson distribution with parameter . We show the following.

In the derivation of , we need the following two mathematical tools. The statement is one of the definitions of the mathematical constant . In the statement , the integer in the numerator is greater than the integer in the denominator. It says that whenever we work with such a ratio of two factorials, the result is the product of with the smaller integers down to . There are exactly terms in the product.

The following is the derivation of .

In , we have . The reason being that the numerator is a polynomial where the leading term is . Upon dividing by and taking the limit, we get 1. Based on , we have . For the last limit in the derivation we have .

We conclude with some comments. As the above derivation shows, the Poisson distribution is at heart a binomial distribution. When we divide the unit time interval into more and more subintervals (as the subintervals get more and more granular), the resulting binomial distribution behaves more and more like the Poisson distribution.

The three assumtions used in the derivation are called the Poisson postulates, which are the underlying assumptions that govern a Poisson process. Such a random process describes the occurrences of some type of events that are of interest (e.g. the arrivals of cars in our example) in a fixed period of time. The positive constant indicated in Assumption 2 is the parameter of the Poisson process, which can be interpreted as the rate of occurrences of the event of interest (or rate of changes, or rate of arrivals) in a unit time interval, meaning that the positive constant is the mean number of occurrences in the unit time interval. The derivation in shows that whenever a certain type of events occurs according to a Poisson process with parameter , the counting variable of the number of occurrences in the unit time interval is distributed according to the Poisson distribution as indicated in .

If we observe the occurrences of events over intervals of length other than unit length, say, in an interval of length , the counting process is governed by the same three postulates, with the modification to Assumption 2 that the rate of changes of the process is now . The mean number of occurrences in the time interval of length is now . The Assumption 2 now states that for any very short time interval of length (and that is also a subinterval of the interval of length under observation), the probability of having one occurrence of event in this short interval is . Applyng the same derivation, it can be shown that the number of occurrences () in a time interval of length has the Poisson distribution with the following probability mass function.

A random experiment resulting in two distinct outcomes (success or failure) is called a Bernoulli trial (e.g. head or tail in a coin toss, whether or not the birthday of a customer is the first of January, whether an insurance claim is above or below a given threshold etc). Suppose a series of independent Bernoulli trials are performed until reaching the rth success where the probability of success in each trial is . Let be the number of failures before the occurrence of the rth success. The following is the probablity mass function of .

Be definition, the survival function and cdf of are:

For each positive integer , let be the number of successes in performing a sequence of independent Bernoulli trials where is the probability of success. In other words, has a binomial distribution with parameters and .

If the random experiment requires more than failures to reach the rth success, there are at most successes in the first trails. Thus the survival function of is the same as the cdf of a binomial distribution. Equivalently, the cdf of is the same as the survival function of a binomial distribution. We have the following:

Remark
The relation is analogous to the relationship between the Gamma distribution and the Poisson distribution. Recall that a Gamma distribution with shape parameter and scale parameter , where is a positive integer, can be interpreted as the waiting time until the nth change in a Poisson process. Thus, if the nth change takes place after time , there can be at most arrivals in the time interval . Thus the survival function of this Gamma distribution is the same as the cdf of a Poisson distribution. The relation is analogous to the following relation.

A previous post on the negative binomial distribution is found here.

The negative binomial distribution arises naturally from a probability experiment of performing a series of independent Bernoulli trials until the occurrence of the rth success where r is a positive integer. From this starting point, we discuss three ways to define the distribution. We then discuss several basic properties of the negative binomial distribution. Emphasis is placed on the close connection between the Poisson distribution and the negative binomial distribution.

________________________________________________________________________

Definitions
We define three versions of the negative binomial distribution. The first two versions arise from the view point of performing a series of independent Bernoulli trials until the rth success where r is a positive integer. A Bernoulli trial is a probability experiment whose outcome is random such that there are two possible outcomes (success or failure).

Let be the number of Bernoulli trials required for the rth success to occur where r is a positive integer. Let is the probability of success in each trial. The following is the probability function of :

The idea for is that for to happen, there must be successes in the first trials and one additional success occurring in the last trial (the th trial).

A more common version of the negative binomial distribution is the number of Bernoulli trials in excess of r in order to produce the rth success. In other words, we consider the number of failures before the occurrence of the rth success. Let be this random variable. The following is the probability function of :

The idea for is that there are trials and in the first trials, there are failures (or equivalently successes).

In both and , the binomial coefficient is defined by

where is a positive integer and is a nonnegative integer. However, the right-hand-side of can be calculated even if is not a positive integer. Thus the binomial coefficient can be expanded to work for all real number . However must still be nonnegative integer.

For convenience, we let . When the real number , the binomial coefficient in can be expressed as:

where is the gamma function.

With the more relaxed notion of binomial coefficient, the probability function in above can be defined for all real number r. Thus the general version of the negative binomial distribution has two parameters r and , both real numbers, such that . The following is its probability function.

Whenever r in is a real number that is not a positive integer, the interpretation of counting the number of failures until the occurrence of the rth success is no longer important. Instead we can think of it simply as a count distribution.

The following alternative parametrization of the negative binomial distribution is also useful.

The parameters in this alternative parametrization are r and . Clearly, the ratio takes the place of in . Unless stated otherwise, we use the parametrization of .
________________________________________________________________________

What is negative about the negative binomial distribution?
What is negative about this distribution? What is binomial about this distribution? The name is suggested by the fact that the binomial coefficient in can be rearranged as follows:

The calculation in can be used to verify that is indeed a probability function, that is, all the probabilities sum to 1.

In , we take . The step above uses the following formula known as the Newton’s binomial formula.

________________________________________________________________________

The Generating Function
By definition, the following is the generating function of the negative binomial distribution, using :

where . Using a similar calculation as in , the generating function can be simplified as:

As a result, the moment generating function of the negative binomial distribution is:

________________________________________________________________________

Independent Sum

One useful property of the negative binomial distribution is that the independent sum of negative binomial random variables, all with the same parameter , also has a negative binomial distribution. Let be an independent sum such that each has a negative binomial distribution with parameters and . Then the sum has a negative binomial distribution with parameters and .

Note that the generating function of an independent sum is the product of the individual generating functions. The following shows that the product of the individual generating functions is of the same form as , thus proving the above assertion.

________________________________________________________________________

Mean and Variance
The mean and variance can be obtained from the generating function. From and , we have:

Note that . Thus when the sample data suggest that the variance is greater than the mean, the negative binomial distribution is an excellent alternative to the Poisson distribution. For example, suppose that the sample mean and the sample variance are 3.6 and 7.1. In exploring the possibility of fitting the data using the negative binomial distribution, we would be interested in the negative binomial distribution with this mean and variance. Then plugging these into produces the negative binomial distribution with and .
________________________________________________________________________

The Poisson-Gamma Mixture
One important application of the negative binomial distribution is that it is a mixture of a family of Poisson distributions with Gamma mixing weights. Thus the negative binomial distribution can be viewed as a generalization of the Poisson distribution. The negative binomial distribution can be viewed as a Poisson distribution where the Poisson parameter is itself a random variable, distributed according to a Gamma distribution. Thus the negative binomial distribution is known as a Poisson-Gamma mixture.

In an insurance application, the negative binomial distribution can be used as a model for claim frequency when the risks are not homogeneous. Let has a Poisson distribution with parameter , which can be interpreted as the number of claims in a fixed period of time from an insured in a large pool of insureds. There is uncertainty in the parameter , reflecting the risk characteristic of the insured. Some insureds are poor risks (with large ) and some are good risks (with small ). Thus the parameter should be regarded as a random variable . The following is the conditional distribution of (conditional on ):

Suppose that has a Gamma distribution with scale parameter and shape parameter . The following is the probability density function of .

Then the joint density of and is:

The unconditional distribution of is obtained by summing out in .

Note that the integral in the fourth step in is 1.0 since the integrand is the pdf of a Gamma distribution. The above probability function is that of a negative binomial distribution. It is of the same form as . Equivalently, it is also of the form with parameter and .

The variance of the negative binomial distribution is greater than the mean. In a Poisson distribution, the mean equals the variance. Thus the unconditional distribution of is more dispersed than its conditional distributions. This is a characteristic of mixture distributions. The uncertainty in the parameter variable has the effect of increasing the unconditional variance of the mixture distribution of . The variance of a mixture distribution has two components, the weighted average of the conditional variances and the variance of the conditional means. The second component represents the additional variance introduced by the uncertainty in the parameter (see The variance of a mixture).

________________________________________________________________________

The Poisson Distribution as Limit of Negative Binomial
There is another connection to the Poisson distribution, that is, the Poisson distribution is a limiting case of the negative binomial distribution. We show that the generating function of the Poisson distribution can be obtained by taking the limit of the negative binomial generating function as . Interestingly, the Poisson distribution is also the limit of the binomial distribution.

In this section, we use the negative binomial parametrization of . By replacing for , the following are the mean, variance, and the generating function for the probability function in :

Let r goes to infinity and goes to zero and at the same time keeping their product constant. Thus is constant (this is the mean of the negative binomial distribution). We show the following:

The right-hand side of is the generating function of the Poisson distribution with mean . The generating function in the left-hand side is that of a negative binomial distribution with mean . The following is the derivation of .

We now focus on the limit in the exponent.

The middle step in uses the L’Hopital’s Rule. The result in is obtained by combining and .

________________________________________________________________________

Reference

1. Klugman S.A., Panjer H. H., Wilmot G. E. Loss Models, From Data to Decisions, Second Edition., Wiley-Interscience, a John Wiley & Sons, Inc., New York, 2004

Example 1
Suppose that the arrivals of customers in a gift shop at an airport follow a Poisson distribution with a mean of per 10 minutes. Furthermore, suppose that each arrival can be classified into one of three distinct types – type 1 (no purchase), type 2 (purchase under $20), and type 3 (purchase over $20). Records show that about 25% of the customers are of type 1. The percentages of type 2 and type 3 are 60% and 15%, respectively. What is the probability distribution of the number of customers per hour of each type?

Example 2
Roll a fair die times where is random and follows a Poisson distribution with parameter . For each , let be the number of times the upside of the die is . What is the probability distribution of each ? What is the joint distribution of ?

In Example 1, the stream of customers arrive according to a Poisson distribution. It can be shown that the stream of each type of customers also has a Poisson distribution. One way to view this example is that we can split the Poisson distribution into three Poisson distributions.

Example 2 also describes a splitting process, i.e. splitting a Poisson variable into 6 different Poisson variables. We can also view Example 2 as a multinomial distribution where the number of trials is not fixed but is random and follows a Poisson distribution. If the number of rolls of the die is fixed in Example 2 (say 10), then each would be a binomial distribution. Yet, with the number of trials being Poisson, each has a Poisson distribution with mean . In this post, we describe this Poisson splitting process in terms of a “random” multinomial distribution (the view point of Example 2).

________________________________________________________________________

Suppose we have a multinomial experiment with parameters , , , where

• is the number of multinomial trials,
• is the number of distinct possible outcomes in each trial (type 1 through type ),
• the are the probabilities of the possible outcomes in each trial.

Suppose that follows a Poisson distribution with parameter . For each , let be the number of occurrences of the type of outcomes in the trials. Then are mutually independent Poisson random variables with parameters , respectively.

The variables have a multinomial distribution and their joint probability function is:

where are nonnegative integers such that .

Since the total number of multinomial trials is not fixed and is random, is not the end of the story. The probability in is only a conditional probability. The following is the joint probability function of :

To obtain the marginal probability function of , , we sum out the other variables () in and obtain the following:

Thus we can conclude that , , has a Poisson distribution with parameter . Furrthermore, the joint probability function of is the product of the marginal probability functions. Thus we can conclude that are mutually independent.

________________________________________________________________________
Example 1
Let be the number of customers per hour of type 1, type 2, and type 3, respectively. Here, we attempt to split a Poisson distribution with mean 30 per hour (based on 5 per 10 minutes). Thus are mutually independent Poisson variables with means , , , respectively.

Example 2
As indicated earlier, each , , has a Poisson distribution with mean . According to , the joint probability function of is simply the product of the six marginal Poisson probability functions.

The above distribution is said to be a Poisson distribution with parameter . The Poisson distribution is usually used to model the random number of events occurring in a fixed time interval. As will be shown below, . Thus the parameter is the rate of occurrence of the random events; it indicates on average how many events occur per unit of time. Examples of random events that may be modeled by the Poisson distribution include the number of alpha particles emitted by a radioactive substance counted in a prescribed area during a fixed period of time, the number of auto accidents in a fixed period of time or the number of losses arising from a group of insureds during a policy period.

Each of the above examples can be thought of as a process that generates a number of arrivals or changes in a fixed period of time. If such a counting process leads to a Poisson distribution, then the process is said to be a Poisson process.

We now discuss some basic properties of the Poisson distribution. Using the Taylor series expansion of , the following shows that is indeed a probability distribution.

The generating function of the Poisson distribution is (see The generating function). The mean and variance can be calculated using the generating function.

The Poisson distribution can also be interpreted as an approximation to the binomial distribution. It is well known that the Poisson distribution is the limiting case of binomial distributions (see [1] or this post).

One application of is that we can use Poisson probabilities to approximate Binomial probabilities. The approximation is reasonably good when the number of trials in a binomial distribution is large and the probability of success is small. The binomial mean is and the variance is . When is small, is close to 1 and the binomial variance is approximately . Whenever the mean of a discrete distribution is approximately equaled to the mean, the Poisson approximation is quite good. As a rule of thumb, we can use Poisson to approximate binomial if and .

As an example, we use the Poisson distribution to estimate the probability that at most 1 person out of 1000 will have a birthday on the New Year Day. Let and . So we use the Poisson distribution with . The following is an estimate using the Poisson distribution.

Another useful property is that the independent sum of Poisson distributions also has a Poisson distribution. Specifically, if each has a Poisson distribution with parameter , then the independent sum has a Poisson distribution with parameter . One way to see this is that the product of Poisson generating functions has the same general form as (see The generating function). One interpretation of this property is that when merging several arrival processes, each of which follow a Poisson distribution, the result is still a Poisson distribution.

For example, suppose that in an airline ticket counter, the arrival of first class customers follows a Poisson process with a mean arrival rate of 8 per 15 minutes and the arrival of customers flying coach follows a Poisson distribution with a mean rate of 12 per 15 minutes. Then the arrival of customers of either types has a Poisson distribution with a mean rate of 20 per 15 minutes or 80 per hour.

A Poisson distribution with a large mean can be thought of as an independent sum of Poisson distributions. For example, a Poisson distribution with a mean of 50 is the independent sum of 50 Poisson distributions each with mean 1. Because of the central limit theorem, when the mean is large, we can approximate the Poisson using the normal distribution.

In addition to merging several Poisson distributions into one combined Poisson distribution, we can also split a Poisson into several Poisson distributions. For example, suppose that a stream of customers arrives according to a Poisson distribution with parameter and each customer can be classified into one of two types (e.g. no purchase vs. purchase) with probabilities and , respectively. Then the number of “no purchase” customers and the number of “purchase” customers are independent Poisson random variables with parameters and , respectively. For more details on the splitting of Poisson, see Splitting a Poisson Distribution.

Reference

1. Feller W. An Introduction to Probability Theory and Its Applications, Third Edition, John Wiley & Sons, New York, 1968

Note that the derivative of at each order is a multiple of a Poisson probability. Thus the Poisson distribution is coded by the function . Because of this reason, such a function is called a generating function (or probability generating function). This post discusses some basic facts about the generating function (gf) and its cousin, the moment generating function (mgf). One important characteristic is that these functions generate probabilities and moments. Another important characteristic is that there is a one-to-one correspondence between a probability distribution and its generating function and moment generating function, i.e. two random variables with different cumulative distribution functions cannot have the same gf or mgf. In some situations, this fact is useful in working with independent sum of random variables.

________________________________________________
The Generating Function
Suppose that is a random variable that takes only nonegative integer values with the probability function given by

The idea of the generating function is that we use a power series to capture the entire probability distribution. The following defines the generating function that is associated with the above sequence , .

Since the elements of the sequence are probabilities, we can also call the generating function of the probability distribution defined by the sequence in . The generating function is defined wherever the power series converges. It is clear that at the minimum, the power series in converges for .

We discuss the following three properties of generating functions:

1. The generating function completely determines the distribution.
2. The moments of the distribution can be derived from the derivatives of the generating function.
3. The generating function of a sum of independent random variables is the product of the individual generating functions.

The Poisson generating function at the beginning of the post is an example demonstrating property 1 (see Example 0 below for the derivation of the generating function). In some cases, the probability distribution of an independent sum can be deduced from the product of the individual generating functions. Some examples are given below.

________________________________________________
Generating Probabilities
We now discuss the property 1 indicated above. To see that generates the probabilities, let’s look at the derivatives of :

By letting above, all the terms vanishes except for the constant term. We have:

Thus the generating function is a compact way of encoding the probability distribution. The probability distribution determines the generating function as seen in . On the other hand, and demonstrate that the generating function also determines the probability distribution.

________________________________________________
Generating Moments
The generating function also determines the moments (property 2 indicated above). For example, we have:

Note that . Thus the higher moment can be expressed in terms of and where .
________________________________________________
More General Definitions
Note that the definition in can also be interpreted as the mathematical expectation of , i.e., . This provides a way to define the generating function for random variables that may take on values outside of the nonnegative integers. The following is a more general definition of the generating function of the random variable , which is defined for all where the expectation exists.

________________________________________________
The Generating Function of Independent Sum
Let be independent random variables with generating functions , respectively. Then the generating function of is given by the product .

Let be the generating function of the independent sum . The following derives . Note that the general form of generating function is used.

The probability distribution of a random variable is uniquely determined by its generating function. In particular, the generating function of the independent sum that is derived in is unique. So if the generating function is of a particular distribution, we can deduce that the distribution of the sum must be of the same distribution. See the examples below.

________________________________________________
Example 0
In this example, we derive the generating function of the Poisson distribution. Based on the definition, we have:

Example 1
Suppose that are independent random variables where each has a Bernoulli distribution with probability of success . Let . The following is the generating function for each .

Then the generating function of the sum is . The following is the binomial expansion:

By definition , the generating function of is:

Comparing and , we have

The probability distribution indicated by and is that of a binomial distribution. Since the probability distribution of a random variable is uniquely determined by its generating function, the independent sum of Bernoulli distributions must ave a Binomial distribution.

Example 2
Suppose that are independent and have Poisson distributions with parameters , respectively. Then the independent sum has a Poisson distribution with parameter .

Let be the generating function of . For each , the generating function of is . The key to the proof is that the product of the has the same general form as the individual .

The generating function in is that of a Poisson distribution with mean . Since the generating function uniquely determines the distribution, we can deduce that the sum has a Poisson distribution with parameter .

Example 3
In rolling a fair die, let be the number shown on the up face. The associated generating function is:

The generating function can be further reduced as:

Suppose that we roll the fair dice 4 times. Let be the sum of the 4 rolls. Then the generating function of is

The random variable ranges from 4 to 24. Thus the probability function ranges from to . To find these probabilities, we simply need to decode the generating function . For example, to find , we need to find the coefficient of the term in the polynomial . To help this decoding, we can expand two of the polynomials in .

Based on the above polynomials, there are three ways of forming . They are: , , . Thus we have:

To find the other probabilities, we can follow the same decoding process.

________________________________________________
Remark
The probability distribution of a random variable is uniquely determined by its generating function. This fundamental property is useful in determining the distribution of an independent sum. The generating function of the independent sum is simply the product of the individual generating functions. If the product is of a certain distributional form (as in Example 1 and Example 2), then we can deduce that the sum must be of the same distribution.

We can also decode the product of generating functions to obtain the probability function of the independent sum (as in Example 3). The method in Example 3 is quite tedious. But one advantage is that it is a “machine process”, a pretty fool proof process that can be performed mechanically.

The machine process is this: Code the individual probability distribution in a generating function . Then raise it to . After performing some manipulation to , decode the probabilities from .

As long as we can perform the algebraic manipulation carefully and correctly, this process will be sure to provide the probability distribution of an independent sum.

________________________________________________
The Moment Generating Function
The moment generating function of a random variable is on all real numbers for which the expected value exists. The moments can be computed more directly using an mgf. From the theory of mathematical analysis, it can be shown that if exists on some interval , then the derivatives of of all orders exist at . Furthermore, it can be show that .

Suppose that is the generating function of a random variable. The following relates the generating function and the moment generating function.

________________________________________________

Reference

1. Feller W. An Introduction to Probability Theory and Its Applications, Third Edition, John Wiley & Sons, New York, 1968