{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 1.6 - Beginner - Distributions\n", "\n", "COMET Team
*Valeria Zolla, Colby Chambers, Jonathan Graves* \n", "2023-01-12\n", "\n", "## Outline\n", "\n", "### Prerequisites\n", "\n", "- Introduction to Jupyter\n", "- Introduction to R\n", "- Introduction to Visualization\n", "- Central Tendency\n", "\n", "### Outcomes\n", "\n", "After completing this notebook, you will be able:\n", "\n", "- Understand and work with Probability Density Functions (PDFs) and\n", " Cumulative Density Functions (CDF)\n", "- Use tables to find joint, marginal, and conditional probabilities\n", "- Interpret uniform, normal, and $t$ distributions\n", "\n", "### References\n", "\n", "- [Introduction to Probability and Statistics Using\n", " R](https://mran.microsoft.com/snapshot/2018-09-28/web/packages/IPSUR/vignettes/IPSUR.pdf)\n", "\n", "# Introduction\n", "\n", "This notebook will explore the concept of distributions, both in terms\n", "of their functional forms for probability and how they represent\n", "different sets of data.\n", "\n", "Let’s first load the 2016 Census from Statistics Canada, which we will\n", "consult throughout this lesson." ], "id": "cdcb42e9-6195-41b2-b8bf-884e69641e45" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# loading in our packages\n", "library(tidyverse)\n", "library(haven)\n", "library(digest)\n", "\n", "source(\"beginner_distributions_tests.r\")" ], "id": "00cbcd38-a1ae-4e87-b6f9-6ce3216d76e1" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# reading in the data\n", "\n", "census_data <- read_dta(\"../datasets_beginner/01_census2016.dta\")\n", "\n", "# cleaning up factors\n", "census_data <- as_factor(census_data)\n", "\n", "# cleaning up missing data\n", "census_data <- filter(census_data, !is.na(census_data$wages))\n", "census_data <- filter(census_data, !is.na(census_data$mrkinc))\n", "\n", "# inspecting the data\n", "glimpse(census_data)" ], "id": "c2c7dc60-47df-4cdf-9e3e-1f5b0e89cdf9" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have our data set ready on stand-by for analysis, let’s\n", "start looking at distributions as a concept more generally.\n", "\n", "# Part 1: Distribution Functions - The Basics\n", "\n", "## What is a Probability?\n", "\n", "The probability of an event is a number that indicates the likelihood of\n", "that event happening.\n", "\n", "When the possible values of a certain event are discrete (e.g. `1,2,3`\n", "or `adult, child`), we refer to this as the **frequency**.\n", "\n", "When the possible values are continuous (e.g. any number between `0.5`\n", "and `3.75`), we refer to this as the **density**.\n", "\n", "There is a difference between *population* probabilities and *empirical*\n", "or *sample* probabilities. Generally, when we talk about distributions\n", "we will be referring to *population* objects: but there are also sample\n", "versions as well, which are often easier to think about.\n", "\n", "For instance, let’s say we have a dataset with 5,000 observations and a\n", "variable called `birthmonth` which records the month of birth of every\n", "participant captured in the dataset. If 500 people in the data were born\n", "in October, then `birthmonth==\"October`” would have an *empirical*\n", "probability of occurring in an observation 10% of the time. We can’t be\n", "sure what the population probability would be, unless we knew more about\n", "the population.\n", "\n", "## What is a Random Variable?\n", "\n", "A **random variable** is a variable whose possible values are numerical\n", "outcomes of a random phenomenon, such as rolling a dice. A random\n", "variable can be either discrete or continuous.\n", "\n", "- A **discrete random variable** is one which may take on only a\n", " finite number of distinct values (e.g the number of children in a\n", " family).\n", "\n", " - In this notebook we see that (`agegrp` in the data) is an\n", " example of that.\n", "\n", "- A **continuous random variable** is one which takes an infinite\n", " number of possible values and can be *measured* rather than merely\n", " *categorized*. (e.g height, weight, or how much people earn).\n", "\n", " - In the data, we can see that `wages` and `mrkinc` are great\n", " examples of continuous random variables.\n", "\n", "## What is the Probability Distribution?\n", "\n", "A **probability distribution** refers to the pattern or arrangement of\n", "probabilities in a population. These are usually described as\n", "*functions*: to indicate the probability of that event occurring. As we\n", "explained above, there is a difference between *population* and *sample*\n", "distributions:\n", "\n", "- A *population* distribution (which is the typical way we describe\n", " these) describes population probabilities\n", "\n", "- An *empirical* or *sample* distribution reports describes empirical\n", " probabilities from within a particular sample\n", "\n", "Note: we typically use *empirical* distribution as a way to learn about\n", "the *population* distribution, which is what we’re primarily interested\n", "in.\n", "\n", "Distribution functions come in several standard forms; let’s learn about\n", "them.\n", "\n", "# Probability Density Functions (PDFs)\n", "\n", "**Probability Density Functions** are also sometimes referred to as PDFs\n", "or probability mass functions. We usually use lower case letters like\n", "$f$ or $p$ to describe these functions.\n", "\n", "## 1. Discrete PDF:\n", "\n", "> ” The probability distribution of a discrete random variable is the\n", "> list of all possible values of the variable and their probabilities\n", "> which sum to 1.”\n", ">\n", "> \\- Econometrics with R\n", "\n", "**Probability Density Function (PDF)**, also referred to as **density**\n", "or **frequency**, is the probability of occurrence of all the different\n", "values of a variable.\n", "\n", "Suppose a random variable X may take k different values, with the\n", "probability that $X = x_{i}$ defined to be $P(X = x_{i}) = p_{i}$. The\n", "probabilities $p_{i}$ must satisfy the following:\n", "\n", "1. For each i: $0% \n", " group_by(agegrp) %>%\n", " summarize(Count = n(),\n", " Frequency = n()/sample_size*100) # creates two variables in our table\n", "\n", "table2" ], "id": "89773133-7b52-48a2-a772-f394b3e974ad" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let’s try creating a graph. Since a PDF has a finite number of\n", "distinct values of which we measure their frequency, we can use a bar\n", "graph (See *Introduction to Visualization* for instruction)." ], "id": "1236f811-1340-49ca-9459-30fe4a6551da" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plot <- ggplot(data = table2, # this declares the data for the chart; all variable names are in this data\n", " aes(# this is a list of the aesthetic features of the chart\n", " x = agegrp, # for example, the x-axis will be \"year\"\n", " y = Frequency # the y-axes will be expenditure-based real GDP per capita\n", " ),\n", " ) \n", "plot1 <- plot + geom_col() + \n", " theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1))\n", "\n", "plot1" ], "id": "f4d82f1a-2521-4d3c-b489-c76cca9e1b77" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2. Continuous PDF:\n", "\n", "> “Since a continuous random variable takes on a continuum of possible\n", "> values, we cannot use the concept of a probability distribution as\n", "> used for discrete random variables.”\n", ">\n", "> \\- Econometrics with R\n", "\n", "Unlike a discrete variable, a continuous random variable is not defined\n", "in specific values. Instead, it is defined over intervals of values, and\n", "is represented by the area under a curve (in calculus, this is the\n", "integral).\n", "\n", "The curve, which represents the probability function is also called a\n", "**density** curve and it must satisfy the following:\n", "\n", "1\\. The curve has no negative values $p(x) > 0$ for all $x$ (the\n", "probability of observing a value can’t be negative)\n", "\n", "2\\. The total area under the curve is equal to 1\n", "\n", "Let’s imagine a random variable that can take any value over an interval\n", "of real numbers. The probability of observing values between $a$ and $b$\n", "is the area between the density curve and the area between $a$ and $b$:\n", "\n", "$$\n", "\\mathrm{P}(a \\le X \\le b) = \\left(\\int_{a}^{b} f(x) \\; dx\\right)\n", "$$\n", "\n", "Since the number of values which may be assumed by the random variable\n", "is infinite, the probability of observing any single value is equal to\n", "0.\n", "\n", "To visualize a continuous PDF:\n", "\n", "- We will use graphs rather than tables as we need to visualize the\n", " entire continuum of possible values to be represented in the graph.\n", "\n", "- Since the probability of observing values between $a$ and $b$ is the\n", " area underneath the curve, therefore a continuous PDF should be\n", " visualized as a line graph instead of a bar graphs or scatterplots.\n", "\n", "Suppose we would like to visualize a continuous empirical PDF for all\n", "wages between 25000 and 75000:" ], "id": "085646a5-95f3-4725-9aae-409a5fa0d158" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "density <- density(census_data$wages)\n", "plot(density)\n", "\n", "# Telling R how to read our upper and lower bounds\n", "l <- min(which(density$x >= 25000))\n", "h <- max(which(density$x < 75000))\n", "\n", "# Visualizing our specified range in red \n", "polygon(c(density$x[c(l, l:h, h)]),\n", " c(0, density$y[l:h], 0),\n", " col = \"red\")" ], "id": "13e395ac-3b30-4748-928b-ae49ae2b84ac" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Cumulative Density Function (CDF)\n", "\n", "When we have a variable which is rankable, we can define a related\n", "object: the **Cumulative Density Function (CDF)**.\n", "\n", "- The CDF for both discrete and continuous random variables is the\n", " probability that the random variable is less than or equal to a\n", " particular value.\n", "- Hence, the CDF must necessarily be an increasing function. Think of\n", " the example of rolling a dice:\n", " - $F(1)$ would indicate the the probability that a 1 was rolled.\n", "\n", " - $F(2)$ would indicate the probability that a 2 **or lower** was\n", " rolled.\n", "\n", " - Evidently, $F(2)$ would be greater than $F(1)$.\n", "- A CDF can only take values between 0 and 1.\n", " - 0 or (0%) is the probability that the random variable is less or\n", " equal to the smallest value of the variable.\n", " - 1 or (100%) is the total probability that the random variable is\n", " less or equal to the biggest value of the variable\n", "- Therefore, if we have a variable $X$ that can have take the value of\n", " $x$ the CDF is is the probability that $X$ will take a value less\n", " than or equal to $x$.\n", "\n", "Since we use the lowercase $f(y)$ to represent the PDF of $y$, we use\n", "the uppercase $F(y)$ to represent the CDF of $y$.\n", "\n", "Mathematically, since $f_{X}(x)$ denotes the probability density\n", "function of $X$, then the probability that $X$ falls between $a$ and $b$\n", "where $a \\leq b$ is:\n", "\n", "$$\n", "\\mathrm{P}(a \\leq X \\leq b) = \\left(\\int_{a}^{b} f_{X}(x) \\; dx\\right)\n", "$$\n", "\n", "We know that the entire X variable falls between 2 values if the\n", "probability of x falling in between them is 1. Therefore $X$’s CDF curve\n", "is:\n", "\n", "$$\n", "\\mathrm{P}(−∞ \\le X \\le ∞) = \\left(\\int_{−∞}^{∞} f_{X}(x) \\; dx\\right) = 1\n", "$$\n", "\n", "Below we’ve used a scatter plot to visualize empirical CDF of the\n", "continuous variable `wages`. From the graph below we can tell that most\n", "people earn between 0-200000 as the probability of people’s wages being\n", "less than or equal to 200000 is over 80%." ], "id": "7981a90e-8f9e-4de0-a8f8-2bcf540b4e41" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "p <- ecdf(census_data$wages)\n", "\n", "# plot CDF\n", "plot(p)" ], "id": "182d795f-a285-4efa-892a-8e63b7d1f609" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Joint Probability Distribution\n", "\n", "So far, we’ve looked at distributions for single random variables.\n", "However, we can also use **joint distributions** to analyze the\n", "probability of multiple random variables taking on certain values.\n", "\n", "In this case, the **joint distribution** is the probability distribution\n", "on all possible values that $X$ and $Y$, can take on.\n", "\n", "Lets suppose both $X$ and $Y$ are discrete random variable which can\n", "take on values from 1-3, with the following joint probability table (X\n", "on vertical axis, and Y on horizontal):\n", "\n", "| | 1 | 2 | 3 |\n", "|-----|-----|-----|-----|\n", "| 1 | 0 | 1/6 | 1/6 |\n", "| 2 | 1/6 | 0 | 1/6 |\n", "| 3 | 1/6 | 1/6 | 0 |\n", "\n", "Above, we’ve created a joint distribution for the two discrete random\n", "variables, $X$ and $Y$, with the total probability adding up to 1.\n", "\n", "Every joint distribution can be represented by a PDF and CDF, just like\n", "single random variables. The formal notation of a PDF for two jointly\n", "distributed random variables is below.\n", "\n", "$$f(x, y) = Prob (X = x, Y = y)$$\n", "\n", "where $f(x, y)$ is the joint probability density that the random\n", "variable $X$ takes on a value of $x$, and the random variable $Y$ takes\n", "on a value of $y$.\n", "\n", "The PDF for jointly distributed random variables are the values recorded\n", "in the example table above. For example the PDF of an immigrant only\n", "speaking french $Prob(X=1, Y =2) = 1/6$.\n", "\n", "The CDF for jointly distributed random variables follows the same logic\n", "as with single variables though this time it represents the probability\n", "of multiple variables taking on values less than those specified all at\n", "once.\n", "\n", "This might not make sense for two discrete random variables such as\n", "`immstat` and `kol`, but this would be much more useful if both of those\n", "variables are continuous (i.e. `wages` and `mrkinc`). The formal\n", "notation of a CDF for two jointly distributed random variables is below.\n", "\n", "$$\n", "F(x, y) = Prob({X \\leq x}, {Y \\leq y})\n", "$$\n", "\n", "where $F(x, y)$ is the joint cumulative probability that the random\n", "variable $X$ takes on a value less than or equal to $x$ and the random\n", "variable $Y$ takes on a value less than or equal to $y$ simultaneously.\n", "\n", "# Marginal Probability Distribution\n", "\n", "The **marginal distribution** is the probability density function for\n", "each individual random variable. If we add up all of the joint\n", "probabilities from the same row or the same column, we get the\n", "probability of one random variable taking on a series of different\n", "values. We can represent the marginal probability density function as\n", "follows:\n", "\n", "$$\n", "f_{x}(x) = \\sum_{y} Prob(X = x, Y = y)\n", "$$\n", "\n", "where we sum across all possible joint probabilities of $X$ and $Y$ for\n", "a given value of $X$ and $Y$.\n", "\n", "If we wanted the marginal empirical probability distribution function of\n", "$X$,we would need to find the marginal probability for all possible\n", "values of $X$.\n", "\n", "For $X=1$, the marginal probability is the sum of all joint\n", "probabilities in that corresponding row: 1/6 + 1/6 = 1/3\n", "\n", "# Conditional Probability Distribution\n", "\n", "The **conditional distribution** function indicates the probability of\n", "seeing a host of values for one random variable conditional on a\n", "specified value of another random variable, provided that the two random\n", "variables are jointly distributed.\n", "\n", "Below is the formula to find the conditional probability density\n", "function:\n", "\n", "$$\n", "f(x | y) = \\frac {Prob ((X = x) \\bigcap (Y = y))} {Prob(Y = y)}\n", "$$\n", "\n", "- Where $f(x | y)$ represents the conditional probability that the\n", " random variable $X$ will take on a value of $x$ when the random\n", " variable of $Y$ takes on a value of $y$." ], "id": "af7ad5fc-388e-4d1c-8ca2-84a8728fe465" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "" ], "id": "bb645f60-91f0-4247-b580-ee1add0b159c" }, { "cell_type": "markdown", "metadata": {}, "source": [ "- The $\\bigcap$ symbol simply represents the case that both $X$ = $x$\n", " and $Y$ = $y$ simultaneously (a joint probability) - we can read\n", " this symbol as “given that”.\n", "\n", "- Note that the marginal probability that $Y = y$ must not be 0 as\n", " that would make the conditional probability undefined.\n", "\n", "Let’s say we want to find the conditional probability of $X=1$ given\n", "$Y=2$. Recall the unconditional probability that $Y=1$ = 1/3. The\n", "conditional probability, given $Y=1$ will therefore be the probability\n", "of $X=1$ AND $Y=2$ divided by the probability that $Y=2$: (1/6) / (1/3)\n", "= 1/2.\n", "\n", "One important point to consider is that of **statistical independence of\n", "random variables**.\n", "\n", "- Two random variables are independent if and only if their joint\n", " probability of occurrence equals the product of their marginal\n", " probabilities for all possible combinations of values of the random\n", " variables.\n", "- In mathematical notation, this means that two random variables are\n", " statistically independent if and only if:\n", "\n", "$$\n", "f(x, y) = f_{x}(x) f_{y}(y)\n", "$$\n", "\n", "- We can check for statistical independence of our jointly distributed\n", " random variables, $X$ and $Y$, referring to our table up above.\n", "- We can determine whether these variables are independent by\n", " multiplying combinations of marginal probabilities to see if they\n", " match the joint probability in the corresponding cell.\n", "\n", "Until now, we have referred to the joint, marginal and conditional\n", "distribution of two discrete random variables; however, ***one or both\n", "of these variables can be continuous***.\n", "\n", "We focused on discrete random variables since they are much easier to\n", "represent in table format. (Creating a table of joint probabilities for\n", "two jointly distributed continuous random variables would produce near\n", "infinite cells, each with a joint probability of about 0!)\n", "\n", "While the same logic for discrete variables applies to continuous random\n", "variables, we often refer to mathematical formulas when finding the\n", "marginal and conditional probability functions for continuous random\n", "variables, since their PDFs and CDFs can be represented by mathematical\n", "functions.\n", "\n", "Note: ***we can have more than two jointly distributed random\n", "variables***. While it is possible to represent the probability of 3 or\n", "more variables taking on certain values at once, it is hard to represent\n", "that graphically or in table format. That is why we have stuck to\n", "investigating two jointly distributed random variables in this notebook.\n", "\n", "Now is your turn to work on some exercises which will test your\n", "understanding of the material presented in this notebook!\n", "\n", "# Exercise 1\n", "\n", "Let the random variable $X$ denote the time (in hours) a person waits\n", "for their flight. This person can wait up to 2 hours for this flight.\n", "\n", "## Question 1\n", "\n", "Is $X$ a discrete or continuous random variable?" ], "id": "abdb4e3f-c3af-46cb-b994-edcc9534c1ed" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "answer_1 <- \"...\" # your answer of \"discrete\" or \"continuous\" in place of ...\n", "\n", "test_1()" ], "id": "08a951eb-9e89-4092-bada-b6bf24c1132c" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Explain your reasoning here:\n", "\n", "## Question 2\n", "\n", "Say a potential probability density function representing this random\n", "variable (from the above flight example) is the following:\n", "\n", "$$ \n", "f(x) = \\begin{cases}\n", "x & \\text{if } 0 \\leq x \\leq 1,\\\\\n", "2 - x & \\text{if } 1 \\leq x \\leq 2,\\\\\n", "0 & \\text{otherwise}\n", "\\end{cases}\n", "$$\n", "\n", "Is this a valid PDF?" ], "id": "dcadf8b4-837e-4e76-abd8-692126c92e44" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "answer_2 <- \"...\" # your answer of \"yes\" or \"no\" in place of ...\n", "\n", "test_2()" ], "id": "546875c9-2289-4274-b651-82887a9d20c7" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Explain your reasoning here:\n", "\n", "## Question 3\n", "\n", "What is the probability of a person waiting up to 1.5 hours for their\n", "flight? Answer to 3 decimal places. **Hint**: this is not the same as\n", "the probability of waiting precisely 1.5 hours." ], "id": "bf3063ae-154b-46c8-830a-2dc11f55745d" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here\n", "\n", "answer_3 <- ... # your answer for the cumulative probability (in decimal format, i.e. 95% = 0.95) here\n", "\n", "test_3()" ], "id": "a4630381-f387-423f-a8a8-f1192c616105" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Exercise 2\n", "\n", "Let’s return to our `joint_table` for the joint distribution of discrete\n", "random variables `immstat` and `kol`." ], "id": "57012cad-9048-45df-840d-5c81eaa870c4" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "joint_table" ], "id": "418f4fa1-0c0d-429b-b358-31f63b5a238e" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Question 1\n", "\n", "What is the probability that someone is both an immigrant and knows both\n", "English and French? Answer to 3 decimal places." ], "id": "8e059609-4ff1-47f5-8b88-3713ec43b3a6" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "answer_4 <- ... # your answer for the probability (in decimal format, i.e. 95% = 0.95) here\n", "\n", "test_4()" ], "id": "33cceed8-6e1b-43ab-a71c-73b14e1f71fb" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Question 2\n", "\n", "What is the probability that someone is an immigrant given that they\n", "know only English? Answer to 3 decimal places." ], "id": "3492797f-0760-4123-8491-d8d47034599b" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here\n", "\n", "answer_5 <- ... # your answer for the probability (in decimal format, i.e. 95% = 0.95) here\n", "\n", "test_5()" ], "id": "84d646e5-c2ee-47de-b3d8-3ecf9771dbc2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Question 3\n", "\n", "Why is it difficult to graph a joint probability distribution function\n", "(either density or cumulative) for these two variables in Jupyter? Which\n", "type of probability density function can we easily graph for jointly\n", "distributed random variables?\n", "\n", "Explain your reasoning here:\n", "\n", "# Exercise 3\n", "\n", "Let the random variable $Y$ be uniformly distributed on the range of\n", "values \\[20, 80\\].\n", "\n", "## Question 1\n", "\n", "What is the probability of $Y$ taking on the value of 30? Answer to 3\n", "decimal places. You may use a graph to help you." ], "id": "b3aa8054-1ce2-44ba-baa2-cd4e348f3615" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here\n", "\n", "answer_6 <- ... # your answer for the probability (in decimal format, i.e. 95% = 0.95) here\n", "\n", "test_6()" ], "id": "54fe6364-34e4-4734-b70c-2b16e8517374" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Question 2\n", "\n", "What is the probability of $Y$ taking on a value of 60 or more? Answer\n", "to 3 decimal places." ], "id": "99277fc7-a3f6-4b6b-9d69-95bccabfae45" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "answer_7 <- ... # your answer for the probability (in decimal format, i.e. 95% = 0.95) here\n", "\n", "test_7()" ], "id": "829ca222-b875-4386-aec5-2c357aefac82" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Question 3\n", "\n", "What would happen to this probability if $Y$ was expanded to be\n", "uniformly distributed on the range of values \\[20, 100\\]?" ], "id": "2bc955dd-fdce-421d-a23f-19305a398ea2" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "answer_8 <- \"...\" # your answer of \"it would increase\" or \"it would decrease\" in place of \"...\"\n", "\n", "test_8()" ], "id": "777b6a2b-dd0e-4b75-aa8a-f4e5e14a1c31" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Explain your reasoning here:\n", "\n", "# Exercise 4\n", "\n", "Now let $Z$ be a normally distributed random variable representing the\n", "length of a piece of classical music (in minutes), with a mean of 5 and\n", "standard deviation of 1.5.\n", "\n", "## Question 1\n", "\n", "What is the probability that a given piece will last between 3 and 7\n", "minutes? Answer to 3 decimal places. You may use code to help you." ], "id": "f35996e8-1461-4efe-8ecc-56acf893f7a4" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here\n", "\n", "answer_9 <- ... # your answer for the probability (in decimal format, i.e. 95% = 0.95) here\n", "\n", "test_9()" ], "id": "c1063478-61b4-4cc2-a4c3-fb27fdd493bd" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Question 2\n", "\n", "If $Z$ were to remain normally distributed and have the same standard\n", "deviation, but the mean piece length was changed to 3 minutes, how would\n", "this probability change?" ], "id": "41c66739-19fb-4272-a9f3-5041c6e995df" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "answer_10 <- \"...\" # your answer of \"it would increase\" or \"it would decrease\" in place of \"...\"\n", "\n", "test_10()" ], "id": "8ae9b935-8797-401c-92ff-aa064a1c3520" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Explain your reasoning here:\n", "\n", "## Question 3\n", "\n", "Returning to our original $Z$ variable (with mean 5), if the standard\n", "deviation were to decrease to 1, how would this probability change?" ], "id": "ec846ddc-4a61-4443-a041-e1c7874a6ac8" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "answer_11 <- \"...\" # your answer of \"it would increase\" or \"it would decrease\" in place of \"...\"\n", "\n", "test_11()" ], "id": "33236baa-97f9-4557-8fc1-a1cc007aaed9" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Explain your reasoning here:\n", "\n", "# Part 2: Parametric Distributions\n", "\n", "While all of the examples we used were for *empirical* distributions as\n", "we don’t know what the *population* distributions are. However, many\n", "statistics *do* have known distributions which are very important to\n", "understand.\n", "\n", "Let’s look at the three most famous examples of distributions:\n", "\n", "- uniform distribution\n", "- normal (or Gaussian) distribution\n", "- student $t$-distribution\n", "\n", "These are called **parametric** distributions because they can be\n", "described by a set of numbers called *parameters*. For instance, the\n", "normal distribution’s two *parameters* are the mean and standard\n", "deviation.\n", "\n", "All the parametric distributions explained in this module are analyzed\n", "using four R commands. The four commands will start with the prefixes:\n", "\n", "- `d` for “density”: it produces the probability density function\n", " (PDF)\n", "- `p` for “probability”: it produces the cumulative distribution\n", " function (CDF)\n", "- `q` for “quantile”: it produces the inverse cumulative distribution\n", " function, also called the quantile function\n", "- `r` for “random”: generates random numbers from a particular\n", " parametric distribution\n", "\n", "## Uniform Distribution\n", "\n", "A continuous variable has a **uniform distribution** if all values have\n", "the same likelihood of occurring.\n", "\n", "- An example of a random event with a uniform distribution is rolling\n", " a dice as it equally likely to roll any of the six numbers." ], "id": "b55076a5-44da-4c7a-a261-257af421fdbf" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "" ], "id": "47e04cb1-60de-4346-a5c1-d5b099b918cb" }, { "cell_type": "markdown", "metadata": {}, "source": [ "- The variable’s density curve is therefore a rectangle, with constant\n", " height across the interval and 0 height elsewhere.\n", "\n", "- Since the area under the curve must be equal to 1, the length of the\n", " interval determines the height of the curve.\n", "\n", "Let’s see with a variable what this kind of distribution might look\n", "like.\n", "\n", "- First, we will generate random values from this distribution using\n", " the function `runif()`.\n", "- This command is written as `runif(n, min = , max = )`, where `n` is\n", " the number of observations, and `max` and `min` provide the interval\n", " between which the random variables are picked from.\n", "\n", "### Simulation" ], "id": "f0c7dfed-db43-4d65-bc42-dd1ad6bf66c9" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "example_unif <- runif(10000, min = 10, max = 100)\n", "hist(example_unif, freq = FALSE, xlab = 'x', xlim = c(0,100), main = \"Empirical PDF for uniform random values on [0,100]\")" ], "id": "d50dc0d7-2126-4bf8-8218-3eaa00c39d84" }, { "cell_type": "markdown", "metadata": {}, "source": [ "- While each number within the specified range is equally likely to be\n", " drawn, by random chance, some ranges of numbers are drawn more\n", " frequently others, hence the bars are not all the exact same height.\n", "\n", "- The shape of the distribution will change each time you re-run the\n", " previous code cell.\n", "\n", "Knowing the distribution, we can now plot and visualize the data!\n", "\n", "For instance, suppose we have a uniform random variable $X$ defined on\n", "the interval $(10,50)$.\n", "\n", "- Since the interval has a width of 40, the curve must have a height\n", " of $\\frac{1}{40} = 0.024$ over the interval and 0 elsewhere.\n", "\n", "- The probability that $X \\leq 25$ is the area between 10 and 25, or\n", " $(25-10)\\cdot 0.025 = 0.375$.\n", "\n", "### PDF\n", "\n", "The `dunif()` function calculates the uniform probability density\n", "function for a variable and can also calculate a specific value’s\n", "density." ], "id": "29b063db-078d-4d70-9297-f93535bf99e7" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "range <- seq(0, 100, by = 1) # creating a variable with a uniform distribution\n", "ex.dunif <- dunif(range, min = 10, max = 60) # calculating the PDF of the variable \"range\"\n", "plot(ex.dunif, type = \"o\") # plotting the PDF" ], "id": "cc1ac081-536e-4861-a4cb-3582f5bd9a2f" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### CDF\n", "\n", "The `punif()` function calculates the uniform cumulative distribution\n", "function for the set of values." ], "id": "eb2e104f-26b8-4442-aef2-4c08bd47ca61" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x_cdf <- punif(range, # Vector of quantiles\n", " min = 10, # Lower limit of the distribution (a)\n", " max = 50, # Upper limit of the distribution (b)\n", " lower.tail = TRUE, # If TRUE, probabilities are P(X <= x), or P(X > x) otherwise\n", " log.p = FALSE) # If TRUE, probabilities are given as log\n", "plot(x_cdf, type = \"l\")" ], "id": "ca2f131f-913c-4dee-9029-b3b4dc57190b" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `qunif()` function calculates, based on the cumulative probability,\n", "where a specific value is located in the distribution of density and\n", "helps us access the quantile distribution probability values from the\n", "data." ], "id": "1e5cf756-f5ff-4311-bedb-e4f9abb3b781" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "quantiles <- seq(0, 1, by = 0.01)\n", "y_qunif <- qunif(quantiles, min = 10, max = 50) \n", "plot(y_qunif, type = \"l\")" ], "id": "6a5f9ea0-02ff-4873-a82d-a003b0b71e1f" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Normal (Gaussian) Distribution\n", "\n", "We first saw the normal distribution in the *Central Tendency notebook*.\n", "The normal distribution is fundamental to many statistic processes as\n", "many random variables in natural and social sciences are normally\n", "distributed (e.g, Height, SAT scores all follow a normal distribution).\n", "We refer to this type of distribution as “normal” because it’s\n", "distribution is symmetrical and bell-shaped.\n", "\n", "A normal distribution is **parameterized** by its mean $\\mu$ and its\n", "standard deviation $\\sigma$, and it is expressed as $N(\\mu,\\sigma)$. We\n", "cannot calculate the normal distribution without knowing the mean and\n", "the standard deviation.\n", "\n", "The PDF has a complex equation, which can be written as:\n", "\n", "$$\n", "f(x; \\mu, \\sigma) = \\displaystyle \\frac{x^{-(x-\\mu)^{2}/(2\\sigma^{2})}}{\\sigma\\sqrt{2\\pi}}\n", "$$\n", "\n", "- A **standard normal distribution** is a special normal distribution\n", " since it has a mean equal to zero and a standard deviation equal to\n", " 1 ($\\mu=0$ and $\\sigma=1$), hence, $N(0,1)$:\n", "\n", "- Standard normal variables are often denoted by $Z$\n", "\n", "- Standard normal PDF is denoted by $\\phi$\n", "\n", "- Standard normal CDF is denoted by $\\Phi$\n", "\n", "To generate simulated normal random variables, we can use the\n", "`rnorm()`function, which is similar to the `runif()` function.\n", "\n", "### Simulation" ], "id": "13474a88-b189-4a0b-926b-e05ebb32b914" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " x <- rnorm(10000, # number of observations\n", " mean = 0, # mean\n", " sd = 1) # sd\n", " hist(x, probability=TRUE) # the command hist() creates a histogram using variable x,\n", " xx <- seq(min(x), max(x), length=100)\n", " lines(xx, dnorm(xx, mean=0, sd=1))" ], "id": "83770a22-ef29-4cfc-a55b-390e88f440a0" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### PDF\n", "\n", "As with the uniform distribution, we can use `dnorm` to plot the\n", "standard normal pdf." ], "id": "a09c2374-cfd8-4190-b220-f3efb022b896" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " # create a sequence of 100 equally spaced numbers between -4 and 4\n", " x <- seq(-4, 4, length=100)\n", "\n", " # create a vector of values that shows the height of the probability distribution\n", " # for each value in x\n", " y <- dnorm(x)\n", "\n", " # plot x and y as a scatterplot with connected lines (type = \"l\") and add\n", " # an x-axis with custom labels\n", " plot(x,y, type = \"l\", lwd = 2, axes = FALSE, xlab = \"\", ylab = \"\")\n", " axis(1, at = -3:3, labels = c(\"-3s\", \"-2s\", \"-1s\", \"mean\", \"1s\", \"2s\", \"3s\"))" ], "id": "d3a9d548-4837-4e7f-b8b3-70b8746a468e" }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have used the random values generated to observe its bell shaped\n", "distribution. This is a standard normal PDF because the mean is zero and\n", "the standard deviation is one.\n", "\n", "We can also change the numbers of mean and sd in the `rnorm()` command\n", "to make the distribution not standard.\n", "\n", "### CDF\n", "\n", "- The `pnorm()` function can 1) give the entire CDF curve of a\n", " normally distributed random *variable* 2) give the probability of a\n", " normally distributed random *number* to be less than the value of a\n", " given number." ], "id": "e9de55ca-ba59-44ee-8d2f-cf4400ec036a" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " curve(pnorm(x), \n", " xlim = c(-3.5, 3.5), \n", " ylab = \"Probability\", \n", " main = \"Standard Normal Cumulative Distribution Function\")" ], "id": "2290a251-6db6-4732-8fbf-0f641d125149" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " pnorm(27.4, mean=50, sd=20) # gives you the CDF at that specific location\n", " pnorm(27.4, 50, 20)" ], "id": "e32e6fa0-95cd-431a-9218-e79c0bc0cf0d" }, { "cell_type": "markdown", "metadata": {}, "source": [ "- The `qnorm()` function can create a percent point function (ppf),\n", " which is the inverse curve of the cumulative distribution function.\n", " The `qnorm()` function gives the inverse of the CDF by taking the\n", " density value and giving a number with a matching cumulative value.\n", " - The CDF of a specific value is the probability of a normally\n", " distributed value of a random variable to be less than the value\n", " of a **given number**.\n", " - To create the ppf, we start with that probability and use the\n", " `qnorm()` function to compute the corresponding **given number**\n", " for the cumulative distribution. Hence, `qnorm()` will calculate\n", " the area before it is X% of the sample." ], "id": "a9d08a52-af1e-40cb-98e1-105744da3a14" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " curve(qnorm(x), \n", " xlim = c(0, 1), \n", " xlab = \"Probability\",\n", " ylab = \"x\", \n", " main = \"Quantile (inverse CDF) Function\")" ], "id": "b188336f-8767-4d62-8e55-8e9c6b0defa4" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " qnorm(0.95, mean=100, sd=15)" ], "id": "b4daaa81-f71c-43ae-bde6-b89bd87baacc" }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Finally, the function `dnorm()` gives the height of the probability\n", " distribution at each point for a given mean and standard deviation.\n", "\n", " - Since the height of the pdf curve is the density, `dnorm()` can\n", " also be used to calculate the entire density curve, as observed\n", " in the command *lines(xx, dnorm(xx, mean=0, sd=1))*" ], "id": "e8cf5951-bc67-4937-af37-eb0c3c1e1ecb" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " dnorm(100, mean=100, sd=15)" ], "id": "f337c991-83fd-40b7-9c04-b065b70d776d" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Student’s $t$-Distribution\n", "\n", "The **Student’s** $t$-distribution is a continuous distribution that\n", "occurs when we estimate the sampling distribution of a normally\n", "distributed population with a small sample size and an uknown standard\n", "deviation. This is an important concept that we will explore in a later\n", "module.\n", "\n", "- The $t$-distribution is based on the number of observations and the\n", " degrees of freedom.\n", "\n", "- A degree of freedom ($\\nu$) is the maximum number of logically\n", " independent values, which is the number of values that need to be\n", " known in order to know all of the values. For example, let’s say you\n", " have 3 values with an average of 5. If you sample two of the values\n", " and they turn out to be 4, and 5, even without sampling the final\n", " value, you know that the final value is 6. Hence, there is no\n", " freedom in the last value.\n", "\n", "- In the case of the $t$-distribution, the degree(s) of freedom can be\n", " represented as $\\nu = n-1$, with $n$ being the sample size.\n", "\n", "- When $\\nu$ is large, the $t$-distribution begins to look like a\n", " standard normal distribution.\n", "\n", "- This approximation between standard normal and $t$-distribution can\n", " start being noticed around $\\nu \\geq 30$.\n", "\n", "As with the uniform and normal distribution, to generate random values\n", "that together have a t-distribution we add the prefix `r` to the name of\n", "the distribution, `rt()`.\n", "\n", "### Simulation" ], "id": "ff9a3d51-e545-4125-bc2c-ce6b0b87e946" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " n <- 100\n", " df <- n - 1\n", " samples <- rt(n, df)\n", " hist(samples,breaks = 20, freq = FALSE)\n", " xx <- seq(min(samples), max(samples), length=100)\n", " lines(xx, dt(xx, df))" ], "id": "ab250442-0bcf-4ded-8137-fce8dfdc2431" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Although the t-distribution is bell-shaped and symmetrical like the\n", "normal distribution, it is not as thin as a normal distribution. Hence,\n", "the data is more spread out than a normal distribution—this is a\n", "characteristic explained by the central limit theorem (CLT) and the law\n", "of large numbers (LLN), which we will explore in future modules.\n", "\n", "### PDF\n", "\n", "The function `dt()` calculates the PDF or the density of a particular\n", "variable, depending on the sample size and degrees of freedom.\n", "\n", "In the examples shown below we use the variable `ex.tvalues` which is a\n", "sequence of numbers ranging from -4 to 4 with increments of 0.01.\n", "Therefore there are 800 numbers generated with the degrees of freedom of\n", "799." ], "id": "8cdf060f-9c0c-4730-997e-03d008fc399e" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " ex.tvalues <- seq(- 4, 4, by = 0.01) # generating a sequence of number \n", " ex_dt <- dt(ex.tvalues, df = 799) # calculating the PDF\n", " plot(ex_dt, type=\"l\") " ], "id": "5acb8305-03ce-4233-b4a3-2062d3bae6bc" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### CDF\n", "\n", "The `pt()` function calculates the entire CDF curve of a t-distributed\n", "random *variable* and gives the probability of a t-distributed random\n", "*number* that is less that the value of a given number." ], "id": "aa801c81-1986-4afc-a53c-bcf8803a471d" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " ex_pt <- pt(ex.tvalues, df = 799) # calculating CDF\n", " plot(ex_pt, type = \"l\") " ], "id": "299b0a4a-1797-4bde-bb82-738f4fa617f3" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `qnorm()` function takes the probability value and gives a number\n", "whose cumulative value matches the probability value. This function can\n", "also create a percent point function (ppf)." ], "id": "792c99db-ea0a-4923-822c-332e041797a8" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " ex.qtvalues <- seq(0, 1, by = 0.01) # generating a sequence of number \n", " ex_qt <- qt(ex.qtvalues, df = 99) # calculating the ppf\n", " plot(ex_qt, type = \"l\") # plotting the ppf " ], "id": "0f60d5a6-93d3-4ace-a0f6-369166225e59" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Beyond these three common distributions, there are many other types of\n", "distributions such as chi-square distribution or f-distribution. In rare\n", "cases we may have variables that do not fit a distribution. This could\n", "happen because the data is being distributed sporadically and can\n", "therefore not be approximated by any common distribution. In these\n", "cases, we describe it as a non-parametrical distribution.\n", "\n", "# Part 3: Exercises\n", "\n", "## Exercise 12\n", "\n", "Which of the following random variables are most likely to be\n", "uniformally distributed.\n", "\n", "**A.** The height of a UBC student \n", "**B.** The wages of a UBC student \n", "**C.** The birthday of a UBC student" ], "id": "8c8b0c06-af16-428b-9324-5b7a465e8ea9" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Enter your answer here as \"A\", \"B\", or \"C\"\n", "\n", "answer_12 <- \"...\"\n", "test_12(answer_12)" ], "id": "4cc9bab2-905b-46db-8a5b-dd3266b7076d" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 13\n", "\n", "Which of the following random variables are most likely to be normally\n", "distributed.\n", "\n", "**A.** The height of a UBC student \n", "**B.** The grades of a particular course \n", "**C.** The birthday of a UBC student" ], "id": "62a252f3-ee5d-4a0c-a143-ff66de583fc5" }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Enter your answer here as \"A\", \"B\", or \"C\"\n", "\n", "answer_13 <- \"...\"\n", "test_13(answer_13)" ], "id": "aaa30d6e-4ee8-45e8-b4da-4b00034f48f2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 14\n", "\n", "Given our uniform distribution `example_unif`, find $F(72)$. Note that\n", "you don’t need to calculate the exact probability given the\n", "distribution. You only need to know that this random variable is\n", "uniformly distributed for values between 10 and 100." ], "id": "90639d37-65d7-48e9-962f-d9081b2f4418" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Enter your answer as an integer below. Your answer should only have one decimal place. \n", "\n", "answer_14 <- ...\n", "test_14()" ], "id": "3cb3fd7b-6583-4f9a-8be4-2663351000dd" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 15\n", "\n", "Assume we have a standard normal distribution. Find $F(0)$" ], "id": "adf6fa3b-a375-4ecc-bf61-51f7d06c0fb2" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Enter your answer as an integer below. Your answer should only have one decimal place\n", "\n", "answer_15 <- ...\n", "test_15()" ], "id": "0646bc0d-297b-43dd-b719-05446b061cab" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 16\n", "\n", "Let’s assume we have a students $t-$distribtion that is nearly\n", "coincident to the corresponding normal distribution. What must be true?\n", "\n", "**A.** The degrees of freedom parameter must be very large. \n", "**B.** The degrees of freedom parameter must be very small. \n", "**C.** The degrees of freedom parameter must be equal to the mean of the\n", "normal distribution." ], "id": "a35400a2-2cc8-4697-9dba-0051493e9595" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Enter your answer here as \"A\", \"B\", or \"C\"\n", "\n", "answer_16 <- \"...\"\n", "test_16(answer_16)" ], "id": "71b2dd92-e1b7-42fa-be5c-c232634f1a50" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "ir", "display_name": "R", "language": "r" } } }