{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 07 - Conducting Within Group Analysis\n",
        "\n",
        "Marina Adshade, Paul Corcuera, Giulia Lo Forte, Jane Platt  \n",
        "2024-05-29\n",
        "\n",
        "## Prerequisites\n",
        "\n",
        "1.  Be able to effectively use Stata do-files and generate log-files.\n",
        "2.  Be able to change your directory so that Stata can find your files.\n",
        "3.  Import data sets in csv and dta format.\n",
        "4.  Save data files.\n",
        "\n",
        "## Learning Outcomes\n",
        "\n",
        "1.  Create new variables using the command `egen`.\n",
        "2.  Know when to use the pre-command `by` and when to use `bysort`.\n",
        "3.  Use the command `collapse` to create a new data set of summary\n",
        "    statistics.\n",
        "4.  Change a panel data set to a cross-sectional data set using the\n",
        "    command `reshape`.\n",
        "\n",
        "## 7.0 Intro"
      ],
      "id": "7d838652-1191-42fc-9997-b29e8b384ef7"
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [],
      "source": [
        "import stata_setup\n",
        "stata_setup.config('C:\\Program Files\\Stata18/','se')"
      ],
      "id": "168b5f36"
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        ">>> import sys\n",
        ">>> sys.path.append('/Applications/Stata/utilities') # make sure this is the same as what you set up in Module 01, Section 1.3: Setting Up the STATA Path\n",
        ">>> from pystata import config\n",
        ">>> config.init('se')"
      ],
      "id": "aa79e7ec"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 7.1 Introduction to Working Within Groups\n",
        "\n",
        "There are times when we will want to analyze our data while considering\n",
        "observations as a part of a group. Consider some of the following\n",
        "examples:\n",
        "\n",
        "-   We would like to know the average wages of workers by educational\n",
        "    grouping, in each year of the data.\n",
        "-   We would like to know the standard deviation of men and women’s\n",
        "    earnings, by geographic region.\n",
        "-   We would like to know the top quintile of wealth, by birth cohort.\n",
        "\n",
        "In this module, we will go over how to calculate these statistics using\n",
        "the fake data set introduced in the previous lecture. Recall that this\n",
        "data set is simulating information of workers in the years 1982-2012 in\n",
        "a fake country where a training program was introduced in 2003 to boost\n",
        "their earnings.\n",
        "\n",
        "Let’s begin by loading that data set into Stata:"
      ],
      "id": "1dbec98f-f229-4533-8fee-62948ecec665"
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "clear *\n",
        "* cd \" \"\n",
        "use \"fake_data.dta\", clear"
      ],
      "id": "421d1376"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 7.2 Generating Variables using `generate`\n",
        "\n",
        "When we are working on a particular project, it is important to know how\n",
        "to create variables that are computed for a group rather than an\n",
        "individual or an observation. For instance, we may have a data set that\n",
        "is divided by individual and by year. We might want the variables to\n",
        "show us the statistics of a particular individual throughout the years\n",
        "or the statistics of all individuals each year.\n",
        "\n",
        "Stata provides a function to easily compute such statistics. The key to\n",
        "this analysis is the pre-command `by`. A pre-command is simply a prefix\n",
        "that tells Stata how we want it to run the command. In the case of `by`,\n",
        "we tell Stata to run the command on the subsets of data. The only\n",
        "requirement to using this pre-command is to ensure that the data is\n",
        "sorted the correct way.\n",
        "\n",
        "Let’s take a look at our data by using the `browse` command we learned\n",
        "in [Module\n",
        "5](https://comet.arts.ubc.ca/docs/Research/econ490-pystata/05_Opening_Datasets.html)."
      ],
      "id": "31e18a23-b099-49dc-9e5c-3711e27ca5c5"
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "5b0919b1"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can tell here that the data is sorted by the variable *workerid*.\n",
        "\n",
        "We use the pre-command `by` alongside the command `generate` to develop\n",
        "these group-compounded variables.\n",
        "\n",
        "If we use variables other than *workerid* (the variable by which the\n",
        "data is sorted) to group our new variable, we will not be able to\n",
        "generate the new variable. We can see this error when we run the command\n",
        "below."
      ],
      "id": "08ea0c72-fb7d-45f9-acb5-9dae70d43546"
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop var_one //recall that capture drop tells Stata to ignore any errors if var_one does not exist, and to drop it if it does\n",
        "by year: generate var_one = 1 "
      ],
      "id": "d340bac0"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If we want to group by year, Stata expects us to sort the data such that\n",
        "all observations corresponding to the same year are next to each other.\n",
        "We can use the `sort` pre-command as follows."
      ],
      "id": "25d7dc59-1963-43f1-8eec-59279bc4726b"
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "sort year "
      ],
      "id": "4c074e0f"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let’s take a look at our data now."
      ],
      "id": "e45e1517-6813-4827-80f8-acc562632977"
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "fb5f122c"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let’s try the command above again, now with the sorted data."
      ],
      "id": "dc18e770-424f-4eaa-9833-0caaa4e5ccff"
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop var_one \n",
        "by year: generate var_one = 1 "
      ],
      "id": "a4050c0e"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that the data is sorted by year, the code works!\n",
        "\n",
        "We could have also used the pre-command `bysort` instead of sorting the\n",
        "data with `sort` and then using `by`. Everything is done in one step!\n",
        "\n",
        "Let’s sort the data, so it is reverted back to the same ordering scheme\n",
        "as when we started (by *workerid*), and generate our new variable again.\n",
        "\n",
        "Stata also lets us sort by two variables. The following block of code\n",
        "tells Stata to first `sort` the data by *workerid*, and then within each\n",
        "*workerid*, to sort the data by *year*."
      ],
      "id": "e78081e8-1b57-467b-963a-03ebc3cf703f"
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "sort workerid year "
      ],
      "id": "42cfda80"
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop var_one \n",
        "bysort year: generate var_one = 1 "
      ],
      "id": "fbbaec1e"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The variable we have created is not interesting by any means. It simply\n",
        "takes the value of 1 everywhere. In fact, we haven’t done anything that\n",
        "we couldn’t have done with `gen var_one=1`. We can see this by using the\n",
        "`summarize` command."
      ],
      "id": "b1b2df5f-e571-4850-b415-f6131451465a"
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "summarize var_one"
      ],
      "id": "e67a2ca1"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You may not be aware, but Stata records the observation number as a\n",
        "hidden variable (a scalar) called <i>\\_n</i> and the total number of\n",
        "observations as <i>\\_N</i>.\n",
        "\n",
        "Let’s take a look at these by creating two newvariables: one that is the\n",
        "observation number and one that is the total number of observations."
      ],
      "id": "0c26d6b1-92cf-44cb-bdf7-54435044be95"
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop obs_number \n",
        "generate obs_number = _n \n",
        "\n",
        "capture drop tot_obs\n",
        "generate tot_obs = _N"
      ],
      "id": "5879a446"
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "7874676c"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As expected, the numbering of observations is sensitive to the way that\n",
        "the data is sorted! The cool thing is that whenever we use the\n",
        "pre-command `by`, the scalars <i>\\_n</i> and <i>\\_N</i> record the\n",
        "observation number and total number of observations for each group\n",
        "separately. Let’s check that below:"
      ],
      "id": "5d40daa8-744f-4438-8308-3cb1c0ffee31"
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop obs_number \n",
        "bysort workerid: generate obs_number = _n \n",
        "\n",
        "capture drop tot_obs\n",
        "bysort workerid: generate tot_obs = _N"
      ],
      "id": "eba7949c"
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "62838f0a"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, some workers are observed only 2 times in the data (they\n",
        "were only surveyed in two years), whereas other workers are observed 8\n",
        "times (they were surveyed in 8 years). By knowing (and recording in a\n",
        "variable) the number of times a worker has been observed, we can do some\n",
        "analysis based on this information. For example, in some cases you might\n",
        "be interested in keeping only workers who are observed across all time\n",
        "periods. In this case, you could use the command:"
      ],
      "id": "566f810d-e8a2-48fd-9bf8-4491e248e92d"
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "keep if tot_obs==8"
      ],
      "id": "2863f3f5"
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "0e389b83"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 7.3 Generating Variables Using Extended Generate\n",
        "\n",
        "The command `egen` is used whenever we want to create variables which\n",
        "require access to some functions (e.g. mean, standard deviation, min).\n",
        "The basic syntax works as follows:\n",
        "\n",
        "``` stata\n",
        " bysort groupvar: egen new_var = function() , options\n",
        "```\n",
        "\n",
        "Let’s see an example where we create a new variable called\n",
        "*avg_earnings*, which is the mean of earnings for every worker. We will\n",
        "need to reload our data since we dropped many observations above when we\n",
        "used the `keep` command."
      ],
      "id": "c3b1be10-6bc5-4df3-842e-9b3d2e7eeb7f"
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "clear *\n",
        "use \"fake_data.dta\", clear"
      ],
      "id": "5aea4978"
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop avg_earnings\n",
        "bysort workerid: egen avg_earnings = mean(earnings)"
      ],
      "id": "3f31d40a"
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop total_earnings\n",
        "bysort workerid: egen total_earnings = total(earnings)"
      ],
      "id": "aae2710b"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "By definition, these commands will create variables that use information\n",
        "across different observations. You can check the list of available\n",
        "functions by writing `help egen` in the Stata command window.\n",
        "\n",
        "In this documentation, we can see that there are some functions that do\n",
        "not allow for `by`. For example, suppose we want to create the total sum\n",
        "across different variables in the same row. We do this below by taking\n",
        "the sum of *start_year*, *region*, and *treated*."
      ],
      "id": "73ed7bc7-e967-4924-9b9a-03a253f5ea1f"
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "cap drop sum_of_vars\n",
        "egen sum_of_vars = rowtotal(start_year region treated)"
      ],
      "id": "4fae7f98"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The variable we are creating for the example has no particular meaning,\n",
        "but what we need to notice is that the function `rowtotal()` only sums\n",
        "the non-missing values in our variables. This means that if there is a\n",
        "missing value in any of the three variables, the sum only occurs between\n",
        "the two variables that do not have the missing value. We could also\n",
        "write this command as `gen sum_of_vars = start_year + region + treated`;\n",
        "however, if there is a missing value (`.`) in *start_year*, *region*, or\n",
        "*treated*, then the generated value for *sum_of_vars* will also be a\n",
        "missing value. The answer lies in the missing observations. If we sum\n",
        "any number with a missing value (`.`), then the sum will also be missing\n",
        "when using `generate`, but not when using `egen`.\n",
        "\n",
        "Just as with `sort`, we can also use `by` with multiple variables. Doing\n",
        "so tells Stata to run the command over all combinations of subgroups.\n",
        "Here will use *year* and *region* in one command. This tells Stata to\n",
        "generate a new variable for each *year*-*region* combination."
      ],
      "id": "706586f6-96ad-4706-850b-9ef6f45ec0fc"
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop regionyear_earnings\n",
        "bysort year region : egen regionyear_earnings = total(earnings)"
      ],
      "id": "d741bb91"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "What this command gives us is a new variable that records total earnings\n",
        "in each region for every year.\n",
        "\n",
        "## 7.4 Collapsing Data\n",
        "\n",
        "We can also compute statistics at some group level with the `collapse`\n",
        "command. `collapse` is extremely useful whenever we want to apply sample\n",
        "weights to our data (we will learn more about this in [Module\n",
        "11](https://comet.arts.ubc.ca/docs/Research/econ490-pystata/11_Linear_Reg.html)).\n",
        "Sample weights cannot be applied using `egen` but are often extremely\n",
        "important when using micro data. These weights allow us to manipulate\n",
        "our data to better reflect the true composition of the data when the\n",
        "authorities that collected the data might have over-sampled some\n",
        "segments of the population.\n",
        "\n",
        "The syntax is:\n",
        "\n",
        "``` stata\n",
        "collapse (statistic1) new_name = existing_variable (statistic2) new_name2 = existing_variable2 ... [pweight =     weight_variable], by(group) \n",
        "```\n",
        "\n",
        "We can find a full list of possible statistics that `collapse` can take\n",
        "by running the command `help collapse`. We can also learn more about\n",
        "using weights by typing `help weight`.\n",
        "\n",
        "Let’s suppose we want to create a data set at the region-year level\n",
        "using information in the current data set, but we want to use the sample\n",
        "weights that were provided with our data (*sample_weight*). First, we\n",
        "decide which statistics we want to keep from the original data set. For\n",
        "the sake of explanation, let’s suppose we want to keep the average\n",
        "earnings, the variance of earnings, and the total employment. We will\n",
        "have three new variables: *avg_earnings*, *sd_earnings*, *tot_emp*. We\n",
        "write the following:"
      ],
      "id": "f3353415-5655-4e2b-8a7a-cf5b4f93ee6a"
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "collapse (mean) avg_earnings = earnings (sd) sd_earnings = earnings (count) tot_emp = earnings [pweight = sample_weight], by(region year)"
      ],
      "id": "95078538"
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "af2ebacb"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Warning:** When we use `collapse`, Stata will produce a new data set\n",
        "with the results, and in the process it drops the data set that was\n",
        "loaded at the time the command was run. If we need to keep the original\n",
        "data, be certain to save the file before running this command.\n",
        "\n",
        "## 7.5 Reshaping\n",
        "\n",
        "We have collapsed our data and so we need to import the data again to\n",
        "gain access to the full data set."
      ],
      "id": "6fdcd232-2704-4a1c-b141-dc1fcb7bb887"
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "clear *\n",
        "\n",
        "use \"fake_data.dta\", clear"
      ],
      "id": "53310cff"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Notice that the nature of this particular data set is panel form;\n",
        "individuals have been followed over many years. Sometimes we are\n",
        "interested in working with a cross section (i.e. we have 1 observation\n",
        "per worker which includes all of the years). Is there a simple way to go\n",
        "back and forth between these two? Yes!\n",
        "\n",
        "The command’s name is `reshape` and has two main forms: `wide` and\n",
        "`long`. `wide` data is cross-sectional in nature, whereas `long` is the\n",
        "usual panel.\n",
        "\n",
        "Suppose we want to record the earnings of workers while keeping the\n",
        "information across years. This entails transforming our panel data into\n",
        "a cross sectional data set. We want one observation per worker, where\n",
        "each observation has all of the years. This is a `wide` transformation."
      ],
      "id": "aede0639-85c7-4a8a-a44e-48619c50a207"
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "reshape wide earnings region age start_year sample_weight quarter_birth, i(workerid) j(year)"
      ],
      "id": "ca43f8eb"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Warning:** This command acts on all of the variables in our data set.\n",
        "If we don’t include them in the list, Stata will assume that they do not\n",
        "vary across *i* (in this case *workerid*). If we don’t check this\n",
        "beforehand, we may get an error message!"
      ],
      "id": "9fdd2878-1520-4ad3-8bc4-7ac0a79770b7"
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "337c4525"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There are so many missing values in the data! Should we worry? Not at\n",
        "all. As a matter of fact, we learned at the beginning of this module\n",
        "that many workers are not observed across all years. That’s what these\n",
        "missing values are representing.\n",
        "\n",
        "Notice that the variable *year* which was part of the command line (the\n",
        "`j(year)` part) has disappeared. We now have one observation per worker,\n",
        "with their information recorded across years in a cross-sectional way.\n",
        "\n",
        "How do we go from a `wide` data set to a regular panel form? We will use\n",
        "the `reshape long` command. Note that to do this, we need to specify the\n",
        "prefix variables. These are formally known as `stubs` in Stata. They are\n",
        "the variables that all share the same prefix (in this case, *year*),\n",
        "that will be transformed into one variable. When we write `j(year)`,\n",
        "Stata will create a new variable called *year*."
      ],
      "id": "92eff5ae-b664-4221-8839-2af1e96bec6a"
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "reshape long earnings region age  start_year sample_weight, i(workerid) j(year) "
      ],
      "id": "2e4e9901"
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "c426ca00"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Notice that we now have an observation for every worker in every year,\n",
        "although we know some workers are only observed in a subset of these.\n",
        "This is known as a **balanced panel**.\n",
        "\n",
        "To retrieve the original data set, we get rid of such observations with\n",
        "missing values."
      ],
      "id": "3dc37a55-e9c8-4322-a87b-bd79b6ac270a"
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "keep if !missing(earnings)"
      ],
      "id": "2888efb8"
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "%browse 10"
      ],
      "id": "1c099a08"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 7.6 Errors\n",
        "\n",
        "### 7.6.1. Sort\n",
        "\n",
        "To develop group-compounded variables, we first need to ensure that we\n",
        "sort the observations by the variable. Not sorting the obserations will\n",
        "return an error code."
      ],
      "id": "e3f94532-5185-48ee-a738-6d744c109489"
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop var\n",
        "by sex: generate var = _n"
      ],
      "id": "729f1468"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The correct method of of generating compounded variables is below:"
      ],
      "id": "a1fd69ca-6d29-4089-8f1c-561c14af585a"
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "capture drop var\n",
        "bysort sex: generate var = _n"
      ],
      "id": "44884250"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Take a look at it below:"
      ],
      "id": "68a90f6f-fcc1-41c1-9d0d-8b69553e33b2"
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "summarize var"
      ],
      "id": "34646651"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 7.6.2. Reshape Error\n",
        "\n",
        "Reshaping data can be tricky and doing so incorrectly can cause many\n",
        "variables to be dropped in the proccess. The command `reshape error` can\n",
        "be used to identify the issues encountered when reshaping data."
      ],
      "id": "d64cca7f-27cd-4e62-8f3f-02852cb4c9dc"
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "clear *\n",
        "use \"fake_data.dta\", clear"
      ],
      "id": "d3380f2a"
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "reshape wide earnings sex, i(year) j(workerid)"
      ],
      "id": "01457bec"
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%stata\n",
        "\n",
        "reshape error"
      ],
      "id": "78588331"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Can you tell what the error is here?\n",
        "\n",
        "## 7.7 Wrap Up\n",
        "\n",
        "In this module, we have covered some very useful skills that will be\n",
        "useful for exploring data sets. Namely, these skills will help us both\n",
        "prepare data for empirical analysis (i.e. turning cross sectional data\n",
        "into panel data) and create summary statistics that illustrate our\n",
        "results. In the next module, we will look at how to work with multiple\n",
        "data sets simultaneously and merge them into one.\n",
        "\n",
        "## 7.8 Wrap-up Table\n",
        "\n",
        "| Command | Function |\n",
        "|----------------------------------|--------------------------------------|\n",
        "| `by` | It is a pre-command used to Repeat Stata command on subsets of the data |\n",
        "| `generate` | It generates variables |\n",
        "| `sort` | It sorts data |\n",
        "| `summary` | It summarizes statistics of a data set |\n",
        "| `_n` | It records the observation number |\n",
        "| `_N` | It records the total number of observations for each group separately |\n",
        "| `drop` | It drops variables or observations |\n",
        "| `keep` | It keeps variables or observations that satisfy a specified condition |\n",
        "| `egenerate` | It create variables that require access to some functions |\n",
        "| `rowtotal()` | It sums non-missing values for each observation of a list of variables |\n",
        "| `collapse` | It makes a data set of a summary of statistics |\n",
        "| `reshape` | It converts data from wide to long and vice versa |\n",
        "\n",
        "## References\n",
        "\n",
        "[Reshape data from wide format to long\n",
        "format](https://www.youtube.com/watch?v=Bx9kVdkr9oY) <br> [(Non\n",
        "StataCorp) How to group data in STATA with SORT and\n",
        "BY](https://www.youtube.com/watch?v=nEOyH0AFKHc) [Syntax for\n",
        "pre-commands](https://www.stata.com/manuals/u11.pdf)"
      ],
      "id": "9408c19b-5653-42fb-820b-f5588e905fbe"
    }
  ],
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "path": "/usr/local/share/jupyter/kernels/python3"
    },
    "language_info": {
      "name": "python",
      "codemirror_mode": {
        "name": "ipython",
        "version": "3"
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    }
  }
}