Skip to content

Instantly share code, notes, and snippets.

@viniciusmss

viniciusmss/Sensitivity to Hidden Bias.ipynb

Created March 11, 2020 08:49
Show Gist options
  • Save viniciusmss/a156c3f22081fb5c690cdd58658f61fa to your computer and use it in GitHub Desktop.
Save viniciusmss/a156c3f22081fb5c690cdd58658f61fa to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Sensitivity to Hidden Bias.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<span style=\"background-color:#b83c3c;padding: 5px 20px 5px 20px;line-height:30px;color:white;font-weight: bold; border-radius: 25px\">Overview</span>\n",
"\n",
"Run the code cells below one-by-one and answer the questions as you progress."
]
},
{
"cell_type": "code",
"execution_count": 98,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:54.272159Z",
"start_time": "2020-03-11T08:32:54.213Z"
}
},
"outputs": [],
"source": [
"# If necessary\n",
"# install.packages(Matching) \n",
"# install.packages(rbounds) \n",
"# install.packages(sensitivitymv) \n",
"\n",
"# Imports\n",
"library(Matching) # The usual library with matching functions and the lalonde dataset\n",
"library(rbounds) # The simplest library for sensitivity analysis\n",
"library(sensitivitymv) # A more up-to-date options\n",
"\n",
"# Load the data\n",
"data(lalonde)\n",
"\n",
"# Set random seed\n",
"set.seed(2020)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Propensity Score Matching\n",
"\n",
"Let us check the sensitivity to hidden bias after running a very simple propensity score matching procedure."
]
},
{
"cell_type": "code",
"execution_count": 99,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:54.325017Z",
"start_time": "2020-03-11T08:32:54.219Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Estimate... 2624.3 \n",
"AI SE...... 802.19 \n",
"T-stat..... 3.2714 \n",
"p.val...... 0.0010702 \n",
"\n",
"Original number of observations.............. 445 \n",
"Original number of treated obs............... 185 \n",
"Matched number of observations............... 185 \n",
"Matched number of observations (unweighted). 344 \n",
"\n"
]
}
],
"source": [
"# Specify a simple propensity score model\n",
"glm1 <- glm(treat~age + educ + black + hisp + married + nodegr + re74 + re75, \n",
" family=binomial, data=lalonde)\n",
"\n",
"# Store variables\n",
"X <- glm1$fitted\n",
"Y <- lalonde$re78\n",
"Tr <- lalonde$treat\n",
"\n",
"# Run Matching\n",
"mout <- Match(Y=Y, Tr=Tr, X=X, M=1)\n",
"summary(mout)"
]
},
{
"cell_type": "code",
"execution_count": 100,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:55.443543Z",
"start_time": "2020-03-11T08:32:54.222Z"
},
"scrolled": true
},
"outputs": [],
"source": [
"# Check the level of balance\n",
"mb <- MatchBalance(treat~age + educ + black + hisp + married + nodegr + re74 + re75, \n",
" data=lalonde, match.out=mout, print.level=0)"
]
},
{
"cell_type": "code",
"execution_count": 101,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:55.476452Z",
"start_time": "2020-03-11T08:32:54.226Z"
}
},
"outputs": [],
"source": [
"# Receives match balance output and prints the variable and value of smallest p-value\n",
"balanceprint <- function(mb) cat(sprintf(\"Smallest p-value.\\nVar\\t: %s\\np-val\\t: %.4f\", \n",
" mb$AMsmallestVarName, as.numeric(mb$AMsmallest.p.value)))"
]
},
{
"cell_type": "code",
"execution_count": 102,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:55.515926Z",
"start_time": "2020-03-11T08:32:54.229Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Smallest p-value.\n",
"Var\t: re74\n",
"p-val\t: 0.0000"
]
}
],
"source": [
"balanceprint(mb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<span class=\"minerva-question\" style=\"background-color:#b83c3c;padding: 5px 20px 5px 20px;line-height:30px;color:white;font-weight: bold; border-radius: 25px\">Note #1</span>\n",
"\n",
"The `rbounds` library contains the `psens` function, which returns the range of significance values for different levels of $\\Gamma$. The output is very similar to Table 2 in Rosenbaum (2005)."
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:55.564796Z",
"start_time": "2020-03-11T08:32:54.233Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<dl>\n",
"\t<dt>$Gamma</dt>\n",
"\t\t<dd>1.5</dd>\n",
"\t<dt>$GammaInc</dt>\n",
"\t\t<dd>0.1</dd>\n",
"\t<dt>$pval</dt>\n",
"\t\t<dd>1e-04</dd>\n",
"\t<dt>$msg</dt>\n",
"\t\t<dd>'Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \\n'</dd>\n",
"\t<dt>$bounds</dt>\n",
"\t\t<dd><table>\n",
"<thead><tr><th scope=col>Gamma</th><th scope=col>Lower bound</th><th scope=col>Upper bound</th></tr></thead>\n",
"<tbody>\n",
"\t<tr><td>1.0 </td><td>1e-04 </td><td>0.0001</td></tr>\n",
"\t<tr><td>1.1 </td><td>0e+00 </td><td>0.0008</td></tr>\n",
"\t<tr><td>1.2 </td><td>0e+00 </td><td>0.0063</td></tr>\n",
"\t<tr><td>1.3 </td><td>0e+00 </td><td>0.0286</td></tr>\n",
"\t<tr><td>1.4 </td><td>0e+00 </td><td>0.0875</td></tr>\n",
"\t<tr><td>1.5 </td><td>0e+00 </td><td>0.1976</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</dd>\n",
"\t<dt>$note</dt>\n",
"\t\t<dd>'Note: Gamma is Odds of Differential Assignment To\\n Treatment Due to Unobserved Factors \\n'</dd>\n",
"</dl>\n"
],
"text/latex": [
"\\begin{description}\n",
"\\item[\\$Gamma] 1.5\n",
"\\item[\\$GammaInc] 0.1\n",
"\\item[\\$pval] 1e-04\n",
"\\item[\\$msg] 'Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \\textbackslash{}n'\n",
"\\item[\\$bounds] \\begin{tabular}{r|lll}\n",
" Gamma & Lower bound & Upper bound\\\\\n",
"\\hline\n",
"\t 1.0 & 1e-04 & 0.0001\\\\\n",
"\t 1.1 & 0e+00 & 0.0008\\\\\n",
"\t 1.2 & 0e+00 & 0.0063\\\\\n",
"\t 1.3 & 0e+00 & 0.0286\\\\\n",
"\t 1.4 & 0e+00 & 0.0875\\\\\n",
"\t 1.5 & 0e+00 & 0.1976\\\\\n",
"\\end{tabular}\n",
"\n",
"\\item[\\$note] 'Note: Gamma is Odds of Differential Assignment To\\textbackslash{}n Treatment Due to Unobserved Factors \\textbackslash{}n'\n",
"\\end{description}\n"
],
"text/markdown": [
"$Gamma\n",
": 1.5\n",
"$GammaInc\n",
": 0.1\n",
"$pval\n",
": 1e-04\n",
"$msg\n",
": 'Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \\n'\n",
"$bounds\n",
": \n",
"Gamma | Lower bound | Upper bound | \n",
"|---|---|---|---|---|---|\n",
"| 1.0 | 1e-04 | 0.0001 | \n",
"| 1.1 | 0e+00 | 0.0008 | \n",
"| 1.2 | 0e+00 | 0.0063 | \n",
"| 1.3 | 0e+00 | 0.0286 | \n",
"| 1.4 | 0e+00 | 0.0875 | \n",
"| 1.5 | 0e+00 | 0.1976 | \n",
"\n",
"\n",
"\n",
"$note\n",
": 'Note: Gamma is Odds of Differential Assignment To\\n Treatment Due to Unobserved Factors \\n'\n",
"\n",
"\n"
],
"text/plain": [
"\n",
" Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \n",
" \n",
"Unconfounded estimate .... 1e-04 \n",
"\n",
" Gamma Lower bound Upper bound\n",
" 1.0 1e-04 0.0001\n",
" 1.1 0e+00 0.0008\n",
" 1.2 0e+00 0.0063\n",
" 1.3 0e+00 0.0286\n",
" 1.4 0e+00 0.0875\n",
" 1.5 0e+00 0.1976\n",
"\n",
" Note: Gamma is Odds of Differential Assignment To\n",
" Treatment Due to Unobserved Factors \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"psens(mout, Gamma=1.5, GammaInc=.1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<span class=\"minerva-question\" style=\"background-color:#b83c3c;padding: 5px 20px 5px 20px;line-height:30px;color:white;font-weight: bold; border-radius: 25px\">Question #1</span>\n",
"\n",
"What does the table above tell you about the sensitivity to hidden bias of the ATE in this matched dataset?\n",
"\n",
"_Your response_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Genetic Matching"
]
},
{
"cell_type": "code",
"execution_count": 104,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:55.611675Z",
"start_time": "2020-03-11T08:32:54.237Z"
}
},
"outputs": [],
"source": [
"attach(lalonde)\n",
"Y <- re78\n",
"Tr <- treat\n",
"X <- cbind(age, educ, black, hisp, married, nodegr, re74, re75)\n",
"detach(lalonde)"
]
},
{
"cell_type": "code",
"execution_count": 105,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:59.722424Z",
"start_time": "2020-03-11T08:32:54.242Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Smallest p-value.\n",
"Var\t: re74\n",
"p-val\t: 0.1198"
]
}
],
"source": [
"genout <- GenMatch(Tr=Tr, X=X, estimand=\"ATT\", M=1,\n",
" pop.size=100, max.generations=10, wait.generations=1, print.level=0)\n",
"\n",
"mout <- Match(Y=Y, Tr=Tr, X=X, estimand=\"ATT\", Weight.matrix=genout)\n",
"\n",
"mb <- MatchBalance(treat~age + educ + black + hisp + married + nodegr + re74 + re75, \n",
" data=lalonde, match.out=mout, print.level=0)\n",
"\n",
"balanceprint(mb)"
]
},
{
"cell_type": "code",
"execution_count": 106,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:59.753346Z",
"start_time": "2020-03-11T08:32:54.246Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<table>\n",
"<thead><tr><th scope=col>Gamma</th><th scope=col>Lower bound</th><th scope=col>Upper bound</th></tr></thead>\n",
"<tbody>\n",
"\t<tr><td>1.0 </td><td>0.0113</td><td>0.0113</td></tr>\n",
"\t<tr><td>1.1 </td><td>0.0018</td><td>0.0501</td></tr>\n",
"\t<tr><td>1.2 </td><td>0.0002</td><td>0.1428</td></tr>\n",
"\t<tr><td>1.3 </td><td>0.0000</td><td>0.2948</td></tr>\n",
"\t<tr><td>1.4 </td><td>0.0000</td><td>0.4797</td></tr>\n",
"\t<tr><td>1.5 </td><td>0.0000</td><td>0.6568</td></tr>\n",
"</tbody>\n",
"</table>\n"
],
"text/latex": [
"\\begin{tabular}{r|lll}\n",
" Gamma & Lower bound & Upper bound\\\\\n",
"\\hline\n",
"\t 1.0 & 0.0113 & 0.0113\\\\\n",
"\t 1.1 & 0.0018 & 0.0501\\\\\n",
"\t 1.2 & 0.0002 & 0.1428\\\\\n",
"\t 1.3 & 0.0000 & 0.2948\\\\\n",
"\t 1.4 & 0.0000 & 0.4797\\\\\n",
"\t 1.5 & 0.0000 & 0.6568\\\\\n",
"\\end{tabular}\n"
],
"text/markdown": [
"\n",
"Gamma | Lower bound | Upper bound | \n",
"|---|---|---|---|---|---|\n",
"| 1.0 | 0.0113 | 0.0113 | \n",
"| 1.1 | 0.0018 | 0.0501 | \n",
"| 1.2 | 0.0002 | 0.1428 | \n",
"| 1.3 | 0.0000 | 0.2948 | \n",
"| 1.4 | 0.0000 | 0.4797 | \n",
"| 1.5 | 0.0000 | 0.6568 | \n",
"\n",
"\n"
],
"text/plain": [
" Gamma Lower bound Upper bound\n",
"1 1.0 0.0113 0.0113 \n",
"2 1.1 0.0018 0.0501 \n",
"3 1.2 0.0002 0.1428 \n",
"4 1.3 0.0000 0.2948 \n",
"5 1.4 0.0000 0.4797 \n",
"6 1.5 0.0000 0.6568 "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"psens(mout, Gamma=1.5, GammaInc=.1)$bounds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<span class=\"minerva-question\" style=\"background-color:#b83c3c;padding: 5px 20px 5px 20px;line-height:30px;color:white;font-weight: bold; border-radius: 25px\">Note #2</span>\n",
"\n",
"Rosenbaum kept pushing the state of the art and there are now more intricate ways of running sensitivity analysis. The shortcoming of `rbounds` is that is doesn't appropriately take care of ties (i.e., when one treated unit is matched to more than one control unit). The package `sensitivitymv` contains a more modern implementation in the function `senmv`, but it is a little harder to use it in conjunction with matching because of the structure of the inputs it expects. You can read the document [here](https://cran.r-project.org/web/packages/sensitivitymv/sensitivitymv.pdf) or check the help file, but the code below will do the job for you."
]
},
{
"cell_type": "code",
"execution_count": 107,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:59.781664Z",
"start_time": "2020-03-11T08:32:54.249Z"
}
},
"outputs": [],
"source": [
"# SKIP THIS CELL\n",
"# Generate the table of matched set outcomes\n",
"make_ymat<-function(matches, y){\n",
" \n",
" # Get indices of treated and control units\n",
" treated <- unique(matches[, 1]) # Remove repeated indices\n",
" controls <- matches[, 2] # Keep repeated indices\n",
"\n",
" # Get outcomes of control units\n",
" y_ctrls <- y[controls]\n",
" \n",
" # Create a table to check how many matches for each control \n",
" trt_table <- table(treated)\n",
" \n",
" # Get number of sets and size of largest set\n",
" n_sets <- length(trt_table)\n",
" max_ctrls <- max(trt_table)\n",
" \n",
" # Smallest table necessary is number of sets vs. size of largest set\n",
" y_mat <- matrix(NA, n_sets, max_ctrls + 1)\n",
" \n",
" m <- 0 # Auxiliary indexer that will run linearly through the matches sets\n",
" \n",
" # For each set\n",
" for (i in 1:n_sets){\n",
" \n",
" y_mat[i, 1] <- y[treated[i]] # Get treated outcome\n",
" \n",
" # Get controls outcomes\n",
" y_mat[i, 2:(1+trt_table[i])] <- y_ctrls[(m+1):(m+trt_table[i])]\n",
" \n",
" # Advance the indexer\n",
" m <- m + trt_table[i]\n",
" }\n",
" y_mat\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:32:59.872415Z",
"start_time": "2020-03-11T08:32:54.252Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<dl>\n",
"\t<dt>$pval</dt>\n",
"\t\t<dd>5.5991461266558e-05</dd>\n",
"\t<dt>$deviate</dt>\n",
"\t\t<dd>3.86304379385991</dd>\n",
"\t<dt>$statistic</dt>\n",
"\t\t<dd>2623.20707027027</dd>\n",
"\t<dt>$expectation</dt>\n",
"\t\t<dd>0</dd>\n",
"\t<dt>$variance</dt>\n",
"\t\t<dd>461111.427440323</dd>\n",
"</dl>\n"
],
"text/latex": [
"\\begin{description}\n",
"\\item[\\$pval] 5.5991461266558e-05\n",
"\\item[\\$deviate] 3.86304379385991\n",
"\\item[\\$statistic] 2623.20707027027\n",
"\\item[\\$expectation] 0\n",
"\\item[\\$variance] 461111.427440323\n",
"\\end{description}\n"
],
"text/markdown": [
"$pval\n",
": 5.5991461266558e-05\n",
"$deviate\n",
": 3.86304379385991\n",
"$statistic\n",
": 2623.20707027027\n",
"$expectation\n",
": 0\n",
"$variance\n",
": 461111.427440323\n",
"\n",
"\n"
],
"text/plain": [
"$pval\n",
"[1] 5.599146e-05\n",
"\n",
"$deviate\n",
"[1] 3.863044\n",
"\n",
"$statistic\n",
"[1] 2623.207\n",
"\n",
"$expectation\n",
"[1] 0\n",
"\n",
"$variance\n",
"[1] 461111.4\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Prepare input for the senmv function\n",
"# Note that we are passing the output of genetic matching\n",
"# instead of from matching itself. Otherwise it won't work!\n",
"ymat <- make_ymat(genout$matches, Y) \n",
"\n",
"# Explaining method='t' is beyond our scope here.\n",
"senmv.out <- senmv(y=ymat, gamma=1, method='t')\n",
"senmv.out"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:33:00.233413Z",
"start_time": "2020-03-11T08:32:54.254Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Gamma: 1.0\tp-val: 0.00006\n",
"Gamma: 1.1\tp-val: 0.00029\n",
"Gamma: 1.2\tp-val: 0.00111\n",
"Gamma: 1.3\tp-val: 0.00334\n",
"Gamma: 1.4\tp-val: 0.00826\n",
"Gamma: 1.5\tp-val: 0.01759\n",
"Gamma: 1.6\tp-val: 0.03314\n",
"Gamma: 1.7\tp-val: 0.05649\n",
"Gamma: 1.8\tp-val: 0.08868\n",
"Gamma: 1.9\tp-val: 0.12992\n",
"Gamma: 2.0\tp-val: 0.17963\n"
]
}
],
"source": [
"# You also need to try different gammas yourself\n",
"for (gamma in seq(1, 2, 0.1)) \n",
" cat(sprintf(\"Gamma: %.1f\\tp-val: %.5f\\n\", \n",
" gamma, senmv(y=ymat, gamma=gamma, method='t')$pval))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<span class=\"minerva-question\" style=\"background-color:#b83c3c;padding: 5px 20px 5px 20px;line-height:30px;color:white;font-weight: bold; border-radius: 25px\">Question #2</span>\n",
"\n",
"What do you observe now? How robust is the ATE after genetic matching?\n",
"\n",
"_Your response_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Comparison\n",
"\n",
"Let us compare propensity score matching with genetic matching, using the same `senmv` function so that we are comparing the output of the same procedure. We will need to pass the propensity scores to genetic matching to generate the table of matched outcomes. "
]
},
{
"cell_type": "code",
"execution_count": 110,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:33:02.071235Z",
"start_time": "2020-03-11T08:32:54.257Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Smallest p-value.\n",
"Var\t: re74\n",
"p-val\t: 0.0040"
]
}
],
"source": [
"genout <- GenMatch(Tr=Tr, X=glm1$fitted, estimand=\"ATT\", M=1,\n",
" pop.size=100, max.generations=10, wait.generations=1, print.level=0)\n",
"\n",
"mout <- Match(Y=Y, Tr=Tr, X=glm1$fitted, estimand=\"ATT\", Weight.matrix=genout)\n",
"\n",
"mb <- MatchBalance(treat~age + educ + black + hisp + married + nodegr + re74 + re75, \n",
" data=lalonde, match.out=mout, print.level=0)\n",
"\n",
"balanceprint(mb)"
]
},
{
"cell_type": "code",
"execution_count": 111,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:33:02.345948Z",
"start_time": "2020-03-11T08:32:54.260Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Gamma: 1.0\tp-val: 0.00002\n",
"Gamma: 1.1\tp-val: 0.00009\n",
"Gamma: 1.2\tp-val: 0.00038\n",
"Gamma: 1.3\tp-val: 0.00124\n",
"Gamma: 1.4\tp-val: 0.00332\n",
"Gamma: 1.5\tp-val: 0.00759\n",
"Gamma: 1.6\tp-val: 0.01528\n",
"Gamma: 1.7\tp-val: 0.02772\n",
"Gamma: 1.8\tp-val: 0.04611\n",
"Gamma: 1.9\tp-val: 0.07131\n",
"Gamma: 2.0\tp-val: 0.10373\n"
]
}
],
"source": [
"ymat <- make_ymat(genout$matches, Y) \n",
"for (gamma in seq(1, 2, 0.1)) \n",
" cat(sprintf(\"Gamma: %.1f\\tp-val: %.5f\\n\", \n",
" gamma, senmv(y=ymat, gamma=gamma, method='t')$pval))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<span class=\"minerva-question\" style=\"background-color:#b83c3c;padding: 5px 20px 5px 20px;line-height:30px;color:white;font-weight: bold; border-radius: 25px\">Question #3</span>\n",
"\n",
"Compare the balance and robustness of the propensity score and genetic matching solutions. Does a higher balance p-value imply necessarily imply an estimate that is more robust? Why or why not?\n",
"\n",
"_Your response_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Balance vs. Robustness\n",
"\n",
"If you were curious about checking the relationship between the p-values for balance and robustness, you could just run the procedure above many many times. Unfortunately, the code we are using the generate the outcome matrix is quite inefficient, so even for 100 times it takes a while."
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:43:11.851417Z",
"start_time": "2020-03-11T08:40:39.527Z"
}
},
"outputs": [],
"source": [
"ITERS <- 100\n",
"balance.pvals <- numeric(ITERS)\n",
"robust.pvals <- numeric(ITERS)\n",
"\n",
"for (i in 1:ITERS) {\n",
" \n",
" # Genetic Matching\n",
" genout <- GenMatch(Tr=Tr, X=X, estimand=\"ATT\", M=1,\n",
" pop.size=20, max.generations=10, wait.generations=1, print.level=0)\n",
"\n",
" # Matching\n",
" mout <- Match(Y=Y, Tr=Tr, X=X, estimand=\"ATT\", Weight.matrix=genout)\n",
"\n",
" # Balance p-value\n",
" mb <- MatchBalance(treat~age + educ + black + hisp + married + nodegr + re74 + re75, \n",
" data=lalonde, match.out=mout, print.level=0)\n",
"\n",
" # Save\n",
" balance.pvals[i] <- as.numeric(mb$AMsmallest.p.value)\n",
"\n",
" # Create matrix of matched outcomes\n",
" ymat <- make_ymat(genout$matches, Y) \n",
"\n",
" # Save robustness p-value for gamma = 2\n",
" robust.pvals[i] <- senmv(y=ymat, gamma=2, method='t')$pval\n",
" \n",
"}\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 113,
"metadata": {
"ExecuteTime": {
"end_time": "2020-03-11T08:43:12.968413Z",
"start_time": "2020-03-11T08:41:12.599Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\ndAHeZh94AAAgAElEQVR4nO3di3qivAJG4YCI1vFw/3c7Ah5QEJF8SUiy3mfvjj0R7c8qENCa\nCwBrJvQdAFJASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEC\nhAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQE\nCBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQ\nEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIg\nQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBI\ngAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQEiCwPKR/\nu8o0qvrflyGAyPyew9KQzmVv2I2TIYBAPIZUm+Lv2N46HQpTuxgCCMRjSIU5Pm4fTeFiCCAQ\njyG97EZO71MSEiLDFgkQ8HuMdDi1tzhGQmo8hnTZ9GbtyrOTIYAwfIZ0+Ve355GKavftPNLi\nIYAgvIa0piEAJUICBAKEtC9MuXc7BOCZz5COlSn2lx2XCCE9HkM6tgXVZnu+nCozuU0iJETG\nY0jb5txR3Z2JPZvSxRBAIN4vETJV7x31EEAg3kP66/bpuEQISfG6a7e9X85w3nKJEJLi84l9\nxWN/zoxskCyfbgiE5PU8Un3Pp5jcHkm3SFQJHxK/sqGtiJTgXOohaRcHfJB2SObtX8CRUCH5\nOY8kCIljLMxBSNML4BgLs6S9a2d9jMQxFuZJPSS7LQrHWJgp8ZAsj3EICTP5fc2G2a/9vXgI\nLULCTLz29ySOkTAPr/09iVk7zMMrrX7BeSTMwWt/AwJskQABXvsbEOC1vwEBXvsbEEj+ygbA\nB0ICBAgJECAkQICQAAFCAgQICRAgJECAkAABQgIECAkQICRAgJAAAUICBAgJECAkQICQAAFC\nAgQICRAgJECAkAABQgIECAkQICRAgJAAAUICBAgJECAkQICQAAFCAgQICRAgJECAkAABQgIE\nCAkQICRAgJAAAUICBAgJECAkQICQAAFCAgQICRAgJECAkAABQgIECAkQyD4kY6gW9jIPqa2I\nlGAt95D8Dodk5R2SefsXWIiQfI6HZBGSz/GQrLxD4hgJIrmHxKwdJDIPifNI0Mg+JECBkAAB\nQgIECAkQICRAgJAAAUJKApP4oRFSAjitHB4hJYALncIjpPhx6e0KEFL8CGkFCCl+hLQChJQA\njpHCI6QEMGsXHiElgfNIoRESIEBIgAAhAQKEBAgQEiBASMBcE5OjhATMM3m6LqeQONkCG5MX\nkOQTEqf/YWX6ksaMQnK2ZGSBkF4WSUlYZm0h7UtjqoPTIaYWSUhYaC3HSN3xyca0aidDzFgk\nIWGhtczatXehNvX5cjnVZu9iiO/LpCMst47zSO2dKMy5uX02pYshvg7PrB3c8B3SfU2eXqM5\nj4TI+A5pew+pcDEEEIjXkKrd/mD+rjfP9fRsAyEhMl5D6rQ3i7OLIYBAfJ5HOh73+6pqpxzq\nyY4ICbHJ5soGwCVCAgR8hnTeGrO5XRwUYvobcMZjSOeinWuouoUQElLiMaT2sqDzvti0CyEk\npMRjSEX3jaeiPBESEuP96u/rRmmzISQkxmNIpbmfPCo3hIS0eAxpb7a3WyezISQkxef0d/2o\n5zByIbbpWzoEEIbXE7LH6n7rtGWLhJRwZQMgQEiAACEBAqFCYtYOSSEkQIBdO0CAkAABQgIE\nvIb0b1d1T0mq/7kaAgjC5xP7yt41QBsnQwCBeH1iX/F3bG+dDgWva4ekeH1i3/Fx+8grrSIp\nAZ7YN3xHNgQQCFskQMDvMdLh1N7iGAmp8Tn9venN2pW89jdS4vc8Ut2eRyqqHeeRkBaubAAE\nCAkQICRAgJAAAUKKCq9UtlaEFJGXvwuPVSGkiJjeW6wLIcXDvP2LFSGkeBDSihFSPAhpxQgp\nIhwjrVeEIeU7Bcys3XpFF1LeK1O+v0TWLr6QfN0H4AexhcQBN1aJkAABQgIEYguJYySsUnwh\nZT1rh7WKLiSmgLFGEYYErA8hAQIZh8QuInSyDYlJCyjlG5KvgZCFXEPSndhlDxEXQrIdiT1E\ntAhJsRxCyl6uIYkK4NI/dPINSbJPRkjoZBuSZpaAkNDJOCQJjpHQIiQ7zNqhRUi2OI+ECyEB\nEoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKE\nBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIJB9SLx0NxQyD4k/JgGN3EPyOxySlXdI\n/ME9iGQcknkeHhESLGUb0svRESHBUr4hDd4Cy+Ua0n1T9G3WjtlxzJJ5SP0DpbGvimV2nN5D\nyz6kGV+1+nU0mt4TlmtIsxqJZVIvkt6Tlm9IM36LRxJSJHczbdmGNOe4IpI1NJK7mbZcQlp2\nNB7HPhMhrUAeIS09Go/kKD6O3tOWSUiLlxrFvHIkvScti5Be932iaONHKT6muGQXEr+94UJ+\nIbkYANnLIqRePcxwwYlMQnru1DkaAZnLI6Tn0TghwQmvIf3bVc3V1qaq/7ka4iuOkeCCx5DO\npXnaOBliBmbt4ILHkGpT/B3bW6dDYeolQ0hOl3DOBXoeQyrM8XH7aIrfh2BjgtXyGNJLAtM9\nfAjJanjAoYi2SEy4Yb38HiMdTu2tZcdIhIT18jn9venN2pXnn4cgJKyX3/NIdXseqah2i84j\ncYz0FTOSocR0ZQOzdl/wAwonppD4hfsFm+xwfIZ03hqzOdwWsmD6e65cc+MgMiCflwgV3YV2\n3UKchZTv/g0hBeR1+nt/rWlftJfZOQzJegmxIqSAvJ6Qbf85FeXJYUi/rU1p7QXm+zskvACX\nCJ03m5WElNpeYGqPJyYeQyrN/SRsuVlJSLaDrU5aW9iY2Ia0L687a6Upv5xhbb/UbG+3Tmaz\nhmMkjikgYxnSoQminY2bUVL9qOcw8pvT9P1+r3rLuczcvyEkyFiGtDF/l6MpL39fnvLaOVb3\nW6ftCs4jERJkLENqVtljcyW3dt/c06qd3jESQhGEVJlDrCGN7QWa20fpCz+w3rU7Hprn6M3b\ntVs0hFODXnoVkRLms59sMGbXrHSHHxfi8BjJhhm8BWawnv7unuta/v26kHWGdC/IV0nsQaYi\nrqdReBrYV0jsQaaDkEYG9haSl1Hgg0VIxqjOoSrulXRknx1RUhK8hhT6tb9n3Eufs3aElBCP\nu3ahX/t7Zh3+ziMRUkI8hiR47W8r6zsiWd89wlKqkP5VYx99Yf3a33ZW+PufWbt02IZUzz9G\nsn7tbzvmPvCaVtx13RssZxnSs6PvVzasYYtkmpltVl7IWYZUmL/LxpxOmxnPR7J97W9b7X7U\n7S2gJbj6e3fdGh3nXLRq+drftnpbI0qCmCCkQ/MyW/POI9m99re1xxEJIUHMMqTqumt3MuXl\nXxxXNqxw5g5psH8axW2Xbfvx6xdwtqJz5gZu2E5/75r3tmZ67sBuCOmCOXMDJ7K7+pszN3DB\nMqTJubflWNcRGdtZu82PTzH/fQggApYhNRd0f3tOxAKEhMjYHiOddteWyp14F4+QEBnBZMOp\nLox4F4+QEBnNrN0+maeaA4sotkjt3t2vr8f10xDAykmOkYr6pLo/I0MA6yeYtdsya4fsWZ9H\nku7SjQ3hC5c8wAJXNtxG5CI82LCebPhrnmK0FV/fECCkQOMiEbYh3Z/1+v1FhBYP4QNPVIId\n6xc/KZqN0aFoniWrQ0iIjPWLn3SvDNT8HVkhQkJkBK/Z8HpDgmMkRMZ61+6+RZIeJDFrh8hY\nP9W8PUb6V0j/hCznkRAb6107J38jiVUakSEkQCC7Fz8BXCAkQICQJjEDgXlUIcV+HmkUc+KY\ni5AmcJYWc7Fr9xnXDWE2QvqMkDAbIX1GSJhNddFqMfk3YW2GCIhjJMwlCumU1mTDbda7ecsE\nOOawCOnwcnVQ3M9Heh39OettupaC3h3EwGaLVPY7kr4mV+CQnm/ZucM8qmMkraBrbm+OgekG\nzMSs3cfBCQnz2Ya0vx4bnUrxnh0hITaWIbV/1bxoDpI4RkLOLEPamL/2FYT+jPS55iuatXvc\nBCYIJhuOpk7totXeySPOI2EOQUiVOYQNyde6TlP4yHrX7ngwxSXkrp2vvS/28jDBfrLBmF2z\nggX7G7K+5gOYd8AE6+nvojlCumj/8uUv98rXDDUz4ZgS/QlZQsIaENLKxkGcog+JYySsgfX0\nt/xVVt+H+Pq1L7Np7qaombXDhPhD6sczurLL7hvnkfCRZtfu32YNf/ryvqKb1w9ePG5IaC1X\nomOks9la35UvQ3z9HnP9NtO71LS/KF+XPlzY+8uUarIh/LV25vnEB/Py0Z+WaLVJYT4iX6KQ\n9ib0qwiZt/+/Len7EpuE7DYpzJBnTDbZsJPdpcvye9XbKr0t6dsS+wkREn4mCqncy+7R+xC/\nfEvvOOl1UV8XeP/2pcP3v42QMhT/CdnX7xkc48zbW7vn9vsmqT8gx0j5SiikT8XMmT9YGtL7\n2eAPdwHJsw7pr7quqJX0SRS255GWDzicPZ/1fb1tEhllyjakze0gaQ0nZO1HnLkf+NwEvf0b\nkl3D/AawZBlSbYpmY3Qogs/a2Y54S+j7CjU2vxd+JbScuGef1JZlSIU5tv8eg59Hsh9z5opk\nBm9XEVLvre/vxkUw/f1+Q2LF/0lf21nLGmhX9Hp+H8TLetfuvkWqNfdnOMTKvIW0kn0iQgrN\ndrKhao+R/hXSa1bX/F90MM+wgowIKTyLkMyrwPfKm7Xszb3iGCkwQvrVWvbmXjFrF1g6Vzb4\ns469uXecRwqKkAAB+0uEmmsbKu3rQxISYqO6REj60t+EhNhYhrR/XCIkfUISISEyliGVjxOy\npeb+DIcAIsAlQoCAbIu06otWmdyFYzkcI3G6Ec7lMGvHBTBwzv48UrX280hckgn3MriygZDg\nnmVIlfRpSKNDyBZGSHBHNf2txTESImM9/X3+9dv35feX7wo+a8d8OX5jGdK52vyb/X3tN96m\n+aZ3CQOfR2K+HL+y3rWb/8S+9ktqU1+3Yad6+ryT/TrMn2eBV75DKrp9wfP0tXm26/Btk7Iw\nJmYn8DOP098vO0zT6/i3Ib4lYu5fsyglQsLPfIe0vYc0eW3el0y+HcKY2xuz7M4SEn7mNaRq\ntz+Y5iKIcz092/B9ezP5Nebt/7/iGAm/8hrS42DKmGJy2nzWft/nL7IOiVk7/MjnJULH435f\nVe2UQz19+mk6pO9/obK3W7dwvoGM8JPorrV7ziFMftG9BSd3lszwLr6Q5k0jPPYhtXfscmHH\nD2NiC+m+Es87cfXpazhdC7EYQ/oWwtezTFabFCbHMSLOkCa/4msmdpsUQsKIANPfMy4p+nay\ntfcFgyXNOsv05UumEBJGeAxprwnpZYMz3PzMO8v0ZZBJHCNhyOt5pGLuS6TMOo/0/ELPIXmf\ntWO6ff28HiPN/gOZc4cYaWJGJtabFL8rNtPtMfA72bC/vZ7k6GIX/NWysWq+ZxLZmsmuZAxi\nm7Ub/7qXkGZkEtO+EpMbUYg7pP5va9ObgRh+f0zpvCKkKMQe0vNKh8vEdiiynbkXhBSFyEN6\nbGqmDySiPsyI+s5nI1RIdk81//gN498Y9y/1mDen+UgupNEDpKWLXYl4D/DyEf2u3es3jP/2\n7j5oog0J65dISI+rwse/3d2Tk4BWMiG9XoI3/llCgiteQ/q3q9rrFqr6y+scLxmiv+823Ldz\n+MRzwGtI57J3DdD05auLryf98P2xzzZg9TyGVJvir7vU7nQobF7X7us3Do+RbJcLTPMYUtG7\nYvXLX0FfHtKHSQVOasItjyG9Pv9Oex6pv+Cxb2bWDm4ltkWaWCQZwSG/x0iHU3vL3TESEIbP\n6e9Nb9auXP7a38D6+D2PVLfnkYpq5+A8EhBQKlc2AEEREiBASIAAIQEChAQIJBLSx9OtnIeF\nF0mE9LwA6K2b9xcKN5fhTUAgjZBubwdX1PWvVe19kivvoJZCSC8v2ND/5pdnT5jRm+MLJDL8\nKKmQ3r+7/4He7S/PTuKvTeB3hPRhcf5WbXY0U5BCSK8vITTctzP9D34NyfuzaXnSYQrSCOn2\nO32wSvZ/2c89RjKDQy23vIcLF5II6b72j+wk9f9GxeOTEztT5vmK/L/eh4UIKQmJhPT4xqlj\njTnnkUzvf34QUhISC0kxcLdN8jjZ0HuLWBHSyMD8jVj8ipBGB/Z7BziPFD9CGhuZ9Ro/IqS3\nkdnPwhKENBibjPA7QgIECAkQICRAgJAAAULyh2mMhBGSL0ysJ42QfOFUb9IIyRMu8k5bHiGt\n4OiEkNKWQ0irODohpLRlEZKDZcZ6L+BIBiGtZFuwiu0iXCEkf1ZwpAZXCAkQyCAkjk7gXhYh\ncXQC13IIiaMTOJdHSNnjN4lrhJQB9m3dI6QMMNviHiGlj/l/DwgpfYTkASGlj5A8IKQMcIzk\nHiFlgFk797IPKY8zLHk8ypAyDymp39XUElBKIfVXpJl/us/p0QN/Zikj6YT08oeXp1aq/h+T\nfXxI/6B8r9jMKASVUEgf3k594T0742CV97xiM8cdVjIh9VekyZVq+IXm5ZtEfK/YhBRW3iHd\nvtrFxoOQ8pJaSN1u2tRiXkMyzwOkuEPiGCmsZELqNi63o525x0iXy2AGT8b3is2sXVAJhfTY\nuJjZs3b90eSTDd5XbM4jBZROSL0VyXxZqd4+abHK/zIMUpZSSMuPSxau8uxN4Y6QLHB8j7uk\nQvK8ZjPjjIfEQvK6r0VIeEgrJL8H+ISEh9RC8opjJNwRkgVm7XBHSFZ610UQVNYISYJtU+4I\nSYKjpdwRkgLzd9kjJAVCyh4hKRBS9ghJgmOk3BGSBLN2uSMkEc4j5S3FkFin4V16IbGXhQAS\nDMl6CQgh8v2I5EKKbyY68jVII/r9iBxDWtWaG/0apBH9fkR+IU2suSEKi34NkohvP+JdciF9\nXTM/fj7ItiH+NUgi/h9DgiFN9/D5P1mQbUP8a5BE/D+G9EL6sof28T9ZmP+W8a9BGtHv4aYY\n0qxFrySk+NcgjejnXLyG9G9XmUZV/3M1xHef1txQIcW+Bqmsai71dx5DOpfmaeNkiDk+rrmh\ntg2Rr0FoeQypNsXfsb11OhSmdjHEPB/WXLYNWM5jSIU5Pm4fTeFiCEtsG7CUx5Be1tLpVZb1\nGZFhiwQI+D1GOpzaW4GPkQA5n9Pfm96sXXl2MgQQht/zSHV7HqmodgHPI/2G+QfMkt2VDT9h\nRhwzEdIUrt/BTIQ0gStKMRchTSAkzEVIEwgJc3m9suGFiyHUOEbCTB5D2ocNaclENrN2mMnn\nrt2xmH7yhGCIz0tcmATnkTCL12Ok4/SFQYohppdIFHDE72TDvnfd6mCxc/f7lmDaAG5lMmtH\nSHCLkACBTELiGAluZRMSE9lwKVRIwvNIM+cmmMiGQ9GHxKYGaxD9rh0HP1iD2ENiOg6rQEiA\ngN/XbNC/9jchYRU8huTmtb85RsIa+H1dOwev/c2sHdbAY0iuXmmVE0QIz+szZD+9IxsCCCSB\nLdKqsbnMhN9jpNxe+5sDuGz4nP7O77W/mVLMht/zSPG99rcVTnLlI/YrG1aNkPJBSA4RUj4I\nySWOkbJBSC4xa5cNQnKL80iZICRAgJAAAUICBAgJECAkQICQAAFCAgQICRDIPiTOmEIh85C4\nhgcauYfkdzgkK8mQZu+u8TwHiCQY0g+7a4QEkRRDmr8IQoJIeiH9FAfHSNDIPSRm7SCReUic\nR4JGeiF92l2jGDiUYkhju2vsw8GpBEMa3fgwqwCnkgzp8wIpCW4QEiBASIBAJiFxjAS3sgmJ\nWTu4lEtInEeCU/mEBDiUdkhsheBJyiFxXARvkg5JuCxgUsIhce4I/hASIEBIgEDCIXGMBH+S\nDolZO/iSckicR4I3aYcEeEJIgAAhAQKEBAjkERKTDnAsh5CYBodzWYQ0XCabKGilF1LTyEsn\nw0uF2ERBLbWQmjxM9+Z9Yb2QLAcB3iUXUvt/01/GICSuZoVcYiGZt//3lva+QSIkCCUbknns\n5A0OiQgJcqmG9F6Ref8qOoJSYiE9Nkf3t6PLYtYOasmF9Jy1M7fljC2M80jQSi2k3nmkqZAA\nrfRCenw/IcGfNEO6FURH8CXVkMamvQFnEg2pP+MAuJdsSIBPhAQIEBIgQEiAACEBAoQECBAS\nIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgkFlIPEEJbqQW0kQpxry9KDggk1ZIE08uN3fs\nN8KBxEL6/N2PV43kCAwOJBWSefv35VP3v1HBJgkOEBIg4D+kfWlMdXAyxJeQuv8x2QAHPIbU\nrcGb7pC/djPE5+9uN0XvfxUTEPEdUm3q8+Vyqs3e2RCfZ+26N4Ce75AKc25un03pYog555EA\nB3yHdF+Vp1dp1ndExndI23tIhYshgEC8hlTt9gfzd715rqdnGwgJkfEakrkfphhTnF0MAQTi\n8zzS8bjfV1U75VBPdkRIiE1SVzYAoRASIOA1pH+7qj1Kqup/roYAgvAY0rk0TxsnQwCBeAyp\nNsXfsb11OhRMfyMpHkMqzPFx+8gJWSTF+9XfY+/IhgACYYsECPg9Rjqc2lscIyE1Pqe/N71Z\nu3JwaYPpWzoEEIbf80h1ex6pqHacR0JauLIBECAkQGClIQGRWbCWa1LxP+ZKRknqwTBK4PEI\niVHyHCXceP7GJCRG8YiQohiGUdY5Srjx/I1JSIzikYcn9gnHXN8oST0YRvE/3g9P7JONucZR\nknowjOJ/vB+e2Ccb8zeExCgeeXgahWzM3xASo3jk4Yl9sjF/Q0iM4hFbpCiGYZR1jmI/3g9P\n7JON+RtCYhSPHD2xD8iLhyf2AenjyUKAACEBAoQECBASIEBIgAAhAQKEBAgQEiBASIAAIQEC\nhAQIEBIgQEiAACEBAh5CqgtT1OfxDww+52KUxS+MPmeYy2VvPn9OP4rwwQxG2Zc+/sM8R3H4\nWM5bY7bHD/fABfchdc8ALEc/MPici1GOwv9eI3f4eF+ywwfzHEX4YAaj1O0HivP4PdCP4vKx\nFO0HjqOfc8J5SP9McbwcC/Nv5AODzzkZ5Wgq6+V/GubSvGc+fc7BKLoHMxjlaLbnZsu3dfpY\neqM4fCx1s/y6Xb7usUxyHlJtDte3f2Y38oHB55yMshcs/9Mw14Vvbqu4wwfTG0X3YAajVN0I\nzUAOH0tvFIePpTDn2yDCxzLJeUiVaV4jpffLp/eBweecjLI3e+vlfxrmYur7i5E5fDC9UXQP\n5tP9bQZy+Vieozh/LO2rW+keyyTnIRnT/+f1A4PPORmlMoft9WjTeoyxYS7H9w+6eDC9UXQP\n5sP9PTcvQO3ysTxHcf1Y6jZU3WOZvgtuF7+OkJa8QPm8YYYfdLXyPUJSPZgP93ff7Ak5D6kd\nxe1j+TPdq8QRkm6Uv+uvwFqxHxE+JN2DGb+/p6L6+Dn9KC4fy74q2uMiQlKN0jkrJkDDh9RR\nPJjRUc7F5uPn5KPc3nH1WC6XbdNoKiEV74+j94HB55yMcqP4SY7e4dt7Dh/Mt/d0o2zKz5+T\nj3LjapSm0UL5WCY5D6mbNDm9z6ednrN2J9nk0PgoN4qf5Ogdfhy9OHswvVFG31ONcio3p8/3\nQD7KjZvH8li07rFMch7Srp3GPzxfHrz3gcHnnIzSnVOQ/CRH7/BtRXD4YHqj6B7McJTD47jf\n5WN5juLwsdwXXSofyyTnIYW/sqFufobn7rycepjGbRV3emXDYxTdgxmMcnrOnzl8LL1RHD6W\n9sqGc9UcI6VyZcOlfM5xditD7wOlbGJ6YpRzd92V5DfSYJjeDYcP5nlD+GDeR9ma57Vv7h5L\nbxSHj+V2rZ14JZviPqRze/FtN5h5+0DvpuNRSs1J9MEwvRsOH8zbKJoH8z6K6YXk7rG8j+Lm\nsbSXfN8WrXssU9yHBGSAkAABQgIECAkQICRAgJAAAUICBAgJECAkQICQAAFCAgQICRAgJECA\nkAABQgIECAkQICRAgJAAAUICBAgJECAkQICQAAFCAgQICRAgJECAkAABQgIECAkQICRAgJAA\nAUICBAgJECAkQICQAhv/m94O/pi9g0XiiZ9uYISUBn66gRFSGvjpBkZIaeCnG9h1/a7vf3T7\nUJnbzXat7797qkyxa7+oLszm1N7al6bYjyzpbMr2A6U5Dxa5MWZz8PPI8kJIgRlzXdWva/f1\n5s606ku31r+8WzQ3m5I2zY3ifL3Vfl/7jfcl7W4f2JimtNP15tsi9927+7E7AiuEFNi1iuPl\nWJi/5ub1zV+75WjevLy7OV8rKJv3r7e2TRmH5tZ5Yw7DJf21ye2un3pbZGGOzbtlqAebMEIK\nzLQlHEz1/MClf0Bze/ff7WbV3DqbornVbJbO/W98LKlNpXxfxv1L4AAhBXZLpvvndNhteiG9\nvdul8PzGm5Elba/7dqd2h+51GfV1R/J49PK4ckNIgfVD2jzKGHv3h5D+Xfft6nYr9raMXXOs\nVZw8PbacEFJgL9uRcn84Pdb6t3eHIX1e0qUom/+NLONyqEuOkRwgpMC6w5/uyKb5j/Fc69/e\n7d5uesdIh/ElbS/NPty+nXAYLONyGUkQ9viZBnafazt0JRw3vZBe3r1PYG/O10zqZvLt+o3X\n96uRJTXttHMRb8sou0k8tkh6hBSYMdvmIKbJob4d9Px7zAz03+0fOLXnkbrDn/aA5z5Hfl9S\nk0x7gultGX+P9yBGSIF11yN0Fy1ck9r8e+7lvb17e9vMvD2ubDDb0+Mz1zfV9ZCoW+zfbXtz\nO2YAAACBSURBVMfvbRntlQ105AAhJYNDn5D44SeDkELih58MQgqJH34yCCkkfviAACEBAoQE\nCBASIEBIgAAhAQKEBAgQEiBASIAAIQEChAQIEBIgQEiAACEBAoQECBASIEBIgAAhAQKEBAgQ\nEiBASIAAIQEChAQIEBIgQEiAACEBAoQECPwHiIONvX4k64AAAAAASUVORK5CYII=",
"text/plain": [
"plot without title"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot(balance.pvals, robust.pvals)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<span class=\"minerva-question\" style=\"background-color:#b83c3c;padding: 5px 20px 5px 20px;line-height:30px;color:white;font-weight: bold; border-radius: 25px\">Question #4</span>\n",
"\n",
"Why do you see? Does lower balance entail lower robustness? Does higher balance entail higher robustness? Why or why not?\n",
"\n",
"_Your response_"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "R [conda env:renv]",
"language": "R",
"name": "conda-env-renv-r"
},
"language_info": {
"codemirror_mode": "r",
"file_extension": ".r",
"mimetype": "text/x-r-source",
"name": "R",
"pygments_lexer": "r",
"version": "3.5.1"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment