Instructions: this week’s exercise is called 4_congress.R. Download or copy-paste that file into a new R script, then open it and run it with the working directory set as the IDA folder. If you download the script, make sure that your browser preserved its .R file extension.
This week's applications use a combination of plyr and ggplot2 functions to show how vectorization works to aggregate and reshape data in R. We do not need to cover all options, but we can survey some essential ones, as Neil Saunders and Jeffrey Breen did in tutorials that help realizing how useful these operations are to manipulate datasets with R.
The exercise looks at estimates of Congressional ideology.
The data for our first application are estimates of Congressional ideology compiled by Boris Shor for the U.S. House of Representatives in 2012. The data are downloaded in XLSX and saved in a plain text comma-separated file. The variables stored in the dataset are documented in the online codebook) and stored in a data.frame object:
# Check result.
head(data[, 1:9])
stdist st party pid full.name incumbent crp.id npat.id score
1 AK1 AK D -1 Sharon Cissna 0 N00034750 15863 -0.69
2 AK1 AK R 1 Donald Young 1 N00007999 26717 0.26
3 AL1 AL R 1 Josiah Bonner 1 N00025330 27522 0.45
4 AL2 AL D -1 Therese Ford 0 N00033968 135114 -0.69
5 AL2 AL R 1 Martha Roby 1 N00030768 71604 0.41
6 AL3 AL D -1 John Harris 0 N00034222 80329 -0.86
We now have loaded some data where each row is a single observation, namely a member of the U.S. House of Representatives in 2012. Each member has been assigned an ideal point estimate of his or her ideological stance in Congress, from liberal (-) to conservative (+). This methodology provides a measure of party polarization in Congress, if we aggregate Congressmen by party.
Aggregating by party involves doing the following: determine all possible values taken by the party variable (and limiting ourselves to the first two in this case), get the data for just one party at a time, and compute the average ideological score in that group. This whole operation can be tiresomely carried with a loop, as below, but that is precisely the kind of approach that we will avoid later on.
# The naive approach: loop over levels, subset the data, compute the scores.
for (i in levels(data$party)) {
# Create a subset of the data for that party.
d = subset(data, party == i)
# Compute the mean ideal estimate point.
cat(i, nrow(d), mean(d$score, na.rm = TRUE), "\n")
}
D 344 -0.8941
R 374 0.8174
X 4 -0.0675
This loop first determines the vector of party names D, R, X (Democrat, Republican, Independent) and then painfully extracts the number of observations and mean ideological score for each party. This type of looping is easily avoided by using the vector of party names to split the data, apply a mean function to the splitted parts and combine it back to shape:
The exercise shows how to use the tapply() function, which returns one mean value (third argument) based on ideological score (first argument) for each group of observations formed by party membership (second argument). The tapply() syntax is pretty straightforward, but it can also follow two alternative syntaxes using aggregate or by() with data frames:
# Simple aggregation with tapply().
tapply(data$score, data$party, mean)
D R X
-0.8941 0.8174 -0.0675
# The by() version for data frames with factors.
by(data$score, data$party, mean)
data$party: D
[1] -0.8941
--------------------------------------------------------
data$party: R
[1] 0.8174
--------------------------------------------------------
data$party: X
[1] -0.0675
# The aggregate() version with formula notation.
aggregate(score ~ party, data = data, FUN = "mean")
party score
1 D -0.8941
2 R 0.8174
3 X -0.0675
We can decompose what is happening here by calling another apply() function that allows to explain each step. In the sapply() example below, we write up an arbitrary function that extracts the mean ideological scores of one party after the other in the vector of unique party names. This example shows how much simplification work is being done in a command like tapply().
sapply(levels(data$party), function(x) {
this.party = which(data$party == x)
mean(data$score[this.party])
})
D R X
-0.8941 0.8174 -0.0675
Aggregating by groups also works visually, as we will see several times during the course from next week onwards, as we look into ggplot2 syntax for plots where we want to visually discriminate some groups. The exercise ends on an example of that principle to stack the distributions of Congressional ideology by party. Distributions are important statistical functions studied later in the course.
The graph implicitly groups observations by assigning different fill and colour attributes to each group of variable party. These groups are assigned colors that were previously extracted from ColorBrewer palette Set1, using the brewer.pal() function of the RColorBrewer package to extract the nine set colors and select the blue (#2), red (#1) and grey (#9) tints.
We continue to look at ideology in the U.S. Congress by plotting the DW-NOMINATE index, using an example adapted from David Sparks. Unlike the cross-sectional estimate for 2012 that we just used, this index is available for several years and therefore requires a more advanced reshaping strategy that splits the data by three groups: Congressional session, year, and party.
Exploring this dataset is easy if you need simple measures like raw frequencies. Here, for instance, is the number of different observations in the data for each major party affiliation, and the average DW-NOMINATE score in each group, obtained through formula notation with the aggregate() function.
# Raw frequencies (N) by party.
table(dw$majorParty)
Democrat Other Republican
17131 4008 15495
# Mean DW-NOMINATE score by party.
aggregate(dwnom1 ~ majorParty, dw, mean)
majorParty dwnom1
1 Democrat -0.3068
2 Other 0.1677
3 Republican 0.3350
The exercise show more ways to do calculations by groups at that stage and then turns to the ddply() function from the plyr package to aggregate the observations by major party within each congressional session, which makes for a more complex pattern than those seen so far. The crux is the .variables separator that groups the observations by Congress and major party.
# David Sparks' transformation to session-party measurements, using plyr for the
# ddply() function and Hmisc for the weighted wtd.functions.
dw.aggregated <- ddply(.data = dw,
.variables = .(cong, majorParty),
.fun = summarise,
Median = wtd.quantile(dwnom1, 1/bootse1, 1/2),
q25 = wtd.quantile(dwnom1, 1/bootse1, 1/4),
q75 = wtd.quantile(dwnom1, 1/bootse1, 3/4),
q05 = wtd.quantile(dwnom1, 1/bootse1, 1/20),
q95 = wtd.quantile(dwnom1, 1/bootse1, 19/20),
N = length(dwnom1))
What happens here is a three-dimensional split of the data, first by Congressional session, then by political party. This operation is one of seven possible transformations in such settings. Hadley Wickham's plyr package is a set of commands like ddply() to implement these transformations, which he calls “The Split-Apply-Combine Strategy for Data Analysis” and illustrates as goes:
The cube slices show the different split-ups that can be accomplished when you combine three variables (dimensions). In our example, the result is a dataset that contains a single observation by party (Democrat, Republican, Independent) and by Congress (1-111th), with its value being the median DW-NOMINATE score of that group. The results below are the results for the most recent Congress:
# Median DW-NOMINATE score of parties in the 111th Congress.
dw.aggregated[dw.aggregated$cong == 111, ]
cong majorParty Median q25 q75 q05 q95 N
271 111 Democrat -0.333 -0.435 -0.232 -0.594 -0.029 262
272 111 Republican 0.606 0.510 0.698 0.242 0.819 183
The following plot derives from the combination of this strategy to a ggplot setting, where filling and colouring elements to reflect membership to a certain grouping in the data is straightforward. For more plots of Congressional ideology, check what David Sparks has done with reshaping to cast, melt and mutate a dataset containing identical data.
Next week: Clusters.