Lesson 5. How to handle missing data or no data values in R - NA and NAN


This lesson covers how to work with no data values in R.

Learning objectives

At the end of this activity, you will be able to:

  • Understand why it is important to make note of missing data values.
  • Be able to define what a NA value is in R and how it is used in a vector.

What you need

You need R and RStudio to complete this tutorial. Also we recommend that you have an earth-analytics directory setup on your computer with a /data directory within it.

Missing data - no data values

Sometimes, our data are missing values. Imagine a spreadsheet in Microsoft Excel with cells that are blank. If the cells are blank, we don’t know for sure whether those data weren’t collected, or someone forgot to fill them in. To account for data that are missing (not by mistake) we can put a value in those cells that represents no data.

The R programming language uses the value NA to represent missing data values.


planets <- c("Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus",
             "Neptune", NA)

The default setting for most base functions that read data into R is to interpret NA as a missing value.

Let’s have a closer look at this using the boulder_precip data that we’ve used in the previous lessons. Please download the data again as there have been some changes made!

# download file
download.file("https://ndownloader.figshare.com/files/9282364",
              "data/boulder-precip.csv",
              method = "libcurl")

Then we can open the data.

# import data but don't specify no data values - what happens?
boulder_precip <- read.csv(file = "data/boulder-precip.csv")

str(boulder_precip)
## 'data.frame':	18 obs. of  3 variables:
##  $ X     : int  756 757 758 759 760 761 762 763 764 765 ...
##  $ DATE  : chr  "2013-08-21" "2013-08-26" "2013-08-27" "2013-09-01" ...
##  $ PRECIP: num  0.1 0.1 0.1 0 0.1 1 2.3 9.8 1.9 1.4 ...

In the example below, note how a mean value is calculated differently depending upon on how NA values are treated when the data are imported.

# view mean values
mean(boulder_precip$PRECIP)
## [1] 1.055556
mean(boulder_precip$TEMP)
## Warning in mean.default(boulder_precip$TEMP): argument is not numeric or
## logical: returning NA
## [1] NA

Notice that we are able to calculate a mean value for PRECIP but TEMP returns a NA value. Why? Let’s plot our data to figure out what might be going on.

# are there data in the TEMP column of our data?
boulder_precip$TEMP
## NULL
# plot the data with ggplot
ggplot(data = boulder_precip, aes(x = DATE, y = TEMP)) +
  geom_point() +
  labs(title = "Temperature data for Boulder, CO")
## Error in FUN(X[[i]], ...): object 'TEMP' not found

quick plot of temperature

Looking at our data, it appears as if we have some extremely large negative values hovering around -1000. However why did our mean return NA?

When performing mathematical operations on numbers in R, most functions will return the value NA if the data you are working with include missing or nodata values.

Returning NA values allows you to see that you have missing data in your dataset. You can then decide how you want to handle the missing data. Youcan add the argument na.rm=TRUE to calculate the result while ignoring the missing values.

heights <- c(2, 4, 4, NA, 6)
mean(heights)
## [1] NA
max(heights)
## [1] NA
mean(heights, na.rm = TRUE)
## [1] 4
max(heights, na.rm = TRUE)
## [1] 6

Let’s try to add the na.rm argument to our code mean calculation on the temperature column above.

# calculate mean usign the na.rm argument
mean(boulder_precip$PRECIP)
## [1] 1.055556
mean(boulder_precip$TEMP, na.rm = TRUE)
## Warning in mean.default(boulder_precip$TEMP, na.rm = TRUE): argument is not
## numeric or logical: returning NA
## [1] NA

Data tip: The functions, is.na(), na.omit(), and complete.cases() are all useful for figuring out if your data has assigned (NA) no-data values. See below for examples.

So now you have successfully calculated the mean value of both precipitation and temperature in our spreadsheet. However does the mean temperature value (NA make sense looking at the data? It seems a bit low - we know that there aren’t temperature values of -200 here in Boulder, Colorado!

Remembering the plot above we noticed that we had some values that were close to -1000. Looking at the summary below we see the exact minimum value is -999.

# calculate mean usign the na.rm argument
summary(boulder_precip$TEMP, na.rm = TRUE)
## Length  Class   Mode 
##      0   NULL   NULL

Finding & assigning no data values

Sometimes, you’ll find a dataset that uses another value for missing data. In some disciplines, for example -999, is frequently used. If there are multiple types of missing values in your dataset, you can extend what R considers a missing value when it reads the file in using the “na.strings” argument.

Below use the na.strings argument on our data. Notice that we can tell R that there are several potential ways that our data documents nodata values.

You can provide R with a vector of missing date values as follows:

c("NA", " ", "-999")

Thus R will assign any calls with the values of nothing "", NA or -999 to NA. This should solve all of our missing data problems!

# import data but specify no data values - what happens?
boulder_precip_na <- read.csv(file = "data/boulder-precip.csv",
                     na.strings = c("NA", " ", "-999"))
boulder_precip_na$TEMP
## NULL

Does our new plot look better?

# are there data in the TEMP column of our data?
boulder_precip$TEMP
## NULL
# plot the data with ggplot
ggplot(data = boulder_precip_na, aes(x = DATE, y = TEMP)) +
  geom_point() +
  labs(title = "Temperature data for Boulder, CO",
       subtitle = "missing data accounted for")
## Error in FUN(X[[i]], ...): object 'TEMP' not found

Plot of temperature with missing data accounted for

Optional challenge

  • Question: Why, in the the example above did mean(boulder_precip$avg_temp) return a value of NA?
  • Question: Why, in the the example above did mean(boulder_precip$avg_temp, na.rm = TRUE) also return a value of NA?
# Extract those elements which are not missing values.
heights[!is.na(heights)]
## [1] 2 4 4 6

# Returns the object with incomplete cases removed. The returned object is atomic.
na.omit(heights)
## [1] 2 4 4 6
## attr(,"na.action")
## [1] 4
## attr(,"class")
## [1] "omit"

# Extract those elements which are complete cases.
heights[complete.cases(heights)]
## [1] 2 4 4 6

Challenge activity

  • Question: Why does the following piece of code return a warning?
sample <- c(2, 4, 4, "NA", 6)
mean(sample, na.rm = TRUE)
## Warning in mean.default(sample, na.rm = TRUE): argument is not numeric or
## logical: returning NA
## [1] NA
  • Question: Why does the warning message say the argument is not numeric?

Leave a Comment