Working with Imported Data - the Base R#

Learning Objectives#

Set up the working directory and import data into R.
Explore and manipulate data frames to extract insights.
Perform basic data cleaning, including handling dates and missing values.

Now you have mastered the basics of R. But we really want to learn how to import data! So let us do that.

We have a CSV file of the Atlantic Hurricane Database (HURDAT2) 1851-2022. The data is released by the National Hurricane Center (NHC) at NOAA. Use this link to download: https://tufts.box.com/shared/static/gdqc9tdv7334622tkco37oyg9t2fyeaf.csv

But how do we import this data so we can use it in R?

Working Directory#

All relative file paths in R are relative to the current working directory, which might or might not be the location of the R script, depending on how RStudio was originally launched. If RStudio was launched by double-clicking on the R script or by right-clicking on the R script and then selecting Open With > RStudio, the default working directory will be the location of the R script. Otherwise the default working directory will either be your Documents directory (Windows) or your home directory (macOS and Linux).

We can use getwd() and dir() to explore the current working directory.

getwd()     # The current location of the working directory.
dir()       # The files in the working directory.

'/home/runner/work/guides/guides/source/source/r'

'00_Getting_started_with_R.md'
'01_Intro_to_RStudio.md'
'02_intro-r_script excution and basics.Rmd'
'03_working_with_imported_data.Rmd'
'04_working_with_imported_data_tidyverse.Rmd'
'animation.gif'
'atlantic.csv'
'Fast_Loops_and_Parallel_Processing.md'
'index.md'
'r_data_dataviz'
'r-data-viz_files'

You can use the %in% operator to ensure that your working directory contains data file (atlantic.csv).

"atlantic.csv" %in% dir()

TRUE

If the statement above returns TRUE, you are all set. But if it returns FALSE, you need to change your working directory. Go to the top menu bar: Session > Set Working Directory > To Source File Location.

We can ensure the working directory is set correctly by either re-running the commands from above or clicking on the Files tab in the lower-right panel and then selecting More > Go To Working Directory and exploring the results.

Importing Data#

The atlantic.csv data file is in comma-separated values (CSV) format. (Note that the specifics of this format are outlined in standard RFC 4180.) This is a very common data format and you can easily import it in R as follows.

hurrdata <- read.csv("atlantic.csv")

The read.csv() function has numerous additional optional arguments that we can use to specify how exactly a data file should be read in and interpreted.

To investigate those, we can use the help() function or the ? operator.

?read.csv

Note that CSV files are different from Excel spreadsheets (XLS or XLSX files) and R does not contain the functionality to import the latter by default. An external community-developed package must be used to import Excel files. Installing and loading external packages is discussed later in this script.

Exploring Data#

We see that a new variable “hurrdata” has been added to the environment.

head(hurrdata)

A data.frame: 6 × 23
	ID	Name	Date	Time	Event	Status	Latitude	Longitude	Maximum.Wind	Minimum.Pressure	⋯	Radii.NW.34kt	Radii.NE.50kt	Radii.SE.50kt	Radii.SW.50kt	Radii.NW.50kt	Radii.NE.64kt	Radii.SE.64kt	Radii.SW.64kt	Radii.NW.64kt	Maximum.Wind.Radius
	<chr>	<chr>	<int>	<int>	<chr>	<chr>	<chr>	<chr>	<int>	<int>	⋯	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<int>
1	AL011851	UNNAMED	18510625	0		HU	28.0N	94.8W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
2	AL011851	UNNAMED	18510625	600		HU	28.0N	95.4W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
3	AL011851	UNNAMED	18510625	1200		HU	28.0N	96.0W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
4	AL011851	UNNAMED	18510625	1800		HU	28.1N	96.5W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
5	AL011851	UNNAMED	18510625	2100	L	HU	28.2N	96.8W	80	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999
6	AL011851	UNNAMED	18510626	0		HU	28.2N	97.0W	70	-999	⋯	-999	-999	-999	-999	-999	-999	-999	-999	-999	-999

This shows us a preview of the fist couple rows of the file. You can also click on the data table under Environment > Data.

We can use summary() to get descriptive statistics.

summary(hurrdata)

      ID                Name                Date               Time       
 Length:53976       Length:53976       Min.   :18510625   Min.   :   0.0  
 Class :character   Class :character   1st Qu.:19180804   1st Qu.: 600.0  
 Mode  :character   Mode  :character   Median :19640609   Median :1200.0  
                                       Mean   :19554281   Mean   : 910.8  
                                       3rd Qu.:19961020   3rd Qu.:1800.0  
                                       Max.   :20221111   Max.   :2330.0  
    Event              Status            Latitude          Longitude        
 Length:53976       Length:53976       Length:53976       Length:53976      
 Class :character   Class :character   Class :character   Class :character  
 Mode  :character   Mode  :character   Mode  :character   Mode  :character  
                                                                            
                                                                            
                                                                            
  Maximum.Wind    Minimum.Pressure Radii.NE.34kt    Radii.SE.34kt   
 Min.   :-99.00   Min.   :-999.0   Min.   :-999.0   Min.   :-999.0  
 1st Qu.: 35.00   1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0  
 Median : 45.00   Median :-999.0   Median :-999.0   Median :-999.0  
 Mean   : 52.57   Mean   :-149.4   Mean   :-795.1   Mean   :-796.4  
 3rd Qu.: 70.00   3rd Qu.: 995.0   3rd Qu.:-999.0   3rd Qu.:-999.0  
 Max.   :165.00   Max.   :1024.0   Max.   : 780.0   Max.   : 720.0  
 Radii.SW.34kt    Radii.NW.34kt  Radii.NE.50kt    Radii.SE.50kt   
 Min.   :-999.0   Min.   :-999   Min.   :-999.0   Min.   :-999.0  
 1st Qu.:-999.0   1st Qu.:-999   1st Qu.:-999.0   1st Qu.:-999.0  
 Median :-999.0   Median :-999   Median :-999.0   Median :-999.0  
 Mean   :-801.4   Mean   :-799   Mean   :-806.1   Mean   :-806.4  
 3rd Qu.:-999.0   3rd Qu.:-999   3rd Qu.:-999.0   3rd Qu.:-999.0  
 Max.   : 660.0   Max.   : 600   Max.   : 360.0   Max.   : 300.0  
 Radii.SW.50kt    Radii.NW.50kt    Radii.NE.64kt    Radii.SE.64kt   
 Min.   :-999.0   Min.   :-999.0   Min.   :-999.0   Min.   :-999.0  
 1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0  
 Median :-999.0   Median :-999.0   Median :-999.0   Median :-999.0  
 Mean   :-807.7   Mean   :-807.1   Mean   :-809.2   Mean   :-809.3  
 3rd Qu.:-999.0   3rd Qu.:-999.0   3rd Qu.:-999.0   3rd Qu.:-999.0  
 Max.   : 360.0   Max.   : 390.0   Max.   : 180.0   Max.   : 250.0  
 Radii.SW.64kt    Radii.NW.64kt    Maximum.Wind.Radius
 Min.   :-999.0   Min.   :-999.0   Min.   :-999.0     
 1st Qu.:-999.0   1st Qu.:-999.0   1st Qu.:-999.0     
 Median :-999.0   Median :-999.0   Median :-999.0     
 Mean   :-809.7   Mean   :-809.5   Mean   :-973.2     
 3rd Qu.:-999.0   3rd Qu.:-999.0   3rd Qu.:-999.0     
 Max.   : 180.0   Max.   : 300.0   Max.   : 400.0     

We might wonder - what type of data is hurrdata? We imported it from csv, but how is it stored?

class(hurrdata)

'data.frame'

We can see that it is a “data.frame”, commonly referred to as a data frame. A data frame is a table where you have observations as rows and variables as columns. Data frames have some great features for working with data and are the go-to for R data storage. You can think of them almost as spreadsheets.

Working with Data Frames#

Let us say we want to access the maximum wind speed of the sixth observation. We can do this in multiple ways. Knowing that “Maximum.Wind” is the ninth column:

hurrdata[[6, 9]]    # [[row, column]]
hurrdata[[9]][6]    # [[column]][row]

NOTE: When using a data frame, we must use [[]] to access variable values. [] will just give us a subset of the data frame, not the values themselves.

Alternatively, we could use the column name:

hurrdata[["Maximum.Wind"]][6]
hurrdata$Maximum.Wind[6]

Note the dollar sign ($). This is a special operator that allows us to access data.frame variables (columns) based on their name.

The “$” operator is the preferred way of accessing data.frame variables based on their name, as it removes the complexity of when to use [[]] vs [] and does not require quotation marks … given your column names do not contain spaces.

Note how the read.csv() function automatically replaced spaces with periods in the column names to accommodate this. Underscores are also an acceptable alternative to spaces and other functions might use those instead of periods.

Selecting Data Based on Conditions#

Note how the summary() function from before did not provide much information on the columns containing textual (character/string) data. These columns can be analyzed further using the table() function. Can you guess what it does?

table(hurrdata$Name)

It creates a frequency table of all the unique values in the column! But what if we wanted to analyze hurricanes only with a specific name?

We can select rows based on a condition by combining logical operators with [] to subset all the rows where the logical operator returns TRUE. For example, all the hurricanes named Nicole can be extracted as follows.

hurrdata[hurrdata$Name == "NICOLE", ]

Note that we leave the column index blank to select all the columns.

Alternatively, we could extract values from one column based on the values of another column. For example, the maximum wind speed for all the hurricanes named Nicole could be obtained as follows.

max(hurrdata$Maximum.Wind[hurrdata$Name == "NICOLE"])

120

Data Cleaning: Dates & Strings#

Let us say we want to analyze maximum wind speed by year. Note how the date is stored as a number in YYYYMMDD format. This notation is great for sorting but very inconvenient for analysis.

We can use the “$” operator to easily extract the Data column as follows.

hurrdata$Date

Passing this column to the as.character() function will convert all the numbers into text (character data). Note the quotes around the new values.

as.character(hurrdata$Date)

Let us store these new character (string) representations of the dates in a new variable and use the substr() function to extract the year and the month.

date_strings <- as.character(hurrdata$Date)

Extract the year from the date string (position 1-4).

hurrdata$Year <- substr(date_strings, start = 1, stop = 4)

Extract the month from the date string (position 5-6).

hurrdata$Month <- substr(date_strings, 5, 6)

Convert both new variables to numeric to accommodate further analysis.

hurrdata$Month <- as.numeric(hurrdata$Month)
hurrdata$Year <- as.numeric(hurrdata$Year)

Note that this is a somewhat hack-y way of dealing with dates. It is suitable for simple conversions like this, but it is highly encouraged to use external packages specifically designed to work with dates for more complex tasks.

Data Cleaning: Missing Data#

Why are some wind speeds negative? If we looked into the metadata, we would find that we should have removed these! We can check on the negative data:

min(hurrdata$Maximum.Wind)

-99

We can combine [] with logical operators like before to find all the values less than zero (< 0) and replace them with “NA”, which means “No Data” in R.

hurrdata$Maximum.Wind[hurrdata$Maximum.Wind < 0] <- NA

We could also delete the whole observation, but this is bad practice! Let’s check on the results. It should print “NA”.

min(hurrdata$Maximum.Wind)

<NA>

To get the minimum wind speed excluding the NA values, we must call the min() function with na.rm = TRUE to instruct the function to ignore the NA values.

min(hurrdata$Maximum.Wind, na.rm = TRUE)

Sampling the Data#

We can also sample the data.frame to de-clutter the scatter plot. This is a two-step process in base R. First we use the sample() function to randomly select a desired quantity of row indexes/numbers for our data frame.

sample(nrow(hurrdata), 1000)

40398
40944
46501
53353
30485
28719
9675
40264
27787
16740
29617
34354
36087
3529
4513
33329
17997
1868
25392
10009
34883
39735
33828
37221
4572
35513
15482
46639
18423
14400
43862
3783
51029
4814
47937
17974
52866
39550
37752
9552
11343
35098
23577
38495
221
33499
34749
10818
43985
13936
8440
1453
48293
45831
8360
35450
17520
25789
40504
11517
32611
38064
42580
53162
30583
44203
52266
20150
38601
49857
51453
47340
50996
32895
10763
13043
53349
42314
435
47401
17216
38246
11726
22994
9261
51932
4885
24988
41585
9175
44805
40077
9309
48988
7247
5357
31580
24715
52643
50591
12648
50473
48196
5469
8823
43521
50397
9916
12010
4366
39408
14521
24286
48870
2911
20266
12936
18172
2187
2206
52830
5617
52347
28866
3846
52263
5023
46021
15188
38713
21343
3771
11676
41960
42998
16299
13431
6076
198
40252
2316
24151
35073
48429
6320
30193
9073
44646
43769
8498
22654
32526
53011
6745
6578
41566
13816
23420
10367
33229
50085
51807
33066
44690
14530
13149
25778
52675
17040
26069
51922
12066
6454
16598
30240
25955
14621
48877
5198
31355
12796
26979
17681
3844
11417
44592
46926
51661
28658
25210
48101
14120
8986
10782
27196
46650
15135
17098
11104
45626
⋯
43027
6208
4113
27859
23390
13559
36156
46168
30508
42473
17658
6951
16072
29835
45965
50521
11761
29223
30708
26856
5529
47348
51568
20825
16546
41207
44160
45707
22552
30279
24665
53590
13101
51131
46944
23595
9658
22813
51785
18601
11702
46171
16331
1949
52149
7072
52882
37126
29531
49427
31333
31974
13521
10301
511
43348
45853
36776
28875
28016
23524
20462
12687
33096
42493
2923
48409
16963
23548
3193
38638
10908
33182
18506
4100
51309
43714
19103
1834
11371
26563
38551
31089
37108
36157
51273
10158
39599
24471
4701
5952
1207
13016
44236
36159
41102
26905
53775
28839
31161
920
36791
30455
50694
10954
28555
29621
44173
2098
34491
7635
1509
13285
48841
10683
23961
44092
14192
23234
43914
17748
13136
5609
46550
36682
9233
36579
30176
50318
8287
48443
42225
14498
49797
9242
27143
13849
8981
31983
13406
40115
50383
50872
26437
24278
37117
38036
31442
50961
15786
9040
34114
41815
42464
32692
21259
21830
40813
6895
23076
34311
25068
49246
49708
4714
51002
6520
13441
29742
12430
48201
10195
46776
12021
44666
26588
31411
1791
10113
16906
24914
4185
45665
46295
14442
46661
43820
38558
1420
30326
31506
1318
8804
37569
29453
39196
28920
19964
52188
38859

And then we use [] to extract those rows from the data frame.

hurrdata2 <- hurrdata[sample(nrow(hurrdata), 1000), ]

To Get Support: - Please email tts-research@tufts.edu for questions and requests.