Climate Coding Challenge

Climate change is impacting the way people live around the world

Learning Goals:

• Analyze temperature data over time
• Parse date information so that it is represented as a datetime type
• Use operators to convert to different units
• Resample time-series data to different frequencies

Part 1: Overview

Higher highs, lower lows, storms, and smoke – we’re all feeling the effects of climate change. In this workflow, you will take a look at trends in temperature over time in Karachi, Pakistan.

Conversation Starter

In a few sentences, how is climate change affecting your home?

What the fork?! Who wrote this?

Below is a scientific Python workflow. But something’s wrong – The code won’t run! Your task is to follow the instructions below to clean and debug the Python code below so that it runs.

Tip

Don’t worry if you can’t solve every bug right away. We’ll get there! If you are working on one bug for more than about 10 minutes, it’s time to ask for help.

At the end, you’ll repeat the workflow for a location and measurement of your choosing.

Alright! Let’s clean up this code.

Check out our demo video!

DEMO: Get Started with Open Reproducible Science by ESIIL

Before we get started, let’s define some parameters. You can use these if you want to change how the workflow runs from the top:

id = 'shortcourse'
ncei_filename = 'ncei-climate-karachi.csv'
project_name = 'Karachi Climate'
location = 'Karachi, Pakistan'
station_id = 'PKM00041780'
start_date = '1942-10-01'
end_date = '2024-09-30'
data_type = 'TAVG'

STEP 2: Wrangle your data

Python packages let you use code written by experts around the world

Because Python is open source, lots of different people and organizations can contribute (including you!). Many contributions are in the form of packages which do not come with a standard Python download.

Read More: Packages need to be installed and imported.

Learn more about using Python packages. How do you find and use packages? What is the difference between installing and importing packages? When do you need to do each one? This article on Python packages will walk you through the basics.

In the cell below, someone was trying to import the pandas package, which helps us to work with tabular data such as comma-separated value or csv files. But something’s wrong!

Try It: Import packages

1. Correct the typo below to properly import the pandas package under its alias pd.
2. Run the cell to import the libraries you’ll need for this workflow.

# Import libraries
import earthpy
import holoviews as hv
import hvplot.pandas
import pandsa as pd

# Import libraries
import earthpy
import holoviews as hv
import hvplot.pandas
import pandas as pd

Download the practice data

Next, lets download some climate data from Karachi, Pakistan to practice with.

1. It is surrounded by quotes – that means Python will interpret it as a character string, or text, rather than which makes sense for a URL.
2. The URL is too long to display as one line on most screens. We’ve put parentheses around it so that we can easily split it into multiple lines by writing two strings – one on each line.
3. We replaced the figshare identifier for this dataset with 'FIGSHARE_ID_HERE'. You’ll have to replace that with the real identifier, ?meta:params.figshare_id

However, we still have a problem - we can’t get the URL back later on because it isn’t saved in a variable. In other words, we need to give the url a name so that we can request in from Python later (sadly, Python has no ‘hey what was that thingy I typed yesterday?’ function).

Read More: Names/variables in Python

One of the most common challenges for new programmers is making sure that your results are stored so you can use them again. In Python, this is called naming, or saving a variable. Learn more in this hands-on activity on using variables from our learning portal.

Try It: Save the URL for later

1. Replace Project Name Here with the actual project name, Karachi Climate.
2. Replace data-folder-name-here with a descriptive name for your data folder.
3. Run the cell. Can you find the data on your computer?

# Set up project folders
project = earthpy.Project(
    'Project Name Here',
    dirname='data-folder-name-here')

# Download data
project.get_data()

# Check where the data ended up
project.project_dir

# Set up project folders
project = earthpy.Project(project_name)

# Download data
project.get_data()

# Check where the data ended up
project.project_dir

Downloading from https://ndownloader.figshare.com/files/55245161

PosixPath('/home/runner/.local/share/earth-analytics/karachi-climate')

If you are on GitHub Codespaces, you should be able to see your data in your Explorer tab.

You can find the Explorer tab on the left hand side of the screen. Your data should be in the data folder mounted there.

You can also take a look at your data using bash, either in your terminal or here in your Jupyter notebook:

%%bash
ls $(project.project_dir)

bash: line 1: project.project_dir: command not found

_climate-01-machine-readable.qmd
_climate-11-python-as-a-calculator.qmd
_climate-31-overview.qmd
_climate-32-wrangle.qmd
_climate-33-units.qmd
_climate-34-plot.qmd
_climate-35-trend-line.qmd
_climate-98-download.qmd
_climate-99-portfolio.qmd
annual_climate.html
climate-download-css.qmd
climate-download-eda.qmd
climate-download-shortcourse.qmd
climate-download-shortcourse.quarto_ipynb
climate-eda.qmd
climate-shortcourse.qmd
climate-shortcourse.quarto_ipynb
climate-stars.qmd

The pandas library you imported can download data from the internet directly into a type of Python object called a DataFrame. In the code cell below, you can see an attempt to do just this. But there are some problems…

Try It: Fix some code!

1. Leave a space between the # and text in the comment, capitalize it, and try to make it more informative

2. Make any changes needed to get this code to run. HINT: The my_url variable doesn’t exist - you need to replace it with the variable name you chose.

3. Modify the .read_csv() function call to include the following parameters:

   • index_col='DATE' – this sets the DATE column as the index. Needed for subsetting and resampling later on
   • parse_dates=True – this lets python know that you are working with time-series data, and values in the indexed column are date time objects
   • na_values=['NaN'] – this lets python know how to handle missing values

4. Clean up the code by using expressive variable names, expressive column names, PEP-8 compliant code, and descriptive comments

#download
climate_df = pd.read_csv(
    project.project_dir / 'filename.csv',
    #index_col='something')
climate_df

# Download the climate data
climate_df = pd.read_csv(
    project.project_dir / ncei_filename,
    index_col='DATE',
    parse_dates=True,
    na_values=['NaN'])
climate_df

	STATION	TAVG
DATE
1942-10-01	PKM00041780	81
1942-10-02	PKM00041780	81
1942-10-03	PKM00041780	84
1942-10-04	PKM00041780	84
1942-10-05	PKM00041780	84
...	...	...
2024-09-26	PKM00041780	87
2024-09-27	PKM00041780	87
2024-09-28	PKM00041780	86
2024-09-29	PKM00041780	87
2024-09-30	PKM00041780	87

19371 rows × 2 columns

Tip

Check out the type() function below - you can use it to check that your data is now in DataFrame type object.

# Check that the data was imported into a pandas DataFrame
type(climate_df)

Clean up your DataFrame

Try It: Get rid of unwanted columns

You can use double brackets ([[ and ]]) to select only the columns that you want from your DataFrame:

1. Change some_column_name to the Temperature column name.
2. Give the DataFrame a more descriptive name.
3. Add a properly formatted comment to describe what this code is doing.

Warning

Column names are text values, not variable names, so you need to put them in quotes!

climate_df = climate_df[['some_column_name']]
climate_df

# Clean up the DataFrame
climate_df = climate_df[[f'{data_type}']]
climate_df

	TAVG
DATE
1942-10-01	81
1942-10-02	81
1942-10-03	84
1942-10-04	84
1942-10-05	84
...	...
2024-09-26	87
2024-09-27	87
2024-09-28	86
2024-09-29	87
2024-09-30	87

19371 rows × 1 columns

STEP 3: Convert units

It’s important to keep track of the units of all your data. You don’t want to be like the NASA team who crashed a probe into Mars because different teams used different units)!

Use labels to keep track of units for you and your collaborators

One way to keep track of your data’s units is to include the unit in data labels. In the case of a DataFrame, that usually means the column names.

Try It: Add units to your column name

A big part of writing expressive code is descriptive labels. Let’s rename the columns of your dataframe to include units. Complete the following steps:

1. Replace dataframe with the name of your DataFrame, and dataframe_units with an expressive new name.
2. Check out the documentation for GCHNd data. We downloaded data with “standard” units; find out what that means for both temperature and precipitation.
3. Replace 'TOBS' with the temperature column name in your data, and 'TOBS_UNIT' with a column name that includes the correct unit.

dataframe_units = dataframe.rename(columns={
    'TOBS': 'TOBS_UNIT',
})

dataframe_units

climate_u_df = climate_df.rename(columns={
    f'{data_type}': 'temp_f',
})
climate_u_df

	temp_f
DATE
1942-10-01	81
1942-10-02	81
1942-10-03	84
1942-10-04	84
1942-10-05	84
...	...
2024-09-26	87
2024-09-27	87
2024-09-28	86
2024-09-29	87
2024-09-30	87

19371 rows × 1 columns

For scientific applications, it is often useful to have values in metric units

Try It: Convert units

The code below attempts to convert the data to Celcius, using Python mathematical operators, like +, -, *, and /. Mathematical operators in Python work just like a calculator, and that includes using parentheses to designat the order of operations. The equation for converting Fahrenheit temperature to Celcius is:

\[ T_C = (T_F - 32) * \frac{5}{9} \]

This code is not well documented and doesn’t follow PEP-8 guidelines, which has caused the author to miss an important error!

Complete the following steps:

1. Replace dataframe with the name of your DataFrame.
2. Replace 'old_temperature' with the column name you used; Replace 'new_temperature' with an expressive column name.
3. THERE IS AN ERROR IN THE CONVERSION MATH - Fix it!

dataframe_units['new_temperature']= dataframe_units['old_temperature']-32*5/9
dataframe_units

climate_u_df['temp_c'] = (
    (climate_u_df['temp_f'] - 32) 
    * 5/9)

climate_u_df

	temp_f	temp_c
DATE
1942-10-01	81	27.222222
1942-10-02	81	27.222222
1942-10-03	84	28.888889
1942-10-04	84	28.888889
1942-10-05	84	28.888889
...	...	...
2024-09-26	87	30.555556
2024-09-27	87	30.555556
2024-09-28	86	30.000000
2024-09-29	87	30.555556
2024-09-30	87	30.555556

19371 rows × 2 columns

Looking for an Extra Challenge?

Using the code below as a framework, write and apply a function that converts to Celcius. You should also rewrite this function name to be more expressive.

# Convert units with a function
def convert(temperature):
    """Convert temperature to Celcius"""
    return temperature # Put your equation in here

dataframe['TEMP_C'] = (
    dataframe['TEMP_F'].apply(convert))

def convert_f_to_c(temperature_f):
    """Convert temperature to Celcius"""
    return (temperature_f - 32) * 5/9

climate_u_df['temp_c'] = (
    climate_u_df['temp_f'].apply(convert_f_to_c))

STEP 4: Plot your results

Plot the temperature column vs time to explore the data

Plotting in Python is easy, but not quite this easy:

climate_u_df.plot()

Looks like we have both temperature units on the same plot, and it’s hard to see what it is because it’s missing labels!

Label your plot

Source: https://xkcd.com/833

Make sure each plot has:

• A title that explains where and when the data are from
• x- and y- axis labels with units where appropriate
• A legend where appropriate

When plotting in Python, you’ll always need to add some instructions on labels and how you want your plot to look.

Try It: Plot your data

1. Change dataframe to your DataFrame name.
2. Change y= to the name of your temperature column name.
3. Use the title, ylabel, and xlabel parameters to add key text to your plot.
4. Adjust the size of your figure using figsize=(x,y) where x is figure width and y is figure height

Tip

Labels have to be a type in Python called a string. You can make a string by putting quotes around your label, just like the column names in the sample code (eg y='temperature').

# Plot the data using .plot
climate_u_df.plot(
    y='the_temperature_column',
    title='Title Goes Here',
    xlabel='Horizontal Axis Label Goes Here',
    ylabel='Vertical Axis Label Goes Here')

# Plot the data using .plot
climate_u_df.plot(
    y='temp_c',
    title=f'Daily Temperature in {location}',
    xlabel='Date',
    ylabel='Temperature ($^\circ$C)')

Looking for an Extra Challenge?

There are many other things you can do to customize your plot. Take a look at the pandas plotting galleries and the documentation of plot to see if there’s other changes you want to make to your plot. Some possibilities include:

• Remove the legend since there’s only one data series
• Increase the figure size
• Increase the font size
• Change the colors
• Use a bar graph instead (usually we use lines for time series, but since this is annual it could go either way)
• Add a trend line

Not sure how to do any of these? Try searching the internet, or asking an AI!

Clean up time series plots by resampling

You may notice that your plot looks a little “fuzzy”. This happens when Python is trying to plot a value for every date, but the resolution of the image is too low to actually do that. You can address this issue by resampling the data, or summarizing it over a time period of your choice. In this case, we will resample annually, giving us one data point per year.

Try It: Resample

1. Set the frequency of your final data by replacing DT_OFFSETwith a Datetime Offset Code. Check out the table in the pandas datetime documentation to find the one you want (we recommend the start of the year).
2. Choose how to summarize each year of data by replacing agg_method_here with a method that will calculate the average annual value. Check out the pandas resampling documentation for a list of common built-in options.

ann_climate_df = climate_u_df.resample('DT_OFFSET').agg_method_here()
ann_climate_df

ann_climate_df = climate_u_df.resample('YS').mean()
# Store for later
%store ann_climate_df
ann_climate_df

Stored 'ann_climate_df' (DataFrame)

	temp_f	temp_c
DATE
1942-01-01	74.597826	23.665459
1943-01-01	78.654795	25.919330
1944-01-01	78.423497	25.790832
1945-01-01	77.786301	25.436834
1946-01-01	76.164474	24.535819
...	...	...
2020-01-01	81.229508	27.349727
2021-01-01	81.617729	27.565405
2022-01-01	81.257618	27.365343
2023-01-01	81.391185	27.439547
2024-01-01	83.018939	28.343855

83 rows × 2 columns

Try It: Plot Annual Data

1. Try plotting your new DataFrame in the cell below. Can you see what is going on more clearly now? Don’t forget to adjust your labels!

# Plot the annual data

# Plot the annual data using .plot
ann_climate_df.plot(
    y='temp_c',
    title=f'Annual Average Temperature in {location}',
    xlabel='Year',
    ylabel='Temperature ($^\circ$C)'
)

Reflect and Respond: Interpret your plot

1. Create a new Markdown cell below this one.

2. In the new cell, answer the following questions using a bulleted list in Markdown – what are 2 things you notice about this data? What physical phenomena or data anomaly could be causing each one?

Check specific values with an interactive plot

You can use the .hvplot() method with similar arguments to create an interactive plot.

Try It: Interactive Plot

1. Copy your plotting code into the cell below.
2. Replace .plot in your code with .hvplot

Now, you should be able to hover over data points and see their values!

# Plot the annual data interactively

# Plot the annual data using .hvplot
ann_climate_plot = ann_climate_df.hvplot(
    y='temp_c',
    title=f'Annual Average Temperature in {location}',
    xlabel='Year',
    ylabel='Temperature (deg. C)'
)
ann_climate_plot

Try It: Explore the data

1. Create a new Markdown cell below this one.
2. Hover over the lowest point on your plot. What is the overall maximum annual average temperature?

BONUS: Save your work

You will need to save your analyses and plots to tell others about what you find.

Try It: Save Your Plot

Just like with any other type of object in Python, if you want to reuse your work, you need to give it a name.

1. Go back to your hvplot code, and give your plot a name by assigning it to a variable. HINT: if you still want your plot to display in your notebook, make sure to call its name at the end of the cell.
2. Replace my_plot with the name you gave to your plot.
3. Replace 'my_plot.html' with the name you want for your plot. If you change the file extension, .html, to .png, you will get an image instead of an interactive webpage, provided you have the necessary libraries installed.

Once you run the code, you should see your saved plot in your files – go ahead and open it up.

Warning

If you are working in GitHub Codespaces, right-click on your file and download it to view it.

hv.save(my_plot, 'my_plot.html')

hv.save(ann_climate_plot, 'annual_climate.html')