Object-Oriented Programming in Python

Course Description

You’ve done your analysis, built your report, and trained a model. What’s next? Well, if you want to deploy your model into production, your code will need to be more reliable than exploratory scripts in a Jupyter notebook. Writing Functions in Python will give you a strong foundation in writing complex and beautiful functions so that you can contribute research and engineering skills to your team. You’ll learn useful tricks, like how to write context managers and decorators. You’ll also learn best practices around how to write maintainable reusable functions with good documentation. They say that people who can do good research and write high-quality code are unicorns. Take this course and discover the magic!

Best Practices

The goal of this course is to transform you into a Python expert, and so the first chapter starts off with best practices when writing functions. You’ll cover docstrings and why they matter and how to know when you need to turn a chunk of code into a function. You will also learn the details of how Python passes arguments to functions, as well as some common gotchas that can cause debugging headaches when calling functions.

Docstrings

Crafting a docstring

You’ve decided to write the world’s greatest open-source natural language processing Python package. It will revolutionize working with free-form text, the way numpy did for arrays, pandas did for tabular data, and scikit-learn did for machine learning.

The first function you write is count_letter(). It takes a string and a single letter and returns the number of times the letter appears in the string. You want the users of your open-source package to be able to understand how this function works easily, so you will need to give it a docstring. Build up a Google Style docstring for this function by following these steps.

• Copy the following string and add it as the docstring for the function: Count the number of times `letter` appears in `content`.

• Now add the arguments section, using the Google style for docstrings. Use str to indicate a string.

• Add a returns section that informs the user the return value is an int.

• Finally, add some information about the ValueError that gets raised when the arguments aren’t correct.

# Add a docstring to count_letter()
def count_letter(content, letter):
  """Count the number of times `letter` appears in `content`."""
  if (not isinstance(letter, str)) or len(letter) != 1:
    raise ValueError('`letter` must be a single character string.')
  return len([char for char in content if char == letter])

def count_letter(content, letter):
  """Count the number of times `letter` appears in `content`.

  # Add a Google-style arguments section
  Args:
    content (str): The string to search.
    letter (str): The letter to search for.
  """
  if (not isinstance(letter, str)) or len(letter) != 1:
    raise ValueError('`letter` must be a single character string.')
  return len([char for char in content if char == letter])

def count_letter(content, letter):
  """Count the number of times `letter` appears in `content`.

  Args:
    content (str): The string to search.
    letter (str): The letter to search for.

  # Add a returns section
  Returns:
    int
  """
  if (not isinstance(letter, str)) or len(letter) != 1:
    raise ValueError('"letter" must be a single character string.')
  return len([char for char in content if char == letter])

def count_letter(content, letter):
  """Count the number of times `letter` appears in `content`.

  Args:
    content (str): The string to search.
    letter (str): The letter to search for.

  Returns:
    int

  # Add a section detailing what errors might be raised
  Raises:
    ValueError: If `letter` is not a one-character string.
  """
  if (not isinstance(letter, str)) or len(letter) != 1:
    raise ValueError('`letter` must be a single character string.')
  return len([char for char in content if char == letter])

Retrieving docstrings

You and a group of friends are working on building an amazing new Python IDE (integrated development environment – like PyCharm, Spyder, Eclipse, Visual Studio, etc.). The team wants to add a feature that displays a tooltip with a function’s docstring whenever the user starts typing the function name. That way, the user doesn’t have to go elsewhere to look up the documentation for the function they are trying to use. You’ve been asked to complete the build_tooltip() function that retrieves a docstring from an arbitrary function.

You will be reusing the count_letter() function that you developed in the last exercise to show that we can properly extract its docstring.

Begin by getting the docstring for the function count_letter(). Use an attribute of the count_letter() function.

Now use a function from the inspect module to get a better-formatted version of count_letter()’s docstring.

Now create a build_tooltip() function that can extract the docstring from any function that we pass to it.

# Get the "count_letter" docstring by using an attribute of the function
docstring = count_letter.__doc__

border = '#' * 28
print('{}\n{}\n{}'.format(border, docstring, border))

import inspect

# Inspect the count_letter() function to get its docstring
docstring = inspect.getdoc(count_letter)

border = '#' * 28
print('{}\n{}\n{}'.format(border, docstring, border))

import inspect

def build_tooltip(function):
  """Create a tooltip for any function that shows the
  function's docstring.

  Args:
    function (callable): The function we want a tooltip for.

  Returns:
    str
  """
  # Get the docstring for the "function" argument by using inspect
  docstring = inspect.getdoc(function)
  border = '#' * 28
  return '{}\n{}\n{}'.format(border, docstring, border)

print(build_tooltip(count_letter))
print(build_tooltip(range))
print(build_tooltip(print))

Docstrings to the rescue!

Some maniac has corrupted your installation of numpy! All of the functions still exist, but they’ve been given random names. You desperately need to call the numpy.histogram() function and you don’t have time to reinstall the package. Fortunately for you, the maniac didn’t think to alter the docstrings, and you know how to access them. numpy has a lot of functions in it, so we’ve narrowed it down to four possible functions that could be numpy.histogram() in disguise: numpy.leyud(), numpy.uqka(), numpy.fywdkxa() or numpy.jinzyxq().

Examine each of these functions’ docstrings in the IPython shell to determine which of them is actually numpy.histogram().

• numpy.leyud()
• numpy.uqka()
• numpy.fywdkxa()
• numpy.jinzyxq()

DRY and “Do One Thing”

Extract a function

While you were developing a model to predict the likelihood of a student graduating from college, you wrote this bit of code to get the z-scores of students’ yearly GPAs. Now you’re ready to turn it into a production-quality system, so you need to do something about the repetition. Writing a function to calculate the z-scores would improve this code.

# Standardize the GPAs for each year
df['y1_z'] = (df.y1_gpa - df.y1_gpa.mean()) / df.y1_gpa.std()
df['y2_z'] = (df.y2_gpa - df.y2_gpa.mean()) / df.y2_gpa.std()
df['y3_z'] = (df.y3_gpa - df.y3_gpa.mean()) / df.y3_gpa.std()
df['y4_z'] = (df.y4_gpa - df.y4_gpa.mean()) / df.y4_gpa.std()

Note: df is a pandas DataFrame where each row is a student with 4 columns of yearly student GPAs: y1_gpa, y2_gpa, y3_gpa, y4_gpa

• Finish the function so that it returns the z-scores of a column.
• Use the function to calculate the z-scores for each year (df['y1_z'], df['y2_z'], etc.) from the raw GPA scores (df.y1_gpa, df.y2_gpa, etc.).

# edited/added
import pandas as pd
df = pd.read_csv("students.csv")

def standardize(column):
  """Standardize the values in a column.

  Args:
    column (pandas Series): The data to standardize.

  Returns:
    pandas Series: the values as z-scores
  """
  # Finish the function so that it returns the z-scores
  z_score = (column - column.mean()) / column.std()
  return z_score

# Use the standardize() function to calculate the z-scores
df['y1_z'] = standardize(df.y1_gpa)
df['y2_z'] = standardize(df.y2_gpa)
df['y3_z'] = standardize(df.y3_gpa)
df['y4_z'] = standardize(df.y4_gpa)

Split up a function

Another engineer on your team has written this function to calculate the mean and median of a sorted list. You want to show them how to split it into two simpler functions: mean() and median()

def mean_and_median(values):
  """Get the mean and median of a sorted list of `values`

  Args:
    values (iterable of float): A list of numbers

  Returns:
    tuple (float, float): The mean and median
  """
  mean = sum(values) / len(values)
  midpoint = int(len(values) / 2)
  if len(values) % 2 == 0:
    median = (values[midpoint - 1] + values[midpoint]) / 2
  else:
    median = values[midpoint]

  return mean, median

Write the mean() function.

Write the median() function.

def mean(values):
  """Get the mean of a sorted list of values

  Args:
    values (iterable of float): A list of numbers

  Returns:
    float
  """
  # Write the mean() function
  mean = sum(values) / len(values)
  return mean

def median(values):
  """Get the median of a sorted list of values

  Args:
    values (iterable of float): A list of numbers

  Returns:
    float
  """
  # Write the median() function
  midpoint = int(len(values) / 2)
  if len(values) % 2 == 0:
    median = (values[midpoint - 1] + values[midpoint]) / 2
  else:
    median = values[midpoint]
  return median

Pass by assignment

Mutable or immutable?

The following function adds a mapping between a string and the lowercase version of that string to a dictionary. What do you expect the values of d and s to be after the function is called?

def store_lower(_dict, _string):
  """Add a mapping between `_string` and a lowercased version of `_string` to `_dict`

  Args:
    _dict (dict): The dictionary to update.
    _string (str): The string to add.
  """
  orig_string = _string
  _string = _string.lower()
  _dict[orig_string] = _string

d = {}
s = 'Hello'

store_lower(d, s)

• d = {}, s = 'Hello'
• d = {}, s = 'hello'
• d = {'Hello': 'hello'}, s = 'Hello'
• d = {'Hello': 'hello'}, s = 'hello'
• d = {'hello': 'hello'}, s = 'hello'

Best practice for default arguments

One of your co-workers (who obviously didn’t take this course) has written this function for adding a column to a pandas DataFrame. Unfortunately, they used a mutable variable as a default argument value! Please show them a better way to do this so that they don’t get unexpected behavior.

def add_column(values, df=pandas.DataFrame()):
  """Add a column of `values` to a DataFrame `df`.
  The column will be named "col_<n>" where "n" is
  the numerical index of the column.

  Args:
    values (iterable): The values of the new column
    df (DataFrame, optional): The DataFrame to update.
      If no DataFrame is passed, one is created by default.

  Returns:
    DataFrame
  """
  df['col_{}'.format(len(df.columns))] = values
  return df

• Change the default value of df to an immutable value to follow best practices.
• Update the code of the function so that a new DataFrame is created if the caller didn’t pass one.

# Use an immutable variable for the default argument
def better_add_column(values, df=None):
  """Add a column of `values` to a DataFrame `df`.
  The column will be named "col_<n>" where "n" is
  the numerical index of the column.

  Args:
    values (iterable): The values of the new column
    df (DataFrame, optional): The DataFrame to update.
      If no DataFrame is passed, one is created by default.

  Returns:
    DataFrame
  """
  # Update the function to create a default DataFrame
  if df is None:
    df = pandas.DataFrame()
  df['col_{}'.format(len(df.columns))] = values
  return df

Context Managers

If you’ve ever seen the “with” keyword in Python and wondered what its deal was, then this is the chapter for you! Context managers are a convenient way to provide connections in Python and guarantee that those connections get cleaned up when you are done using them. This chapter will show you how to use context managers, as well as how to write your own.

Using context managers

The number of cats

You are working on a natural language processing project to determine what makes great writers so great. Your current hypothesis is that great writers talk about cats a lot. To prove it, you want to count the number of times the word “cat” appears in “Alice’s Adventures in Wonderland” by Lewis Carroll. You have already downloaded a text file, alice.txt, with the entire contents of this great book.

• Use the open() context manager to open alice.txt and assign the file to the file variable.

# Open "alice.txt" and assign the file to "file"
with open('alice.txt') as file:
  text = file.read()

n = 0
for word in text.split():
  if word.lower() in ['cat', 'cats']:
    n += 1

print('Lewis Carroll uses the word "cat" {} times'.format(n))

The speed of cats

You’re working on a new web service that processes Instagram feeds to identify which pictures contain cats (don’t ask why – it’s the internet). The code that processes the data is slower than you would like it to be, so you are working on tuning it up to run faster. Given an image, image, you have two functions that can process it:

• process_with_numpy(image)
• process_with_pytorch(image)

Your colleague wrote a context manager, timer(), that will print out how long the code inside the context block takes to run. She is suggesting you use it to see which of the two options is faster. Time each function to determine which one to use in your web service.

• Use the timer() context manager to time how long process_with_numpy(image) takes to run.
• Use the timer() context manager to time how long process_with_pytorch(image) takes to run.

# edited/added
import numpy as np
import time
import contextlib
def get_image_from_instagram():
  return np.random.rand(84, 84)
def _process_pic(n_sec):
  print('Processing', end='', flush=True)
  for i in range(10):
    print('.', end='' if i < 9 else 'done!\n', flush=True)
    time.sleep(n_sec)
def process_with_pytorch(p):
  _process_pic(0.0328)
def process_with_numpy(p):
  _process_pic(0.1521)
@contextlib.contextmanager
def timer():
  """Time how long code in the context block takes to run."""
  t0 = time.time()
  try:
      yield
  except:
    raise
  finally:
    t1 = time.time()
    print('Elapsed: {:.2f} seconds'.format(t1 - t0))

image = get_image_from_instagram()

# Time how long process_with_numpy(image) takes to run
with timer():
  print('Numpy version')
  process_with_numpy(image)

# Time how long process_with_pytorch(image) takes to run
with timer():
  print('Pytorch version')
  process_with_pytorch(image)

Writing context managers

The timer() context manager

A colleague of yours is working on a web service that processes Instagram photos. Customers are complaining that the service takes too long to identify whether or not an image has a cat in it, so your colleague has come to you for help. You decide to write a context manager that they can use to time how long their functions take to run.

• Add a decorator from the contextlib module to the timer() function that will make it act like a context manager.
• Send control from the timer() function to the context block.

# Add a decorator that will make timer() a context manager
@contextlib.contextmanager
def timer():
  """Time the execution of a context block.

  Yields:
    None
  """
  start = time.time()
  # Send control back to the context block
  yield
  end = time.time()
  print('Elapsed: {:.2f}s'.format(end - start))

with timer():
  print('This should take approximately 0.25 seconds')
  time.sleep(0.25)

A read-only open() context manager

You have a bunch of data files for your next deep learning project that took you months to collect and clean. It would be terrible if you accidentally overwrote one of those files when trying to read it in for training, so you decide to create a read-only version of the open() context manager to use in your project.

The regular open() context manager:

• takes a filename and a mode ('r' for read, 'w' for write, or 'a' for append)
• opens the file for reading, writing, or appending
• yields control back to the context, along with a reference to the file
• waits for the context to finish
• and then closes the file before exiting

Your context manager will do the same thing, except it will only take the filename as an argument and it will only open the file for reading.

• Yield control from open_read_only() to the context block, ensuring that the read_only_file object gets assigned to my_file.
• Use read_only_file’s .close() method to ensure that you don’t leave open files lying around.

@contextlib.contextmanager
def open_read_only(filename):
  """Open a file in read-only mode.

  Args:
    filename (str): The location of the file to read

  Yields:
    file object
  """
  read_only_file = open(filename, mode='r')
  # Yield read_only_file so it can be assigned to my_file
  yield read_only_file
  # Close read_only_file
  read_only_file.close()

with open_read_only('my_file.txt') as my_file:
  print(my_file.read())

Advanced topics

Context manager use cases

Which of the following would NOT be a good opportunity to use a context manager?

• A function that starts a timer that keeps track of how long some block of code takes to run.
• A function that prints all of the prime numbers between 2 and some value n.
• A function that connects to a smart thermostat so that it can be programmed remotely.
• A function that prevents multiple users from updating an online spreadsheet at the same time by locking access to the spreadsheet before every operation.

Scraping the NASDAQ

Training deep neural nets is expensive! You might as well invest in NVIDIA stock since you’re spending so much on GPUs. To pick the best time to invest, you are going to collect and analyze some data on how their stock is doing. The context manager stock('NVDA') will connect to the NASDAQ and return an object that you can use to get the latest price by calling its .price() method.

You want to connect to stock('NVDA') and record 10 timesteps of price data by writing it to the file NVDA.txt.

You will notice the use of an underscore when iterating over the for loop. If this is confusing to you, don’t worry. It could easily be replaced with i, if we planned to do something with it, like use it as an index. Since we won’t be using it, we can use a dummy operator, _, which doesn’t use any additional memory.

• Use the stock('NVDA') context manager and assign the result to nvda.
• Open a file for writing with open('NVDA.txt', 'w') and assign the file object to f_out so you can record the price over time.

# edited/added
class MockStock:
    def __init__(self, loc, scale):
        self.loc = loc
        self.scale = scale
        self.recent = list(np.random.laplace(loc, scale, 2))
    def price(self):
        sign = np.sign(self.recent[1] - self.recent[0])
        # 70% chance of going same direction
        sign = 1 if sign == 0 else (sign if np.random.rand() > 0.3 else -1 * sign)
        new = self.recent[1] + sign * np.random.rand() / 10.0
        self.recent = [self.recent[1], new]
        return new
@contextlib.contextmanager
def stock(symbol):
    base = 140.00
    scale = 1.0
    mock = MockStock(base, scale)
    print('Opening stock ticker for {}'.format(symbol))
    yield mock
    print('Closing stock ticker')

# Use the "stock('NVDA')" context manager
# and assign the result to the variable "nvda"
with stock('NVDA') as nvda:
  # Open 'NVDA.txt' for writing as f_out
  with open('NVDA.txt', 'w') as f_out:
    for _ in range(10):
      value = nvda.price()
      print('Logging ${:.2f} for NVDA'.format(value))
      f_out.write('{:.2f}\n'.format(value))

Changing the working directory

You are using an open-source library that lets you train deep neural networks on your data. Unfortunately, during training, this library writes out checkpoint models (i.e., models that have been trained on a portion of the data) to the current working directory. You find that behavior frustrating because you don’t want to have to launch the script from the directory where the models will be saved.

You decide that one way to fix this is to write a context manager that changes the current working directory, lets you build your models, and then resets the working directory to its original location. You’ll want to be sure that any errors that occur during model training don’t prevent you from resetting the working directory to its original location.

• Add a statement that lets you handle any errors that might occur inside the context.
• Add a statement that ensures os.chdir(current_dir) will be called, whether there was an error or not.

def in_dir(directory):
  """Change current working directory to `directory`,
  allow the user to run some code, and change back.

  Args:
    directory (str): The path to a directory to work in.
  """
  current_dir = os.getcwd()
  os.chdir(directory)

  # Add code that lets you handle errors
  try:
    yield
  # Ensure the directory is reset,
  # whether there was an error or not
  finally:
    os.chdir(current_dir)

Decorators

Decorators are an extremely powerful concept in Python. They allow you to modify the behavior of a function without changing the code of the function itself. This chapter will lay the foundational concepts needed to thoroughly understand decorators (functions as objects, scope, and closures), and give you a good introduction into how decorators are used and defined. This deep dive into Python internals will set you up to be a superstar Pythonista. More on Decorators

Functions are objects

Building a command line data app

You are building a command line tool that lets a user interactively explore a dataset. We’ve defined four functions: mean(), std(), minimum(), and maximum() that users can call to analyze their data. Help finish this section of the code so that your users can call any of these functions by typing the function name at the input prompt.

Note: The function get_user_input() in this exercise is a mock version of asking the user to enter a command. It randomly returns one of the four function names. In real life, you would ask for input and wait until the user entered a value.

• Add the functions std(), minimum(), and maximum() to the function_map dictionary, like we did with mean().
• The name of the function the user wants to call is stored in func_name. Use the dictionary of functions, function_map, to call the chosen function and pass data as an argument.

# edited/added
import random
def get_user_input(prompt='Type a command: '):
    command = random.choice(['mean', 'std', 'minimum', 'maximum'])
    print(prompt)
    print('> {}'.format(command))
    return command
def mean(data):
    print(data.mean())
def std(data):
    print(data.std())
def minimum(data):
    print(data.min())
def maximum(data):
    print(data.max())
def load_data():
    df = pd.DataFrame()
    df['height'] = [72.1, 69.8, 63.2, 64.7]
    df['weight'] = [198, 204, 164, 238]
    return df

# Add the missing function references to the function map
function_map = {
  'mean': mean,
  'std': std,
  'minimum': minimum,
  'maximum': maximum
}

data = load_data()
print(data)

func_name = get_user_input()

# Call the chosen function and pass "data" as an argument
function_map[func_name](data)

Reviewing your co-worker’s code

Your co-worker is asking you to review some code that they’ve written and give them some tips on how to get it ready for production. You know that having a docstring is considered best practice for maintainable, reusable functions, so as a sanity check you decide to run this has_docstring() function on all of their functions.

def has_docstring(func):
  """Check to see if the function 
  `func` has a docstring.

  Args:
    func (callable): A function.

  Returns:
    bool
  """
  return func.__doc__ is not None

Call has_docstring() on your co-worker’s load_and_plot_data() function.

• Check if the function as_2D() has a docstring.

• Check if the function log_product() has a docstring.

# edited/added
def has_docstring(func):
    """Check to see if the function 
    `func` has a docstring.
    Args:
        func (callable): A function.
    Returns:
        bool
    """
    return func.__doc__ is not None
def load_and_plot_data(filename):
    """Load a data frame and plot each column.
    Args:
        filename (str): Path to a CSV file of data.
    Returns:
        pandas.DataFrame
    """
    df = pd.load_csv(filename, index_col=0)
    df.hist()
    return df
def as_2D(arr):
    """Reshape an array to 2 dimensions"""
    return np.array(arr).reshape(1, -1)
def log_product(arr):
    return np.exp(np.sum(np.log(arr)))

# Call has_docstring() on the load_and_plot_data() function
ok = has_docstring(load_and_plot_data)

if not ok:
  print("load_and_plot_data() doesn't have a docstring!")
else:
  print("load_and_plot_data() looks ok")
  
  # Call has_docstring() on the as_2D() function
ok = has_docstring(as_2D)

if not ok:
  print("as_2D() doesn't have a docstring!")
else:
  print("as_2D() looks ok")
  
# Call has_docstring() on the log_product() function
ok = has_docstring(log_product)

if not ok:
  print("log_product() doesn't have a docstring!")
else:
  print("log_product() looks ok")

Returning functions for a math game

You are building an educational math game where the player enters a math term, and your program returns a function that matches that term. For instance, if the user types “add”, your program returns a function that adds two numbers. So far you’ve only implemented the “add” function. Now you want to include a “subtract” function.

• Define the subtract() function. It should take two arguments and return the first argument minus the second argument.

def create_math_function(func_name):
  if func_name == 'add':
    def add(a, b):
      return a + b
    return add
  elif func_name == 'subtract':
    # Define the subtract() function
    def subtract(a, b):
      return a - b
    return subtract
  else:
    print("I don't know that one")
    
add = create_math_function('add')
print('5 + 2 = {}'.format(add(5, 2)))

subtract = create_math_function('subtract')
print('5 - 2 = {}'.format(subtract(5, 2)))

Scope

Understanding scope

What four values does this script print?

x = 50

def one():
  x = 10

def two():
  global x
  x = 30

def three():
  x = 100
  print(x)

for func in [one, two, three]:
  func()
  print(x)

• 50, 30, 100, 50
• 10, 30, 30, 30
• 50, 30, 100, 30
• 10, 30, 100, 50
• 50, 50, 50, 50

Modifying variables outside local scope

Sometimes your functions will need to modify a variable that is outside of the local scope of that function. While it’s generally not best practice to do so, it’s still good to know how in case you need to do it. Update these functions so they can modify variables that would usually be outside of their scope.

Add a keyword that lets us update call_count from inside the function.

Add a keyword that lets us modify file_contents from inside save_contents().

Add a keyword to done in check_is_done() so that wait_until_done() eventually stops looping.

call_count = 0

def my_function():
  # Use a keyword that lets us update call_count 
  global call_count
  call_count += 1
  
  print("You've called my_function() {} times!".format(
    call_count
  ))
  
for _ in range(20):
  my_function()
  
def read_files():
  file_contents = None
  
  def save_contents(filename):
    # Add a keyword that lets us modify file_contents
    nonlocal file_contents
    if file_contents is None:
      file_contents = []
    with open(filename) as fin:
      file_contents.append(fin.read())
      
  for filename in ['1984.txt', 'MobyDick.txt', 'CatsEye.txt']:
    save_contents(filename)
    
  return file_contents

print('\n'.join(read_files()))

def wait_until_done():
  def check_is_done():
    # Add a keyword so that wait_until_done() 
    # doesn't run forever
    global done
    if random.random() < 0.1:
      done = True
      
  while not done:
    check_is_done()

done = False
wait_until_done()

print('Work done? {}'.format(done))

Closures

Checking for closure

You’re teaching your niece how to program in Python, and she is working on returning nested functions. She thinks she has written the code correctly, but she is worried that the returned function won’t have the necessary information when called. Show her that all of the nonlocal variables she needs are in the new function’s closure.

• Use an attribute of the my_func() function to show that it has a closure that is not None.

• Show that there are two variables in the closure.

• Get the values of the variables in the closure so you can show that they are equal to [2, 17], the arguments passed to return_a_func().

def return_a_func(arg1, arg2):
  def new_func():
    print('arg1 was {}'.format(arg1))
    print('arg2 was {}'.format(arg2))
  return new_func
    
my_func = return_a_func(2, 17)

# Show that my_func()'s closure is not None
print(my_func.__closure__ is not None)

def return_a_func(arg1, arg2):
  def new_func():
    print('arg1 was {}'.format(arg1))
    print('arg2 was {}'.format(arg2))
  return new_func
    
my_func = return_a_func(2, 17)

print(my_func.__closure__ is not None)

# Show that there are two variables in the closure
print(len(my_func.__closure__) == 2)

def return_a_func(arg1, arg2):
  def new_func():
    print('arg1 was {}'.format(arg1))
    print('arg2 was {}'.format(arg2))
  return new_func
    
my_func = return_a_func(2, 17)

print(my_func.__closure__ is not None)
print(len(my_func.__closure__) == 2)

# Get the values of the variables in the closure
closure_values = [
  my_func.__closure__[i].cell_contents for i in range(2)
]
print(closure_values == [2, 17])

Closures keep your values safe

You are still helping your niece understand closures. You have written the function get_new_func() that returns a nested function. The nested function call_func() calls whatever function was passed to get_new_func(). You’ve also written my_special_function() which simply prints a message that states that you are executing my_special_function().

You want to show your niece that no matter what you do to my_special_function() after passing it to get_new_func(), the new function still mimics the behavior of the original my_special_function() because it is in the new function’s closure.

Show that you still get the original message even if you redefine my_special_function() to only print “hello”.

Show that even if you delete my_special_function(), you can still call new_func() without any problems.

Show that you still get the original message even if you overwrite my_special_function() with the new function.

def my_special_function():
  print('You are running my_special_function()')
  
def get_new_func(func):
  def call_func():
    func()
  return call_func

new_func = get_new_func(my_special_function)

# Redefine my_special_function() to just print "hello"
def my_special_function():
  print("hello")

new_func()

def my_special_function():
  print('You are running my_special_function()')
  
def get_new_func(func):
  def call_func():
    func()
  return call_func

new_func = get_new_func(my_special_function)

# Delete my_special_function()
del(my_special_function)

new_func()

def my_special_function():
  print('You are running my_special_function()')
  
def get_new_func(func):
  def call_func():
    func()
  return call_func

# Overwrite `my_special_function` with the new function
my_special_function = get_new_func(my_special_function)

my_special_function()

Decorators

Using decorator syntax

You have written a decorator called print_args that prints out all of the arguments and their values any time a function that it is decorating gets called.

Decorate my_function() with the print_args() decorator by redefining the my_function variable.

Decorate my_function() with the print_args() decorator using decorator syntax.

# edited/added
def print_args(func):
    sig = inspect.signature(func)
    def wrapper(*args, **kwargs):
        bound = sig.bind(*args, **kwargs).arguments
        str_args = ', '.join(['{}={}'.format(k, v) for k, v in bound.items()])
        print('{} was called with {}'.format(func.__name__, str_args))
        return func(*args, **kwargs)
    return wrapper

def my_function(a, b, c):
  print(a + b + c)

# Decorate my_function() with the print_args() decorator
my_function = print_args(my_function)

my_function(1, 2, 3)

# Decorate my_function() with the print_args() decorator
@print_args
def my_function(a, b, c):
  print(a + b + c)

my_function(1, 2, 3)

Defining a decorator

Your buddy has been working on a decorator that prints a “before” message before the decorated function is called and prints an “after” message after the decorated function is called. They are having trouble remembering how wrapping the decorated function is supposed to work. Help them out by finishing their print_before_and_after() decorator.

• Call the function being decorated and pass it the positional arguments *args.
• Return the new decorated function.

def print_before_and_after(func):
  def wrapper(*args):
    print('Before {}'.format(func.__name__))
    # Call the function being decorated with *args
    func(*args)
    print('After {}'.format(func.__name__))
  # Return the nested function
  return wrapper

@print_before_and_after
def multiply(a, b):
  print(a * b)

multiply(5, 10)

More on Decorators

Now that you understand how decorators work under the hood, this chapter gives you a bunch of real-world examples of when and how you would write decorators in your own code. You will also learn advanced decorator concepts like how to preserve the metadata of your decorated functions and how to write decorators that take arguments.

Real-world examples

Print the return type

You are debugging a package that you’ve been working on with your friends. Something weird is happening with the data being returned from one of your functions, but you’re not even sure which function is causing the trouble. You know that sometimes bugs can sneak into your code when you are expecting a function to return one thing, and it returns something different. For instance, if you expect a function to return a numpy array, but it returns a list, you can get unexpected behavior. To ensure this is not what is causing the trouble, you decide to write a decorator, print_return_type(), that will print out the type of the variable that gets returned from every call of any function it is decorating.

• Create a nested function, wrapper(), that will become the new decorated function.
• Call the function being decorated.
• Return the new decorated function.

def print_return_type(func):
  # Define wrapper(), the decorated function
  def wrapper(*args, **kwargs):
    # Call the function being decorated
    result = func(*args, **kwargs)
    print('{}() returned type {}'.format(
      func.__name__, type(result)
    ))
    return result
  # Return the decorated function
  return wrapper
  
@print_return_type
def foo(value):
  return value
  
print(foo(42))
print(foo([1, 2, 3]))
print(foo({'a': 42}))

Counter

You’re working on a new web app, and you are curious about how many times each of the functions in it gets called. So you decide to write a decorator that adds a counter to each function that you decorate. You could use this information in the future to determine whether there are sections of code that you could remove because they are no longer being used by the app.

• Call the function being decorated and return the result.
• Return the new decorated function.
• Decorate foo() with the counter() decorator.

def counter(func):
  def wrapper(*args, **kwargs):
    wrapper.count += 1
    # Call the function being decorated and return the result
    return func(*args, **kwargs)
  wrapper.count = 0
  # Return the new decorated function
  return wrapper

# Decorate foo() with the counter() decorator
@counter
def foo():
  print('calling foo()')
  
foo()
foo()

print('foo() was called {} times.'.format(foo.count))

Decorators and metadata

Preserving docstrings when decorating functions

Your friend has come to you with a problem. They’ve written some nifty decorators and added them to the functions in the open-source library they’ve been working on. However, they were running some tests and discovered that all of the docstrings have mysteriously disappeared from their decorated functions. Show your friend how to preserve docstrings and other metadata when writing decorators.

• Decorate print_sum() with the add_hello() decorator to replicate the issue that your friend saw - that the docstring disappears.

• To show your friend that they are printing the wrapper() function’s docstring, not the print_sum() docstring, add the following docstring to wrapper():

"""Print 'hello' and then call the decorated function."""

• Import a function that will allow you to add the metadata from print_sum() to the decorated version of print_sum().

• Finally, decorate wrapper() so that the metadata from func() is preserved in the new decorated function.

def add_hello(func):
  def wrapper(*args, **kwargs):
    print('Hello')
    return func(*args, **kwargs)
  return wrapper

# Decorate print_sum() with the add_hello() decorator
@add_hello
def print_sum(a, b):
  """Adds two numbers and prints the sum"""
  print(a + b)
  
print_sum(10, 20)
print_sum_docstring = print_sum.__doc__
print(print_sum_docstring)

def add_hello(func):
  # Add a docstring to wrapper
  def wrapper(*args, **kwargs):
    """Print 'hello' and then call the decorated function."""
    print('Hello')
    return func(*args, **kwargs)
  return wrapper

@add_hello
def print_sum(a, b):
  """Adds two numbers and prints the sum"""
  print(a + b)
  
print_sum(10, 20)
print_sum_docstring = print_sum.__doc__
print(print_sum_docstring)

# Import the function you need to fix the problem
from functools import wraps

def add_hello(func):
  def wrapper(*args, **kwargs):
    """Print 'hello' and then call the decorated function."""
    print('Hello')
    return func(*args, **kwargs)
  return wrapper
  
@add_hello
def print_sum(a, b):
  """Adds two numbers and prints the sum"""
  print(a + b)
  
print_sum(10, 20)
print_sum_docstring = print_sum.__doc__
print(print_sum_docstring)

from functools import wraps

def add_hello(func):
  # Decorate wrapper() so that it keeps func()'s metadata
  @wraps(func)
  def wrapper(*args, **kwargs):
    """Print 'hello' and then call the decorated function."""
    print('Hello')
    return func(*args, **kwargs)
  return wrapper
  
@add_hello
def print_sum(a, b):
  """Adds two numbers and prints the sum"""
  print(a + b)
  
print_sum(10, 20)
print_sum_docstring = print_sum.__doc__
print(print_sum_docstring)

Measuring decorator overhead

Your boss wrote a decorator called check_everything() that they think is amazing, and they are insisting you use it on your function. However, you’ve noticed that when you use it to decorate your functions, it makes them run much slower. You need to convince your boss that the decorator is adding too much processing time to your function. To do this, you are going to measure how long the decorated function takes to run and compare it to how long the undecorated function would have taken to run. This is the decorator in question:

def check_everything(func):
  @wraps(func)
  def wrapper(*args, **kwargs):
    check_inputs(*args, **kwargs)
    result = func(*args, **kwargs)
    check_outputs(result)
    return result
  return wrapper

• Call the original function instead of the decorated version by using an attribute of the function that the wraps() statement in your boss’s decorator added to the decorated function.

# edited/added
def check_inputs(a, *args, **kwargs):
  for value in a:
    time.sleep(0.01)
  print('Finished checking inputs')
def check_outputs(a, *args, **kwargs):
  for value in a:
    time.sleep(0.01)
  print('Finished checking outputs')
def check_everything(func):
  @wraps(func)
  def wrapper(*args, **kwargs):
    check_inputs(*args, **kwargs)
    result = func(*args, **kwargs)
    check_outputs(result)
    return result
  return wrapper

@check_everything
def duplicate(my_list):
  """Return a new list that repeats the input twice"""
  return my_list + my_list

t_start = time.time()
duplicated_list = duplicate(list(range(50)))
t_end = time.time()
decorated_time = t_end - t_start

t_start = time.time()
# Call the original function instead of the decorated one
duplicated_list = duplicate.__wrapped__(list(range(50)))
t_end = time.time()
undecorated_time = t_end - t_start

print('Decorated time: {:.5f}s'.format(decorated_time))
print('Undecorated time: {:.5f}s'.format(undecorated_time))

Decorators that take arguments

Run_n_times()

In the video exercise, I showed you an example of a decorator that takes an argument: run_n_times(). The code for that decorator is repeated below to remind you how it works. Practice different ways of applying the decorator to the function print_sum(). Then I’ll show you a funny prank you can play on your co-workers.

def run_n_times(n):
  """Define and return a decorator"""
  def decorator(func):
    def wrapper(*args, **kwargs):
      for i in range(n):
        func(*args, **kwargs)
    return wrapper
  return decorator

Add the run_n_times() decorator to print_sum() using decorator syntax so that print_sum() runs 10 times.

Use run_n_times() to create a decorator run_five_times() that will run any function five times.

Here’s the prank: use run_n_times() to modify the built-in print() function so that it always prints 20 times!

# edited/added
def run_n_times(n):
  """Define and return a decorator"""
  def decorator(func):
    def wrapper(*args, **kwargs):
      for i in range(n):
        func(*args, **kwargs)
    return wrapper
  return decorator

# Make print_sum() run 10 times with the run_n_times() decorator
@run_n_times(10)
def print_sum(a, b):
  print(a + b)
  
print_sum(15, 20)

# Use run_n_times() to create the run_five_times() decorator
run_five_times = run_n_times(5)

@run_five_times
def print_sum(a, b):
  print(a + b)
  
print_sum(4, 100)

# Modify the print() function to always run 20 times
print = run_n_times(20)(print)

print('What is happening?!?!')

HTML Generator

You are writing a script that generates HTML for a webpage on the fly. So far, you have written two decorators that will add bold or italics tags to any function that returns a string. You notice, however, that these two decorators look very similar. Instead of writing a bunch of other similar looking decorators, you want to create one decorator, html(), that can take any pair of opening and closing tags.

def bold(func):
  @wraps(func)
  def wrapper(*args, **kwargs):
    msg = func(*args, **kwargs)
    return '<b>{}</b>'.format(msg)
  return wrapper

def italics(func):
  @wraps(func)
  def wrapper(*args, **kwargs):
    msg = func(*args, **kwargs)
    return '<i>{}</i>'.format(msg)
  return wrapper

Return the decorator and the decorated function from the correct places in the new html() decorator.

Use the html() decorator to wrap the return value of hello() in the strings <b> and </b> (the HTML tags that mean “bold”).

Use html() to wrap the return value of goodbye() in the strings <i> and </i> (the HTML tags that mean “italics”).

Use html() to wrap hello_goodbye() in a DIV, which is done by adding the strings <div> and </div> tags around a string.

def html(open_tag, close_tag):
  def decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
      msg = func(*args, **kwargs)
      return '{}{}{}'.format(open_tag, msg, close_tag)
    # Return the decorated function
    return wrapper
  # Return the decorator
  return decorator

# Make hello() return bolded text
@html('<b>', '</b>')
def hello(name):
  return 'Hello {}!'.format(name)

print(hello('Alice'))

# Make goodbye() return italicized text
@html('<i>', '</i>')
def goodbye(name):
  return 'Goodbye {}.'.format(name)
  
print(goodbye('Alice'))

# Wrap the result of hello_goodbye() in <div> and </div>
@html('<div>', '</div>')
def hello_goodbye(name):
  return '\n{}\n{}\n'.format(hello(name), goodbye(name))
  
print(hello_goodbye('Alice'))

Timeout(): a real world example

Tag your functions

Tagging something means that you have given that thing one or more strings that act as labels. For instance, we often tag emails or photos so that we can search for them later. You’ve decided to write a decorator that will let you tag your functions with an arbitrary list of tags. You could use these tags for many things:

• Adding information about who has worked on the function, so a user can look up who to ask if they run into trouble using it.
• Labeling functions as “experimental” so that users know that the inputs and outputs might change in the future.
• Marking any functions that you plan to remove in a future version of the code.
• Etc.

• Define a new decorator, named decorator(), to return.
• Ensure the decorated function keeps its metadata.
• Call the function being decorated and return the result.
• Return the new decorator.

def tag(*tags):
  # Define a new decorator, named "decorator", to return
  def decorator(func):
    # Ensure the decorated function keeps its metadata
    @wraps(func)
    def wrapper(*args, **kwargs):
      # Call the function being decorated and return the result
      return func(*args, **kwargs)
    wrapper.tags = tags
    return wrapper
  # Return the new decorator
  return decorator

@tag('test', 'this is a tag')
def foo():
  pass

print(foo.tags)

Check the return type

Python’s flexibility around data types is usually cited as one of the benefits of the language. It can sometimes cause problems though if incorrect data types go unnoticed. You’ve decided that in order to ensure your code is doing exactly what you want it to do, you will explicitly check the return types in all of your functions and make sure they’re returning what you expect. To do that, you are going to create a decorator that checks if the return type of the decorated function is correct.

Note: assert is a keyword that you can use to test whether something is true. If you type assert condition and condition is True, this function doesn’t do anything. If condition is False, this function raises an error. The type of error that it raises is called an AssertionError.

• Start by completing the returns_dict() decorator so that it raises an AssertionError if the return type of the decorated function is not a dictionary.

• Now complete the returns() decorator, which takes the expected return type as an argument.

def returns_dict(func):
  # Complete the returns_dict() decorator
  def wrapper(*args, **kwargs):
    result = func(*args, **kwargs)
    assert type(result) == dict
    return result
  return wrapper

@returns_dict
def foo(value):
  return value

try:
  print(foo([1,2,3]))
except AssertionError:
  print('foo() did not return a dict!')
  
def returns(return_type):
  # Write a decorator that raises an AssertionError if the
  # decorated function returns a value that is not return_type
  def decorator(func):
    def wrapper(*args, **kwargs):
      result = func(*args, **kwargs)
      assert type(result) == return_type
      return result
    return wrapper
  return decorator
  
@returns(dict)
def foo(value):
  return value

try:
  print(foo([1,2,3]))
except AssertionError:
  print('foo() did not return a dict!')

Great job!

Great job!

Great job! You’ve covered a lot in this course.

Chapter 1 - Best Practices

In the first chapter, you learned how to make high-quality functions by giving them docstrings and by making sure that they only do one thing. Remembering the acronym DRY, or “Don’t Repeat Yourself”, helped you notice when you needed to pull part of your code into a reusable function. You also learned about how Python passes arguments to functions and the difference between mutable and immutable variables.

Chapter 2 - Context Managers

In the chapter on context managers, you learned how to use the keyword “with” to enter and then exit a context. You also learned how to write your own context managers by using the contextmanager() decorator.

Chapter 3 - Decorators

You also spent a lot of time in this course understanding decorators: how they work, how to use them, and how to write decorators of your own.

Chapter 4 - More on Decorators

Finally, in chapter 4, you learned how to use functools.wraps() to make sure your decorated functions maintain their metadata.

Chapter 4 - More on Decorators

And you learned how to write decorators that take arguments.

Thank you!

It has been an honor to spend this time with you. I wanted to let you know that I’ll be donating a portion of the proceeds from this course to WiMLDS, a non-profit that helps support and promote women in the fields of machine learning and data science. I’ll be looking forward to hearing about all of the amazing things you do with Python in the future!