Lab Scrape Employees
Update:
It appears if there are too many requests to the Ridgewater URL with the same User-Agent, then anti-bot measures kick in and you will get a 403 Forbidden error.
Websites that describe how to overcome this:
How to customize Your User-Agent with Python Requests
How to Effectively Use User Agents for Web Scraping
The fix is to use your own User-Agent.
Go to Chrome tools, Console, and type navigator.userAgent at the > (prompt)
You will get a response with the User-Agent of your machine. Use it in your Python code.
Here’s how you can check and get the user agent using your browser’s console:
Open the developer tools in Google Chrome, Microsoft Edge, Mozilla Firefox, Safari or any other browser. You can use F12 or Ctrl+Shift+I on Windows/Linux, or Cmd+Option(⌘)+I on macOS. Switch to the Console tab.
Type navigator.userAgent in the console and press Enter (or Ctrl+Enter). The console will return a string which is your browser’s user agent.
This lab is to be done on your own.
- Create a python program named scrape_employees.py that does the following:
- Comments at the top of the module with your name, date, and description. 1 point.
- Put this website url into a variable: http://ridgewater.edu/contact-us/staff-directory/ 1 points.
- Set up the context that is needed to read https pages (secure pages). 2 points.
- Set up the headers dictionary. 1 point.
- Use urllib.request.Request to request a website. 1point.
- Use urllib.request.urlopen to open the website 1 point.
- Use
r'<td>\s*<strong>([^,<]+),\s*([^<]+)</strong>\s*</td>\s*<td>([^<]+)</td>\s*<td>([^<]*)</td>'
- as the regular expression to find all email addresses. 2 points.
- Open a new file named 'employees.txt'. 1 point.
- Iterate through the regular expression matches to extract the individual employee last name, first name, and their job title. 2 point.
- Write the employees and their job title to the screen. 2 point
- Write the employees and their job title to the text file in CSV format. Notice the job title is enclosed by quotes. 5 points
- Close the text file. 1point
Submit all these files to the D2L dropbox:
- The scrape_employees.py code file.
- The employees.txt file.
- A screenshot in .jpg or .png format showing the start of the run as output.
- A screenshot in .jpg or .png format showing the end of the run as output.
- A screenshot showing your employees.txt file.
The image below shows an example start run of the program.

The image below shows an example end run of the program.

The image below shows an example employees.txt file.