{% include toc.html %}
This lesson takes the frequency pairs created in Counting Frequencies and outputs them to an HTML file.
Here you will learn how to output data as an HTML file using Python. You will also learn about string formatting. The final result is an HTML file that shows the keywords found in the original source in order of descending frequency, along with the number of times that each keyword appears.
obo.pyIf you do not have these files from the previous lesson, you can download programming-historian-6, a zip file from the previous lesson
In the previous lesson, you learned how to embed the message "Hello World!" in HTML tags, write the result to a file and open it automatically in the browser. A program that puts formatting codes around something so that it can be used by another program is sometimes called a wrapper. What we're going to do now is develop an HTML wrapper for the output of our code that computes word frequencies. We're also going to add some helpful, dynamic metadata to supplement the frequency data collected in Counting Frequencies.
The distinction between data and metadata is crucial to information science. Metadata are data about data. This concept should already be very familiar to you, even if you haven't heard the term before. Consider a traditional book. If we take the text of the book to be the data, there are a number of other characteristics which are associated with that text, but which may or may not be explicitly printed in the book. The title of the work, the author, the publisher, and the place and date of publication are metadata that are typically printed in the work. The place and date of writing, the name of the copy editor, Library of Congress cataloging data, and the name of the font used to typeset the book are sometimes printed in it. The person who purchased a particular copy may or may not write their name in the book. If the book belongs in the collection of a library, that library will keep additional metadata, only some of which will be physically attached to the book. The record of borrowing, for example, is usually kept in some kind of database and linked to the book by a unique identifier. Libraries, archives and museums all have elaborate systems to generate and keep track of metadata.
When you're working with digital data, it is a good idea to incorporate metadata into your own files whenever possible. We will now develop a few basic strategies for making our data files self-documenting. In our wrapper, we want to include dynamic information about the file, such as the time and date it was created, as well as an HTML title that is relevant to the file. In this case we could just give it a name ourselves, but when we start working with multiple files, automatically creating self-documenting files will save a lot of time, so we’ll practice now. And for that, we'll have to learn to take advantage of a few more powerful string formatting options.
Python includes a special formatting operator that allows you to insert one string into another one. It is represented by a percent sign followed by an "s". Open a Python shell and try the following examples.
frame = 'This fruit is a %s'
print(frame)
-> This fruit is a %s
print(frame % 'banana')
-> This fruit is a banana
print(frame % 'pear')
-> This fruit is a pear
There is also a form which allows you to interpolate a list of strings into another one.
frame2 = 'These are %s, those are %s'
print(frame2)
-> These are %s, those are %s
print(frame2 % ('bananas', 'pears'))
-> These are bananas, those are pears
In these examples, a %s in one string indicates that another string is
going to be embedded at that point. There are a range of other string
formatting codes, most of which allow you to embed numbers in strings in
various formats, like %i for integer (eg. 1, 2, 3), %f for
floating-point decimal (eg. 3.023, 4.59, 1.0), and so on. Using this
method we can input information that is unique to the file.
Let's bundle some of the code that we've already written into functions.
One of these will take a URL and return a string of lowercase text from
the web page. Copy this code into the obo.py module.
# Given a URL, return string of lowercase text from page.
def webPageToText(url):
import urllib.request, urllib.error, urllib.parse
response = urllib.request.urlopen(url)
html = response.read().decode('UTF-8')
text = stripTags(html).lower()
return text
We're also going to want a function that takes a string of any sort and
makes it the body of an HTML file which is opened automatically in
Firefox. This function should include some basic metadata, like the time
and date that it was created and the name of the program that created
it. Study the following code carefully, then copy it into the obo.py
module.
If you are using a Mac, make sure you include the proper file path in the filename variable on the 2nd last line to reflect where you're saving your files.
# Given name of calling program, a url and a string to wrap,
# output string in html body with basic metadata and open in Firefox tab.
def wrapStringInHTMLMac(program, url, body):
import datetime
from webbrowser import open_new_tab
now = datetime.datetime.today().strftime("%Y%m%d-%H%M%S")
filename = program + '.html'
f = open(filename,'w')
wrapper = """<html>
<head>
<title>%s output - %s</title>
</head>
<body><p>URL: <a href=\"%s\">%s</a></p><p>%s</p></body>
</html>"""
whole = wrapper % (program, now, url, url, body)
f.write(whole)
f.close()
#Change the filepath variable below to match the location of your directory
filename = 'file:///Users/username/Desktop/programming-historian/' + filename
open_new_tab(filename)
# Given name of calling program, a url and a string to wrap,
# output string in html body with basic metadata
# and open in Firefox tab.
def wrapStringInHTMLWindows(program, url, body):
import datetime
from webbrowser import open_new_tab
now = datetime.datetime.today().strftime("%Y%m%d-%H%M%S")
filename = program + '.html'
f = open(filename,'w')
wrapper = """<html>
<head>
<title>%s output - %s</title>
</head>
<body><p>URL: <a href=\"%s\">%s</a></p><p>%s</p></body>
</html>"""
whole = wrapper % (program, now, url, url, body)
f.write(whole)
f.close()
open_new_tab(filename)
***
Note that this function makes use of the string formatting operator about which we just learned. If you are still having trouble with this idea, take a look at the HTML file that opened in your new Firefox tab and you should see how this worked. If you're still stuck, take a look at the
URL: http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33
in the HTML file and trace back how the program knew to put the URL value there.
The function also calls the Python datetime library to determine the
current time and date. Like the string formatting operator %s, this
library uses the % as replacements for values. In this case, the %Y %m %d %H %M %S represents year, month, date, hour, minute and second
respectively. Unlike the %s, the program will determine the value of
these variables for you using your computer's clock. It is important to
recognize this difference.
This date metadata, along with the name of the program that called the
function, is stored in the HTML title tag. The HTML file that is created
has the same name as the Python program that creates it, but with a
.html extension rather than a .py one.
Now we can create another version of our program to compute frequencies.
Instead of sending its output to a text file or an output window, it
sends the output to an HTML file which is opened in a new Firefox tab.
From there, the program's output can be added easily as bibliographic
entries to Zotero. Type or copy the following code into your text
editor, save it as html-to-freq-3.py and execute it, to confirm that
it works as expected.
Use either obo.wrapStringInHTMLMac() or obo.wrapStringInHTMLWindows() as appropriate for your system.
# html-to-freq-3.py
import obo
# create sorted dictionary of word-frequency pairs
url = 'http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33'
text = obo.webPageToText(url)
fullwordlist = obo.stripNonAlphaNum(text)
wordlist = obo.removeStopwords(fullwordlist, obo.stopwords)
dictionary = obo.wordListToFreqDict(wordlist)
sorteddict = obo.sortFreqDict(dictionary)
# compile dictionary into string and wrap with HTML
outstring = ""
for s in sorteddict:
outstring += str(s)
outstring += "<br />"
obo.wrapStringInHTMLMac("html-to-freq-3", url, outstring)
Note that we interspersed our word-frequency pairs with the HTML break
tag <br\>, which acts as a newline. If all went well, you should see
the same word frequencies that you computed in the last section, this
time in your browser window.
To follow along with future lessons it is important that you have the
right files and programs in your "programming-historian" directory. At
the end of each chapter you can download the "programming-historian" zip
file to make sure you have the correct code. If you are following along
with the Mac / Linux version you may have to open the obo.py file and
change "file:///Users/username/Desktop/programming-historian/" to the
path to the directory on your own computer.