• Web programming is a very broad topic
• We're going to break it up into five parts:
  1. HTTP basics
  2. Client programming:
     • HTML Forms
     • Javascript
  3. Server as a client:
     • Downloading and analyzing content from the web
  4. Server programming:
     • CGI
  5. Putting it all together:
     • AJAX
• Our focus is on web programming
• See the Software Carpentry lecture for a more general discussion of Internet programming

• The ability to fetch and parse content from the web is an essential part of modern bioinformatics
  • For example, using NCBI eutils to pull data from Entrez or PubMed.
• In this context, your server program becomes a client to someone else's server

• Opening sockets, constructing HTTP requests, and parsing responses is tedious
  • So most languages provide libraries to do the work for you
  • In Python, that library is called urllib
• urllib.urlopen(URL) does what your browser would do if you gave it the URL
  • Parse it to figure out what server to connect to
  • Connect to that server
  • Send an HTTP request
  • Returns an object that looks like a file, from which to read response data

• Read a page the easy way

  import urllib
  
  instream = urllib.urlopen("http://www.third-bit.com/greeting.html")
  lines = instream.readlines()
  instream.close()
  for line in lines:
      print line,
  

• Note: readlines wouldn't do the right thing if the thing being read was an image
  • Might try to convert “line endings”
  • Use read to grab the bytes in that case

• A web spider is a program that can explore the web on its own
  • Fetch a page, extract all the external links, visit those pages…
  • That, a search engine, and a few billion dollars, and you're Google

$ python spider.py http://www.google.ca
http://groups.google.ca/grphp?hl=en&tab=wg&ie=UTF-8
http://news.google.ca/nwshp?hl=en&tab=wn&ie=UTF-8
http://scholar.google.com/schhp?hl=en&tab=ws&ie=UTF-8
http://www.google.ca/fr
			

import sys, urllib, re

url = sys.argv[1]
instream = urllib.urlopen(url)
page = instream.read()
instream.close()

links = re.findall(r'href=\"[^\"]+\"', page)
temp = set()
for x in links:
    x = x[6:-1]    # strip off 'href="' and '"'
    if x.startswith('http://'):
        temp.add(x)
links = list(temp)
links.sort()
for x in links:
    print x

• Sometimes want to provide extra information as part of a URL
  • Example: when searching on Google, have to specify what the search terms are
• Could do this as part of the URL
  • Amazon puts ISBNs in URLs
• More flexible to add parameters to the URL
  • http://www.google.ca?q=Python searches for pages related to Python
  • "?" separates the parameters from the rest of the URL
  • If there are multiple parameters, they are separated from each other by "&"
    • E.g., http://www.google.ca/search?q=Python&client=firefox

• What if you want to include "?" or "&" in a parameter?
  • Same problem (and solution) as including a quote in a string, or <> in XML
• URL encode special characters using "%" followed by a 2-digit hexadecimal code
  • And replace spaces with "+"

  Character	Encoding
  "#"	%23
  "$"	%24
  "%"	%25
  "&"	%26
  "+"	%2B
  ","	%2C
  "/"	%2F
  ":"	%3A
  ";"	%3B
  "="	%3D
  "?"	%3F
  "@"	%40
  Table 3: URL Encoding

• To search Google for “grade = A+”, use http://www.google.ca/search?q=grade+%3D+A%2B
• urllib has functions to make this easy
  • urllib.quote(str) replaces special characters in str with escape sequences
  • urllib.unquote(str) replaces escape sequences with characters
  • urllib.urlencode(params) takes a dictionary and constructs the entire query parameter string

  import urllib
  print urllib.urlencode({'surname' : 'Von Neumann', 'forename' : 'John'})
  

  surname=Von+Neumann&forename=John
  

• Suppose you want to write a script that actually does search Google
  • Construct a URL: easy
  • Send it and read the response: no problem
  • Parse the response: there's a lot of junk on the page…
• Many first-generation web applications relied on screen scraping
  • “Parse” the HTML with regular expressions
• Hard to get right if the page layout is complex
  • And whenever the layout changes, the application breaks
• Now, there's a better way...

Figure 5: Web Services

• Modern web services separate data from presentation
  • When a client sends a request, it indicates that it wants machine-readable XML, rather than human-readable HTML
    • Much easier to parse
    • Much less likely to change over time
• Many web services use the Simple Object Access Protocol (SOAP) standard
  • Despite its name, it's anything but simple
  • Luckily, there are libraries to hide the details for most widely-used web services

• Amazon has defined an API for web services
  • You need to get a license key in order to use it
    • They're free
    • But they allow Amazon to throttle requests to one per second per client
• PyAmazon turns parameters into URL, and converts the XML reply into Python objects

import sys, amazon

# Format multiple authors' names nicely.
def prettyName(arg):
    if type(arg) in (list, tuple):
        arg = ', '.join(arg[:-1]) + ' and ' + arg[-1]
    return arg

if __name__ == '__main__':

    # Get information.
    key, asin = sys.argv[1], sys.argv[2]
    amazon.setLicense(key)
    items = amazon.searchByASIN(asin)

    # Handle errors.
    if not items:
        print 'Nothing found for', asin
    if len(items) > 1:
        print len(items), 'items found for', asin

    # Display information.
    item = items[0]
    productName = item.ProductName
    ourPrice = item.OurPrice
    authors = prettyName(item.Authors.Author)
    print '%s: %s (%s)' % (authors, productName, ourPrice)

	$ python findbook.py 123ABCDEFGHIJKL4MN56 0974514071 Greg Wilson: Data Crunching : Solve Everyday Problems Using Java, 
Python, and more. ($18.87)

• Note: much more code devoted to creating human-readable output than to getting the information
• Note 2: this was based on version 3.0 of the amazon web service interface, which has since been deprecated

• Users want to make the web do different things
  • How to let them write programs that handle HTTP requests?
• Option #1: Require them to write socket-level code
  • Complicated and error-prone
  • Can only have one program listening to a socket at a time
• Option #2: have the web server accept the HTTP request, and then run the user's code
  • Recompiling the web server every time someone wants to add functionality would be a pain
  • So define a protocol that lets web servers run other programs

• The Common Gateway Interface (CGI) protocol specifies:
  • How a web server passes information to a program
  • How that program passes information back to the web server
• CGI does not specify:
  • A particular language
    • You can use Fortran, the shell, C, Java, Perl, Python…
  • How the web server figures out what program to run
    • Each web server has its own rules
    • We'll (briefly) talk about Apache's

• Web server runs the CGI by creating a new process


Figure 5: CGI Data Processing Cycle

• Web server passes some information to the CGI process through environment variables

  Name	Purpose	Example
  REQUEST_METHOD	What kind of HTTP request is being handled	GET or POST
  SCRIPT_NAME	The path to the script that's executing	/cgi-bin/post_photo.py
  QUERY_STRING	The query parameters following "?" in the URL	name=mydog.jpg&expires=never
  CONTENT_TYPE	The type of any extra data being sent with the request	img/jpeg
  CONTENT_LENGTH	How much extra data is being sent with the request (in bytes)	17290
  Table 4: Important CGI Environment Variables

• The web server may also send CONTENT_LENGTH bytes to the CGI on standard input
  • E.g., when a file is being uploaded

• The CGI program sends data back to the web server by printing it to standard output
• The web server then forwards this directly to the client
  • Which means that the CGI program is responsible for creating headers
• Note: none of this works unless the web server has been configured to run the CGI
  • By default, modern servers won't do this unless they're told they can

• Clients and servers need a way to specify data types to each other
  • Remember, bytes are just bytes: the browser doesn't magically know how to interpret them
• Multipurpose Internet Mail Extensions standard specifies how to do this
  • Organizes data types into families, and provides a two-part name for each type
  • Use the "Content-Type" header to specify the MIME type of the data being sent

Family	Specific Type	Describes
Text	text/html	Web pages
Image	image/jpeg	JPEG-format image
Audio	audio/x-mp3	MP3 audio file
Video	video/quicktime	Apple Quicktime video format
Application-specific data	application/pdf	Adobe PDF document
Table 5: Example Mime Types

• Simplest possible CGI pays no attention to query parameters or extra data
  • Just prints HTML to standard output, to be relayed to the client
  • Along with a Content-Type header to tell the client to expect HTML…
  • …and a blank line to separate the headers from the data

#!/usr/bin/env python

# Headers and an extra blank line
print 'Content-type: text/html'
print

# Body
print '<html><body><p>Hello, CGI!</p></body></html>'

• Invoke it by going to http://plato.cgl.ucsf.edu/bmi280/cgi-bin/hello_cgi.py
  • By convention, CGI programs are put in a cgi-bin directory
• Browser displays the simple HTML page generated by the program

Figure 6: Basic CGI Output

• But the whole point of CGI is to generate content dynamically
  • E.g., show a list of environment variables and their values
• You'll use this frequently when debugging…

#!/usr/bin/env python

import os, cgi

# Headers and an extra blank line
print 'Content-type: text/html'
print

# Body
print '<html><body>'
keys = os.environ.keys()
keys.sort()
for k in keys:
    print '<p>%s: %s</p>' % (cgi.escape(k), cgi.escape(os.environ[k]))
print '</body></html>'

Figure 7: Environment Variable Output

• Each <input/> element has a name attribute
  • These become the names of the parameters that the client sends to the server
  • The input elements' values are the parameters' values
• Submitting the form shown above with default values produces:
  • os.environ['REQUEST_METHOD']: "POST"
  • os.environ['SCRIPT_NAME']: "/cgi-bin/simple_form.py"
  • os.environ['CONTENT_TYPE']: "application/x-www-form-urlencoded"
  • os.environ['REQUEST_LENGTH']: "80"
  • Standard input: sequence=GATTACA&search_type=Similarity+match&program=FROG-11&program=Bayes-Hart

• Could handle form data directly
  • Read and parse environment variables
  • Read extra data from standard input
• But the mechanics are the same each time, so use Python's cgi module instead
  • Defines a dictionary-like object called FieldStorage
    • Keys are parameter names
    • Values are either strings (if there's a single value assocatied with the parameter) or lists (if there are many)
• When a FieldStorage object is created, it reads and stores information contained in the URL and environment
  • Which means that a CGI program should only ever create one
• Program can read extra data from sys.stdin

• Example: show the parameters send to a script

  #!/usr/bin/env python
  import cgi
  
  print 'Content-type: text/html'
  print
  print '<html><body>'
  form = cgi.FieldStorage()
  for key in form.keys():
      value = form.getvalue(key)
      if isinstance(value, list):
          value = '[' + ', '.join(value) + ']'
      print '<p>%s: %s</p>' % (cgi.escape(key), cgi.escape(value))
  print '</body></html>'
  

  URL	Value of a	Value of b
  http://www.third-bit.com/swc/show_params.py?a=0	"0"	None
  http://www.third-bit.com/swc/show_params.py?a=0&b=hello	"0"	"hello"
  http://www.third-bit.com/swc/show_params.py?a=0&b=hello&a=22	[0, 22]	"hello"
  Table 6: Example Parameter Values

• During development, add import cgitb; cgitb.enable() to the top of the program
  • cgitb is the CGI traceback module
  • When enabled, it will create a web page showing a stack trace when something goes wrong in your script
• Testing whether a FieldStorage value is a string or a list is tedious
  • In almost all cases, you'll know whether to expect one value or many
  • Use FieldStorage.getfirst(name) to get the unique value
    • Returns the first, if there are many
  • FieldStorage.getlist(name) always returns a list of values
    • Empty list if there's no data associated with name
    • If there's only one value, get a single-item list

• Often want to change the data a server is managing, as well as read it
  • Update a description of an experiment, change your preferred email address, etc.
• The industrial-strength solution is to use a three-tier architecture
  • 

    Figure 9: Three Tier Architecture

  • CGI program stuffs parameters from HTTP requests into SQL queries
  • Runs the queries
  • Translates results into HTML to send back to the client

• Simple programs can often get away with using files
  • The CGI program re-reads the file each time it processes a request
  • And re-writes it if there have been any updates
• Example: append messages to a web page
  • Old messages are saved in a file, one per line

  Hi, is anyone reading this site?
  I was wondering the same thing.
  I wasn't sure if we were supposed to post here.
  Good point.  Is there way to delete messages?
  

• Script checks the incoming parameters to decide what to do
  • If newmessage is there, append it, and display results
  • If newmessage isn't there, someone's visiting the page, rather than submitting the form

  # Get existing messages.
  infile = open('messages.txt', 'r')
  lines = [x.rstrip() for x in infile.readlines()]
  infile.close()
  
  # Add more data?
  form = cgi.FieldStorage()
  if form.has_key('newmessage'):
      lines.append(form.getfirst('newmessage'))
      outfile = open('messages.txt', 'w')
      for line in lines:
          print >> outfile, line
      outfile.close()

• AJAX (Asynchronous Javascript And XML) provides:
  • Increased usability (on-page server interaction)
  • Client-side state
  • Enhanced user experience (possibly)
• Basic AJAX idea:
  1. Javascript in browser sends a request to server using XMLHttpRequest()
  2. Server CGI processes request and sends back response, usually as an XML document
  3. Javascript in browser receives response, parses the XML and (using DOM) extracts the information
  4. Results are presented to the user, or used to modify the interface in some way

Method	Description
abort()	Cancels the current request
getAllResponseHeaders()	Returns the complete set of http headers as a string
getResponseHeader("headername")	Returns the value of the specified http header
open("method","URL",async,"username","password")	Specifies the method, URL, and other optional attributes of a request The method parameter can have a value of "GET", "POST", or "PUT" (use "GET" when requesting data and use "POST" when sending data (especially if the length of the data is greater than 512 bytes. The URL parameter may be either a relative or complete URL. The async parameter specifies whether the request should be handled asynchronously or not. true means that script processing carries on after the send() method, without waiting for a response. false means that the script waits for a response before continuing script processing
send(content)	Sends the request
setRequestHeader("label", "value")	Adds a label/value pair to the http header to be sent
Table 7: XMLHttpRequest Methods

Property	Description
onreadystatechange	An event handler for an event that fires at every state change
readyState	Returns the state of the object: 0 = uninitialized 1 = loading 2 = loaded 3 = interactive 4 = complete
responseText	Returns the response as a string
responseXML	Returns the response as XML. This property returns an XML document object, which can be examined and parsed using W3C DOM node tree methods and properties
status	Returns the HTTP status as a number (e.g. 404 for "Not Found" or 200 for "OK")
statusText	Returns the HTTP status as a string (e.g. "Not Found" or "OK")
Table 8: XMLHttpRequest Properties

• The server implementation is just a CGI
• Arguments can be handled using the python cgi module
• Be sure to return a proper XML document if you want to use XMLHttpRequest.responseXML:

#! /usr/bin/python
import cgi
import sys

print "Content-type: text/xml"
print ""
# We want this to be interpreted as HTML by the client
print '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">'

print '<html xmlns="http://www.w3.org/1999/xhtml">'

• Note that like any CGI, we need to send the Content-type, followed by a blank line
• In this example, we want the browser to parse this as XHTML, could be any arbitrary XML, though

• Consider the Example Application
• It is using SVG for the graphics, HTML forms for the input, and AJAX to query the backend database and populate the tables
• There is also a fair amount of JavaScript and CSS trickery going on
• The application is made up of 4 files:
  • bmi280/bmi280.svg: the XHTML+SVG file that makes up the front-end
  • bmi280/css/bmi280.css: the stylesheet for both the XHTML and SVG
  • bmi280/js/bmi280.js: the JavaScript that drives the application
  • cgi-bin/getBmi280Table.py: the server-side component