Asynchronous URL Fetch for Google App Engine

There are times when you want to do remote logging or a remote API call. You may be okay with losing some updates for the tradeoff of adding little or no overhead for each call. For this case the asynchronous URL Fetch is your solution for Google App Engine. In the case I show below, the call is made and the returned result is never checked. See the GAE documentation on doing async calls which are started early in a request and then checked later in the request.

Some things to note when playing around with it is that it will not be truly asynchronous in the SDK version. In fact, if you use the code below, nothing will happen because in the SDK the call is actually made when you wait on the result. AppScale uses a modified SDK of the GAE and will suport asynchronous fetches in version 1.5.

Make sure to catch exceptions when put into production. The code below is pseudo code on GAE versus environments that allow threads.

GAE Method

from google.appengine.api import urlfetch
def url_async_post(url, argsdic):
    if isProductionGAE:
        # This will not work on the dev server for GAE, dev server must only use
        # synchronous calls
        rpc = urlfetch.create_rpc(deadline=10)
        urlfetch.make_fetch_call(rpc, url, payload=urllib.urlencode(argsdic), method=urlfetch.POST)
    else:
        raise

def call_remote(api_key, account, urlpath):
    argsdict = {"apikey":api_key,
               "accountid":account}
    url_async_post(urlpath, argsdict)
    return True
 
Threaded Method
This is how to do it for an environment that allows threads:

import threading
def my_threaded(callback=lambda *args, **kwargs: None, daemonic=True):
  """Decorate a function to run in its own thread and report the result
  by calling callback with it. Code yanked from stackoverflow.com"""
  def innerDecorator(func):
    def inner(*args, **kwargs):
      target = lambda: callback(func(*args, **kwargs))
      t = threading.Thread(target=target)
      t.setDaemon(daemonic)
      t.start()
    return inner
  return innerDecorator

@my_threaded()
def threaded_url_post(url, argsdic):
  self.url_post(url, argsdic)

def url_post(url, argsdic):
  import socket
  socket.setdefaulttimeout(5) #timeout value
  url_values = ""
  if argsdic:
    url_values = urllib.urlencode(argsdic)

  req = urllib2.Request(url, url_values)
  output = ""
  response = urllib2.urlopen(req)
  output = response.read()

  return output

§


Posterous theme by Cory Watilo