%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% The header
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[titlepage,11pt]{article}
\usepackage[a4paper]{geometry}

\sloppy

\usepackage{microtype}
\usepackage{hyperref}
\usepackage{cite}

\bibliographystyle{is-unsrt}

% section on new page
\let\stdsection\section
\renewcommand\section{\newpage\stdsection}

\title{Using Google App Engine}
\author{Bohuslav Kabrda}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% end of the header
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% the document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% hyphenations
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\hyphenation{Datastore}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% end of hyphenations
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\maketitle

\section*{Introduction}
The aim of this paper is to provide a description of using \emph{Google App Engine} (or
\emph{GAE}) with Python. It contains all important information about using the \emph{webapp}
framework and the \emph{Datastore}. I concentrated on making this paper shorter than the Google
tutorials and references \cite{gae_reference}, while heavily drawing from them. Therefore,
this paper is not meant to be innovative -- it is rather a compilation of information that
I thought were important for creating real applications for \emph{GAE}.

I added my own examples and tried to find and explain problems that are left unexplained
or not talked about in Google references.

\newpage
\tableofcontents
\newpage


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{What Is Google App Engine?}
\emph{GAE} is a mechanism which lets web applications run on Google infrastructure.
It is based on cloud computing and it spans applications over multiple servers.
It was first released in April 2008 (as a beta version). The main purpose of this technology
is to let the web developers to focus on programming itself, while deploying,
scaling and database management is done automatically.

Applications may be written using Java technologies (which means either Java
with JavaEE technologies or any language that can run inside the Java JVM -- for
example Ruby -- can be used) or using Python.

\emph{GAE} provides applications with a persistent storage, the so called \emph{Datastore},
that has the ability to scale automatically according to the way it is being used. It can be
used with JPA/JDO or Python modeling API.

All applications run in a \emph{Sandbox}, which means that they have some limitations (for example
they cannot write to the file system, only read from it), on the other hand, this ensures that
they can scale more efficiently and automatically.


\subsection{Java Runtime Environment}
The JRE used provides applications with most standard Java web development tools (such as JSP,
JPA/JDO, JavaMail, etc\ldots, but neither EJB nor JDBC).

JRE on Google App Engine currently uses Java 6, the JVM runs inside the already mentioned
\emph{Sandbox}.


\subsection{Python Runtime Environment}
Java App Engine uses optimized Python interpreter (currently version 2.5) and it also provides
developers with a special web application framework (called \emph{webapp}) and tools for accessing
the stored data. Other web frameworks such as
\emph{Django}\footnote{See \url{http://www.djangoproject.com/}} may also be
used.

Due to the \emph{Sandbox} restrictions, the code must be written in Python exclusively, extensions
written in C language are not supported.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Datastore}
The \emph{Datastore} provides applications running on Google App Engine with the only option to
store the application data.

The \emph{Datastore}:
\begin{itemize}
	\item scales ``on demand'' (which means it scales by monitoring amounts of stored
				data and number of queries).
	\item supports developers with transactions (using \emph{optimistic concurrency control}\footnote{
		\emph{Optimistic concurrency control} is a concurrency control method, in which
			the transaction assumes that it will not be interrupted while executing, therefore it
			doesn't lock the table. Before committing, it checks that the data haven't been modified
			by any other transactions. If they have been modified, the transaction rolls back.
		}).
	\item is strongly consistent.
\end{itemize}

The \emph{Datastore} is not:
\begin{itemize}
	\item traditional relational database -- entities (data objects) are stored and loaded based
		on values of their properties.
	\item based on tables -- the entities are schemaless, their structure is provided by
	  the application code.
\end{itemize}


\subsection{A Closer Look Inside -- BigTable}
This section draws from \cite{bigtable}.

The Datastore is built on \emph{BigTable}, a technology based on \emph{Google File System}
(\emph{GFS}). \emph{BigTable} is designed to be a very fast and extremely scalable DBMS.
It is, for example, used for storing data for Google projects like web indexing or Google Earth.


\subsubsection{BigTable Data Model}
\emph{BigTable} is basically a multi-dimensional sorted map. It is indexed by a row key, column
key and a timestamp, each value is an uninterpreted array of bytes.

The row keys (names of the rows) are arbitrary strings and each operation under a single row
key is atomic. The data is maintained in lexicographic order (ordered by the row key). The range
of the rows is partitioned, making each range (called \emph{tablet}) a basic unit of distribution
and load balancing.

The column keys are divided into sets called \emph{column families}. The \emph{column families}
are the basic units of access control. All data in one family typically have the same type and are
compressed together. The column keys are used in the format \emph{family:qualifier}, thus forming
a kind of a multi-level hashtable.

The timestamp dimension of BigTable is used for keeping different versions of data.


\subsubsection{BigTable Technologies}
As already mentioned, \emph{BigTable} is based on \emph{GFS}. It is designed to operate
on a cluster and depends on a cluster management system, which manages resources and schedules
jobs. \emph{BigTable} also relies on distributed lock service \emph{Chubby}, that uses the
\emph{Paxos} algorithm\footnote{Basically, the participants of the process have to agree on the
result. See \cite{paxos} for more information.}.

\subsection{Datastore API}
The primary subject of storing is an entity (also called data object, \emph{DO}). Each entity has
one or more properties of several basic data types, including a reference to another entity.

The datastore API is discussed in the chapter \ref{datastore_python}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Python Applications on Google App Engine}
This section shows how to develop few simple applications for Google App Engine using
Python. You will need Python 2.5 interpreter, \emph{GAE} SDK and a Python editor
(i suggest using Eclipse with the Pydev module).


\subsection{Setting Up the Working Environment}
Install the Python interpreter\footnote{Download from URL
\url{http://www.python.org/download/releases/2.5/}} GoogleAppEngineSDK\footnote{Download from URL
\url{http://code.google.com/intl/cs/appengine/downloads.html\#Google_App_Engine_SDK_for_Python}}
and Eclipse\footnote{Download from URL
\url{http://www.eclipse.org/downloads/}} with Pydev\footnote{Download from URL
\url{http://pydev.org/download.html}}.
In Eclipse, add the directory containing GoogleAppEngineSDK directory with libraries (e.g.
\texttt{google\_appengine\textbackslash lib} to Python Libraries -- use
\begin{tt}
Window $ \rightarrow $ Pydev $ \rightarrow $ Interpreter - Python $ \rightarrow $ New Folder 
\end{tt}.
The environment is now ready and you can create your first project.


\subsection{Hello, World!}
In Eclipse, choose
\begin{tt}
File $ \rightarrow $ New $ \rightarrow $ Other $ \rightarrow $ Pydev $ \rightarrow $ 
Pydev Google App Engine Project
\end{tt}
then click the Next button. Type \texttt{helloworld} as the project name. Then choose grammar
version 2.5 and click Next again. On the next page, browse to the folder with your instalation of
\emph{GAE} (e.g. \texttt{C:\textbackslash Program Files\textbackslash Google\textbackslash
google\_appengine\textbackslash}) and again click the Next button. Select ``Hello world''
template and click the Finish button. The project is now created.


\subsubsection{Examining the Project}
As you can see, one directory (\texttt{src}) containing two files (\texttt{app.yaml} and
\texttt{helloworld.py}) was created. The \texttt{helloworld.py} is just a simple script that
outputs the \texttt{Content-type} header and a line ``Hello world'':

\begin{verbatim}
print 'Content-Type: text/plain'
print ''
print 'Hello, world!'
\end{verbatim}

The second file is more interesting for us. It is a file written in \emph{YAML}\footnote{
``A human-readable data serialization format'', see \url{http://www.yaml.org/}}
and we will call it the configuration file:

\begin{verbatim}
application: sample-app
version: 1
runtime: python
api_version: 1

handlers:
- url: /.*
  script: helloworld.py
\end{verbatim}

The line \texttt{application: sample-app} is called the application identifier. The version line
contains the number of a version -- it is very important to use it properly, because after
uploading newer version of the application to the App Engine, you can always go back to the older
versions. Runtime environment is, of course, Python and the version of the API is set to
\texttt{1} (currently, there is only this version, but more are to come in the future).
The \texttt{handlers} section is responsible for mapping URLs to the scripts (or images,
stylesheets etc\ldots). Note, that regular expressions can be used in mapping patterns.


\subsubsection{Running the Application}
Run the \emph{GAE} Launcher (which was installed with GoogleAppEngineSDK) and add the ``Hello,
World!'' application (\texttt{File $ \rightarrow $ Add existing application} and input the path
to the \texttt{src} in the \texttt{helloworld} directory of the application. Select the project,
click \texttt{Run} and browse \texttt{http://localhost:8080/} in your browser. You should see the
``Hello, world!'' greeting.


\subsection{The webapp Framework}
There are several Python frameworks for web development, like \emph{Django}\footnote{\url{
http://www.djangoproject.com/}}, \emph{CherryPy}\footnote{\url{http://www.cherrypy.org/}} or
\emph{web.py}\footnote{\url{http://webpy.org/}}. Google, however, have decided to make their own
simple framework embedded in GoogleAppEngineSDK (which means you don't have to
install it, everything is ready to go, you only need to include it in the source files).
This framework is called \emph{webapp} and we will be using it throughout this whole tutorial.
We will also use some \emph{Django} functionality - namely the HTML \emph{templates}\footnote{
See \url{http://docs.djangoproject.com/en/dev/ref/templates/}}.

The application written in \emph{webapp} has three important parts:
\begin{itemize}
	\item a part that processes request and builds response -- it consists of one or more classes
	  extending \texttt{RequestHandler}
	\item an instance of \texttt{WSGIApplication} which routes the requests to handlers based on URL
	\item a main routine that runs the \texttt{WSGIApplication}
\end{itemize}

\subsubsection{Creating a Sophisticated Application}

Rewrite the \texttt{helloworld.py} file to look like this:

\begin{verbatim}
import os
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
from google.appengine.ext.webapp.util import run_wsgi_app

class MainPage(webapp.RequestHandler)
    def get(self):
        template_values = {
            'title': 'Hello, world!',
            'header': 'Hello, sophisticated world!'
        }
        path = os.path.join(os.path.dirname(__file__),
            'index.html')
        self.response.out.write(template.render(path,
            template_values))

class ErrorPage(webapp.RequestHandler):
    def get(self):
        template_values = {
            'title': 'Error!',
            'header': 'Yikes, this page doesn\'t exist!'
        }
        path = os.path.join(os.path.dirname(__file__),
            'index.html')
        self.response.out.write(template.render(path,
            template_values))

#the action done by the MainPage handler is
#mapped to '/' and to '/sayhello' URLs
application = webapp.WSGIApplication([('/', MainPage),
                                      ('/sayhello', MainPage),
                                      ('/.*', ErrorPage)],
                                      debug=True)

def main():
    run_wsgi_app(application)

if __name__ == "__main__":
    main()
\end{verbatim}

Then change the \texttt{app.yaml}:

\begin{verbatim}
application: sample-app
version: 1
runtime: python
api_version: 1

handlers:
- url: /stylesheets/.*
  static_dir: stylesheets

- url: /.*
  script: helloworld.py
\end{verbatim}

After altering the configuration file, create a new directory in \texttt{src}, called
\texttt{stylesheets} and create a simple stylesheet \texttt{style.css} in it:

\begin{verbatim}
body {
  background-color: brown;
}

h1 {
  color: white;
}
\end{verbatim}

Finally, create the \texttt{index.html} \emph{template} in the \texttt{src} directory:

\begin{verbatim}
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
  "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
  charset=UTF-8">
<link rel="stylesheet" type="text/css"
  href="/stylesheets/style.css">
<title>{{ title }}</title>
</head>
<body>
<h1>{{ header }}</h1>
{% ifequal title 'Error!' %}
	<b>This is so sad...</b>
{% else %}
	<b>You made it!</b>
{% endifequal %}
</body>
</html>
\end{verbatim}

Save all modified and created files and try to reload the \texttt{http://localhost:8080/} page.
Note, that you don't need to restart the application in Google App Engine Launcher, it reloads
automatically.


\subsubsection{What Does This Do?}
How does this code work? When an HTTP request is sent to the server, the requested URL is checked
against the \texttt{app.yaml} file, resolving the name of file to use for creating a response. The
response can be made by serving a static file or running a script. It is important to know, that
the URL patterns are tested from top to bottom as written in the configuration file, which means
that if we had switched the two patterns, the \texttt{helloworld.py} script would have been used
everytime, and the second pattern would have never come to use (and the stylesheet wouldn't have
been loaded). So, as you can see, if the client is asking for a file inside the
\texttt{stylesheets} folder, it gets the file directly, otherwise the \texttt{helloworld.py}
script is run and the result is returned.

Probably the most interesting thing on this example is the HTML file \emph{template} and how we
have used it. If you take a closer look at the \texttt{helloworld.py} file, you will notice, that
the two handlers (\texttt{MainPage} and \texttt{ErrorPage}) don't output any HTML code directly.
They ather save the parameters to a \emph{dictionary}\footnote{A Python data structure}, which is
then used by the \emph{template} to make the response page.

The \texttt{index.html} file is a simple HTML template written using \emph{Django}
functionality implemented in \emph{webapp}. You can use \emph{templates} to output various data
computed by request handlers. \emph{Templates} also support basic condition handling and loops.


\subsubsection{Conclusion of This Basic Example}
There are several important things to notice about this example:
\begin{itemize}
	\item The order of patterns in \texttt{app.yaml} is important.
	\item The order of patterns in \texttt{application} variable in the script file is also
	  important (for the same reason).
	\item You have to discriminate between scripts and static files.
	\item You can use \emph{Django} \emph{templates} when writing applications in \emph{webapp} 
    framework.
\end{itemize}


\subsection{Limitations}
This section lists all important limitiations for applications running on \emph{GAE}.

\begin{enumerate}
	\item Request handler has a limited amount of time to process request (currently 30 seconds),
	  when this deadline is reached, the handler is interrupted.
	\item The application cannot write to the filesystem (\emph{Sandbox} restriction).
	\item The application cannot connect to any host directly, it has to use the \emph{GAE} URL
		fetch service (\emph{Sandbox} restriction).
	\item The application cannot spawn sub-process or thread (\emph{Sandbox} restriction).
	\item All code has to be in pure Python, no C (or other languages) extensions can be included.
	\item All imports are cached, which means, that global variables are initialized only once,
		when the module is first imported, their value then remains the same between requests.
\end{enumerate}

In the next chapter, we will finally get to the main focus of this work -- using \emph{Datastore}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Using Datastore With Python}
\label{datastore_python}


\subsection{Data Modelling}
\label{modeling}
The Datastore API (namely package \texttt{google.appengine.ext.db}) doesn't need any table
definitions like relational databases do. The format of the stored data is described by
subclasses of the \texttt{db.Model}, \texttt{db.Expando} and \texttt{db.PolyModel} classes.


\subsubsection{Models and Properties}
Let's look at a small example:

\begin{verbatim}
class Chair(db.Model):
    legs = db.IntegerProperty(required = True)
    description = db.StringProperty()
    is_armchair = db.BooleanProperty(required = True)
    created_on = db.DateProperty(auto_now_add = True)
    owned_by = db.UserProperty(auto_current_user_add = True)
    materials = db.ListProperty(basestring)
\end{verbatim}

We have created a class representing a chair with six attributes. The first four attributes
store data as you would expect using common data types. Some of the attributes are marked as
required (which means, that they need to be passed to the constructor when creating an instance).
Attributes that are not required and not set are stored as \texttt{None} values. The
property \texttt{created\_on} will be assigned current date on storing, if uninitialized.
Property \texttt{owned\_by} is a reference to a Google account (it defaults to currently signed
user). With \emph{webapp}, you can even use Google accounts for authentication in your
application. The last property is a list of strings, which can also be defined as
\texttt{db.StringListProperty}. You can now make an instance of the \texttt{Chair} class:

\begin{verbatim}
broken_chair = Chair(legs = 3,
                     is_armchair = False,
                     materials = ['rust', 'wood_worm'],
                     created_on = datetime(1900, 1, 1))
\end{verbatim}

Any attributes that you might assign to the chair that are not specified in the model will not be
saved in the datastore (which could be useful for storing values used only at runtime).

Sometimes you might need to store different attributes for instances of the same class.
That is what the Expando Models are used for. The \texttt{db.Expando} class is a subclass of
\texttt{db.Model}. It can define both fixed and dynamic properties. All properties of each
instance are stored in the database:

\begin{verbatim}
class Person(db.Expando):
    sex = db.StringProperty(required = True,
        choices = set(['female', 'male', 'alien']))
    height = db.IntegerProperty()
    
p = Person(sex = 'female')
p.name = 'Monty'

\end{verbatim}

The attributes defined in the class definitions have the same semantics as in the \texttt{db.Model}
definition, but all the dynamic attributes will be stored as well. Note, that dynamic attribute with
value \texttt{None} is not the same as a dynamic attribute that doesn't exist (which means that the
entity simply doesn't have it).

The last type of model is a Polymorphic Model (\texttt{db.PolyModel}). It is used to create class hierarchy:

\begin{verbatim}
class Food(db.PolyModel):
    calories = db.IntegerProperty(required = True)

class Spam(Food):
    is_tasty = db.BooleanProperty(required = True)
    
class Beans(Food):
    quantity = db.IntegerProperty(required = True)
\end{verbatim}

All \texttt{Spam} and \texttt{Food} instances will now have the \texttt{calories} attribute.
A \emph{Datastore} query (described in \ref{queries}) for Food will return \texttt{Food},
\texttt{Spam} and \texttt{Beans} entities, while a query for \texttt{Spam} will only
return \texttt{Spam}.

Dynamic attributes for \texttt{db.PolyModel} descendants aren't stored.


\subsubsection{Special Properties}

There are few interesting properties, that we will now have a closer look at.

A string can be stored as \texttt{db.StringProperty} (indexed) with maximum length of 500
bytes or as a \texttt{db.TextProperty} with maximum length of 1 megabyte (unindexed).
Binary data can be stored in similar types -- 500 bytes long indexed \texttt{db.ByteString}
and 1 megabyte long unindexed \texttt{db.BlobProperty}.

What you have probably thought of already is how to model entity relations. This can be
achieved by using \texttt{db.ReferenceProperty}:

\begin{verbatim}
class Alien(db.Model):
    has_flying_saucer = db.BooleanProperty(required = True)
    
class Human(db.Model):
    kidnapped_by = db.ReferenceProperty(Alien, required = True)
    
alien = Alien(has_flying_saucer = True)

#these two lines do the same
human1 = Human(kidnapped_by = alien)
human2 = Human(kidnapped_by = alien.key())
\end{verbatim}

The \texttt{human1} and \texttt{human2} have been kidnapped by the same \texttt{alien}, as you
have probably expected. If \texttt{alien} is not in the memory, but only in the \emph{Datastore}
and one of the human entities is used, the \texttt{alien} is fetched automatically. The
\texttt{key()} method will be described in \ref{queries}.

You can use \texttt{db.SelfReferenceProperty} for reffering to an entity of the same kind.

For detailed description of all datatypes, see
\url{http://code.google.com/intl/cs/appengine/docs/python/datastore/typesandpropertyclasses.html}.


\subsection{Storing and Retrieving Data}
Up until now, we haven't stored a single byte of data into the Datastore. To do that, we will
first need to know what a \texttt{Key} is.


\subsubsection{Keys}
\label{keys}

A \emph{key} (represented by a \texttt{Key} object) is a unique identifier for the entity across
all entities for one application. Each \emph{key} has several components:
\begin{itemize}
	\item \emph{path} -- describes parent-child relationship between this entity and another entity
		(we will talk entity relationships in \ref{relationships})
	\item \emph{kind} of the entity
	\item either a \emph{name} assigned to entity by application or an \emph{ID} assigned by the datastore
\end{itemize}

The \texttt{Key} attribute is created upon storing the entity and cannot be changed!


\subsubsection{Creating and Updating Entities}

Creating and updating entity is very simple. The only thing you need is to call the \texttt{put()}
method of \texttt{db.Model}/\texttt{db.Expando}/\texttt{db.PolyModel} subclass instance and the
Datastore API takes care of the rest. When you call this method for an instance that hasn't yet
been stored, it is saved and given a \texttt{Key}. If the entity already exists in Datastore upon
calling \texttt{put()} method, its properties are updated.

The \texttt{put()} method returns the \texttt{Key} of the stored entity.

You can use the \texttt{db.put()} function to put a list of entities (list of \texttt{Key}s
is returned in this case).

Examples:

\begin{verbatim}
class Movie(db.Model):
  title = db.StringProperty(required = True)
  length = db.IntegerProperty()
  director = db.StringProperty()
  
life_of_brian = Movie(title = 'Life of Brian'
                      length = 123)

#store this movie
life_of_brian.put()

#update this movie
life_of_brian.length = 234
life_of_brian.put()

holy_grail = Movie(title = 'Monty Python and the Holy Grail'
                   length = 100)
best_of = Movie(title = 'Best of Monty Python\'s Flying Circus')

#put more movies at once
db.put([holy_grail, best_of])
\end{verbatim}


\subsubsection{Retrieving Entities -- Queries}
\label{queries}

The \emph{Datastore} API provides you with two similar ways of retrieving the stored data: the
\emph{Query} interface and the \emph{GqlQuery} interface.

The \emph{Query} interface lets you get entities using the \texttt{all()} class method of a
\texttt{db.Model}/\texttt{db.Expando} subclass. Calling this method returns a \texttt{Query}
object representing a query for all entities of given kind. You can than adjust this query
by calling its methods: \texttt{filter()}, \texttt{order()}, \texttt{ancestor()}. The
\texttt{ancestor()} method is explained in \ref{relationships}.

Using more than one filter in query means, that objects returned by the query will match
all conditions (in other words, filters are interpreted as a \emph{conjunction} of predicates).
Unfortunatelly, there is no ``\texttt{LIKE}'' operator for matching string properties,
the application has to filter those in the program code.

You can order entities ascending (use property name as a parameter for \texttt{order()}) or
descending (use property name prefixed by \texttt{-} as a parameter for \texttt{order()}).
If the values of the property used for ordering have different types, they are first ordered
by type and then by value.

This kind of query also lets you to pull out only the \texttt{Key}s of the entities by passing
\texttt{keys\_only = True} to its constructor.

Examples:

\begin{verbatim}
query = Movie.all()

#select all movies directed by Steven Spielberg, shorter than
#200 minutes and sort them by title descending
query.filter('length <', 200)
query.filter('director =', 'Stephen Spielberg')
query.order('-title')

#do the same shorter (even on one line)
query.filter('length <', 200)
    .filter('director =', 'Stephen Spielberg')
    .order('-title')
\end{verbatim}

The \emph{GqlQuery} interface lets you use the \emph{GQL} -- an SQL-like query language.
\texttt{GqlQuery} is a class located in \texttt{google.appengine.ext.db}. The constructor
takes a string representation of query and optional parameters for binding.

Examples:

\begin{verbatim}
query = Movie.all()

\begin{verbatim}
#do the same as in the previous example
#use positional parameter binding
query = db.GqlQuery("SELECT * FROM Movie"
                    "WHERE director = :1 AND length < :2"
                    "ORDER BY title DESC",
                    'Steven Spielberg', 200)

#do the same, but use keyword parameter binding
query = db.GqlQuery("SELECT * FROM Movie"
                    "WHERE director = :director AND length < :length"
                    "ORDER BY title DESC",
                    'Steven Spielberg', 200)
\end{verbatim}

You can also use string, boolean and number values as literals in the query string.
There is one more way of creating a query. The equivalent to the above example would be:

\begin{verbatim}
query = Movie.gql("WHERE director = :director AND length < :length"
                  "ORDER BY title DESC",
                  'Steven Spielberg', 200)
\end{verbatim}

The queries (both \texttt{Query} and \texttt{GqlQuery}) do not execute automatically.
You either have to try to use the entities retrieved (use the query as an iterator)
or use the \texttt{fetch()} method of a query object. The fetch method has two parameters:
the \texttt{limit} (maximum number of entities to fetch) and optional parameter \texttt{offset}
(number of entities to skip -- \emph{these entites are not returned by \texttt{fetch()}, but are
loaded into the memory!}). This method returns a list of entities.

Examples:

\begin{verbatim}
#fetch 10 entities, but skip 5
#we get 5 entites as the result of fetch(),
#but all 10 entities are loaded into memory
results = query.fetch(10, 5)
for result in results:
    print "Title: " + result.title

#use query as an iterator
#all entities are fetched
for result in query:
    print "Title: " + result.title
\end{verbatim}

Currently, the \emph{Datastore} allows you to retrive only 1000 entities in one query.

One more way to retrive an entity by using its \texttt{Key}:

\begin{verbatim}
#construct a hyperref somewhere else
movie = Movie(title='Holy Grail')
self.response.write('<a href="/edit?key=%s">%s</a>'
	% (str(movie.key()), movie.name()))


#get the movie key from the parameter
key = self.request.get('key')
#you have to convert the string to the key object
entity = db.get(db.Key(key))
\end{verbatim}

The \texttt{get()} method used without any parameters returns the first entity of the result.
If an entity has a \texttt{db.ReferenceProperty}, you can use the parent entity for result
filtering and you can retrive the referenced entity using this \texttt{db.ReferenceProperty}.

\begin{verbatim}
class Viking(db.Model):
    name = db.StringProperty(required = True)

class BowlOfSpam(db.Model):
    owner = db.ReferenceProperty(Viking, required = True)
    
#lets presume that Viking Harry has a bowl of spam
#stored in the Datastore
harry = Viking(name = 'Harry')

#query for his bowl of spam
query = BowlOfSpam.all()
query.filter('owner = ', harry)

#equivalent:
query = BowlOfSpam.all()
query.filter('owner = ', harry.key())

#this statement returns Harry
query.get().owner

\end{verbatim}

Filters can only contain comparison operators (\texttt{<}, \texttt{>}, \texttt{<=}, \texttt{>=},
\texttt{=}, \texttt{!=}) and the \texttt{IN} operator.

The ``\texttt{!=}'' operator in fact runs two sub-queries -- one with the ``\texttt{!=}'' operator
replaced by ``\texttt{<}'' and one with the ``\texttt{!=}'' operator replaced by ``\texttt{>}''.
The results are then merged and returned. A query can only have one ``\texttt{!=}'' operator.

The ``\texttt{IN}'' operator performs multiple sub-queries -- one for each value in the provided
list (the ``\texttt{IN}'' operator is replaced by ``\texttt{=}''). Again, the results are merged
and returned. A query can contain more ``\texttt{IN}'' operators, but in this case, the query is
run for each combination of the values of provided list (which can result in many sub-queries).

Moreover, the inequality operators (\texttt{<}, \texttt{>}, \texttt{<=}, \texttt{>=}, 
\texttt{!=}) can only be used on one property and if the results are to be sorted,
they must be sorted by the properties used for inequality comparisons first.

A query can currently have a maximum of 30 sub-queries.

\subsubsection{Deleting Entities}

Entities can be deleted by calling the \texttt{delete()} method of \texttt{db.Model} subclass
instance or by using the \texttt{db.delete()} function - it requires a single \texttt{Key} or
entity (or list of \texttt{Key}s or entities):

\begin{verbatim}
query = db.GqlQuery("SELECT * FROM Movie"
                    "WHERE director = :1 AND length < :2"
                    "ORDER BY title DESC",
                    'Steven Spielberg', 200)
results = query.fetch(10)
for result in results:
    result.delete()

#do the same faster
query = db.GqlQuery("SELECT * FROM Movie"
                    "WHERE director = :1 AND length < :2"
                    "ORDER BY title DESC",
                    'Steven Spielberg', 200)
results = query.fetch(10)
db.delete(results)
\end{verbatim}

For a complete reference for creating queries, see
\url{http://code.google.com/intl/cs/appengine/docs/python/datastore/queryclass.html} and
\url{http://code.google.com/intl/cs/appengine/docs/python/datastore/gqlqueryclass.html}.


\subsection{Entity Groups and Relationships}
\label{relationships}

In this chapter, we will take a closer look at entities and entity relationships --
a feature that is pretty much necessary in any kind of application.


\subsubsection{Entity Groups}

Every entity stored in the \emph{Datastore} belongs to an \emph{entity group}. Each \emph{entity
group} has one \emph{root} (an entity without a parent). Other entities may have the
\emph{root} entity set as a parent (which can be done in entity constructor), some other
entities may set these as their parents and so on, forming a \emph{tree}. A \emph{path}
is a chain of entities from an entity to the \emph{root}. All entities in the \emph{path}
of a single entity are called \emph{ancestors}.

What are the \emph{entity groups} good for?

\begin{itemize}
	\item All entities belonging to the same group are stored in the same datastore node.
	\item Only entities in the same \emph{entity group} can be modified in transactions
	  (see \ref{transactions}).
	\item Although it would seem, that putting entities in one entity group makes operations
	  with them faster,
	  \url{http://code.google.com/intl/cs/appengine/docs/python/datastore/keysandentitygroups.html}
	  claims, that doing so has no significant impact on the speed.
	\item The \emph{entity groups} should only be used when you need to change entities in
	  transaction. You should use \texttt{db.ReferenceProperty} otherwise.
\end{itemize}

Note, that entites are not deleted when you delete one of their \emph{ancestors}.


\subsubsection{Filtering Entities By Ancestors}
You can also use the ancestors similarly as filters in queries:

\begin{verbatim}
class A:
  pass

q = A.all()
q.ancestor(ancestor_key)
\end{verbatim}


\subsection{Transactions}
\label{transactions}

A natural requirement when using various DBMSs is the ability to change data in
transactions. A transaction has to be \emph{ACID} -- (atomic, consistent, isolated, durable).


\subsubsection{Using Transactions}
Let's see a simple use of transactions:

\begin{verbatim}
class Counter(db.Model):
  count = db.IntegerProperty()
  
#the function to run in transaction
def increment(key):
  counter = db.get(key)
  counter.count += 1
  counter.put
  
counter_key = Counter.all().get().key()

#run the transaction
db.run_in_transaction(increment, counter_key)
\end{verbatim}

As you can see, the only thing you need is to write an ordinary Python function and than call
\texttt{db.run\_in\_transaction()} function, which takes the function object and its parameters
to pass to the it.

If the function throws an exception, the transaction is rolled back (has no effects on
\emph{Datastore}) and the exception is re-thrown, unless it is a \texttt{Rollback} exception,
which is caught and \texttt{None} value is returned.

If the function returns successfully, the transaction is committed and the function return value
is returned.

The \texttt{db.run\_in\_transaction()} tries to run the function few times if it doesn't succeed
on the first run, which means, that functions used in transactions should not have any side
effects. To adjust the number of function runs, use the
\texttt{db.run\_in\_transaction\_custom\_retries()} function (which is the same as
\texttt{db.run\_in\_transaction()}, but it has an extra parameter which determines the number of
runs).


\subsubsection{Transaction Limitations}
There are several transactions limitations that one needs to be aware of:

\begin{itemize}
	\item Each transaction can only use entities from one \emph{entity group}.
	\item If a query is performed during a transaction, it has to include
	  the \emph{ancestor} filter.
	\item A transaction can contain any Python code, but it should be as fast as possible
	  to minimize the chance of being interrupted by any other \emph{Datastore} operations.
\end{itemize}


\subsection{Indexes}
An \emph{index} is a table that contains potentional results of a query in a desired order
(that means, that \emph{index} is basically a table containing all entities of given kind).
The indexes are defined in \texttt{index.yaml} file. The development server configures the
indexes automatically as it executes queries, that do not have \emph{index} yet. You can
also configure \emph{indexes} manually.

As we have seen in \ref{queries}, the queries are used to retrieve data from the \emph{Datastore}.
Each query has to specify the entity kind and may also specify \emph{filters} and sort orders.


\subsubsection{Explaining Indexes}
Each query run from the application has to have a proper \emph{index} configured in the
\emph{index.yaml}. When the application changes an entity, all indexes containing this entity
are updated accordingly. When the application executes a query, all matching entities are
retrieved from the proper \emph{index}. \emph{Indexes} are stored for all combinations of kind,
\emph{filters} and sort orders used in queries throughout the application.

Let's look at an example query:

\begin{verbatim}
SELECT * FROM Movie WHERE director = 'John Smith'
                      AND title = 'Custom Title'
                    ORDER BY title ASC
\end{verbatim}

\emph{Index} for this query contains columns with \texttt{Key}s, directors and titles, the
rows are sorted by titles ascending. Now, all queries filtering movies by director and title,
asking to sort results by title ascending use this \emph{index}. For example:

\begin{verbatim}
SELECT * FROM Movie WHERE director = 'Someone Else'
                      AND title = 'Document About Spam'
                    ORDER BY title ASC
\end{verbatim}

The \texttt{index.yaml} file for application using these queries would look like this:
\begin{verbatim}
indexes:

- kind: Movie
  properties:
  - name: director
  - name: title
    direction: asc
\end{verbatim}

In \ref{modeling}, it has been pointed out, that it is not the same, when an entity has a property
with \texttt{None} value and when the entity simply doesn't have the property. The reason is,
that an entity without a property is not included in an \emph{index} for a query using that propert
(while the entity with this property of value \texttt{None} is included).

We have also seen, that some properties aren't indexed -- namely \texttt{db.TextProperty} and
\texttt{db.BlobProperty}. This means, that these properties cannot be used in \emph{filters}.
A property can also be marked as non-indexed by passing the \texttt{indexed = False} parameter
to its constructor.

\subsubsection{Exploding Indexes}
The \emph{Datastore} lets you store the \texttt{db.ListProperty}, which could be useful, but it
can have one negative effect called \emph{exploding index}.

When you store a \texttt{db.ListProperty}, an index entry is made for each value of the list in
each \emph{index} using this value. And what is worse -- when an entity has two or more list
properties, all permutations of values are stored in each \emph{index}. And because each entity
has a limited number of index entries, this can couse an application not to work properly
(not speaking of significant performance drop).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}
\emph{GAE} seems to be a big step to making web development easier. The \emph{webapp}
framework and \emph{Datastore} API are simple to learn, yet powerful tools. A simple application
can be made fast and efficiently without a big effort.

On the other hand, there are many downsides. The \emph{webapp} framework (lacking some
important features like redirecting between Request Handlers) can be substituted
by any other Python framework, but the \emph{Datastore} queries seem to have to many
limitations to be used in large and complex applications.

Still, \emph{GAE} is certainly worth trying when you want to see where the modern web development
is going, or you just want to write a simple web application to be run on Google infrastucture.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% bibliography
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliography{biblio}

\end{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% end of the document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
