AngularJS: Reducing Initial Load Time by Preloading Data

2015-12-28 00:00:00 +0000

Background

Most of the web has transitioned into client-side heavy development using various JavaScript frameworks. More and more sites are being built in the “Single Page Application” (SPA) fashion. This puts more work on the user’s browser and less work on the servers. The shift keeps front-end and back-end development separate and isolated, allowing clients to present information in different ways without relying on server-side rendering. Additionally, most of smartphones these days are more powerful than servers that serve thousands of users.

[ Source Code | Demo ]

One of the most common issues with this paradigm is the slow initial load time of the page. Typically, the page is loaded very quickly only to show a spinning icon for some time until the actual content is loaded. Here’s the diagram that explains the request cycle:

Client requests the page. Server responds with some bootstrapped and non-rendered HTML. (green)
Client requests for static assets, including the JavaScript files and HTML templates. (blue)
Application makes one or more HTTP calls to fetch various resources. (yellow)
The loaded resources are displayed to the user.

These steps are sequential. As one can imagine, the initial load time can be very slow if the user has a slow internet connection and/or the user is far away from the website data centers.

By looking at the diagram, let’s assume the api/products/123 is the highest priority HTTP call. We want to show the result of it as soon as it is ready while other HTTP calls may still be pending. If we somehow include the response of this endpoint with the initial HTML page or app.js, the extra HTTP call would not be necessary.

AngularJS and Angular UI Router

AngularJS is a popular and somewhat opinionated JavaScript framework that is backed by Google. (Fun fact: the Angular team sat one floor below my team at Google!)

AngularJS lets you write client-side web applications as if you had a smarter browser. It lets you use good old HTML (or HAML, Jade and friends!) as your template language and lets you extend HTML’s syntax to express your application’s components clearly and succinctly. It automatically synchronizes data from your UI (view) with your JavaScript objects (model) through 2-way data binding.

If you are building an AngularJS 1.x app, you probably are already familiar with AngularUI Router. It is generally more flexible than the standard $route service. I suggest looking into it if you have not done so. The example in this post uses AngularUI Router.

A typical application route setup might look something like this:

$stateProvider
    .state('main', {
        abstract: true,
        resolve: {
            user: function (UserService) {
                return UserService.me();
            }
        }
    .state('main.product', {
        url: '/products/{id}'
        resolve: {
            product: function ($stateParams, ItemService) {
                return ItemService.get($stateParams.id);
            }
        }

The user and product properties are “resolved” (by making HTTP calls for example) before the state transition happens.

Preload (Prefetch) Data

The solution I propose is simple but somewhat ugly to implement. It minimizes the initial load time, which is the best user experience.

When the server receives a request to website.com/products/123, it makes an API call to api/products/123 on behalf of the client. This will be significantly faster than the client calling the API endpoint because the data centers are typically close to each other with much faster network speeds.
The result from api/products/123 is appended to the original as URL parameter (let’s call it _data). The URL change to: website.com/products/123?_data={...}
The client handles the case when this URL parameter is present.

Important: In order for the proposed solution to work, the application needs to have html5Mode enabled. This is because anything after the # portion of the URL is not sent to the server (read more: fragment identifier). The default hashbang method uses the #. You should have strong reasons if you do not already have html5Mode enabled.

In this scenario, we are only working with api/products/123. The solution can be used in 100 different ways, including but not limited to calling multiple APIs or including some placeholder data.

First, let’s make that URL parameter an optional parameter (_data).

class Config {
    constructor($locationProvider, $stateProvider, $urlRouterProvider) {
        $locationProvider.html5Mode(true);

        $urlRouterProvider.otherwise('/');

        $stateProvider
            .state('default', {
                url: '/',
                templateUrl: 'main.html'
            })
            .state('item', {
                url: '/items/{id}?_data',
                controller: 'Item as item',
                templateUrl: 'item.html',
                reloadOnSearch: false
        });
    }
}

It makes sense to remove the _data parameter from the URL as soon as the application reads so the user does not share or bookmark the full URL (and to make the URL look pretty). The reloadOnSearch: false option prevents the state from being reloaded when this happens.

Now, the client logic skips the API call to get product information when the _data parameter is present.

class Item {
    constructor($log, $state, $stateParams, ItemService) {
        this.title = 'Loading..';
        this._log = $log;
        this._state = $state;
        this._stateParams = $stateParams;
        this._ItemService = ItemService;

        this.init();
    }

    init() {
        let context = this;

        this._log.debug(this._stateParams);

        // Pre-fetched data can come as a URL parameter (`_data`).
        var data = angular.fromJson(this._stateParams._data);

        if (data) {
            // Remove `_data` parameter from URL.
            this._state.go('.', {_data: null}, {location: 'replace'});

            return ready(data);
        }

        this._ItemService.fetchItem(this._stateParams.id).then(ready, ready);

        function ready(data) {
            context.title = data.name;
            context.item = data;
        }  
    }
}

class ItemService {
    constructor($timeout) {
        this._timeout = $timeout;
    }

    fetchItem(id) {
      return this._timeout(() => {
          return {id: id, name: 'TV', description: 'Fetched from the backend'};
      }, 1500);
    }
}

This method reduces the number of HTTP requests made by the application. However, I do not recommend doing this everywhere in the application as it introduces logic code both in the back-end endpoint and in the Angular application.

Full source and demo can be found here.

Conclusion

We have looked at one of the biggest concerns against modern client-side heavy development – slower initial load times. We proposed a back-end agnostic solution for the problem in AngularJS that makes use of URL parameters. Code samples for both front-end and back-end have been provided.

Bonus: Flask Back-end Endpoint Handler

Below is the quick implementation of step-1 and step-2 of the proposed solution above, in Flask. I’m sure a similar approach can be implemented in other frameworks.

try:
    from urllib.parse import urlencode
except:
    from urllib import urlencode

import re
import flask as f

app = f.Flask(__name__)
ITEMS_RE = re.compile(r'^items/(\d+)')

@app.route('/app/', defaults={'path': ''})
@app.route('/app/<path:path>')
def index(path):
    """Main application entry. Let's assume our app is served at `/app`."""
    m = ITEMS_RE.match(path)
    if m and not f.request.args.get('_data'):
        data = get_item(int(m.groups()[0]))
        args = dict(f.request.args)
        args['_data'] = f.json.dumps(data)
        url = '{}?{}'.format(f.request.base_url, urlencode(args, doseq=True))
        return f.redirect(url)
    return 'return index.html'


def get_item(ident):
    return {'id': ident, 'name': 'fake', 'description': 'fake api return'}


if __name__ == '__main__':
    app.run(debug=True)

Handling Unexpected WTForms Default Values

2015-11-13 00:00:00 +0000

The WTForms library is great for working with forms and validations. However, it does not work well with JSON form data. Specifically, fields that were not provided in the request are assigned default values. This causes problems when you expect PATCH-like requests. Example:

import wtforms

class MyBaseForm(wtforms.Form):
    pass

class FooForm(MyBaseForm):
    foo = wtforms.StringField('Foo')
    bar = wtforms.IntegerField('Bar')
    is_baz = wtforms.BooleanField('IzBaz')

fd = DummyMultiDict({'foo': 'ayy'})
f = FooForm(fd)

print(f.data)

# # # # #
class DummyMultiDict(dict):
    def getlist(self, key):
        v = self[key]
        return v if isinstance(v, (list, tuple)) else [v]

The above produces:

{'is_baz': False, 'foo': 'ayy', 'bar': None}

This is not necessarily what you want when working with REST APIs or PATCH-like requests. (Again, wtforms is not really meant for that). The default values your code expects may be different from the default values wtforms assigns.

Suppose you have:

def save(foo='No name', bar=42, is_baz=True):
    return db.save(Object(foo=foo, bar=bar, is_baz=is_baz))

fd = DummyMultiDict({'foo': 'ayy'})
f = Foo(fd)

save(**f.data)

In this case, you would expect a new entry in the “database” to be Object(foo='ayy', bar=42, is_baz=True). But the actual entry saved is Object(foo='ayy', bar=None, is_baz=False). All your unit tests fail, your users are confused, and your manager is mad.

Here is a small wrapper around wtforms.Form to better handle the above problem. (There is also wtforms-json, but its interface is quite different and feels wrong.)

JForm

import wtforms


class JForm(wtforms.Form):

    def process(self, *args, **kwargs):
        if args:
            formdata = args[0]
        else:
            formdata = kwargs.get('formdata', None)
        self._formdata = formdata
        super(JForm, self).process(*args, **kwargs)

    @property
    def data(self):
        """Returns form data with fields that were not in request popped."""
        d = super(JForm, self).data
        if self._formdata is None:
            return d
        keys = d.keys()
        keys = keys if isinstance(keys, list) else list(keys)
        for k in keys:
            if k not in self._formdata:
                del d[k]
        return d

When form.data is requested, JForm ignores any property that was not included in the original form creation.

Usage:

import wtforms

class MyBaseForm(JForm):
    pass

class FooForm(MyBaseForm):
    foo = wtforms.StringField('Foo')
    bar = wtforms.IntegerField('Bar')
    is_baz = wtforms.BooleanField('IzBaz')

fd = DummyMultiDict({'foo': 'ayy'})
f = Foo(fd)

print(f.data) # {'foo': 'ayy'}

Happy coding. It also works nicely with other wrappers such as flask-wtf. Simply change to

class MyBaseForm(JForm, flask_wtf.Form):
    pass

Update: I did not want to publish this as a pip-package because … I is lazy.

Sharing AWS Credentials Between Users On a Docker Container

2015-07-18 00:00:00 +0000

TL;DR: AWS CLI configurations and credentials are stored in the user’s home directory by default (~/.aws). This becomes a problem when other users such as root, www-data, nobody, or cron jobs need access to these credentials. This post shows how to get around this in a Docker environment.

Docker is a great application containment tool for building and shipping software of any type. You can learn more about Docker here. We love and use Docker at Humanlink with AWS Elastic Beanstalk.

Being on Elastic Beanstalk means utilizing other amazing Amazon services as well. AWS CLI and boto (for Python developers) are the de-facto tools for interacting with AWS.

There are a few ways to provide credentials to these tools. On production, it is a good idea to serve them as environment variables (such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). During development, however, it is recommended that these values are set via the aws configure command, which in turn places config files under ~/.aws/.

I was planning on having a full-blown example with detailed explanations, but that quickly became a long post and lost its focus.

In short, we do not want to pollute the Dockerfile just for the development environment. When running the Docker image locally, we can mount the ~/.aws directory AND set the $HOME environment variable. Otherwise, $HOME defaults to /root which causes permission problems for non-root users.

Since mounted file permissions are copied over to the Docker container, we need to give read permissions to everyone (on your host machine).

$ chmod 644 ~/.aws/*

We can now run the container:

$ docker run -it --rm \
  -e "HOME=/home" \
  -v $HOME/.aws:/home/.aws \
  myapp

Now the credentials are available system-wide in the Docker container for any user to use and communicate with AWS.

Automatically Download the Latest App Engine SDK

2015-05-20 00:00:00 +0000

TL;DR: Google doesn’t provide a permanent link to the latest SDK; update check endpoint is not reliable; I provide a snippet (view) to work around this.

Google provides an SDK for interacting with App Engine. dev_appserver.py simulates the App Engine Python runtime environment locally. appcfg.py is used to interact with App Engine to do tasks such as deploying a new version of the application.

For continuous deployment / integration scripts such as those for CircleCI, Travis CI, or Jenkins CI, it is usually necessary to download the latest version of the Google App Engine SDK. Unfortunately, Google does not provide a permanent link to the latest version for easy download. It seems like a basic need. I talk about how to programatically get the latest SDK in this post.

UpdateCheck Endpoint

In appengine/tools/sdk_update_checker.py, I found the API endpoint for checking for updates to the SDK: https://appengine.google.com/api/updatecheck.

$ curl https://appengine.google.com/api/updatecheck

release: "1.9.19"
timestamp: 1424415497
api_versions: ['1']
supported_api_versions:
  python:
    api_versions: ['1']
  python27:
    api_versions: ['1']
  go:
    api_versions: ['go1']
  java7:
    api_versions: ['1.0']

Great, this is all we need. Not so fast! But it is useful.

SDK Download URL

On the App Engine SDK downloads, the only link to the latest SDK is something like:

https://storage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.20.zip

Notice the version number is 1.9.20, not 1.9.19.

Using the Google Cloud Storage API, we can list all the files in the appengine-sdks/featured bucket:

https://www.googleapis.com/storage/v1/b/appengine-sdks/o?prefix=featured

Fetch the Latest SDK

Given the two points from above, we can can automatically check for the latest SDK version and download the zip with that version.

# script.bash
# DON'T COPY-PASTE, READ BELOW.

#!/usr/bin/env bash

API_CHECK=https://appengine.google.com/api/updatecheck
SDK_VERSION=$(curl -s $API_CHECK | awk -F '\"' '/release/ {print $2}')
SDK_URL=https://storage.googleapis.com/appengine-sdks/featured/google_appengine_$SDK_VERSION.zip

function download_sdk {
  echo ">>> Downloading..."
  curl -fo $HOME/gae.zip $SDK_URL || exit 1
  unzip -qd $HOME $HOME/gae.zip
}

function upload {
  $HOME/google_appengine/appcfg.py \
      --oauth2 update my-app
}

download_sdk
upload

Easy, right? Ehhhh.

./script.bash

>>> Downloading...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

curl: (22) The requested URL returned error: 404 Not Found

Apparently, the UpdateCheck endpoint does not always have the latest version. Like wut. From my experience, it can take anywhere from 1-7 days for the endpoint to return the latest version. Additionally, once a new version becomes available, previous download links become unavailable. The App Engine team acknowledged the issue five months ago but still has not addressed it.

Revisiting SDK Download URL

Well, obviously at this point the only way to get the latest SDK version is to parse the HTML content of the download page. We could also look through all the files in the featured bucket and find the one correct file to download. However, nobody should do that! Since we don’t necessarily need the latest version to simply deploy the project, we can fallback to a slightly older SDK version (the one UpdateCheck returns).

As indicated on the downloads page, deprecated SDK downloads can be found at the following link:

https://console.developers.google.com/m/cloudstorage/b/appengine-sdks/o/deprecated/1919/google_appengine_1.9.19.zip

which I found out is the same as:

https://storage.googleapis.com/appengine-sdks/deprecated/1919/google_appengine_1.9.19.zip

New script.bash

So now we can write a script that fall-backs to the deprecated SDK URL if the first URL does not work.

# script.bash

#!/usr/bin/env bash

API_CHECK=https://appengine.google.com/api/updatecheck
SDK_VERSION=$(curl -s $API_CHECK | awk -F '\"' '/release/ {print $2}')
# Remove the dots.
SDK_VERSION_S=${SDK_VERSION//./}

SDK_URL=https://storage.googleapis.com/appengine-sdks/
SDK_URL_A="${SDK_URL}featured/google_appengine_${SDK_VERSION}.zip"
SDK_URL_B="${SDK_URL}deprecated/$SDK_VERSION_S/google_appengine_${SDK_VERSION}.zip"

function download_sdk {
  echo ">>> Downloading..."
  curl -fo $HOME/gae.zip $SDK_URL_A || \
      curl -fo $HOME/gae.zip $SDK_URL_B || \
      exit 1
  unzip -qd $HOME $HOME/gae.zip
}

function upload {
  echo ">>> Deploying..."
  $HOME/google_appengine/appcfg.py \
      --oauth2 update my-app
}

download_sdk
upload

If we run the script again, it works as expected:

Tries to download from the features folder first
Otherwise, tries to download from the deprecated folder

Conclusion

We came up with script.bash that automatically downloads the “latest” Google App Engine Python/PHP SDK to make life easier. This is especially useful during automated builds and deployments.

References

0: https://code.google.com/p/googleappengine/issues/detail?id=11604

App Engine and SSL

2015-04-05 00:00:00 +0000

Google App Engine is a great platform for getting things done quickly. However, it can be very unpleasant to work with due to its sandboxed environment and close source code. Basic needs such as installing third-party libraries can be tricky to install as well.

Getting one of the most popular python libraries, python-requests, was particularly tricky to get it running and working with SSL connections. I’ll walk through how I fixed the issue.

Start by adding the library to the project:

# From project root.
pip install -t lib/ requests

The above command pip-installs the requests library into the lib directory. This is where all the third-party libraries can be placed.

Now we need to let App Engine know about this. Create or modify the file appengine_config.py in the root of the project.

from google.appengine.ext import vendor
vendor.add('lib')

appengine_config.py runs when a new instance is created. vendor.add adds the specified path to $PYTHONPATH.

At this point, most third-party libraries work just fine. However, there’s a bit of work that needs to be done to get requests working.

Head over to http://localhost:8000/console and execute:

import requests

r = requests.get('https://httpbin.org/status/200')
print(r.status_code)

In a normal Python environment, the code executes just fine printing a 200 status. But on GAE, the following exception occurs:

Traceback (most recent call last):
  File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/request_handler.py",
line 225, in handle_interactive_request
    exec(compiled_code, self._command_globals)
  File "<string>", line 1, in <module>
  File ".../lib/requests/__init__.py", line 58, in <module>
    from . import utils
  File ".../lib/requests/utils.py", line 26, in <module>
    from .compat import parse_http_list as _parse_list_header
  File ".../lib/requests/compat.py", line 42, in <module>
    from .packages.urllib3.packages.ordered_dict import OrderedDict
  File ".../lib/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

The issue goes away once the ssl library is included in app.yaml:

libraries:
- name: ssl
  version: latest

But wait, there’s more! The code should now work remotely. However, it still doesn’t work on the development server.

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/request_handler.py", line 225, in handle_interactive_request
    exec(compiled_code, self._command_globals)
  File "<string>", line 3, in <module>
  File ".../lib/requests/api.py", line 68, in get
    return request('get', url, **kwargs)
  File ".../lib/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  [...]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 387, in wrap_socket
    ciphers=ciphers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 141, in __init__
    ciphers)
TypeError: must be _socket.socket, not socket

The problem is GAE has a “whitelist” of select standard libraries. SSL (_ssl, _socket) is not one of them. So, we need to tweak the sandbox environment (dangerous) carefully. The below code uses the standard Python socket library instead of the GAE-provided in the development environment. Modify appengine_config.py:

import os

# Workaround the dev-environment SSL
#   http://stackoverflow.com/q/16192916/893652
if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
    import imp
    import os.path
    from google.appengine.tools.devappserver2.python import sandbox

    sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket']
    # Use the system socket.
    psocket = os.path.join(os.path.dirname(os.__file__), 'socket.py')
    imp.load_source('socket', psocket)

INFO     2015-04-04 06:57:28,449 module.py:737] default: "POST / HTTP/1.1" 200 4
INFO     2015-04-04 06:57:46,868 connectionpool.py:735] Starting new HTTPS connection
    (1): httpbin.org

This solution mostly works, except for non-blocking sockets. I haven’t had a need for that yet :)

References

0: Open issue that is 2 years old

1: http://stackoverflow.com/q/16192916/893652