App Engine and SSL05 Apr 2015 Tags: gae Suggest changes
Google App Engine is a great platform for getting things done quickly. However, it can be very unpleasant to work with due to its sandboxed environment and close source code. Basic needs such as installing third-party libraries can be tricky to install as well.
Getting one of the most popular python libraries, python-requests, was particularly tricky to get it running and working with SSL connections. I’ll walk through how I fixed the issue.
Start by adding the library to the project:
# From project root. pip install -t lib/ requests
The above command pip-installs the
requests library into the
directory. This is where all the third-party libraries can be placed.
Now we need to let App Engine know about this. Create or modify the file
appengine_config.py in the root of the project.
from google.appengine.ext import vendor vendor.add('lib')
appengine_config.py runs when a new instance is created.
vendor.add adds the specified path to
At this point, most third-party libraries work just fine. However,
there’s a bit of work that needs to be done to get
Head over to http://localhost:8000/console and execute:
import requests r = requests.get('https://httpbin.org/status/200') print(r.status_code)
In a normal Python environment, the code executes just fine printing a 200 status. But on GAE, the following exception occurs:
Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/request_handler.py", line 225, in handle_interactive_request exec(compiled_code, self._command_globals) File "<string>", line 1, in <module> File ".../lib/requests/__init__.py", line 58, in <module> from . import utils File ".../lib/requests/utils.py", line 26, in <module> from .compat import parse_http_list as _parse_list_header File ".../lib/requests/compat.py", line 42, in <module> from .packages.urllib3.packages.ordered_dict import OrderedDict File ".../lib/requests/packages/__init__.py", line 95, in load_module raise ImportError("No module named '%s'" % (name,)) ImportError: No module named 'requests.packages.urllib3'
The issue goes away once the
ssl library is included in
libraries: - name: ssl version: latest
But wait, there’s more! The code should now work remotely. However, it still doesn’t work on the development server.
Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/request_handler.py", line 225, in handle_interactive_request exec(compiled_code, self._command_globals) File "<string>", line 3, in <module> File ".../lib/requests/api.py", line 68, in get return request('get', url, **kwargs) File ".../lib/requests/api.py", line 50, in request response = session.request(method=method, url=url, **kwargs) [...] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 387, in wrap_socket ciphers=ciphers) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 141, in __init__ ciphers) TypeError: must be _socket.socket, not socket
The problem is GAE has a “whitelist” of select standard libraries.
SSL (_ssl, _socket) is not one of them.
So, we need to tweak the sandbox environment (dangerous) carefully.
The below code uses the standard Python socket library instead of the GAE-provided
in the development environment. Modify
import os # Workaround the dev-environment SSL # http://stackoverflow.com/q/16192916/893652 if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'): import imp import os.path from google.appengine.tools.devappserver2.python import sandbox sandbox._WHITE_LIST_C_MODULES += ['_ssl', '_socket'] # Use the system socket. psocket = os.path.join(os.path.dirname(os.__file__), 'socket.py') imp.load_source('socket', psocket)
INFO 2015-04-04 06:57:28,449 module.py:737] default: "POST / HTTP/1.1" 200 4 INFO 2015-04-04 06:57:46,868 connectionpool.py:735] Starting new HTTPS connection (1): httpbin.org
This solution mostly works, except for non-blocking sockets. I haven’t had a need for that yet :)