{
  "description": "Kanat Bekt Blog",
  "url": "https://bekt.net/",
  "posts": [
  
    {
      "title": "AngularJS: Reducing Initial Load Time by Preloading Data",
      "excerpt": "<h2 id=\"background\">Background</h2>\n",
      "content": "<h2 id=\"background\">Background</h2>\n\n<p>Most of the web has transitioned into client-side heavy development using\nvarious JavaScript frameworks. More and more sites are being built in\nthe “Single Page Application” (SPA) fashion. This puts more work on the user’s browser\nand less work on the servers. The shift keeps front-end and back-end development\nseparate and isolated, allowing clients to present information in different ways without\nrelying on server-side rendering. Additionally, most of smartphones these days\nare more powerful than servers that serve thousands of users.</p>\n\n<p>[ <a href=\"https://plnkr.co/edit/og8aqTSLU6IrpaHqX56O\">Source Code</a>\n| <a href=\"https://run.plnkr.co/plunks/og8aqTSLU6IrpaHqX56O\">Demo</a> ]</p>\n\n<p>One of the most common issues with this paradigm is the slow initial load time\nof the page. Typically, the page is loaded very quickly only to show a spinning\nicon for some time until the actual content is loaded. Here’s the diagram\nthat explains the request cycle:</p>\n\n<p><a href=\"//i.imgur.com/nQjRqXe.png\"><img src=\"//i.imgur.com/nQjRqXel.png\" alt=\"\" /></a></p>\n\n<ol>\n  <li>Client requests the page. Server responds with some bootstrapped and\nnon-rendered HTML. (green)</li>\n  <li>Client requests for static assets, including the JavaScript files and HTML templates. (blue)</li>\n  <li>Application makes one or more HTTP calls to fetch various resources. (yellow)</li>\n  <li>The loaded resources are displayed to the user.</li>\n</ol>\n\n<p>These steps are sequential. As one can imagine, the initial load time\ncan be very slow if the user has a slow internet connection and/or the user\nis far away from the website data centers.</p>\n\n<p>By looking at the diagram, let’s assume the <code>api/products/123</code> is the highest\npriority HTTP call. We want to show the result of it as soon as it is ready\nwhile other HTTP calls may still be pending. If we somehow include the response of\nthis endpoint with the initial HTML page or <code>app.js</code>, the extra HTTP call\nwould not be necessary.</p>\n\n<h2 id=\"angularjs-and-angular-ui-router\">AngularJS and Angular UI Router</h2>\n\n<p><a href=\"https://angularjs.org/\">AngularJS</a> is a popular and somewhat opinionated\nJavaScript framework that is backed by Google.\n(Fun fact: the Angular team sat one floor below my team at Google!)</p>\n\n<blockquote>\n  <p>AngularJS lets you write client-side web applications as if you had a smarter browser.\nIt lets you use good old HTML (or HAML, Jade and friends!) as your template language and\nlets you extend HTML’s syntax to express your application’s components clearly and succinctly.\nIt automatically synchronizes data from your UI (view) with your JavaScript objects (model)\nthrough 2-way data binding.</p>\n</blockquote>\n\n<p>If you are building an <em>AngularJS 1.x</em> app, you probably are already familiar with\n<a href=\"https://github.com/angular-ui/ui-router\">AngularUI Router</a>. It is generally\nmore flexible than the standard <a href=\"https://docs.angularjs.org/api/ngRoute/service/$route\"><code>$route</code></a>\nservice. I suggest looking into it if you have not done so.\nThe example in this post uses AngularUI Router.</p>\n\n<p>A typical application route setup might look something like this:</p>\n\n<pre><code class=\"js\">$stateProvider\n    .state(&#39;main&#39;, {\n        abstract: true,\n        resolve: {\n            user: function (UserService) {\n                return UserService.me();\n            }\n        }\n    .state(&#39;main.product&#39;, {\n        url: &#39;/products/{id}&#39;\n        resolve: {\n            product: function ($stateParams, ItemService) {\n                return ItemService.get($stateParams.id);\n            }\n        }\n</code></pre>\n\n<p>The <code>user</code> and <code>product</code> properties are “resolved” (by making HTTP calls for example)\nbefore the state transition happens.</p>\n\n<h2 id=\"preload-prefetch-data\">Preload (Prefetch) Data</h2>\n\n<p>The solution I propose is simple but somewhat ugly to implement.\nIt minimizes the initial load time, which is the best user experience.</p>\n\n<ol>\n  <li>When the server receives a request to <code>website.com/products/123</code>, it makes\nan API call to <code>api/products/123</code> on behalf of the client.\nThis will be significantly faster than the client calling the API endpoint because\nthe data centers are typically close to each other with much faster network speeds.</li>\n  <li>The result from <code>api/products/123</code> is appended to the original as URL parameter\n(let’s call it <code>_data</code>). The URL change to: <code>website.com/products/123?_data={...}</code></li>\n  <li>The client handles the case when this URL parameter is present.</li>\n</ol>\n\n<p><strong>Important</strong>: In order for the proposed solution to work, the application\nneeds to have\n<a href=\"https://code.angularjs.org/1.4.7/docs/api/ng/provider/$locationProvider#html5Mode\">html5Mode</a>\nenabled. This is because anything after the <code>#</code> portion of the URL is not sent\nto the server (read more: <a href=\"https://en.wikipedia.org/wiki/Fragment_identifier\">fragment identifier</a>).\nThe default hashbang method uses the <code>#</code>. You should have strong reasons if\nyou do not already have html5Mode enabled.</p>\n\n<p>In this scenario, we are only working with <code>api/products/123</code>. The solution\ncan be used in 100 different ways, including but not limited to calling multiple APIs or\nincluding some placeholder data.</p>\n\n<p>First, let’s make that URL parameter an optional parameter (<code>_data</code>).</p>\n\n<pre><code class=\"js\">class Config {\n    constructor($locationProvider, $stateProvider, $urlRouterProvider) {\n        $locationProvider.html5Mode(true);\n\n        $urlRouterProvider.otherwise(&#39;/&#39;);\n\n        $stateProvider\n            .state(&#39;default&#39;, {\n                url: &#39;/&#39;,\n                templateUrl: &#39;main.html&#39;\n            })\n            .state(&#39;item&#39;, {\n                url: &#39;/items/{id}?_data&#39;,\n                controller: &#39;Item as item&#39;,\n                templateUrl: &#39;item.html&#39;,\n                reloadOnSearch: false\n        });\n    }\n}\n</code></pre>\n\n<p>It makes sense to remove the <code>_data</code> parameter from the URL as soon as the\napplication reads so the user does not share or bookmark the full URL\n(and to make the URL look pretty). The <code>reloadOnSearch: false</code> option prevents\nthe state from being reloaded when this happens.</p>\n\n<p>Now, the client logic skips the API call to get product information\nwhen the <code>_data</code> parameter is present.</p>\n\n<pre><code class=\"js\">class Item {\n    constructor($log, $state, $stateParams, ItemService) {\n        this.title = &#39;Loading..&#39;;\n        this._log = $log;\n        this._state = $state;\n        this._stateParams = $stateParams;\n        this._ItemService = ItemService;\n\n        this.init();\n    }\n\n    init() {\n        let context = this;\n\n        this._log.debug(this._stateParams);\n\n        // Pre-fetched data can come as a URL parameter (`_data`).\n        var data = angular.fromJson(this._stateParams._data);\n\n        if (data) {\n            // Remove `_data` parameter from URL.\n            this._state.go(&#39;.&#39;, {_data: null}, {location: &#39;replace&#39;});\n\n            return ready(data);\n        }\n\n        this._ItemService.fetchItem(this._stateParams.id).then(ready, ready);\n\n        function ready(data) {\n            context.title = data.name;\n            context.item = data;\n        }  \n    }\n}\n\nclass ItemService {\n    constructor($timeout) {\n        this._timeout = $timeout;\n    }\n\n    fetchItem(id) {\n      return this._timeout(() =&gt; {\n          return {id: id, name: &#39;TV&#39;, description: &#39;Fetched from the backend&#39;};\n      }, 1500);\n    }\n}\n</code></pre>\n\n<p>This method reduces the number of HTTP requests made by the application.\nHowever, I do not recommend doing this everywhere in the application as it\nintroduces logic code both in the back-end endpoint and in the Angular\napplication.</p>\n\n<p><strong>Full source and demo can be found <a href=\"https://run.plnkr.co/plunks/og8aqTSLU6IrpaHqX56O\">here</a>.</strong></p>\n\n<h2 id=\"conclusion\">Conclusion</h2>\n\n<p>We have looked at one of the biggest concerns against modern client-side heavy\ndevelopment – slower initial load times. We proposed a back-end agnostic solution\nfor the problem in AngularJS that makes use of URL parameters. Code samples for\nboth front-end and back-end have been provided.</p>\n\n<h2 id=\"bonus-flask-back-end-endpoint-handler\">Bonus: Flask Back-end Endpoint Handler</h2>\n\n<p>Below is the quick implementation of step-1 and step-2 of the proposed solution above,\nin <a href=\"http://flask.pocoo.org/\">Flask</a>.\nI’m sure a similar approach can be implemented in other frameworks.</p>\n\n<pre><code class=\"py\">try:\n    from urllib.parse import urlencode\nexcept:\n    from urllib import urlencode\n\nimport re\nimport flask as f\n\napp = f.Flask(__name__)\nITEMS_RE = re.compile(r&#39;^items/(\\d+)&#39;)\n\n@app.route(&#39;/app/&#39;, defaults={&#39;path&#39;: &#39;&#39;})\n@app.route(&#39;/app/&lt;path:path&gt;&#39;)\ndef index(path):\n    &quot;&quot;&quot;Main application entry. Let&#39;s assume our app is served at `/app`.&quot;&quot;&quot;\n    m = ITEMS_RE.match(path)\n    if m and not f.request.args.get(&#39;_data&#39;):\n        data = get_item(int(m.groups()[0]))\n        args = dict(f.request.args)\n        args[&#39;_data&#39;] = f.json.dumps(data)\n        url = &#39;{}?{}&#39;.format(f.request.base_url, urlencode(args, doseq=True))\n        return f.redirect(url)\n    return &#39;return index.html&#39;\n\n\ndef get_item(ident):\n    return {&#39;id&#39;: ident, &#39;name&#39;: &#39;fake&#39;, &#39;description&#39;: &#39;fake api return&#39;}\n\n\nif __name__ == &#39;__main__&#39;:\n    app.run(debug=True)\n</code></pre>\n",
      "tags": null,
      "published": "2015-12-28 00:00:00 +0000",
      "url": "https://bekt.net/p/angular-prefetch/"
    },
  
    {
      "title": "Handling Unexpected WTForms Default Values",
      "excerpt": "<p>The <a href=\"https://github.com/wtforms/wtforms\">WTForms</a> library is great for\nworking with forms and validations. However, it does not work well with\nJSON form data. Specifically, fields that were not provided in the request\nare assigned default values. This causes problems when you expect <code>PATCH</code>-like\nrequests. Example:</p>\n",
      "content": "<p>The <a href=\"https://github.com/wtforms/wtforms\">WTForms</a> library is great for\nworking with forms and validations. However, it does not work well with\nJSON form data. Specifically, fields that were not provided in the request\nare assigned default values. This causes problems when you expect <code>PATCH</code>-like\nrequests. Example:</p>\n\n<pre><code class=\"py\">import wtforms\n\nclass MyBaseForm(wtforms.Form):\n    pass\n\nclass FooForm(MyBaseForm):\n    foo = wtforms.StringField(&#39;Foo&#39;)\n    bar = wtforms.IntegerField(&#39;Bar&#39;)\n    is_baz = wtforms.BooleanField(&#39;IzBaz&#39;)\n\nfd = DummyMultiDict({&#39;foo&#39;: &#39;ayy&#39;})\nf = FooForm(fd)\n\nprint(f.data)\n\n# # # # #\nclass DummyMultiDict(dict):\n    def getlist(self, key):\n        v = self[key]\n        return v if isinstance(v, (list, tuple)) else [v]\n</code></pre>\n\n<p>The above produces:</p>\n\n<pre><code>{'is_baz': False, 'foo': 'ayy', 'bar': None}\n</code></pre>\n\n<p>This is not necessarily what you want when working with REST APIs or PATCH-like\nrequests. (Again, wtforms is not really meant for that). The default values your\ncode expects may be different from the default values wtforms assigns.</p>\n\n<p>Suppose you have:</p>\n\n<pre><code class=\"py\">def save(foo=&#39;No name&#39;, bar=42, is_baz=True):\n    return db.save(Object(foo=foo, bar=bar, is_baz=is_baz))\n\nfd = DummyMultiDict({&#39;foo&#39;: &#39;ayy&#39;})\nf = Foo(fd)\n\nsave(**f.data)\n</code></pre>\n\n<p>In this case, you would expect a new entry in the “database” to be\n<code>Object(foo='ayy', bar=42, is_baz=True)</code>. But the actual entry saved is\n <code>Object(foo='ayy', bar=None, is_baz=False)</code>.\n All your unit tests fail, your users are confused, and your manager is mad.</p>\n\n<p>Here is a small wrapper around <code>wtforms.Form</code> to better handle the above problem. (There is also <a href=\"https://github.com/kvesteri/wtforms-json\">wtforms-json</a>,\nbut its interface is quite different and feels wrong.)</p>\n\n<h2 id=\"jform\">JForm</h2>\n\n<pre><code class=\"py\">import wtforms\n\n\nclass JForm(wtforms.Form):\n\n    def process(self, *args, **kwargs):\n        if args:\n            formdata = args[0]\n        else:\n            formdata = kwargs.get(&#39;formdata&#39;, None)\n        self._formdata = formdata\n        super(JForm, self).process(*args, **kwargs)\n\n    @property\n    def data(self):\n        &quot;&quot;&quot;Returns form data with fields that were not in request popped.&quot;&quot;&quot;\n        d = super(JForm, self).data\n        if self._formdata is None:\n            return d\n        keys = d.keys()\n        keys = keys if isinstance(keys, list) else list(keys)\n        for k in keys:\n            if k not in self._formdata:\n                del d[k]\n        return d\n</code></pre>\n\n<p>When <code>form.data</code> is requested, <code>JForm</code> ignores any property that was not\nincluded in the original form creation.</p>\n\n<h2 id=\"usage\">Usage:</h2>\n\n<pre><code class=\"py\">import wtforms\n\nclass MyBaseForm(JForm):\n    pass\n\nclass FooForm(MyBaseForm):\n    foo = wtforms.StringField(&#39;Foo&#39;)\n    bar = wtforms.IntegerField(&#39;Bar&#39;)\n    is_baz = wtforms.BooleanField(&#39;IzBaz&#39;)\n\nfd = DummyMultiDict({&#39;foo&#39;: &#39;ayy&#39;})\nf = Foo(fd)\n\nprint(f.data) # {&#39;foo&#39;: &#39;ayy&#39;}\n</code></pre>\n\n<p>Happy coding. It also works nicely with other wrappers such as <a href=\"https://github.com/lepture/flask-wtf\">flask-wtf</a>. Simply change to</p>\n\n<pre><code>class MyBaseForm(JForm, flask_wtf.Form):\n    pass\n</code></pre>\n\n<p>Update: I did not want to publish this as a pip-package because … I is lazy.</p>\n",
      "tags": null,
      "published": "2015-11-13 00:00:00 +0000",
      "url": "https://bekt.net/p/wtforms-patch/"
    },
  
    {
      "title": "Sharing AWS Credentials Between Users On a Docker Container",
      "excerpt": "<p><strong>TL;DR</strong>:\nAWS CLI configurations and credentials are stored in the user’s home directory by\ndefault (<code>~/.aws</code>). This becomes a problem when other users such as\n<code>root</code>, <code>www-data</code>, <code>nobody</code>, or cron jobs need access to these credentials.\nThis post shows how to get around this in a Docker environment.</p>\n",
      "content": "<p><strong>TL;DR</strong>:\nAWS CLI configurations and credentials are stored in the user’s home directory by\ndefault (<code>~/.aws</code>). This becomes a problem when other users such as\n<code>root</code>, <code>www-data</code>, <code>nobody</code>, or cron jobs need access to these credentials.\nThis post shows how to get around this in a Docker environment.</p>\n\n<p><a href=\"https://www.docker.com/\">Docker</a> is a great application containment tool for\nbuilding and shipping software of any type. You can learn more about Docker\n<a href=\"https://www.docker.com/whatisdocker\">here</a>. We love and use Docker at\n<a href=\"https://www.humanlink.co\">Humanlink</a> with\n<a href=\"http://aws.amazon.com/elasticbeanstalk/\">AWS Elastic Beanstalk</a>.</p>\n\n<p>Being on Elastic Beanstalk means utilizing other amazing Amazon services as well.\n<a href=\"http://aws.amazon.com/cli/\">AWS CLI</a> and <a href=\"https://github.com/boto/boto\">boto</a>\n(for Python developers) are the de-facto tools for interacting with AWS.</p>\n\n<p>There are a few ways to provide credentials to these tools.\nOn production, it is a good idea to serve them as environment\nvariables (such as <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code>).\nDuring development, however, it is recommended that these values are set via\nthe <code>aws configure</code> command, which in turn places config files under <code>~/.aws/</code>.</p>\n\n<p>I was planning on having a full-blown example with detailed explanations,\nbut that quickly became a long post and lost its focus.</p>\n\n<p>In short, we do not want to pollute the Dockerfile just for the development environment. When running the Docker image locally, we can mount the <code>~/.aws</code> directory AND set the <code>$HOME</code> environment variable. Otherwise, <code>$HOME</code> defaults\nto <code>/root</code> which causes permission problems for non-root users.</p>\n\n<p>Since mounted file permissions are copied over to the Docker container, we\nneed to give read permissions to everyone (on your host machine).</p>\n\n<pre><code class=\"bash\">$ chmod 644 ~/.aws/*\n</code></pre>\n\n<p>We can now run the container:</p>\n\n<pre><code class=\"bash\">$ docker run -it --rm \\\n  -e &quot;HOME=/home&quot; \\\n  -v $HOME/.aws:/home/.aws \\\n  myapp\n</code></pre>\n\n<p>Now the credentials are available system-wide in the Docker container for any\nuser to use and communicate with AWS.</p>\n",
      "tags": null,
      "published": "2015-07-18 00:00:00 +0000",
      "url": "https://bekt.net/p/docker-aws-credentials/"
    },
  
    {
      "title": "Automatically Download the Latest App Engine SDK",
      "excerpt": "<p><strong>TL;DR</strong>:\nGoogle doesn’t provide a permanent link to the latest SDK;\nupdate check endpoint is not reliable;\nI provide a snippet (<a href=\"https://gist.github.com/Bekt/7cb68b12674b282c8d78\">view</a>)\nto work around this.</p>\n",
      "content": "<p><strong>TL;DR</strong>:\nGoogle doesn’t provide a permanent link to the latest SDK;\nupdate check endpoint is not reliable;\nI provide a snippet (<a href=\"https://gist.github.com/Bekt/7cb68b12674b282c8d78\">view</a>)\nto work around this.</p>\n\n<p>Google provides an SDK for interacting with App Engine. <code>dev_appserver.py</code>\nsimulates the App Engine Python runtime environment locally.\n<code>appcfg.py</code> is used to interact with App Engine to do tasks such as deploying\na new version of the application.</p>\n\n<p>For continuous deployment / integration scripts such as those for\n<a href=\"https://circleci.com/docs/deploy-google-app-engine\">CircleCI</a>,\n<a href=\"https://github.com/travis-ci/dpl/issues/94\">Travis CI</a>,\nor <a href=\"https://jenkins-ci.org\">Jenkins CI</a>, it is usually necessary to download\nthe latest version of the Google App Engine SDK. Unfortunately, Google does not\nprovide a permanent link to the latest version for easy download.\nIt seems like a basic need. I talk about how to programatically get the latest\nSDK in this post.</p>\n\n<h2 id=\"updatecheck-endpoint\">UpdateCheck Endpoint</h2>\n\n<p>In <code>appengine/tools/sdk_update_checker.py</code>, I found the API endpoint for\nchecking for updates to the SDK: <code>https://appengine.google.com/api/updatecheck</code>.</p>\n\n<pre><code class=\"bash\">$ curl https://appengine.google.com/api/updatecheck\n\nrelease: &quot;1.9.19&quot;\ntimestamp: 1424415497\napi_versions: [&#39;1&#39;]\nsupported_api_versions:\n  python:\n    api_versions: [&#39;1&#39;]\n  python27:\n    api_versions: [&#39;1&#39;]\n  go:\n    api_versions: [&#39;go1&#39;]\n  java7:\n    api_versions: [&#39;1.0&#39;]\n</code></pre>\n\n<p>Great, this is all we need. Not so fast! But it is useful.</p>\n\n<h2 id=\"sdk-download-url\">SDK Download URL</h2>\n\n<p>On the App Engine SDK <a href=\"https://cloud.google.com/appengine/downloads\">downloads</a>,\nthe only link to the latest SDK is something like:</p>\n\n<p><code>https://storage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.20.zip</code></p>\n\n<p>Notice the version number is 1.9.20, not 1.9.19.</p>\n\n<p>Using the Google Cloud Storage API, we can list all the files in the\n<code>appengine-sdks/featured</code> bucket:</p>\n\n<p><code>https://www.googleapis.com/storage/v1/b/appengine-sdks/o?prefix=featured</code></p>\n\n<h2 id=\"fetch-the-latest-sdk\">Fetch the Latest SDK</h2>\n\n<p>Given the two points from above, we can can automatically check for the latest SDK\nversion and download the zip with that version.</p>\n\n<pre><code class=\"bash\"># script.bash\n# DON&#39;T COPY-PASTE, READ BELOW.\n\n#!/usr/bin/env bash\n\nAPI_CHECK=https://appengine.google.com/api/updatecheck\nSDK_VERSION=$(curl -s $API_CHECK | awk -F &#39;\\&quot;&#39; &#39;/release/ {print $2}&#39;)\nSDK_URL=https://storage.googleapis.com/appengine-sdks/featured/google_appengine_$SDK_VERSION.zip\n\nfunction download_sdk {\n  echo &quot;&gt;&gt;&gt; Downloading...&quot;\n  curl -fo $HOME/gae.zip $SDK_URL || exit 1\n  unzip -qd $HOME $HOME/gae.zip\n}\n\nfunction upload {\n  $HOME/google_appengine/appcfg.py \\\n      --oauth2 update my-app\n}\n\ndownload_sdk\nupload\n</code></pre>\n\n<p>Easy, right? Ehhhh.</p>\n\n<pre><code class=\"bash\">./script.bash\n\n&gt;&gt;&gt; Downloading...\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n\ncurl: (22) The requested URL returned error: 404 Not Found\n</code></pre>\n\n<p>Apparently, the UpdateCheck endpoint does not always have the latest\nversion. Like wut. From my experience, it can take anywhere from 1-7 days for the\nendpoint to return the latest version.\nAdditionally, once a new version becomes available,\nprevious download links become unavailable. The App Engine team acknowledged the <a href=\"https://code.google.com/p/googleappengine/issues/detail?id=11604\">issue</a>\nfive months ago but still has not addressed it.</p>\n\n<h2 id=\"revisiting-sdk-download-url\">Revisiting SDK Download URL</h2>\n\n<p>Well, obviously at this point the only way to get the latest SDK version is to\nparse the HTML content of the download page. We could also look through all\nthe files in the <code>featured</code> bucket and find the one correct file to download.\nHowever, nobody should do that!\nSince we don’t necessarily need the latest version to simply deploy the project,\nwe can fallback to a slightly older SDK version (the one UpdateCheck returns).</p>\n\n<p>As indicated on the downloads page, deprecated SDK downloads can be found at\nthe following link:</p>\n\n<p><code>https://console.developers.google.com/m/cloudstorage/b/appengine-sdks/o/deprecated/1919/google_appengine_1.9.19.zip</code></p>\n\n<p>which I found out is the same as:</p>\n\n<p><code>https://storage.googleapis.com/appengine-sdks/deprecated/1919/google_appengine_1.9.19.zip</code></p>\n\n<h2 id=\"new-scriptbash\">New script.bash</h2>\n\n<p>So now we can write a script that fall-backs to the deprecated SDK URL if the first\nURL does not work.</p>\n\n<pre><code class=\"bash\"># script.bash\n\n#!/usr/bin/env bash\n\nAPI_CHECK=https://appengine.google.com/api/updatecheck\nSDK_VERSION=$(curl -s $API_CHECK | awk -F &#39;\\&quot;&#39; &#39;/release/ {print $2}&#39;)\n# Remove the dots.\nSDK_VERSION_S=${SDK_VERSION//./}\n\nSDK_URL=https://storage.googleapis.com/appengine-sdks/\nSDK_URL_A=&quot;${SDK_URL}featured/google_appengine_${SDK_VERSION}.zip&quot;\nSDK_URL_B=&quot;${SDK_URL}deprecated/$SDK_VERSION_S/google_appengine_${SDK_VERSION}.zip&quot;\n\nfunction download_sdk {\n  echo &quot;&gt;&gt;&gt; Downloading...&quot;\n  curl -fo $HOME/gae.zip $SDK_URL_A || \\\n      curl -fo $HOME/gae.zip $SDK_URL_B || \\\n      exit 1\n  unzip -qd $HOME $HOME/gae.zip\n}\n\nfunction upload {\n  echo &quot;&gt;&gt;&gt; Deploying...&quot;\n  $HOME/google_appengine/appcfg.py \\\n      --oauth2 update my-app\n}\n\ndownload_sdk\nupload\n</code></pre>\n\n<p>If we run the script again, it works as expected:</p>\n<ul>\n  <li>Tries to download from the <code>features</code> folder first</li>\n  <li>Otherwise, tries to download from the <code>deprecated</code> folder</li>\n</ul>\n\n<h2 id=\"conclusion\">Conclusion</h2>\n\n<p>We came up with <code>script.bash</code> that automatically downloads the “latest”\nGoogle App Engine Python/PHP SDK to make life easier. This is especially useful\nduring automated builds and deployments.</p>\n\n<h3 id=\"references\">References</h3>\n<p>0: https://code.google.com/p/googleappengine/issues/detail?id=11604</p>\n",
      "tags": null,
      "published": "2015-05-20 00:00:00 +0000",
      "url": "https://bekt.net/p/gae-sdk/"
    },
  
    {
      "title": "App Engine and SSL",
      "excerpt": "<p>Google App Engine is a great platform for getting things done quickly.\nHowever, it can be very unpleasant to work with due to its sandboxed environment\nand close source code. Basic needs such as installing third-party\nlibraries can be tricky to install as well.</p>\n",
      "content": "<p>Google App Engine is a great platform for getting things done quickly.\nHowever, it can be very unpleasant to work with due to its sandboxed environment\nand close source code. Basic needs such as installing third-party\nlibraries can be tricky to install as well.</p>\n\n<p>Getting one of the most\npopular python libraries, <a href=\"http://docs.python-requests.org/en/latest/\">python-requests</a>,\nwas particularly tricky to get it running and working with SSL connections.\nI’ll walk through how I fixed the issue.</p>\n\n<p>Start by adding the library to the project:</p>\n\n<pre><code class=\"bash\"># From project root.\npip install -t lib/ requests\n</code></pre>\n\n<p>The above command pip-installs the <code>requests</code> library into the <code>lib</code>\ndirectory. This is where all the third-party libraries can be placed.</p>\n\n<p>Now we need to let App Engine know about this. Create or modify the file\n<code>appengine_config.py</code> in the root of the project.</p>\n\n<pre><code class=\"python\">from google.appengine.ext import vendor\nvendor.add(&#39;lib&#39;)\n</code></pre>\n\n<p><code>appengine_config.py</code> runs when a new instance is created. <code>vendor.add</code> adds the specified path to <code>$PYTHONPATH</code>.</p>\n\n<p>At this point, most third-party libraries work just fine. However,\nthere’s a bit of work that needs to be done to get <code>requests</code> working.</p>\n\n<p>Head over to http://localhost:8000/console and execute:</p>\n\n<pre><code class=\"python\">import requests\n\nr = requests.get(&#39;https://httpbin.org/status/200&#39;)\nprint(r.status_code)\n</code></pre>\n\n<p>In a normal Python environment, the code executes just fine printing a\n200 status. But on GAE, the following exception occurs:</p>\n\n<pre><code class=\"python\">Traceback (most recent call last):\n  File\n&quot;/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/request_handler.py&quot;,\nline 225, in handle_interactive_request\n    exec(compiled_code, self._command_globals)\n  File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;\n  File &quot;.../lib/requests/__init__.py&quot;, line 58, in &lt;module&gt;\n    from . import utils\n  File &quot;.../lib/requests/utils.py&quot;, line 26, in &lt;module&gt;\n    from .compat import parse_http_list as _parse_list_header\n  File &quot;.../lib/requests/compat.py&quot;, line 42, in &lt;module&gt;\n    from .packages.urllib3.packages.ordered_dict import OrderedDict\n  File &quot;.../lib/requests/packages/__init__.py&quot;, line 95, in load_module\n    raise ImportError(&quot;No module named &#39;%s&#39;&quot; % (name,))\nImportError: No module named &#39;requests.packages.urllib3&#39;\n</code></pre>\n\n<p>The issue goes away once the\n<a href=\"https://cloud.google.com/appengine/docs/python/sockets/ssl_support\">ssl</a> library is included in <code>app.yaml</code>:</p>\n\n<pre>\n<code class=\"nohighlight\">libraries:\n- name: ssl\n  version: latest\n</code></pre>\n\n<p>But wait, there’s more! The code should now work remotely.\nHowever, it still doesn’t work on the development server.</p>\n\n<pre><code class=\"python\">Traceback (most recent call last):\n  File &quot;/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/request_handler.py&quot;, line 225, in handle_interactive_request\n    exec(compiled_code, self._command_globals)\n  File &quot;&lt;string&gt;&quot;, line 3, in &lt;module&gt;\n  File &quot;.../lib/requests/api.py&quot;, line 68, in get\n    return request(&#39;get&#39;, url, **kwargs)\n  File &quot;.../lib/requests/api.py&quot;, line 50, in request\n    response = session.request(method=method, url=url, **kwargs)\n  [...]\n  File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py&quot;, line 387, in wrap_socket\n    ciphers=ciphers)\n  File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py&quot;, line 141, in __init__\n    ciphers)\nTypeError: must be _socket.socket, not socket\n</code></pre>\n\n<p>The problem is GAE has a “whitelist” of select standard libraries.\nSSL (_ssl, _socket) is not one of them.\nSo, we need to tweak the sandbox environment (dangerous) carefully.\nThe below code uses the standard Python socket library instead of the GAE-provided\nin the development environment. Modify <code>appengine_config.py</code>:</p>\n\n<pre><code class=\"python\">import os\n\n# Workaround the dev-environment SSL\n#   http://stackoverflow.com/q/16192916/893652\nif os.environ.get(&#39;SERVER_SOFTWARE&#39;, &#39;&#39;).startswith(&#39;Development&#39;):\n    import imp\n    import os.path\n    from google.appengine.tools.devappserver2.python import sandbox\n\n    sandbox._WHITE_LIST_C_MODULES += [&#39;_ssl&#39;, &#39;_socket&#39;]\n    # Use the system socket.\n    psocket = os.path.join(os.path.dirname(os.__file__), &#39;socket.py&#39;)\n    imp.load_source(&#39;socket&#39;, psocket)\n</code></pre>\n\n<pre><code class=\"nohighlight\">INFO     2015-04-04 06:57:28,449 module.py:737] default: &quot;POST / HTTP/1.1&quot; 200 4\nINFO     2015-04-04 06:57:46,868 connectionpool.py:735] Starting new HTTPS connection\n    (1): httpbin.org\n</code></pre>\n\n<p>This solution mostly works, except for non-blocking sockets.\nI haven’t had a need for that yet :)</p>\n\n<h3 id=\"references\">References</h3>\n<p>0: <a href=\"https://code.google.com/p/googleappengine/issues/detail?id=9246\">Open issue that is 2 years old</a></p>\n\n<p>1: http://stackoverflow.com/q/16192916/893652</p>\n",
      "tags": null,
      "published": "2015-04-05 00:00:00 +0000",
      "url": "https://bekt.net/p/gae-ssl/"
    }
  
  ]
}
