Erik Zaadi

The tales of a half breed devops and web developer

Handle Proxy 404 in Nginx

| Comments

In our SPA era, when you get a 40x or 500x (Oy vei) error from your proxied backend, you typically want to display a static part of your SPA.

To do this, we can usual a small but usefull nugget from nginx:

HERE BE DRAGONS
1
proxy_intercept_errors on;

This ensures that if the proxied backend returns an error status, nginx will be the one showing the error page.

We need to tell nginx to handle the 404:

Trap 404
1
2
3
4
5
server {
  #...
  error_page 404 = @404;
  #...
}

Notice the use of the @404 location alias, it allows us to redirect as we wish:

404 Location hack
1
2
3
  location @404 {
    return 302 /app-path-to-404;
  }

Here’s a full sample:

Handling proxied 404 in nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
upstream your-api-cluster {
  server localhost:1337 max_fails=3  fail_timeout=10s ;
  keepalive 16;
}


server {
  root /path/to/your/static/site/;

  location @404 {
    return 302 /app-path-to-404;
  }

  error_page 404 = @404;

  location / {
    autoindex off;
    expires 1h;

    #...
  }

  location /api{
    proxy_intercept_errors on;
    proxy_pass http://your-api-cluster/api;
    proxy_redirect   off;
    proxy_set_header Host            $host;
    proxy_set_header X-Real-IP       $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    #...
  }
}

Enjoy!

Long Time No Spamming

| Comments

TL;DR – Yet another ‘didn’t blog in a while due to ENTER EXCUSES HERE’ post..

I’ve moved on

I now work at BigPanda since March, and until now, I didn’t make time to write here..

I did get a chance to write a blog post at the BigPanda Blog about how we use vagrant, which was fun writing.

In the end there are only two reasons I stopped blogging.

  1. Number one: Refactor Syndrome

    REFACTOR ALL THE CODE

    Since the last post, I’ve been trying to rewriting this blog in:

    Here’s another MEME describing how I feel about my progress refactoring:

    Oh cool, this seems simpllllllleeeaaaaaahhhh

  2. Now for the real reason:

Cleaning Up Jenkins Jobs

| Comments

Jenkins rules, but…

It’s a Java XML consuming monster.

Once you get up to a large number of jobs (We are near 300), it gets a bit tough, especially when restarting.

So, every once in a while, I cleanup old unused jobs that are left over.

delete_non_active_jobs.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import urllib2
import os
from shutil import rmtree

def get_active_jobs():
   request = urllib2.Request('http://localhost/api/python?tree=jobs[name]')
   opener = urllib2.build_opener()
   json = opener.open(request)
   return [job['name'] for job in eval(json.read())['jobs']]

def get_jobs_dir():
   return os.walk('.').next()[1]


if __name__ == '__main__':
    jobs = get_active_jobs()
    jobs_in_dir = get_jobs_dir()
    jobs_to_delete = sorted([job for job in jobs_in_dir if job not in jobs]
    for job in jobs_to_delete:
        print "Deleting: '%s'" % job
        rmtree(os.path.abspath(os.path.join(os.curdir, job)))
‘Delete them allz’
1
2
3
4
cd ${JENKINS_HOME}
#note Jenkins must be up
python delete_non_active_jobs.py
service jenkins restart

Assign an Application to All Desktops in Applescript

| Comments

Back to basics, applescript

It’s been a while since I wrote applescript.

Although the capabilities are truly amazing, I never really connected to the tell syntax, a bit to verbose for me.

Anyhow, I’m using slate for automagically position apps when I change monitors (amongs other things), and I needed a way to pin certain applications to all desktops when using multiple monitors, or to a specific desktop when using just one monitor.

You can run this script from the shell (if you chmod it):

Usage
1
assignTo this|all Chrome Adium "App with spaces"

Source

assignToassignTo@github
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
#!/usr/bin/osascript
script assignTo
  on toAllDesktops(appsList)
      forAllChats("All Desktops", appsList)
  end toAllDesktops
  
  
  on toThisDesktop(appsList)
      forAllChats("This Desktop", appsList)
  end toThisDesktop
  
  on forAllChats(toSet, appsList)
      repeat with appName in appsList
          tell application "System Events" to tell UI element appName of list 1 of process "Dock"
              perform action "AXShowMenu"
              click menu item "Options" of menu 1
              click menu item toSet of menu 1 of menu item "Options" of menu 1
          end tell
      end repeat
  end forAllChats
end script
on run argv
    tell application "Finder"
        set scriptName to name of file (path to me) as text
    end tell
  set usage to scriptName & "
 Usage: all|this Application1 ApplicationN"
  if (count of argv) is 0 then
      return usage
  end if
  set what to item 1 of argv
  set apps to rest of items of argv
  if (count of apps) is 0 then
      return usage & "
 At least one application is required!"
  end if
  tell assignTo
      if what is "all" then
          toAllDesktops(apps)
      else if what is "this" then
          toThisDesktop(apps)
      else
          return usage
      end if
  end tell
end run

Enjoy

Return of the Mac

| Comments

After two wonderful years of Linux, mostly with Ubuntu (with a small visit to Fedora), I had the opportunity to switch to a Mac at work. It’s an old 2009 MBP inherited by Vuduchild who left us lately.

At my previous workplace, I had an iMac for about four years, and I’m ashamed to admit that about two and a half years out of those four, I was mainly on the dark windows side of the iMacs bootcamp (NEVER AGAIN). As for the last year and a half out of those four, I started to use Mac OS exclusivly, fell for git and vim, but still developed mainly DONT.NIET apps.

When I started to work at my current workplace, I switched to Linux, and more importantly, I started working with Python and learning a whole lot of *nix os, bash, zsh, you name it.

I got used to tweaking every last detail to my specific needs. Installing Ubuntu only from the bare minimal text mode, and only installing Gnome 3 and the parts I liked.

I started to “live” almost only in two worlds, the terminal (Helloooo tmux) and the browser.

Everything was in instant reach either with an apt-get, ppa or worst(!) case scenario, I’d download and compile it.

And things worked. Even worked smoothly.

Multiple screens worked just as I wanted in Gnome 3 (the secondary screen sticks apps to stay when you switch workspace).

Sure I had some problems, but nothing I wasn’t able to fix.

So why switch back to a Mac?

Curiosity and the fact that it’s shiny.

It’s that simple, I’m a tech geek that could not possibly say no to a new (old) toy.

The facts

When I left the Apple world, I had just installed “Snow Leopard” (After using “Leopard” mostly). I remember there was a new shiny thing called (home)brew I wanted to check out but never got to.

Now we’re up to “Mountain Lion”. Good thing I didn’t join back on {Grumpy,Nyan} cat. There’s an App Store (!).

The good

It’s smooth. Almost to smooth.

Brew and Brew cask Makes life a bliss. Hello apt-get. Never download dmg => Drag to applications again.

Might sound silly, but I’ve missed Google Chrome Canary.

Finally my Magic Trackpad works flawlessly, with VERY smooth gestures.

The Adium Icon. I know, it’s simply awesome.

The MEH

I miss the good old workspaces, having four explicit ones. That’s the way I configured Gnome 3.

Apple, it’s been two years, Y U NO PROPER WINDOW POSITION AND SIZE HANDLING?? (brew cask install slate fixes some of that though).

Montain Lions fullscreen feature isn’t really that helpful.

The WTF

I had to set LC_LANG? REALLYZ? Why did it take me two days to understand that this is what ruins my vim powerline in tmux?!?!?!

Multiple screen handling. Nuff said.

The basics that just doesn’t work. readline -f should work, Y U NO GNU?

The worst

Hey, I want to develop something –> sure no problem.

Download a 1.7 gb download of Xcode, after it’s done, open it, find the downloads tab and click command line tools, which takes some time to download as well.

Then you can install homebrew, and your life is shiny again.

It still annoys me that I need to walk around with a VGA adaptor, HDMI adaptor etc.

Da Summaryz

I came back to the Mac world a different user thanks to Linux.

It was still worth it moving back.

"Return of the mac"

PS, The handcuffs are rather symbolic for moving back to a close{d,r} system..

FDD - Fika Driven Development

| Comments

FDD FTW

There are many different type of development methodologies. TDD, BDD, what have you.

You also got non development centric methodologies such as Pomodoro, which help you make the most of your time.

At our group at work, we have fine tuned a brand new and shiny methodology called:

“FDD” – Fika Driven Development

20120515_114703

Fika is a Swedish custom imported by yours truly.

Every day, at 16:00, it’s Fika O’clock, usually announced by SHOUTING in ascii in our IRC channel.

During the Fika, we have coffee in our kitchen, with something sweet like cookies on the side, and we talk.

The Fika exposes the following crucial exercises we nerds usually don’t use on a daily bases.

  1. We socialize face to face
  2. Talk NOT about work
  3. We actually move physically

In addition, we drink coffee, which is always important.

On a side note, did you know that Sweden is the top of the world in coffee consumption?

We also have morning Fika(s), but this works in a more distributed manner, usually in two or three batches according to arrival times.

We’ve found that using FDD (FTW), we’re more focused, more relaxed.

20130124_162258

Try FDD and let us know how it affects you!

Nose-rapido - a Rapid Feedback Plugin for Nosetests

| Comments

I use a tmux + vim development environment which I find really productive. When hacking on python projects, I like to have a tmux window open with nosetests logs, typically using nose-watch.

Every now and then after saving, I’d switch to that window to see how the tests are doing.

I wanted something a bit more small that would give me immediate feedback in my main window that I use, which is of course the vim window. The feedback should on one hand not take up to much screen estate, nor disrupt my vim-fu focus.

Enter nose-rapido.

nose-rapido is a nosetests plugin, that does one small and simply thingy: It fills your terminal with green (or optionally blue for the colorblinded) or red color, according to your nosetests result.

To install
1
pip install nose-rapido
Usage
1
nosetests --with-rapido
Usage with nose-watch
1
nosetests --with-rapido --with-watch
Changing successfull color to blue
1
nosetests --with-rapido --rapido-blue

E.g:

"Tests passing"

And if you fail your tests:

"Tests failing"

With blue instead of green:

"Tests success with blue"

Enjoy!

Auto Auto Reloading iPython Modules

| Comments

You all use ipython, it’s an awesome tool.

However, it’s kind of annoying to exit and re-enter ipython when you change your python files.

There’s an extension built into ipython called autoreload which can be loaded by entering:

Loading Auto Reload
1
2
%load_ext autoreload
%autoreload 2

Now whenever you edit your file, your objects will be updated in ipython without the need to close and reopen ipython.

But entering those lines each time you pop open ipython isn’t fun either.

Enter ipython profiles!

Create iPython Profile
1
2
3
4
5
6
7
8
9
10
ipython profile create
cat << EOF >> ~/.config/ipython/profile_default/ipython_config.py

#w00t : taken from erikzaadi.com!
 
c.InteractiveShellApp.exec_lines = []
c.InteractiveShellApp.exec_lines.append('%load_ext autoreload')
c.InteractiveShellApp.exec_lines.append('%autoreload 2') 
 
EOF

And BOOM, you’re auto auto reloading python modules L I K E A B O S S !!1

Fast Remote Editing With Vim

| Comments

UPDATED: bash script now even more cool!

There’s a feature in vim of editing files over scp, built in since vim 7.1 (originially the now baked in netrw plugin).

This feature uses scp to copy a local version of the remote file over scp, edit it with vim, and with each save connect via scp and save it to the remote location.

This allows you to edit remote files with your own tailored vim instance (plugin galore!).

The problem is the connection is not reused, and it’s really slow when vim (scp) connects for each write.

To solve that, save this script and chmod a+x it..

vim-scp take two
1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
echo "vim-scp FTW"
if [ $# -ne 2 ]; then
    echo "usage : `basename $0` user@host /path"
    exit 1
fi
COMMAND="ssh $1 -f -N -o ControlMaster=auto -o ControlPath=/tmp/%r@%h:%p"
echo "opening ssh tunnel.."
$COMMAND || exit $?
echo "ssh tunnel active, opening vim.."
vim scp://$1$2
echo "closing ssh tunnel.."
ps -ef | grep "$COMMAND" | grep -v grep | awk '{print $2}' | xargs kill -9
echo "Great Success!"

This script opens a ssh connection without running any command in the background using ssh -f -N.

Then the script calls vim with the wanted path to edit, and finally kills the background ssh process when you’re done editing.

Run

usage
1
vim-scp myuser@some.host.com /dir/path/or_file

And you’re editing remote files with vim blazing fast..

Enjoy!

Three Amigos One Tmux

| Comments

TL;DR: telnet gameoflife.erikzaadi.com 1337

This Saturday (8th of December, 2012), was the second Global Coderetreat, and I was amongst the lucky ones to participate from Tel Aviv.

The event was epic, 30-ish geeks pairing up to 45 minutes of hacking sessions on Conway’s Game Of Life.

And that was just in Tel Aviv, about 3500 people worldwide took part in this Geekathon.

The last session I teamed up with Roy Rothenberg and Pablo Klijnjan (Yes that last name is in Klingon), both my day to day teammates at work.

We had this idea of making a Telnet version of Game Of Life, inspired by the Telnet Nyan Cat ( telnet miku.acm.uiuc.edu ).

We hacked together a working Python version in that session, that I thought would be hilarous if it was available online.

We used a very simply Telnet server using Python’s socket module and most of the time was spent understanding how the f#!@#k to clear the screen:

Clear telnet screen, works for console as well
1
self.conn.send(chr(27) + "[2J") #chr(27) == <ESC>

The problem with the Python version we hacked together was that if you connected more than one Telnet client, it would crash.

During the session we looked at using Twisted to improve the Telnet server, but we had a crappy connection and it took ages for it to download.

Yesterday when I looked at improving the code to make it available online (yes I didn’t delete that sessions’ code :$), I had a look at some Twisted Telnet implementations, but the code was to Twisted for me.

Reactor this, Factory that. No thanks.

So, I decided to hipster up and try it in nodejs, initially setting a 45 minute target as one of the sessions we did at Coderetreat.

"Node.js hipster"

Getting up the Telnet server was easy, a quick npm search telnet found me wez-telnet.

Writing the coffeescript version of The Game Of Life took me most of the time.

I used Mocha as testing framework, which is very smooth. Writing the game and telnet server took me about 55 minutes (while multitasking other stuff).

I tested the Telnet server by connecting from 10 clients from different machines in our network, seem to not even flicker.

When I connected via ConnectBot on my Android, it crashed the server. So, before putting it up online, I needed some kind of daemon that restarts the server if (when) it crashes. This was easy with forever.

Here’s the entire package.json with the scripts for testing, testing continuously, starting and stopping daemon:

package.jsonpackage.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
  "name": "GameOfLifeTelnet",
  "version": "0.0.1",
  "description": "See readme",
  "main": "node_modules/.bin/coffee index.coffee",
  "scripts": {
    "test": "node_modules/.bin/mocha --compilers coffee:coffee-script -R spec tests",
    "watch": "node_modules/.bin/mocha --compilers coffee:coffee-script -w -R min tests",
    "daemon": "NODE_ENV=production node_modules/.bin/forever start -c node_modules/.bin/coffee index.coffee",
    "stop": "NODE_ENV=production node_modules/.bin/forever stop 0"
  },
  "repository": "https://github.com/erikzaadi/GameOfLifeNodeTelnet",
  "dependencies" : {
    "coffee-script"   :  "*",
    "wez-telnet"    :  "*",
    "forever"     : "*"
  },
  "devDependencies": {
    "chai" :  "*",
    "mocha" :  "*"
  },
  "author": "Erik Zaadi",
  "license": "MIT"
}

So to get this working, all you need to do is clone the repo and one npm command and you’re done:

Hacking on the code
1
2
3
git clone https://github.com/erikzaadi/GameOfLifeNodeTelnet
cd GameOfLifeNodeTelnet
npm i

To run tests:

Running Mocha tests
1
2
3
4
#simple test run
npm test
#continuous test run
npm run-script watch

To run as a daemon:

Forever Daemon
1
2
3
4
#starting
npm run-script daemon
#stopping
npm run-script stop

Grab the source code or run the live sample:

run sample
1
telnet gameoflife.erikzaadi.com 1337
Running game of life
1
2
3
4
5
6
7
8
9
✩✩   ✩       ✩      ✩✩        ✩✩   ✩      ✩✩
✩✩    ✩ ✩✩  ✩✩               ✩✩✩✩  ✩✩✩✩ ✩ ✩✩
✩                           ✩✩
✩  ✩            ✩✩✩   ✩  ✩✩    ✩
✩                  ✩          ✩       ✩ ✩        ✩
✩         ✩✩✩             ✩✩   ✩✩            ✩ ✩   ✩
✩ ✩       ✩✩✩     ✩ ✩     ✩✩   ✩✩     ✩    ✩   ✩
✩✩      ✩✩✩                    ✩✩       ✩✩       ✩
✩✩✩

Enjoy and prosper!