Erik Zaadi

The tales of a half breed devops and web developer

S3cmd Is Dead, Long Live S3-cli

| Comments

For those of you that still use Octopress like me, one of the annoying things is deploying to Amazon S3.

I was using s3cmd which is indeed an awesome utility to sync the rendered blog to S3, however it’s slow, it’ll take about 5 minutes to deploy my blog, which is not a huge blog.

While implementing a quick hack that uploads images to Amazon S3, I stumbeled upon a certain Mr Awesome called Andrew Kelley – @andrewrk, who’s node s3 client I used.

In the README for the node-s3-client there was a link to s3-cli.

This is a inplace replace to s3cmd, written in node (yaay!), which works flawlessly with the existing s3cmd configuration, which (amongs other awsome stuff), uploads to S3 in parallel, saving LOADS of time.

I did a quick change to the Octopress Rakefile

Rakefile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
diff --git a/Rakefile b/Rakefile
index fccdd5e..47f3e09 100755
--- a/Rakefile
+++ b/Rakefile
@@ -403,10 +403,10 @@ task :list do
   puts "(type rake -T for more detail)\n\n"
 end

-desc "Deploy with s3cmd"
+desc "Deploy with s3-cli"
 task :s3 do
     puts "S3 FTW"
     cd "#{public_dir}" do
-        system "s3cmd sync --delete-removed . s3://yourbucket.com/"
+        system "s3-cli sync --delete-removed . s3://yourbucket.com/"
     end
 end

And Voila! Deploy time reduced from 5 minutes to 17 seconds (UBER W000TZ!!1)

Stay classy San Diego.

Two Bots One Slack

| Comments

We started evaluating Slack at BigPanda, and it is truly great.

The signup process was super simple, adding integrations is really fast, and all the apps, web and native feels fresh and fast.

Needless to say, one of the first things I did was to hook up on of our Hubot instances that controls our door (see previous post).

With our previous not so hip chat, we did a quick hack, creating an Android Widget that calls our Hubot instance (called Bellboy), opening the door. Yes, a tad bit cumbersome, but this way we used the existing stack on our Raspberry Pi.

Anyhuze, when jumping aboard to Slack, that was the only thing that was not smooth, it appears that Slack’s Hubot adapter does not support Bot to Bot communication.

Sad Robot

That esentially means that all the things Hubot can hear are ignored if they’re not originated from a real user.

A quick Google search said that it might be implemented within a year.

It really bugged me that there wasn’t any hack around this, I really didn’t want to wait for a full year to use our shiny Android Widget to open our door.

Then I found out about SlackBot, which is Slacks native (and somewhat limited) bot.

SlackBot

It appears that this Bot has somehow entered userland (Oh how Tron!), and anything SlackBot says is picked up by Hubot as opposed to by other Bots.

And as making SlackBot talk is a simple matter of a POST request, our Android Widget is back in business!

On another small sidenote on the Slack Hubot Adapter, be sure to see this stackoverflow, you’ll need to change your Hubot’s init script to add the extra -l parameter to make sure that mention works as it should…

Cheerioz

BigPanda’s Raspberry Pi Powered Geekalicous Door

| Comments

We recently moved to a new shiny office at BigPanda.

Before moving in, we renovated parts of the office, including the door. We got a door with an electronic lock, which usually is opened using a code panel.

Alas, we can’t have a simple code panel, we’re way to cool for that!

Like a sir

So we decided to hook up our Raspberry Pi to control the door.

Here’s the Raspberry Pi, all hooked up:

Our Raspberry Pi

Here are the buttons we use:

Buttonz

The black button is the outside doorbell. The white button is what we use to open the door from the inside. Behind the AWESOMEz BigPanda logo there’s a NFC tag that also opens the door for authenticated devices.

To control our electronic lock, we need to pass 24V/12V on or off to the lock. To enable this, we create a digital switch using a MOSFET:

MOSFET

The buttons are connected to the Raspberry PI via simple GPIO connection:

Inside button connection

One of the great things about the Raspberry Pi is that it’s running good old Debian, and has standard connections such as headphone jack and usb. We used this to hook up a simple USB web cam:

USB Webcam

And commodity speakers:

Speakers

The speakers play our custom chosen mp3 files when someone rings the doorbell from the outside and when the door is opened. The Webcam takes a picture whenever someone rings the doorbell.

We control the Raspberry Pi powered door by http site/api and HipChat.

In HipChat, we have a bot powered by Hubot which we use to open the door and to see who’s calling the doorbell.

HipChat

The code is of course open source, up at https://github.com/bigpandaio/bellboy.

Enjoy!

Handle Proxy 404 in Nginx

| Comments

In our SPA era, when you get a 40x or 500x (Oy vei) error from your proxied backend, you typically want to display a static part of your SPA.

To do this, we can usual a small but usefull nugget from nginx:

HERE BE DRAGONS
1
proxy_intercept_errors on;

This ensures that if the proxied backend returns an error status, nginx will be the one showing the error page.

We need to tell nginx to handle the 404:

Trap 404
1
2
3
4
5
server {
  #...
  error_page 404 = @404;
  #...
}

Notice the use of the @404 location alias, it allows us to redirect as we wish:

404 Location hack
1
2
3
  location @404 {
    return 302 /app-path-to-404;
  }

Here’s a full sample:

Handling proxied 404 in nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
upstream your-api-cluster {
  server localhost:1337 max_fails=3  fail_timeout=10s ;
  keepalive 16;
}


server {
  root /path/to/your/static/site/;

  location @404 {
    return 302 /app-path-to-404;
  }

  error_page 404 = @404;

  location / {
    autoindex off;
    expires 1h;

    #...
  }

  location /api{
    proxy_intercept_errors on;
    proxy_pass http://your-api-cluster/api;
    proxy_redirect   off;
    proxy_set_header Host            $host;
    proxy_set_header X-Real-IP       $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    #...
  }
}

Enjoy!

Long Time No Spamming

| Comments

TL;DR – Yet another ‘didn’t blog in a while due to ENTER EXCUSES HERE’ post..

I’ve moved on

I now work at BigPanda since March, and until now, I didn’t make time to write here..

I did get a chance to write a blog post at the BigPanda Blog about how we use vagrant, which was fun writing.

In the end there are only two reasons I stopped blogging.

  1. Number one: Refactor Syndrome

    REFACTOR ALL THE CODE

    Since the last post, I’ve been trying to rewriting this blog in:

    Here’s another MEME describing how I feel about my progress refactoring:

    Oh cool, this seems simpllllllleeeaaaaaahhhh

  2. Now for the real reason:

Cleaning Up Jenkins Jobs

| Comments

Jenkins rules, but…

It’s a Java XML consuming monster.

Once you get up to a large number of jobs (We are near 300), it gets a bit tough, especially when restarting.

So, every once in a while, I cleanup old unused jobs that are left over.

delete_non_active_jobs.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import urllib2
import os
from shutil import rmtree

def get_active_jobs():
   request = urllib2.Request('http://localhost/api/python?tree=jobs[name]')
   opener = urllib2.build_opener()
   json = opener.open(request)
   return [job['name'] for job in eval(json.read())['jobs']]

def get_jobs_dir():
   return os.walk('.').next()[1]


if __name__ == '__main__':
    jobs = get_active_jobs()
    jobs_in_dir = get_jobs_dir()
    jobs_to_delete = sorted([job for job in jobs_in_dir if job not in jobs]
    for job in jobs_to_delete:
        print "Deleting: '%s'" % job
        rmtree(os.path.abspath(os.path.join(os.curdir, job)))
‘Delete them allz’
1
2
3
4
cd ${JENKINS_HOME}
#note Jenkins must be up
python delete_non_active_jobs.py
service jenkins restart

Assign an Application to All Desktops in Applescript

| Comments

Back to basics, applescript

It’s been a while since I wrote applescript.

Although the capabilities are truly amazing, I never really connected to the tell syntax, a bit to verbose for me.

Anyhow, I’m using slate for automagically position apps when I change monitors (amongs other things), and I needed a way to pin certain applications to all desktops when using multiple monitors, or to a specific desktop when using just one monitor.

You can run this script from the shell (if you chmod it):

Usage
1
assignTo this|all Chrome Adium "App with spaces"

Source

assignToassignTo@github
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
#!/usr/bin/osascript
script assignTo
  on toAllDesktops(appsList)
      forAllChats("All Desktops", appsList)
  end toAllDesktops
  
  
  on toThisDesktop(appsList)
      forAllChats("This Desktop", appsList)
  end toThisDesktop
  
  on forAllChats(toSet, appsList)
      repeat with appName in appsList
          tell application "System Events" to tell UI element appName of list 1 of process "Dock"
              perform action "AXShowMenu"
              click menu item "Options" of menu 1
              click menu item toSet of menu 1 of menu item "Options" of menu 1
          end tell
      end repeat
  end forAllChats
end script
on run argv
    tell application "Finder"
        set scriptName to name of file (path to me) as text
    end tell
  set usage to scriptName & "
 Usage: all|this Application1 ApplicationN"
  if (count of argv) is 0 then
      return usage
  end if
  set what to item 1 of argv
  set apps to rest of items of argv
  if (count of apps) is 0 then
      return usage & "
 At least one application is required!"
  end if
  tell assignTo
      if what is "all" then
          toAllDesktops(apps)
      else if what is "this" then
          toThisDesktop(apps)
      else
          return usage
      end if
  end tell
end run

Enjoy

Return of the Mac

| Comments

After two wonderful years of Linux, mostly with Ubuntu (with a small visit to Fedora), I had the opportunity to switch to a Mac at work. It’s an old 2009 MBP inherited by Vuduchild who left us lately.

At my previous workplace, I had an iMac for about four years, and I’m ashamed to admit that about two and a half years out of those four, I was mainly on the dark windows side of the iMacs bootcamp (NEVER AGAIN). As for the last year and a half out of those four, I started to use Mac OS exclusivly, fell for git and vim, but still developed mainly DONT.NIET apps.

When I started to work at my current workplace, I switched to Linux, and more importantly, I started working with Python and learning a whole lot of *nix os, bash, zsh, you name it.

I got used to tweaking every last detail to my specific needs. Installing Ubuntu only from the bare minimal text mode, and only installing Gnome 3 and the parts I liked.

I started to “live” almost only in two worlds, the terminal (Helloooo tmux) and the browser.

Everything was in instant reach either with an apt-get, ppa or worst(!) case scenario, I’d download and compile it.

And things worked. Even worked smoothly.

Multiple screens worked just as I wanted in Gnome 3 (the secondary screen sticks apps to stay when you switch workspace).

Sure I had some problems, but nothing I wasn’t able to fix.

So why switch back to a Mac?

Curiosity and the fact that it’s shiny.

It’s that simple, I’m a tech geek that could not possibly say no to a new (old) toy.

The facts

When I left the Apple world, I had just installed “Snow Leopard” (After using “Leopard” mostly). I remember there was a new shiny thing called (home)brew I wanted to check out but never got to.

Now we’re up to “Mountain Lion”. Good thing I didn’t join back on {Grumpy,Nyan} cat. There’s an App Store (!).

The good

It’s smooth. Almost to smooth.

Brew and Brew cask Makes life a bliss. Hello apt-get. Never download dmg => Drag to applications again.

Might sound silly, but I’ve missed Google Chrome Canary.

Finally my Magic Trackpad works flawlessly, with VERY smooth gestures.

The Adium Icon. I know, it’s simply awesome.

The MEH

I miss the good old workspaces, having four explicit ones. That’s the way I configured Gnome 3.

Apple, it’s been two years, Y U NO PROPER WINDOW POSITION AND SIZE HANDLING?? (brew cask install slate fixes some of that though).

Montain Lions fullscreen feature isn’t really that helpful.

The WTF

I had to set LC_LANG? REALLYZ? Why did it take me two days to understand that this is what ruins my vim powerline in tmux?!?!?!

Multiple screen handling. Nuff said.

The basics that just doesn’t work. readline -f should work, Y U NO GNU?

The worst

Hey, I want to develop something –> sure no problem.

Download a 1.7 gb download of Xcode, after it’s done, open it, find the downloads tab and click command line tools, which takes some time to download as well.

Then you can install homebrew, and your life is shiny again.

It still annoys me that I need to walk around with a VGA adaptor, HDMI adaptor etc.

Da Summaryz

I came back to the Mac world a different user thanks to Linux.

It was still worth it moving back.

"Return of the mac"

PS, The handcuffs are rather symbolic for moving back to a close{d,r} system..

FDD - Fika Driven Development

| Comments

FDD FTW

There are many different type of development methodologies. TDD, BDD, what have you.

You also got non development centric methodologies such as Pomodoro, which help you make the most of your time.

At our group at work, we have fine tuned a brand new and shiny methodology called:

“FDD” – Fika Driven Development

20120515_114703

Fika is a Swedish custom imported by yours truly.

Every day, at 16:00, it’s Fika O’clock, usually announced by SHOUTING in ascii in our IRC channel.

During the Fika, we have coffee in our kitchen, with something sweet like cookies on the side, and we talk.

The Fika exposes the following crucial exercises we nerds usually don’t use on a daily bases.

  1. We socialize face to face
  2. Talk NOT about work
  3. We actually move physically

In addition, we drink coffee, which is always important.

On a side note, did you know that Sweden is the top of the world in coffee consumption?

We also have morning Fika(s), but this works in a more distributed manner, usually in two or three batches according to arrival times.

We’ve found that using FDD (FTW), we’re more focused, more relaxed.

20130124_162258

Try FDD and let us know how it affects you!

Nose-rapido - a Rapid Feedback Plugin for Nosetests

| Comments

I use a tmux + vim development environment which I find really productive. When hacking on python projects, I like to have a tmux window open with nosetests logs, typically using nose-watch.

Every now and then after saving, I’d switch to that window to see how the tests are doing.

I wanted something a bit more small that would give me immediate feedback in my main window that I use, which is of course the vim window. The feedback should on one hand not take up to much screen estate, nor disrupt my vim-fu focus.

Enter nose-rapido.

nose-rapido is a nosetests plugin, that does one small and simply thingy: It fills your terminal with green (or optionally blue for the colorblinded) or red color, according to your nosetests result.

To install
1
pip install nose-rapido
Usage
1
nosetests --with-rapido
Usage with nose-watch
1
nosetests --with-rapido --with-watch
Changing successfull color to blue
1
nosetests --with-rapido --rapido-blue

E.g:

"Tests passing"

And if you fail your tests:

"Tests failing"

With blue instead of green:

"Tests success with blue"

Enjoy!