VHX Dev Blog and Documentation

xxxxxx
menu
xxxxxx
0XXXXo         OXXXXKXXXX0       'KXXX00XXXXKd       ,0XXXXk
;0000O'       c0000O:0000O       ,0000d.x00000o     ;O0000o 
 cxxxxd      .xxxxx;.xxxxd       'xxxxl  :xxxxxo.  cxxxxx;  
  lllll;     cllll: .lllll       .llll:   .clllll;lllllc.   
  .:::::.   '::::;  .:::::.      .::::,     ,:::::::::'     
   .'''''   '''''.   ''''''''''''.''''.      .'''''''.      
    ;cccc:.ccccc,   .cccccccccc: .cccc;      ,ccccccc,      
    .OOOOOOOOOOk    .OOOOO.....  ,OOOOd    .kOOOOOOOOOx.    
     lKKKKKKKKK,    .KKKK0       ;KKKKk   cKKKKK0:kKKKK0,   
      OXXXXXXXo     .XXXXK       ;XXXXk .OXXXXXk.  lXXXXXd  
      .KXXXXXO      .XXXXK       ;XXXXO;KXXXXXl     'KXXXXO.
       lXXXXX,      .KXXXO       .XXXXXXXXXXXc       .0XXXX0

############# DEVELOPMENT BLOG & DOCUMENTATION ##############

Hack Day 0002

by Casey Pugh @ 06/19/14

We had another great hack day recently. Such hacks. To help get our creative juices flowing, we had three special guests stop by the office, all with amazingly successful Kickstarter projects. James George of CLOUDS, Alex Hague of Monikers, and John Yost of White Creek.

Here's the recap:

Michael and I built a page that allows publishers to easily customize their embeds. Customizable Package Embed

Forest made an internal utility that allows us to have more flexibility when it comes to analyizing all our data VHX's data tools

Steven added some magic to the tab of our video page that allows you to see at a glance if a video is playing. Play button favicon

Gina and Charlie massively upgraded our video player to support thumbnail scrubbing. Thumbnail scrubbing

Sagar and Chris built a more robust monitoring system using Librato so we have better and more immediate insights into our platform. A cool Librato graph

Alex stole the entire team's baby photos and framed them for the office. The Ultimate Hack. VHX Babies

Chad reenvisioned what a 10-foot, internal dashboard could look like. It allows us to at a glance to see how we're doing day to day. Dashboard 1 Dashboard 2 Dashboard 3

Fueled by our inspiration of CLOUDS, James, Charlie, and I helped James George test the possibility of integrating CLOUDS directly with VHX. CLOUDS is basically an interactive movie with hours and hours of content. Because of its unique form, it faces a difficult distribution challenge. That said, by the end of the day, we had a working example of CLOUDS authorizing with the VHX API and dynamically pulling in videos on the fly. Crazy! CLOUDS is built with OpenFrameworks and James hopes to release an OpenFrameworks VHX addon to the ofxAddons community. CLOUDS integrating with VHX

Hack day success.

API Caching on Fastly

by Kevin Sheurs @ 05/28/14

We are moving towards a service-oriented architecture to help us grow and scale out the VHX platform. Given the nature of our releases, we tend to see large and sudden increases in traffic to various resources (site visits, video embeds, purchasing, streaming, apps, etc). We're focused on putting our API at the core of this architecture, with its performance and availability as our top priorities. This post digs into how we are utilizing Fastly and their direct Varnish Configuration Language (VCL) access to cache and accelerate our API.

Varnish and HTTP Caching

Varnish is an HTTP reverse proxy cache that is at the core of Fastly's service. When a request comes in to our API it first goes through Varnish and does a lookup in cache for that item. If it's not found, it passes the request to the backend (our API application), and then puts that response in cache for subsequent requests before finally returning the response to the client. The rules around looking up requests and caching responses are based on standard HTTP headers. Generally by default, HEAD and GET requests are cached and POST, PUT, and DELETE requests are not. There are many other factors to determine the correct caching policy (like Set-Cookie, Authorization, etc), but let's look at some specific cache related headers.

The HTTP Cache-Control header is used to specify directives on how clients should cache a response. With the following header for example, clients would cache the response for 1 hour:

response.headers['Cache-Control'] = 'public, max-age=3600'

Since we have both an intermediate proxy (Fastly/Varnish) and the originating client (browser or library), Cache-Control would be respected by both clients. This can become problematic because our dynamic content would now be cached in two places; giving us only partial cache invalidation control.

Fastly has another caching header that precisely addresses this: Surrogate-Control. By setting our Cache-Control header to no-cache and Surrogate-Control to max-age=3600 we can tell originating clients not to cache responses, but Fastly will. Perfect!

response.headers.merge!(
  'Cache-Control'     => 'public, no-cache, no-store, must-revalidate',
  'Surrogate-Control' => 'max-age=3600'
)

Invalidation

As data changes in our system the API needs to immediatly reflect that. Fastly has great instant invalidation capabilities via the HTTP PURGE method. Upon a database write in our system, we queue up purges to be sent to Fastly (see their Github for some useful libraries). In it's simplest form you can do something like the following to purge a resource:

curl -X PURGE "https://api.vhx.tv/packages/1"

However, in addition to that Surrogate-Control header, there is also the Surrogate-Key header. This allows you in your backend response to group content dependencies together based on various keys. So for example, on an API response that includes both site and package data our cache headers would be:

response.headers.merge!(
  'Cache-Control'     => 'public, no-cache, no-store, must-revalidate',
  'Surrogate-Control' => 'max-age=3600',
  'Surrogate-Key'     => 'sites/1 packages/1'
)

Now when package #1 in our system changes, we can issue a single "purge by key" command via the Fastly API for packages/1 that will invalidate all items in cache that are dependent on that data. Basically a wildcard-like approach to cache invalidation - powerful stuff!

VCL and Varying

VCL is the language in Varnish that let's you have low-level control of request handling and caching policies. You can do things like set or change request and response headers on the fly, have specific logic to do a "pass" to the backend or "lookup" in Varnish cache, route incoming requests to different backends per certain criteria, etc.

One way we utilize this is to inspect an incoming request for the VHX session cookie. For the most part our API does application and user identification via Oauth2, but there are a few endpoints that need to return logged-in content based on the session. By having the following, we can easily pass through logged-in requests to our back-end and continue to serve cached logged-out requests via Fastly:

if (req.http.cookie ~ "_session") {
  return(pass);
}

Vary is another cool HTTP header that Varnish respects that let's you cache variations in content under the same "hash" or cache key. Some of the content served via our API is geographically sensitive. A user from Australia may see different content from a user in the United Kingdom.

With the following VCL and HTTP headers we can now properly vary on our dynamic content giving us the benefit of caching and the ease of invalidation via a single purge command.

VCL:

sub vcl_recv {
  set req.http.X-Geo-Country-Code = geoip.country_code;
}

HTTP headers:

response.headers.merge!(
  'X-Geo-Country-Code' => 'AU',
  'Vary' => 'Accept-Encoding, X-Geo-Country-Code'
)

We look forward to iterating on our API and service oriented archtecture and finding more ways Fastly and Varnish can help accelerate our performance!

Meet Sagar Shah

by Sagar Shah @ 04/16/14

Sagar

Sagar is a full-stack software developer at VHX. But, he wasn't always a computer programmer. He actually got his degree in finance and economics at NYU and worked in consulting for three years before fully committing to his interests in computer science. After spending an intensive 3 months at The Flatiron School, Sagar joined VHX in October 2013 as a developer and hasn't looked back since. Here's a quick Q&A with Sagar.

What draws you to programming?

The cliche, but extremely true answer: I really love building things. It's gratifying to know that people are using something that I worked on. Also, I live for those "A'ha!" moments. You could say I'm kind of addicted to it. Because, let's be honest - programming is a struggle. But, when you solve a problem, or finally connect the dots and learn something new, it's the greatest feeling.

What kinds of things have you made recently?

Currently, I'm working on integrating ChromeCast into the VHX watching experience. So, VHX users, get ready to stream your purchases directly to your TV! It's been great working more with Javascript and learning about streaming protocols, subtitles, and the ChromeCast sender/receiver application structure.

I've also spent a lot of time over the past few months working on our publisher blogging/email communication tool, PayPal payouts for our publishers, spruces to our invoicing system, and some number crunching for our stats page.

What is something you discovered recently that you really liked?

One thing that a coworker introduced me to recently while I was writing some specs was a Ruby Gem called Timecop, which makes it much easier to test time dependent code. It literally freezes time.

In your specs, you can add something like this in a before block:

Timecop.freeze(Time.new(2014, 4, 16))

This will mock any Time.now call during the exeuction of those specs. Then, just specify Timecop.return in an after block to return to normal Rspec execution. Look at that, you just went back in time to run your specs, and you didnt even need a flux capacitor!

What's a great movie you've seen recently?

These are some of my recent favorites:

Documentaries: Mitt and Indie Game: The Movie (shoutout to Lisanne Pajot who is now helping other filmmakers successfully distribute through VHX!)

Features: Mud and Joe. Two films with monosyllabic titles set in the deep south about outlaw alpha males making an impact on Tye Sheridan. Basically the same film, but enjoyed both of 'em!

TV: Cosmos. Who doesn't love Neil Degrasse Tyson. Also, Blue Mountain State because it's hilarious.

VHX Hack Day

by Casey Pugh @ 03/31/14

For those of you who don't know what a hack day is: We spend the whole day working on whatever idea we want as long as it contributes to VHX in some capacity. For example, wouldn't it be great to easily export a GIF from your favorite movie scene on VHX? Not exactly revolutionary, but its fun!

Our team is founded by a bunch of hackers and designers with a passion for video. Hack days to allow us to continue to think beyond the scope of traditional video publishing and consumption.

Our most recent hack day yielded some really exciting results. In just under 8 hours:

  • Sagar and Gina integrated Chromecast into our video player. This feature is in beta and we will release it publicly soon! Chromecast
  • Chad redesigned our main blog and our developer blog. We have a lot of opinions on the future of video, and now we will have a pretty, candy-wrapped face.
    VHX blog redesign
  • Chris and Steven made some nice upgardes to our internal chat program, Hipchat. Now whenever someone does something good (or bad) we can give them Karma points. Yay.
  • Jamie integrated VHX into Popcorn Time. Whoa.
  • James and Charlie integrated VHX into Plex.
  • Michael and Kevin optimized the speed and scalability of our embeds. Death Star is fully operational.
  • Kathleen figured out how to empty an entire wine bottle. Turns out you can just drink it.

Just after one day it allowed us to more concretely define our API and make it even more robust. I'm really looking forward to our next hack day, and hope to host one to the public in the near future.

-Casey