Skip to content
Rex St. John

Get in touch

Node.js: Stories From The Trenches

Summary: What follows are my notes from the Node.js: Stories from the Trenches Meetup event hosted by Mixpo in their downtown Seattle office earlier this week.

Originally published on the rexstjohn.com blog

Node.js: Stories From The Trenches
Tags
  • nodejs

Originally published at rexstjohn.com on January 22, 2014. Recovered from the Internet Archive.

Trench Warfare

Summary: What follows are my notes from the  Node.js: Stories from the Trenches Meetup event hosted by Mixpo in their downtown Seattle office earlier this week.

Several Node.js  experts from Rackspace, Nordstrom and Piper (a local cloud based receipt management service)  shared their professional learning in building massive, high volume Node.js applications. Slides from several of the talks are now also available at this link.

Mixpo Plug: Interactive multi-screen video ads. Beer. Ping pong. Push ups. Whats not to love? Mixpo is hiring a  guru-like Sr. Javascript Magician / architect. Apply today: <http://multiscreen.mixpo.com/>.

Speakers Include:

  • Kyle McFarland, Piper
  • Ken Perkins, Rackspace
  • Jason Wilson, David Von Lehman, Nordstrom

Kyle McFarland
CEO / founder of “Piper”
Dealing with private / sensitive data
Piper: Digital receipt solution (DropBox for receipts)

Piper Is Built On:

  • Node.js
  • Express.js
  • Passport.js
  • And exposes receipt data via REST-ful JSON

Node.js Enterprise

  • Passport.js for authentication mechanism as generic middleware
  • Oath2orize (Outh 2.0 server)

Piper & Enterprise

  • “Designed by committee XSD’s (XML schemas)”
  • 16,000 lines of schema (example)
  • Streamline XSDs using Swagger (Wordnik)
  • “JSON data models”
  • Live documentation
  • JSON Schema draft v4

Discussion

Node and PCI compliance  - apparently no one has yet managed PCI compliance on a Node stack, PayPal will probably be the first.

Ken Perkins
Developer Advocate
Rackspace
Running Node.js in Production

“How do you run node in production”

  • If you setup your development environment like production you will have a good time
  • Deterministic infrastructure: puppet, chef, docker will set you up for success
  • Docker = containerized environments (not supposed to run in production)
  • OS tuning for node: fd limits, open sockets
  • Build & deploying Node.js: git, rsync, grunt, ftp
  • Try not to build your own, use robust tools
  • Word of caution about dependencies and nom:
    • Always use semver
    • NPM deploys break often
  • Checking in node_modules is “mostly” ok* (checking in dependencies)

NPM ShrinkWrap:

<https://npmjs.org/doc/shrinkwrap.html>

  • Not perfect
  • No checksum (“, force fucks you”)
  • Still depends on moving parts (registry.npmjs.org)
  • Lossy conversion (npm shrinkwrap > npm install > npm
  • Shrinkwrap yields different nom-shrinkwrap files
  • Running your app
  • Service your-app start (using up start)

Lots of options

Monitoring Strategies

  • Remote ping probes
  • App/SVC level monitoring
  • “How do you deal with it when things do go wrong?”
  • Running node is about mitigating liabilities

Nordstrom Data Labs
Jason & David
“Recommendo Node.js Service”

These guys built a Node.js recommendation engine with two data scientists which handles billions of recommendations. They did it in 2-3 months.

App is built on…

  • REST-ful product recommendations API
  • Live on Nordstrom.com
  • Service Emails: AWS, elastic bean stalk, node.js, DynamoDB
  • 2 billion recommendations to date

“People also viewed….” Type Recommendations In Various Nordstrom Apps

  • 3 million API hits per day

  • 5 servers, 10 auto-scaling
  • 70ms average request latency, 5ms db latency
  • 2 developers, 2 data scientists
  • 1 outage since launch due to unhanded exception

How it was built

  • Continuous integration and deployment from day 1 using Jenkins
  • git push deployment via jenkins server to elastic beanstalk
  • Unit tests -> push to AWS
  • 90% code coverage with tests overall
  • Fewer moving parts
  • Health checks that actually do something (constantly pinging the server to check if its running)

NPM Includes (What they used in their node project)

  • express
  • async (deeply nested callback chains)
  • aws-sdk
  • mocha (unit tests)
  • supertest
  • winston, logging
  • vogels (object data mapping library for dynamo db), similar to mongoose except for DynamoDB specifically
  • lodash (utility belt)
  • lru-cache (in-memory caching, static)

Structure

-social media and data ware house, feed through strategies -> into python rec engine
-> uploads to the cloud -> bulk writes to dynamo db -> cloud front as CDN (see slides, seriously)

Lessons & Pitfalls

  • Zero downtown deployment difficult to achieve
  • Thoroughly explore elastic beanstalk configuration settings
  • Rogue unhandled exception can bring it down, track errors
  • Out of box monitoring is pretty good (Amazon AWS stuff)
  • Autoscale instance count based on traffic patterns (highly predictable)
  • Node exception handling…requires strong understanding of how they propagate through various layers. Errors propagate up through callback chain, proper form is to bubble the error up through to the top.
  • Have health checks which are useful and do something, ping the service every 30 seconds and take it out if it is too slow

Panel Discussion

How do you keep up with security development and changes?

  • Kyle from Piperup.com
    • Have to stay 10 steps ahead
    • Being extremely involved with experts and the community
    • Audit / regular audit strategies
  • Node security initiative (adam baldwin, lyft) is probing all of Node’s security issues to fix them

Single hardest thing about keeping node apps up and running (app, infrastructure level)?

  • Architecture designed for single threaded node apps
  • Node inspector (you have failed at something if you need to use it on production)
  • Performance testing on EC2 in advance
  • jmeter EC2 automate spinning up of instances