How to deal with node_modules in a Dockerized Node application?

Posted on Leave a comment

There are tens of thousands of blogs, articles and forum threads out there on containerizing a Node.js application. We have everything from outright bad advices to informed and tested opinions. This short post is about a very specific aspect of Dockerinzing a Node app, something that is usually not addressed or given as an after-thought in these articles.

Once you have a Node app up and running in Docker, things are well until you – like all good developers – start adding or removing NPM packages. Nasty things happen, such as:

  • packages get installed on your local (host) filesystem but are not available inside the container
  • packages get installed on your container’s filesystem but are not available on your local
  • there’s a mismatch between versions of the same package in host and container
  • permission issues while installing or removing packages

All these can be solved by correctly using Docker volumes. A volume is basically a way to make parts of container’s filesystem available to host and vice-versa. Not understanding volumes properly may also lead to one of the issues listed above.

I had a similar experience in one of my projects. See the fix:
https://github.com/universalnative/un-website/commit/0eaaca98adac46bf9b1553e66d6acd9d731db43d

As noted in the stackoverflow answer cited in the commit:

When docker builds the image, the node_modules directory is created within the worker app directory, and all the dependencies are installed there. Then on runtime the worker app directory from outside docker is mounted into the docker instance (which does not have the installed node_modules), hiding the node_modules you just installed. You can verify this by removing the mounted volume from your docker-compose.yml.

https://stackoverflow.com/a/32785014/1775160

This is a powerful thing to remember. Once you know how volumes work, you’ll be able to better troubleshoot your Node app when things are wrong. For example, when I recently installed a new NPM package in my local I knew I had to do something like this to make it available inside the container as well:

https://github.com/universalnative/un-website/commit/3fa9c9a79c35a3ddbf39079d16b7dbca12105c0d#diff-4e5e90c6228fd48698d074241c2ba760

Why didn’t I just mount my host node_modules to container’s node_modules? Here’s the explanation:

An alternative approach is to mount host node_modules as a volume in container, but that will override container’s own node_modules folder with host’s. Keeping things independent allows for cleaner and easier troubleshooting of installed/missing packages.

This the approach I prefer, which of course is not certified gold. I works for me well. Besides, yarn installing packages each time container comes up does so incrementally (only new packages are fetched and installed). A win-win 🙂

Express API Refkit

Posted on Leave a comment

https://github.com/anuragbhd/express-api-refkit

I frequently create APIs for muse/professional apps, and every time I find myself scrambling to pick the best pieces from my previously built APIs or online repositories.

To streamline this, I recently created this reference kit (mostly for myself) to help me in writing a production-grade Express-based API from scratch. Sharing with this group in hopes that my fellow XTians will benefit from it when writing their own RESTful APIs.

As noted in the docs:

This is NOT a starter template. The purpose of this repository is to provide reference code to anyone looking to create a beautiful, secure and well-tested Node/Express-based API from scratch.

Creating from scratch has the invaluable benefit of learning while doing. Most likely, there will be struggle which will make you better. You will know by heart all parts under the hood and exactly how they interact with each other. Basing your project off of a starter kit takes that away from you. Hence the need for a reference kit (or refkit, as I like to call it).

If you are experienced in Express, feel free to open pull requests to add improvements.

Contributions in any form are welcome 💁‍♂️

Featured image credit Nick Youngson CC BY-SA 3.0 Alpha Stock Images

My first cron job

Posted on Leave a comment

That’s right. Believe it or not. I created my first cron job TODAY! This is what it looks like:

0 4 * * * cd /var/www/mysite/ && node gmailscripts/watch.js my-email-id@gmail.com

It’s a very simple job that calls a Node.js script to renew a watch I have on one of my Gmail accounts. at exactly 4:00am daily. Now, what is a watch or why I am watching my mailbox are beyond the scope of this little blog post. Basically, I have a nifty little Node.js API that I use as a webhook to get notification updates from Gmail whenever a new mail arrives. It then checks if the new mail has a specific subject and from address, which when true instructs the API to call a custom parser to scan the contents of the new mail and return me important bits of information that I then save in a database. Geeky? Perhaps it is. I plan to do a separate blog post on how you can do the same (not setting up cron, but watching Gmail mailboxes for new mail programmatically).

Back to cron again: it’s not that I didn’t know the concept of automated jobs before (in Linux or otherwise), I guess I never really needed to create a scheduled job before. Creating the job was not as much fun as reading the correct way to do it. I followed this Digital Ocean community tutorial, which is now 6 years old but stays relevant today.

On this topic, in my professional life I’ve literally seen people learning about this ‘cool’ tool and then misusing it for all sorts of software development things that can be (and should be) done using some form of publish-subscribe or message queue design pattern.

Hosting meteor.js app in IIS

Posted on 10 Comments

meteorjsMaking reactive and real-time web applications is in fashion these days. Among popular real-time programming frameworks are meteor.js, knockout.js and signalr. Both Knockout and SignalR are developed by Microsoft employees, and integrate seamlessly with Microsoft products. Meteor, on the other hand, is though based on the cross-platform node.js, it is more Linux / Mac-centric than it is Windows. In fact, Meteor’s official documentation is also designed with instructions that pertain to Linux. They don’t even have an official Windows installer for their framework, but depend on a certain Tom Wijsman for that.

It’s true that Linux/Apache is still the dominant server stack, but we still have thousands of corporations that rely on Microsoft’s Windows servers and IIS. And deploying a meteor app on a Windows server is a real pain in the butt because the ever so elegant command meteor bundle doesn’t work in Windows. Last I heard from Tom, implementation of meteor bundle in Windows was in the works. But meteor-win won’t support it until at least v0.5.1.

Although there are workarounds like using a Vagrant VM or msysgit for deploying meteor apps on Windows servers, as I found out through my forum thread, all these workarounds are either too cumbersome or don’t work in all cases. During my constant searching on ways to deploy my meteor app on Windows, I stumbled upon an excellent utility called iisnode, which does that in the best and most elegant way possible.

iisnode is basically meant to host node.js applications is IIS, but again as meteor apps themselves are node.js apps only, using iisnode to host them just works. But some setup and configuration is required before your app is fully hosted.

Continue reading Hosting meteor.js app in IIS