Tackling another design issue…

Probably the most popular module on my HR startup has to be the Recruitment module, which has taken over from the absence tracking module in the earlier days. We currently have over 500 job listings posted on HR Partner, with well over 35,000 candidates being tracked.

All in all, users love using the Recruitment area, however we have had some feedback that it is currently a little clunky to view individual applications. You have to constantly switch between the job listing view screen, which shows all applicants in the various stages of the hiring pipeline, and the main application view screen, and back. This consumes valuable clicks and precious seconds of time - especially when you have hundreds of applicants for a job.

We needed a way to allow our users to quickly step between job applications without the constant screen switching. We also didn’t want to clutter up the screen real estate too much. This was a tough problem to solve. But we had some good UX people on the team, and I wanted all of us to brainstorm and learn how we could achieve this end goal.

Now, currently on the employee view screen in HR Partner, we have this little pop up widget that allows you to skip forwards or backwards to the next 1 or 2 employees quickly without having to go back to the employee list.

Our immediate decision was to perhaps build something very similar in the Applicants view screen as well - to allow the user to jump ahead or back one or two applicants at a time. But then we though that it could be impractical, as applicants weren’t really sorted in any order, and there could be too many times when the user needed to jump from the top to the bottom of the list.

So back to the drawing board we went. We needed something that could be obvious and ‘there’ when needed, and out of sight when not needed.

Solution: We decided to build a sidebar popout widget that the user could call up when needed.

That’s it. The end result was one tiny little icon that is SO easy to miss (which is why he have that popup appearing when our users first go to this screen). An infinitesimal change, but one that can have a big impact when clicked.

You see, clicking the icon slides out a narrow (around 200 pixel wide) sidebar window which shows all applicants who are in the current stage of the job pipeline as well.

Clicking on any candidate name will open up the same Application details window, where you can see their answers to the application form, read their resumes, check their interviews and email history and fill in the scorecards and leave comments. AND - it leaves the sidebar open while doing so, so you quickly jump from candidate to candidate without interruption.

While working on this widget, and based on our customer feedback and even our own experience with using our Recruitment module to hire internal team members, I remember that I was often circulating across 4 or 5 applicants, especially towards the tail end of the recruiting process, when we were trying to narrow down our selection to more suitable people we wanted to shortlist. I recall that as being quite tedious when I had to jump back to the list of 100 odd candidates and try and remember who our top picks were.

So we decided to include a ‘Recent’ tab to the sidebar, which shows you the last 10 candidates that were looked at. Now we can use this tab to quickly cycle between those last few candidates when discussing with the team.

Here is the new sidebar widget working how we expect it to. Doesn’t this look easier than the earlier video with all that back and forth??

Lastly, another common complaint about the Application view screen was the fact that in 90% of instances, the user would want to change the ‘Stage’ that the candidate was at while reading that screen. For example, they might want to move the candidate from the ‘Applied’ stage to the ‘Interview’ stage based on their suitability.

They could certainly do that now, however, it required changing the pull down menu, then scrolling down to the bottom of the application form and clicking ‘Save’. People were often forgetting to click ‘Save’ when they got distracted by the myriad of other information on that screen.

So we decided to introduce an unobtrusive ‘auto save’ whenever that drop down was changed. It ran instantly, in the background, and just popped up a discreet little ‘Saved!’ notification for a few seconds before disappearing again.

Once again, we haven’t changed anything on screen, as our users were completely used to navigating around. We simple introduced quiet background behaviour to do the heavy lifting for them.

I guess that is one of the learning points for me, especially. Good design doesn’t mean throwing out the baby with the bathwater or making drastic changes. It is about minimising the ‘surprise factor’ for the user, but still letting them choose how they want to work and get out of their way when we have to.

It is early days yet, but I think we have gotten close with the excellent work the team have done on this.

Getting important notifications on my iPhone, even with DND turned on.

Photo by BENCE BOROS on Unsplash

As a startup founder of an HR SaaS called HR Partner, I found myself being ‘on call’ virtually 24/7. I had hundreds of customers in countries all over the world, and this resulted in my email and support centre notifications going off around the clock.

Probably the best thing I ever did for my sanity was to turn on ‘Scheduled DND (Do Not Disturb)’ on my iPhone between the hours of 10pm and 6am. I could finally get some decent sleep. (Note: The second best thing I did was to hire a great support team to take care of customer tickets during that time).

However, some of those notifications were generated from our AWS monitoring system, and were caused by critical events such as a server going down, or becoming unavailable due to a DDoS attack or similar. Because we don’t yet have a full DevOps or security team in place, I still deal with those incidents myself.

However, on the iPhone, DND is exactly that - a blanket ‘do not disturb’ which shuts off all notifications from all apps. That was, until iOS 12 came out, which included a feature which allowed ‘critical notifications’ to still get through.

But the problem with that was that critical notifications were deemed as being from government agencies or weather services, that had to send through important notifications about events that could be life threatening. Not all apps could have the ability to post critical notifications.

But then this week, I noticed that a third party notification app called Pushover was actually approved for sending critical notifications. This got me to thinking - could I send certain important event activity from AWS to myself using this service? The answer was YES, and it was quite simple to do so, using and AWS lambda function as the go between.

Here is how you can do it too.

Set up a Pushover account

Head on over to Pushover.net and set yourself up an account. You can stay on the free tier as long as you send less than 7500 messages per month. Seeing as I only expected about 2 or 3 messages per week, based on our normal activities, this is well under their limit and I can stay on the free tier forever.

The have an iPhone app that you will also need to download so you can receive notifications. The cost of this app is USD$5 (via in app purchase once your 7 day trial is over) which is a tiny price to pay for the convenience of critical notifications.

Once you set up an account, you will get a User Key, that you will need, and you will also have to set up an Application on the platform, which will give you an Application API Token, that you will also need for the integration steps below.

To get the application API token, you will need to click on the Application once you create it.

Create a Node.JS Project

Now we can create a small (talking about 20 lines of code) Node.js project which will handle the Pushover notifications from AWS. But wait, why are we creating the project locally first instead of just creating the code in AWS Lambda directly?

Well, the short answer is that we will be using a node module called ‘pushover-notifications’. This isn’t in the standard collection of modules that Lambda has access to, so we need to create the project offline, with all the requisite module dependencies, then ZIP and upload the package to Lambda.

So, create an empty folder on your local drive, and within that folder, run these two commands:

npm init

npm install pushover-notifications

And now, you can create a file called index.js in here, with the following contents:

var Push = require('pushover-notifications')

exports.handler = function(event, context) {

    var p = new Push( {
      user: process.env['PUSHOVER_USER'],
      token: process.env['PUSHOVER_TOKEN'],
    })
    
    var snsMessage = JSON.parse(event.Records[0].Sns.Message);
  
    var alertSubject = snsMessage.AlarmDescription;
    var alertMessage = snsMessage.NewStateValue + "\n" + snsMessage.NewStateReason + "\n" + snsMessage.Trigger.MetricName + " " + snsMessage.Trigger.Namespace;

    var msg = {
      title: alertSubject,
      message: alertMessage,
      priority: 2,
      retry: 30,
      expire: 300,
      url: "https://<link to rebbot server via API>",
      url_title: "Reboot Server"
    }

    p.send( msg, function( err, result ) {
      if ( err ) {
        throw err
      }
    })

};

And once saved, you can ZIP up the entire folder (including the ‘node_modules’ folder and others that may be automatically created in here) and keep the ZIP file aside for the next step.

To explain the code in more detail, the first line is just including the pushover-notifications module. Then the next bit called exports.handler is the standard function that AWS Lambda will call (with the SNS parameters passed in the event variable).

Next, we instantiate an instance of the Push() object called p, which is where we pass the Pushover API token and User token. Note: For security reasons, these tokens are stored as environment variables instead of being embedded within the code. You will need to set up these environment variables as part of setting up the Lambda function in the next section.

The next line after this just extracts the JSON stringified SNS notification details that are sent to the Lambda function from Amazon SNS and stores it in a variable called snsMessage.

Then we set up the subject and actual message body. These can be anything you like, but in this case, we are extracting the description and particular values that have been passed to us from SNS which contain things like the actual error text and service names etc.

Penultimately, we set up an object called msg which contains the packet of data to send to Pushover. This includes the subject and message body we created above, but more importantly, this also specifies the type of message. The priority variable is key. Normal messages are priority 0, but these messages are treated as non-urgent and won’t come through when DND mode is turned on on the phone.

Priority 1 messages are marked ‘critical’ and WILL come through, but priority 2 messages are deemed ‘critical’ AND will also persist until they are acknowledged by the recipient. THIS is the type of message I wanted - as the notifications are critical enough to warrant me responding to them and not just glancing and ignoring them.

For priority 2 messages, you also have to specify the retry period (which I set to 30 seconds) and the expire period (which I set to 5 minutes, or 300 seconds). This means that if I don’t acknowledge the message, it will keep pinging me every 30 seconds for up to 5 minutes. Long enough to wake me from a deep slumber at 2am!

At the bottom of this object, you can also specify a URL and a link name that gets sent with the notification. This is really useful, and I used it to embed a special link that our team has which will call an AWS API Gateway function to reboot the server(s) automatically. Perhaps I will detail how I set this up on another blog post in the future, but for now, lets just stick to the notifications.

Set Up A Lambda Function

Now we will need to set up a serverless function in AWS which will transmit certain SNS (Simple Notification Service) messages to Pushover.

This is all done quite simply and easily via your AWS console. Just choose the ‘Lambda’ function from the AWS ‘Services’ drop down

And in the next section click the ‘Create Function’ button.

Now we choose ‘Author from scratch’ and we give the function a meaningful name such as sendAlertToPushover, and we specify ‘Node.js 12.x’ as the Runtime engine, and click ‘Create Function’.

On the next screen, we won’t actually type in any code. Instead, we will replace the existing code by clicking on the ‘Actions’ button and selecting ‘Upload a .zip file’.

Specify the name of the ZIP file you created containing the Node.js project files that you created in the previous step. This will upload all the files to Lambda and show you the code in the editor window.

Don’t forget to scroll down a little and set up the environment variables that the project needs:

Ok, that should be all for this part of the setup.

Setting Up The SNS Service

Ok, nearly there. Now we need to set up the Amazon SNS (Simple Notification Service) which is the conduit that AWS uses to send messages from one service to another.

From the ‘Services’ menu in the AWS Console again, find the SNS service and select it.

Then in the SNS dashboard, choose ‘Topics’ from the left hand menu, and click on ‘Create Topic’.

Next, we need to give the topic a meaningful name, such as ‘criticalPushoverMessage’ or similar.

Once done, click on ‘Create Topic’ at the bottom of the screen (leave everything else as default for now).

Once you save this topic, you will be taken to this screen, where you can click ‘Create Subscription’ to tell AWS how to handle the topic messages.

And on the subscription creation screen, we need to change the ‘Protocol’ to ‘AWS Lambda’ and then on the ‘Endpoint’ we need to choose the Lambda function that we created in the step above.

Now we can click ‘Create Subscription’ at the bottom of the screen to save it.

That is it! You should see the following confirmation screen.

Make sure the ‘Status’ says Confirmed. If so, you are all set to go.

Choosing Messages To Send

So the last step on the AWS side of things is to specify which messages you want to send over to Pushover to wake you up.

Monitoring Elastic Beanstalk

In our case, we host our servers on Elastic Beanstalk, and this service has a monitoring console where we can set up Alerts which we can send through whenever any events occur ‘outside normal operations’ - these can be things like very high sustained CPU activity, or network traffic dropping to zero, or the server status switching to ‘Degraded’ or ‘Severe’.

In our case, we want to trigger an alarm when the server CPU hits more than 90% for 15 minutes straight, so from the Elastic Beanstalk Monitoring console, we simply click the little ‘bell’ icon on the top right of the ‘CPU Utilization’ graph.

Now on the next screen you can set up the parameters which will trigger the alarm:

Give this alert a meaningful name such as ‘mainServerHighCPU’. It is also a good idea to give it a meaningful description, as this description can be used on the Pushover notification itself.

Then you can specify the monitoring period - we will select 15 minutes, as that is the window that we would like to monitor for the high CPU activity. And we will specify >= 90% as the threshold to look for. Feel free to modify to suit your own needs.

The important thing is in the ‘Notify’ field, you pick the SNS Notification Topic that we set up in the previous step above. Also, you need to tick ‘Alarm’ in the last section as we only want notifications when things hit an ‘Alarm’ state, and not when they go back to ‘OK’ or when they have insufficient data. (You may actually want notification when things go back to normal, so you can select it if you want, but I don’t normally care if things are OK or unclear, only when they break).

That is it! When you click the ‘Add’ button, the alert will be created in Cloudwatch, and the SNS Topic we set up will be listening for any changes in this alert. As soon as the alert conditions are met, the details of the alert will be send to SNS, which will then pass it over to the Lambda function we created, which in turn will send it to Pushover, and to our phones!

Monitoring EC2

You can also create alerts directly on an EC2 instance if you are not using Elastic Beanstalk. They work in a similar way. Here you can go to your EC2 console, then choose an EC2 instance from the list, and click on the ‘Monitoring’ tab.

You can then click on the ‘Create Alarm’ button, and tell the system what sort of alarm you want to set:

Don’t forget to choose the same SNS topic name we created earlier, then you can specify the parameters of the alarm. Here we want to trigger an alert when the Network Out falls below 10,000 bytes for a period of 5 minutes. That would usually mean the server has stopped responding to external requests for some reason.

Receiving Alerts

OK, we are all done. All we have to do is to wait until you have a server issue, then you should receive an alert like this:

The little red triangle next to the alert app signifies that this is a CRITICAL alert.

Note: The first time you receive a critical alert, you will be asked by iOS whether you wish the app to be able to send critical alerts. You have to answer ‘Yes’ to this prompt to receive future critical alerts.

Tapping on the alert should open Pushover where you can see more information on the error.

(Note that the red background denotes a critical alert!)

And further, opening up the alert will show you full details.

Note the link on the bottom of the alert which allows me to reset the server with just one tap. No need for me to even get out of bed to log into my AWS console in the middle of the night. I can get enough information to judge the severity of the alert, and I can choose to reboot the server right there and then to sort the issue out quickly.

Also note that there is a link to acknowledge the alert. This is because I set the alert in the Lambda function to be priority 2, so I have to acknowledge it or it will keep bugging me. If you don’t want to be constantly nagged, set it to priority 1 instead.

Conclusion

So that is it. I hope that this step by step tutorial is useful to you in setting up special alert notifications which will get through even if your phone is set to ‘Do Not Disturb’. Just be careful if you are out at a concert or a meditation camp where you truly don’t want to be disturbed. You can set notification hours within the Pushover app though, or even silence ALL notifications for ‘x’ minutes/hours, to further narrow down when you can be disturbed. Have fun!

Six figures...

Checked my phone this morning to see that we had hit another milestone that I had been holding in the back of my mind for years. We hit six figures in our business bank account. This was after paying all salaries and wages for our growing team (we just had a new developer join our team this week).

“Cashflow positive”. A term I thought I would never see just a couple of years ago. Now we have a “war chest” to reinvest back into our business to grow it to an extent that we will soon see 7, and even 8 figures in our bank. I am more confident of that now than I have ever been before.

But the past few months of hard work are catching up with me, so I have taken a week’s break from work this week. I know my exceptional team will keep things running, and they have been SO supportive of me taking some time off. I intend to come back next week totally recharged, and to begin work on get to that 7+ figures in our account.

The first screen I look at every morning

Part of my morning ritual when I wake up and sit down at my iMac nowadays is to open up this particular screen on our SaaS dashboard:

This shows me all the users who are logged on to my HR SaaS application over the past few hours. This screen always gives me pause to think.

I remember when my startup was just in its infancy, and we only had a few dozen users around the world. When I look at this stack of pins in this real time map, I am reminded that this is no longer a side business or just a hobby business any more. We are now providing a real time critical service for people in all corners of the world (at time of writing, 1685 customers in 73 different countries).

It is a sobering thought, and I keep that in the back of my mind when I make any changes to our system now, to be sure that I don’t do anything that will result in downtime or errors.

This is also motivation for me to one day visit a lot of these countries and get to talk to our users face to face!

(BTW, this real time map widget was built using RethinkDB, Segment, MapBox and VueJS - I may do a separate blog post on the technical aspect of building this, which only took a couple of hours).