Guard clauses and single exit point

Developers are often told to have a single exit point in functions but in my opinion this just makes things more difficult than they have to be.

Instead of having a single exit point I prefer to use guard clauses in most of the functions I write, using the single exit point paradigm only when dealing with resources that have to be manually closed (having a single exit point makes resource handling easier and less prone to error if you don't have something like a finally block to help you).

As a matter of fact, I prefer to use what I call "return early" paradigm, that is, return as soon as the job is done (guard clauses are a specific case of this). That avoids nested conditionals and makes the code simpler and easier to read.

Example

Using nested conditional to have a single exit point:

 String ret = null;
 if (isGreen && isRound && smellsCitric){
   ret = getLime();
 } else {
   if (isGreen && isRound){
     ret = getGreenBall();
   } else {
     if (isGreen && smellsCitric){
       ret = getCitricSoap();
     } else {
       if (isRound && smellsCitric){
         ret = getOrange();
       } else {
         if (isGreen){
           ret = getGreenColor();
         } else {
           ret = getUnknown();
         }
       }
     }
   }
 }

 return ret;

Using the return early paradigm:

 if (isGreen && isRound && smellsCitric){
   return getLime();
 }
 if (isGreen && isRound){
   return getGreenBall();
 }
 if (isGreen && smellsCitric){
   return getCitricSoap();
 }
 if (isRound && smellsCitric){
   return getOrange();
 }
 if (isGreen){
   return getGreenColor();
 } 

 return getUnknown();

AWS Fargate POC

This week I had the opportunity to work on a POC using AWS Fargate to provide the desired scalability to our most critical service: the one which calculates taxes and fiscal information of a commercial operation.

AWS has had the ability to run containers for quite a while and we've tested ECS before with the same goal but we found it a bit cumbersome as we had to create auto-scaling groups, launch configurations and manage the scale out of EC2 instances. This felt wrong somehow, as we had 2 different places to control the size of the cluster: 1) part of it was in the ECS service and 2) part of it was in the EC2 auto-scaling groups.

AWS Fargate solves this by putting all scalability configuration in the ECS service. Now you can manage the services without even thinking about the EC2 instances which back their execution.

Setup a Fargate cluster is a piece of cake using the wizard provided by AWS. There's a lot of information about that in the internet so I won't get into the details of creating a Fargate cluster. Instead, I will focus on the benefits and drawbacks we found of using Fargate to provide the infrastructure to our service.

Our tax calculation service has a very heavy initialization cost because it has to load millions of tax rules and build several indexing structures before start calculating. The whole process can take up to one minute. For this reason we couldn't have a task instantiated for each calculation request. So we chose to have each container running a Tomcat for an indefinite amount of time servicing our tax calculation service.

Our cluster:

• Tasks with 5GB of memory and 2 vCPU each. Each task was a Tomcat running our service that could serve requests for as long as the task was alive.
• Auto scaling was based on requests per target: ALBRequestCountPerTarget at 100
• Auto scaling limits: 1-10 tasks

The load was provided by 3 t2.xlarge EC2 instances using 16 threads each to fire requests to our cluster. The requests were taken from a database of approximately 15000 different tax scenarios read in a round robin fashion.

The test consisted of 278040 scenarios sent to the cluster. Each test thread would send a request and wait for the complete response before sending another request. All tests were started with the cluster running only one task.

We ran 8 tests. The results were:

Worst case: 278040 tax scenarios in 2544 seconds => 109,3 scenarios/s
Best case: 278040 tax scenarios in 491 seconds => 566,3 scenarios/s
Average: 278040 tax scenarios in 945 seconds => 294,2 scenarios/s

Checking the scalability of the cluster:

• 3 of these tests ran with only one task for the whole time because ECS could not provide new task instances due to lack of available capacity (service <service-name> was unable to place a task. Reason: Capacity is unavailable at this time).
• 2 of these tests had more tasks instanced but it took a long time to provision the resources and the impact on the performance was reduced.
• 3 of these tests had new tasks instanced very quickly (as desired) and that showed a great impact in the performance. 

Given the results we had, my opinion is that the use of ECS + Fargate to provide scalability to a heavy load service is feasible. Setting up a cluster and managing it is very simple and the results we had showed that the infrastructure can be very flexible, adapting to different loads quite quickly.

The fact that you could be left on your own when demand on your availability zone increases, however, is a bit worrying to me. In one occasion no new instance could be added to the cluster for about 2 hours even though the load was high. I'm not sure we could support degraded performance for such a long time in production.

For this reason my opinion is that a ECS Fargate cluster is not yet the best choice for a critical service where performance is so importante. I got a good impression from Fargate and I think managing a ECS cluster is now much more intuitive and easier than it was in our first ECS POC. But being left with a cluster that was not able to scale out under heavy load for about 2 hours really kills any possibility of using ECS + Fargate for production of our most critical services for now.

Good news, everyone!

Why Self-Taught Artificial Intelligence Has Trouble With the Real World
https://www.quantamagazine.org/why-self-taught-artificial-intelligence-has-trouble-with-the-real-world-20180221/

Carnegie Mellon Reveals Inner Workings of Victorious AI
https://www.cmu.edu/news/stories/archives/2017/december/ai-inner-workings.html

Simple Rules, Complex Systems and Software Development
https://www.targetprocess.com/blog/2009/03/simple-rules-complex-systems-and/

Understanding Activation Functions in Neural Networks
https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

Creating a user in MongoDB not knowing the admin credentials

Last week I had to setup a staging machine for one of our applications in an environment with a pre-installed MongoDB instance. After creating all the infrastructure in AWS, setting up the environment and configuring the application I got a failed logon attempt to the database. Trying to connect directly (not through the application) resulted in the same error:

 
> mongo --port 27017 -u usuario -p minhasenha --authenticationDatabase novoBD

  Error: 18 { code: 18, ok: 0.0, errmsg: "auth fails" }

OK, it seemed the user didn't have access to the database. In MongoDB authentication is managed at a database level. When trying to connect to the MongoDB using a database, it checks for the provided credentials in the collection <database>.system.users. Solution: create the user in the database.

Well, easy solution if you have the admin credentials, but that was not the case. Workaround: as I had root access to the OS, I disabled MongoDB authentication:

> vim /etc/mongod.conf

  noauth=true
  #auth=true

Then logged in MongoDB and created an admin user:

> mongo --port 27017

  use admin;

  db.createUser(
    {
        user: "admin",
        pwd: "minhasenha",
        roles:["root"]
    });

Logged out MongoDB and enabled authentication back again:

> vim /etc/mongod.conf

  #noauth=true
  auth=true

Logged in MongoDB back with admin user (saved its credentials for future use!) and created the user in the new database:

> mongo --port 27017 -u admin -p minhasenha --authenticationDatabase admin

  use novoBD;

  db.createUser(
    {
        user: "usuario",
        pwd: "minhasenha",
        roles:[{ role: "readWrite", db: "novoBD" }, { role: "dbAdmin", db: "novoBD" }]
    });

After that the application happily connected to MongoDB. If the application is happy, I'm happy too!

Simulating cron environment

Ever added a script to crontab just to discover it didn't run as expected? Well, cron environment is not the same as your user environment and that could make a huge difference depending what your script is doing.

To help me writing scripts taking into account this difference I use the following tip by mmccoo (https://stackoverflow.com/questions/2135478/how-to-simulate-the-environment-cron-executes-a-script-with).

Add this to crontab:

 
30 08 * * * env > ~/cronenv

After it runs, do this:

 
env - `cat ~/cronenv` /bin/sh

This assumes that your cron runs /bin/sh, which is the default regardless of the user's default shell.

If you want to use another shell just change /bin/sh for your favorite shell and don't forget to also change the shell used by cron by adding this to your crontab:

 
SHELL=<your_favorite_shell>

Good news, everyone!

What is backpropagation and what is it actually doing? 
https://www.youtube.com/watch?v=Ilg3gGewQ5U

What Google Learned From Its Quest to Build the Perfect Team
https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html?smid=fb-nytimes&smtyp=cur&_r=0&pagewanted=all

Why everything might have taken so long
https://meteuphoric.wordpress.com/2017/12/31/16417/

Why are your bones not made of steel?
https://www.materialstoday.com/mechanical-properties/news/why-are-your-bones-not-made-of-steel/

You’re Descended from Royalty and So Is Everybody Else
http://nautil.us/issue/56/perspective/youre-descended-from-royalty-and-so-is-everybody-else

ON PROOF AND PROGRESS IN MATHEMATICS
https://arxiv.org/pdf/math/9404236.pdf

Checking the version of your MongoDB instance

If you want to know the exact version of your MongoDB instance, simply use:

 db.version();

It couldn't be simpler!