Skip to Content

Category Archives: Development

Test cases; an asset or a hindrance for the QA testing process?

 

This blog post aims to address a controversial topic within the QA tester community: test cases. When I first read about this topic, I was wondering whether test cases are actually an asset or a hindrance to the QA testing process. The following paragraphs are meant to help us distinguish between the advantages and disadvantages of different test cases. 

On one hand, some people say that it is essential to design a test plan and have the whole list of test cases before starting to test. These people consider test cases an asset. On the other hand, some other people say test cases are actually a hindrance to the QA testing process since the tester would then be biased, thus limiting creativity by focusing on the test cases.

First, I am going to present the information supporting the idea of test cases being an asset. 

 

Test cases are an asset

 

As we know, there are some projects which lack documentation for different reasons; given the circumstances, these teams cannot provide new teammates with the proper information to understand the product. In this case, new people on the team can use test cases as the documentation they need in order to understand the project.

Other people consider test cases important as it shows stakeholders and/or clients the parts of the system which have been tested; in other words, what testers have worked on. From this perspective, test cases are such a nice tool for the QA testers to justify their work.

Moreover, writing test cases before running any tests is also considered good practice so that future testers are able to run the necessary tests once the person who designed the tests is no longer in the project.

For other testers, creating the test cases before even getting their hands onto their new product is great to learn the product. By executing a set of tests, they will get a pretty good hands-on experience without having to use their peers’ time to give them a tour of the product.

 

Test cases are a hindrance

 

For instance, according to James Bach and Aaron Hodder in their article “Test cases are not testing: towards a culture of test performance”, testing cannot be predicted for two reasons, on the one hand, there is always more testing to do than what we can afford to do, and on the other hand, we don’t know where the bugs are until we find one. Creating test cases before testing may bias the tester.

Similar to what happens with recipes, which are not cooking, we need to bear in mind that a test case is not a test, and should avoid using an artifact as the basis for human performance; also keep in mind that with tacit knowledge and skill, the artifacts are not central, and they might not be necessary whatsoever. So the test case may have a role, but the tester is indeed the center of testing.

Taking the example of the recipe into account, the performance of the tester should have substantial freedom to make her own choices from one moment to another. Having said this, if a tester has a test case, it is hard to make decisions at the moment of testing because the tester is then biased by the test cases being followed.

Test cases do not cover as many scenarios as a tester would like, for several reasons. A test case, most of the time, can cover the happy path and the most common paths for a feature, but they lack focus on elaborate scenarios, tricky paths a normal user might follow. Having test cases biases the tester. Testers are then tricked into focusing on the scripts, rather than focusing on learning about the product, or finding bugs. They may execute the test cases to the letter, and they can perfectly carry out this activity without finding any defects, which is the main objective of a tester, not to have bugs on the product.

 

Conclusion

Based on my experience regarding test cases and whether they are convenient or inconvenient for the QA testing process, I do have my own take. In the end, I believe it all comes down to being open to the needs of the product, meaning there will be some times when test cases will be quite useful, but some other times, it’ll be better not to stick to a document. After all, testing is more about how testers perform, rather than the test cases per se.

For instance, if regression testing will take place, the best approach might be to use scripts so the tests are carried out straight forward. If the testing is not about checking the main functionalities in a product, the following scripts might not be the best approach since the tester should be free of biases, the product should be explored in great detail. Important to take into account the fact of keeping documentation once testing is done, even though there are no test cases, the tester should keep notes of the results, for further situations.

 

References

 

0 0 Continue Reading →

How to conquer legacy code and not die trying

As a software engineer, I know how frustrating it can be to work with legacy code, especially when your client has no idea of the level of technical debt you are inheriting, and wants you to deliver bug fixes and new features as soon as possible. If you’re as passionate about software development as I am, you’re supposed to enjoy it, not hate it. That’s why I’m writing this blog post: To share my experience and a key piece of advice about how to deal with it.

The most common issues of working with legacy code are:

  • Having no tests at all, or no useful tests.
  • Outdated dependencies.
  • Poor software architecture.
  • Technical debt.
  • Lack of documentation.

Here are some recommendations about how to deal with it and not to die in the attempt.

Risk Assessment

After performing this assessment, you’ll be aware of all the risks you’re taking (or inheriting). In the end, you’ll have to decide how comfortable you are with the given risks. Therefore it’s super important to know the current state of the project and to be aware of what to expect when it’s time to get your hands on the code.

The goal of this assessment is to learn as much as possible of the current codebase. My recommendation is to focus on the following:

  • Documentation: Unfortunately, most of the time the only documentation that you might find in an existing project is the README file, and even worse, this file is usually not up to date. Look for architecture diagrams, wikis, or any other documentation that can help you to understand the codebase at a glance.
  • Test coverage: Speaking of documentation, tests are considered the best code documentation. You should check for the current test coverage, but more importantly, check the quality of the tests. Sometimes tests check against specific text instead of testing the business logic that drives that text to be displayed. If there are good quality tests, you should be able to have a better sense of the business logic, and moreover, the current architecture.
  • Dependencies: Having a ton of dependencies is not a good sign. I always try to avoid adding a new dependency unless it’s strictly necessary, or the benefits of said dependency exceed the cost of making it happen. Check how outdated dependencies are and how difficult it would be to do upgrades. Pay extra attention when it comes to deprecated versions and you need to do major upgrades.
  • Deployment process: A new feature, bug fix, or improvement is not considered done until it reaches the production environment. You need to understand what the deployment process is since you have to take this into account while giving estimations.
  • Backlog: After getting a little bit familiar with the codebase, you need to know what’s about to come. You might be asked to add a particular feature that might not be that easy to add given the current architecture or the version of the dependencies implemented. The value of knowing the backlog beforehand is to raise any warning flags in the early stages so that you can plan.After going through each of the above points, you should be able to communicate any risks to your client. Set clear expectations and decide whether to move forward or not based on your discoveries.

Buy insurance

If the test coverage isn’t the best, you should ask to work on adding tests before adding new features. You need to be confident enough that you’re not adding new bugs while changing the codebase, and the only way to be sure is to have a proper test suite. Good test coverage is your insurance as a developer.

Refactor on the go

I know that you might be tempted to do a major refactor as soon as you start touching the codebase; however, based on my experience, that is not always the best option. A big refactor can take forever; it’s like a chain reaction. You start with a few files or lines of code, which then scales so quickly that you’re suddenly in a situation where you’ve already refactored hundreds of files with no end in sight.

A better option is to do small refactors on the go. Refactor the code you’re touching based on the features you’re working on.

The approach that I like to follow in this case is one of the SOLID principles, Open-Closed principle, which suggests that a class should be open for extension and closed for modifications. If you can’t extend an existing class or file, then create a new one following this principle, and use it only when it’s required. Avoid changing existing implementations since it can scale quickly, and you might get lost in the refactoring instead of focusing on delivering the feature.

Conclusion

Dealing with legacy code shouldn’t be that painful. It depends on your approach to working with it.

Here are the things that you should do before starting the feature development:

  • Read the existing codebase.
  • Analyze the current backlog.
  • Add tests until the point you feel confident enough.
  • Set clear expectations to your client.


Once you’ve already started the development process, these are the things you should bear in mind:

  • Avoid major refactors; instead, do refactor on the go.
  • Add tests to any single feature or bug-fix you work on.

Let me know your thoughts in the comments section. If you have any other suggestions about how to deal with legacy code, feel free to share it with us.

 

0 5 Continue Reading →

Design thinking for Developers (Workshop, part 2 of 2)

The following video is the core content of the second part of a Desing Thinking workshop imparted by our friend Michael Gharabiklou, Founder/Managing Partner at Quor Studio. You can watch part one here.

 

0 0 Continue Reading →

Design thinking for Developers (Workshop, part 1 of 2)

On his visit to TangoSource’s México’s Headquarters, our friend Michael Gharabiklou, Founder/Managing Partner at Quor Studio, shared his knowledge about Design Thinking with us.

The following video is the core content of a 3-hour workshop and the first part of a 2-video series. It Includes the Design Thinking process and the importance of empathy in this methodology, among other topics.

 

 

0 0 Continue Reading →

React Context may be here to save us all from prop drilling.

What is prop drilling?

Prop drilling is when a prop is passed to a Component only for the sole purpose of passing it down to a child.”

In React, it is pretty common to find yourself writing a lot of boilerplate code for passing a value to a component, only to pass that component down again to a child, and this can be repeated as many times with many children as needed, until the data you are passing down reaches the component where you need it.

Not only this will cause a considerable amount of extra code written, but it will also affect the architectural design of your component, which will now require to have a property that the component itself will not even use, the prop is there just to be passed down the chain until it reaches the final component that will, in fact, use it for something. This issue is even more noticeable in apps written using a global state approach, where all the app state is saved in a global object.

In the previous image, the arrows represent the way the data needs to flow down to reach the components where it is needed; the lower in the three the component that needs the data is, the more drilling needs to be done to reach it.

React Context has always existed within React, it was previously shipped with an experimental context API and it was created with the sole purpose of getting rid of prop drilling, react itself has used it internally for a long time, but it was only until March 2018 that a completely revamp version of the context API that we can now use was released. This new version will not only help us to avoid prop drilling, but it’s now possible to use that context approach to manage our application state. Hey! We can even replace redux and only use vanilla react without any extra library to manage our centralized state architecture.

In the following image, we can see how with React Context, we just make the values available from everywhere in the chain instead of passing them down in the form of properties to other components, we do so by wrapping the components in a context that can be accessed from any position in the tree.

You can now see how there is only one arrow and that we no longer need to be passing down values over and over again to reach our farthest components.

Now with some code examples

Using Prop Drilling to pass down data example:

1) We mount our main element Toggle.

2) Inside Toggle, we can see how the Switch component saves the state and passes it down by using props, on property for printing the button state and OnToggle function for changing the actual state.

You can visit a working demo here: https://codesandbox.io/s/prop-drilling-mpzen

 

Now an Example using React Context:

1) First, we need to create a React Context (React.createContext) with some default values for our state, “on” will be started as false and let’s make onToggle an empty function for now. This created react context, which will give us access to a context provider and a context consumer:

  • ToggleContext.Provider: This will make the context available for every child.
  • ToggleContext.Consumer: When inside a Provider, it will allow us to read the context.

2) Lets make <ToggleContext.Provider> component a bit more intelligent by adding our state and our custom function (toggle) to toggle the switch value between on and off.

In order to do this, we need to initialize our Provider with the “on” and toggle values via its value property and then make sure we pass down all the children inside.

3) This will give us our custom ToggleContexProvider that we can now use to wrap the entire application in, and since Toggle lives inside a Context Provider we can use a Context.Consumer from any child to read the state provided by ToggleContexProvider

4) This is the rest of the code, notice how we use ToggleContext.Consumer for accessing whatever objects we used in values={} for our Custom Provider, also notice how we can use it directly inside the component where we need it, instead of passing it down every time, avoiding prop drilling.

Usage:

Implementation:

In the example above, we used a React Context to wrap the entire state of our small application, but you can use several different contexts for each of your app’s concerns and abstract your logic in different files.

You can visit a working demo here: https://codesandbox.io/s/53pq01x144

 

Conclusion:

Sometimes having a centralized-managed state is very similar to having a globally declared variable, this is often avoided because it leads to having a very confusing data model for your application, answering the question “can I modify this without breaking the app?”, which is sometimes very difficult to answer.

Prop drilling helps with this without any doubt, but we need to keep its downsides in mind. Use prop drilling only when you want to be very specific about how the data flows and you want to make clear dependencies between components, but use react context for normally managing the state in the rest of your application.

If you want to learn more on how to use context, here is a great place to start: https://reactjs.org/docs/context.html

 

References:

https://www.youtube.com/watch?v=u0slAvqjA6E

Code examples inspired by:
https://kentcdodds.com/blog/prop-drilling

0 2 Continue Reading →

Rails basic API authentication with Sorcery and JWT

APIs have become more and more popular due to increasing demand for development of mobile and single page apps, and so the need of sessionless server-side applications, being those that are token-based the most popular ones now days due to their relative easy implementation. Here you have a great example that uses JWT to secure a microservice-based application.

JWT (or jot) stands for JSON Web Token and as it states it is a web token structured in a JSON format. JWTs are composed of 3 parts: the header, the payload, and the signature. In short, the header contains the type of token and the algorithm for the cryptographic signing; the payload (or claim) contains the information wanted to securely send to the server, and the signature which contains the secret key known only by the server and that is used to decrypt the data contained in the payload. It is not the primary objective of this post to explain in detail what JWT is, so if you want to know more about it you can do it here.

JWTs are used in the next way:

  1. The API server generates the JWT.
  2. The JWT is sent to the client.
  3. The client saves the JWT (i.e. in LocalStorage).
  4. The client sends the JWT back –we’ll send it on a request’s header– to the server.
  5. The server uses the JWT’s payload data for X or Y purpose. In our case to identify the user that is performing the request.

 

Besides JWT, we need an authentication strategy to actually identify users. We’ll use Sorcery for this purpose; I recommend you to go ahead and read the documentation if you haven’t done so. You can implement any other authentication library –or build your own– but for this particular case I personally like Sorcery because it offers all basic authentication features and it is not as overkilling as other libraries.

This is how the general authentication flow looks like:

jwt-diagram

Fig. 1 JWT Flow

Conveniently there is a jwt library that we can use to generate user’s token, so go ahead, read its documentation and install it in your project.

Next, we’ll create a service to provide the JWT, another one to validate it, and a last one to authenticate the user.

app/services/jwt/token_provider.rb

app/services/jwt/token_decriptor.rb

app/services/jwt/user_authenticator.rb

NOTE: In this case we are assuming that JWT is in ‘Authorization’ header with format ‘Bearer xxxx.yyyy.zzzz’, that’s why we are splitting @request_headers['Authorization'].

Implementing our services

We need to generate the JWT after user has successfully logged in:

Or signed up:

The token returned in the response should be properly saved in the client so it is sent back to the server in subsequent requests’ headers.

Then we need to authenticate user on future requests. Adding an authentication method on ApplicationController will let us use it on ApplicationController’s children controllers to protect our private endpoints:

And that’s it, our API is now secured using a sessionless approach. From this point on you can add more complexity using jwt and Sorcery‘s methods to, for instance, make token expire after 1 hour or reset user’s password.

Let me know in the comments section below what you think about this strategy.

0 9 Continue Reading →

Dev jobs: A microservices architecture journey

Recently at Tangosource I was assigned to work on an internal project named Dev Jobs (see dev jobs on iOS and dev jobs on Android), where I had the chance to work next to great teammates and implement a microservice architecture with Ruby on Rails, nodeJS, JWT, Ionic, Docker, and several testing tools, including RSpec, Mocha, Frisby, and Protractor. In this first post, I’ll explain the basics of the microservice architecture.

What is a Microservice Architecture ?

Martin Fowler gives a great definition.

[1]“The microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies.”

The Problem

Let’s have some context. We are building a server side application that must support different client side platforms such as:

  • Desktop browsers
  • Mobile browsers
  • Native mobile applications

 

Also, we need to consume an external API service. In our case, Linkedin through Oauth2 for authentication. And the ability for users to chat between each other.

On an architectural matter we still had to decide between monolithic or microservice. So let’s dig a little more on this two.

Monolithic vs MicroServices

Monolithic

If we take this approach, we need to keep the following in mind:

  • Build one application to rule them all.
  • Use one language ( Ruby, PHP, JavaScript, etc. ) we could not take any extra advantages from other programming languages.
  • The architecture needs to be flexible enough to support API through AJAX and render HTML.
  • The development team needs to feel comfortable with the language we picked.
  • Security. We will have a public API to serve mobiles/clients.
  • Best tools for RTCP. So we can have a nice looking chat
  • Testing. build testable code

 

I have seen great monolithic architectures that can support all of this. One of my favorites is RoR engines based or nodeJS services based applications.

MicroServices

In the other hand, if we go for MicroServices we must take in mind the following:

  • Delegate responsibilities in small applications.
  • We can use different languages per service. Which is the best choice per responsibility?
  • How are we going to communicate each service ? Normally with tcp/ssl.
  • How are we going to persist Data through services?
  • Security: Protect the services.
  • Testing: Test all microservices.

 

On a personal note, as a developer working on a microservice platform, I can focus only on the technical details wrapping a specific service. This does not mean I should not understand the whole set of services as a unit.

“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.”

— Melvyn Conway, 1967

Having this in mind, we had to start by making technical decisions, such as how to protect our API, how to manage RTCP, by nature HTTP is stateless which means is sessionless so for our case we had to maintain a session through tokens ( JWT ), at first we though maybe nodeJS for RTCP since nodeJS is strong for DIRT ( Data Intensive Real Time )  apps. But we had not only had to take care of the tech devs but also for dev resources, currently at TangoSource we have strong Ruby on Rails devs so we had to work all together and take advantage of the expertise of each team member, our best bet was to use RoR for business logic, and take advantage of nodeJS. So the combination of RoR and nodeJS makes the solution solid and strong.

Solution

For this particular project, we decided to go for the MicroService and have some fun! The first thing that we needed to figure out was defining the responsibilities per service. So we end up with 4 microservices.

  • Ionic
    • More than a service, it’s the client side of the application. Ionic by itself is a front end SDK for hybrid mobile apps using AngularJS.
  • IDP
    • Responsible for (a) providing identifiers for users looking to interact with a system, (b) asserting to such a system that such an identifier presented by a user is known to the provider (c) possibly providing other information about the user that is known to the provider.
    • This may be achieved via an authentication module, which verifies a security token, and that can be accepted as an alternative to repeatedly and explicitly authenticating a user within a security area.
  • SP
    • In charge of persisting all user’s data, including interaction with the system. This service will contain all the business logic
  • Chat
    • In charge of establishing real-time communication among users within the system.

 

We decided to protect the services with JWT

Architecture

devjobsarchitecture

Each square represents a service, and the lines are how they communicate between each other.

Ionic : This might be the most critical service of them all, but not the most important. We have 3 main components:

  • Ionic server
    • An HTTP server in charge of sending all the assets ( HTML, CSS, JS ) to the next component.
  • Ionic Cordova
    • A Mobile SDK, which provides the user’s interaction with the app, and consumes the rest of the microservices as resources. This component runs on top of 3 different type of devices.
      • Android
      • Ios
      • Browser
  • Ionic App
    • Ionic admin panel, this panel allow us to configure GCM and APNS exposing a protected API.

 

IDP : This service is in charge of giving identity to a component/user. This service will expire a valid token with a component/user’s basic information on the JWT’s payload. Similar to memoization, this service will keep track of sessions using Redis on a basic TLS connection. Thanks to the RFC 7519 standard, this service can scale in other microservice based architectures and be reused.

SP : This is an API/Https service protected with JWT. This micro service is in charge of persisting all user’s interaction/information using MySQL with a standard TLS connection. It also manages the business logic.  

Chat : This microservice will provide real-time communication between users. This RTCP (real time control protocol) is being protected with JWT on the handshake process, and persists the communication activity on MongoDB, a NoSQL database.

Conclusion

Taking these kind of decisions is not simple. It requires a lot of experience in order to compare libraries/languages, you must at least know how they work, and it’s even better if you have used them. Remember that you will not build this by yourself, you are part of a team. It’s smart to sit with your team and discuss each individual’s strengths and weaknesses in order to make the best tech decisions.

DO NOT mix responsibilities of the micro services, instead identify your services. Add all the needed documentation to help the team such as /*Comments */,  READMEs, and Wikis.
A microservice architecture is not always the best approach, and could be painful if you are not careful.

4 9 Continue Reading →

Developer levels: the right skill set for the job.

Back when software was developed in Fortran and C, you needed a lot of knowledge to write anything of significance.  Originally, Fortran lacked even the concept of a data structure [1].  In the 1990s, an array was a contiguous chunk of memory containing a fixed number of structs or of pointers. Alternatively, one could create a struct where one field was a pointer to the next struct in the array, a true linked list.  You had to write all the code that manipulated anything.  You had to decide if an array or a linked list was best for each situation.  You had to decide if you were gonna use Quick Sort or needed a Heap Sort or if a Bubble Sort was more than enough.

Additionally, there have always been design patterns and the insight to know when to use one over another; the need for an understanding of programming languages and their compilers which affect everything we do; as well as the operating systems on which everything is built.  This list is long, going down deep all the way to discrete applied mathematics and probability.  There is a lot one can know which affects the quality of the software we write.

Today, everything has changed.  Languages do more and more.  Frameworks are getting better and better.  With Ruby on Rails, inexpert coders have so much functionality built into Ruby and even more built into Rails.  Given the objects which contain data structure and an astonishing amount of algorithms and code doing so much of what you need. The ORM with migrations, algorithms and code doing almost all the database management you need, and generators which create boilerplate code inside an opinionated MVC layout, one can write a pretty complete web application while writing surprisingly little of the actual code.  It can be quite some time before one really must make a deep technical decision which one used to make from day one.

This has changed the nature of writing software.  Back then, no company would hire someone after nine months of training let alone nine weeks … but that happens now and reasonably so.  It is perhaps analogous to the difference between a mechanic who can work with a car and a mechanical engineer who can design a car.  An upshot of this is the question of how do we categorize people writing software. When asked, I figure software people, particularly those who write applications, generally fall into about 3 categories.  This post is in no way scientific. It is written in a definitive voice but understand these are generalizations. This is just for the sake of starting a discussion and being analytical.  I have no proof.  Your mileage may vary.

Coders

Coders know how to … code.  They may no longer be muggles but they are more Ron Weasley than Hermione Granger.  They might know a few incantations but there’s still a lot of smoke.  They often use examples or starter apps.  When hitting a roadblock, which normally comes with an error message, they will put that error message into a search engine and see what comes up on StackOverflow.  They find a solution that sounds good and try it.  If that one fails, they try another. Eventually one works.  They can’t really own their code because so much of it is still magic to them. But, they can write serviceable code that works the way it should work.  Their knowledge is usually narrow, shallow and limited [2].  It should be said most are voracious and learning quickly about all the stuff they don’t know.  They do well in a collaborative team environment especially when they get to pair.  Many will go on to be solid developers.

 1

Developers

Developers have a much broader understanding than Coders of the frameworks they use and languages in which they are written.  They understand their code.  When it breaks, they generally know why and can often fix it.  They understand objects; how to write a good one and, preferably, how to test it well.  They understand where the magic lies and often how it works.  They have an understanding of the underlying technologies (SQL, HTTP, SSH, Unix, etc.) and how to work with them directly.  As they become more senior they will know more about more technologies and understand how and when to use them.  Their knowledge is still clearly limited.  They might know a lot about one area, technology or layer and know little or nothing about others. They do not have the knowledge and experience to architect complex systems well but can build well thought-out features that cut across much of an application and know how to refactor properly to bring simplicity and elegance to a complicated design.

 2

Engineers

Engineers have deep knowledge of most every layer of the system on which they work from the OS and servers on which things run, the compiler that munches their code, through to the languages they use and the frameworks on which their products are built.  They understand patterns and architectures, can determine which is best at this moment and can explain why.  They know where the magic lies and understand how it is implemented and how to extend or fix that magic.  Whereas they often have a degree in computer science, there is a growing number who do not, who have delved in and taught themselves.  They understand their code as well as the underlying technologies and they also understand the theories behind why things should or should not work a particular way.  This gives them the facility to fully understand and design architectures as well as solid objects and well-thought-out tests.

 4

 

So?

When building a team to build your applications, you must know what mix of skills you need and who can provide those in the same way Ford or Tesla will have a mix of skills on a team creating a new car, some deep engineering skills, some mechanic skills.  There are many senior engineers who can architect and understand objects but have fewer skills at coding than some coders.  Many ambitious coders know the mechanics of a specific technology and can, especially with oversight, use that technology more effectively than others with more overall experience and knowledge.  Personally, I have found pairing two such people (one with more specific tech knowledge and one with more in-depth knowledge) can be very powerful.  They can learn from each other and your team gets stronger faster.

We must understand who is who, what to expect from each, how to help each grow to their potential and how to get the best code our team can build.  These are complicated enough that each should be another post in what I expect will become a series.

It is a brave new world in which we build applications. There are amazing advances in technology that allow us to build a better product faster than we could ever build in the past.  It has created the opportunity for many more to be able to contribute in meaningful ways.

Thanks for reading!

This is the first of a series of guest posts by Ken Decanio, a software engineer, architect and product manager.  If you would like to read more of his work, you can find find it at his blog.

 

 

[1] Ask me about the DD Array someday.

[2] Actually, it is preferable that a Coder’s knowledge be narrow and focused but that’s another post.

 

 

Photo credits:

  1. http://www.libstock.com
  2. https://www.thefirehoseproject.com
  3. http://www.intelliware.com
2 6 Continue Reading →

Benefits of WordPress + Docker in a production environment + troubleshooting guide.

Hello,

Even though we are a Rails development team, we use WordPress for our blog. We love Rails, but WordPress provides a robust solution for our blogging needs. And lately, we have been experimenting with Docker + WordPress.

So, why Docker?

There are quite a few straightforward reasons why we use Docker with WordPress, namely:

  • Docker reduces clutter.
  • We can run a couple of Rails applications and WordPress instances on the same server.
  • It helps with version conflict management in Rails apps
  • It allows for the duplication of a production server into staging for testing purposes.

 

We can “dockerize” an application, run it in our laptops, run tests and then deploy with confidence that it will work. If you are new to Docker, you should read What is Docker and Dockerizing applications. So, the plan is to apply this process to our application. However, I found some issues while building the instances. I would like to share those issues and how I managed to solve them.

Creating a WordPress instance

There are multiple instances ready to use on the Docker hub.

Try a simple search. I first tried this instance but didn’t like the lack of documentation, and that I wasn’t able to save the uploaded files in a persistent volume. Then I tried this other one but wasn’t able to modify wp-config.php without some hacking. So, I built my own WordPress image instead.

Dockerfile

This file is responsible for creating the docker instance. You can check the Dockerfile instance here. Let me explain each line, so that it all makes sense.

I created a base image php-fpm that allows you to run any PHP application. It is based on ubuntu:14.04.

The first line is to avoid mysql to ask for a root user and password. Then I updated the apt-cache and install curl and mysql client. Last, I removed the apt-cache to reduce the instance size.

These are environment variables that you can change when you run the instance. I think is important to provide with sensible defaults that allow you to quickly test the instance without much thought.

The base image php-fpm has an index.php file that allows you to test it out. I removed it to provide WordPress a clean folder.

This actually installs WordPress. It’s designed to be in one line so that Docker can cache it completely

Then I copy WordPress to the correct folder.

I copy a template to generate wp-config.php. When the instance starts I run the 40_parse_wordpress_config.sh init script. I use bash to render the template using ENV variables. For instance, ${DB_NAME} will be converted to *wordpress_development*. I found the template parser script here.

The issue is that PHP doesn’t inherit the environment variables, so I had to modify wp-config.php in order to set all the variables that the application needs.

I prepare the files to be accessible by nginx and PHP

This line allows you to create persistent storage for the uploaded files and other application changes.

These lines prepare the init scripts.  I add them and set as executable scripts.

When the instance starts it will run each init script. The last one is 50_init.sh from php-fpm which runs nginx and php-fpm. All the previous scripts should execute and stop to allow the init process to work.

Docker compose

Previously known as fig, Docker’s Compose allows you to run multiple Docker instances together. Since WordPress requires a database instance, I created a docker-compose.yml file.

The “build” attribute is the Dockerfile folder. “Links” allows you to connect it with mysql. “Ports” refers to the exposed ports. The format is <host port>:<instance port>. “Environment” allows you to setup the instance

This is to run the database. The password in MYSQL_ROOT_PASSWORD should be the same as DB_PASS

Creating a data volume

In order to run it in production, you need to save the uploaded files. You can use a data volume for that.

In docker-compose.yml you need to add the following:

And add another image to docker-compose.yml:

So every file that is uploaded or modified will be saved in the wordpressdata image.

Just be careful when you are cleaning up docker and removing old images because you could delete wordpressdata.

Another nginx instance.

If you run the example docker-compose.yml it will work. However, WordPress will be connected to port 80 and we won’t be able to run another instance. We need to add another nginx to act as a reverse proxy.

I added the following lines to docker-compose.yml:

And removed the ports attribute in WordPress. Now the proxy connects to WordPress using the env variables and can expose multiple applications.

Here’s where we end for now. In conclusion, after having solved the minor issues mentioned above, we found Docker to be a very useful and efficient tool to manage distributed applications. Give it it a try when you can. Let me know if you have any questions about how our instance works, or check out our documentation here.

Thanks for reading!

F.

0 7 Continue Reading →

Long live your app: MongoDB Replication

The scenario

Not so long ago a customer requested that we provide the highest server reliability that we could. The application was pretty big, with about seven servers per environment. We needed to increase reliability in web servers and databases. In this article, I’ll explain how we achieved that from the database side.

While searching for options, we found that MongoDB (which was the database engine we were using) offered several options that increase your database reliability, i.e.:
MongoDB – Sharding
MongoDB – Replication

We went for MongoDB Replication –also known as Replica Set– because our customer needed a solution ASAP, and the configuration needed is minimal. Besides, it offers whole database replication, which we needed to implement other security measures, like backups.

Enough context for now, so let’s jump into the action!

Configuring MongoDB

First, it is important to notice one thing, as the name states, MondoDB’s Replication will be copying your database ‘n’ times (once for each replica member). With that said, to obtain the maximum benefit from this practice you will want to have your replica set distributed in N different physical servers (i.e. Ubuntu servers). We’ll explain how this works below.

Having N servers set and ready only to practice could be expensive and time-consuming. Therefore, we’ll be performing all of the examples on this article in one single server –your computer– to allocate all of the replica’s members. And we will be creating different folders and pretend that each one of them is a path to a different physical server. So let’s create folders that will be hosting each replica’s member. In our case, our DB’s path is under ‘/usr/local/var/mongodb/’, so we’ll create three folders (a replica needs at least three members) under that route:

If you don’t have your database’s path, you can get it by running this in MongoDB’s terminal:

Within the output command above, locate “dbpath” inside “parsed”.

Example:

With our fictitious servers ready, the next step is to create three “mongod” clients (one for each replica set member).

Note: you need to leave each client running, so you might want to run them in background by appending an ‘&’ at the end of each command. If you need to bring those commands to the foreground, then execute ‘fg’ in the corresponding window where your background jobs are running. Here is an example of how to run mongod clients:

Explaining commands:

mongod: executes mongodb.
–port: indicates the port that the mongod client will be listening to.
–dbpath: specifies the path to your application’s database. In our case, this is pointing to the directories previously created.
–replSet: sets the name of the replica set.

Now you can connect to each mongod client to start configuring the replica set!

Configuring your Replica Set

To configure your replica set, first you need to be connected to one client. You do it by executing the ‘mongo’ command in your terminal and append the –port option to indicate which client you want to be connecting to. In our case, we will be connecting to our client with port 27018:

Once connected, we can start configuring our replica set.

Defining Replica Set Configuration

Configuring the replica set is pretty simple and straightforward, all we need is to set some basic parameters. Go to the mongo shell you just opened and type:

Explaining options:

rsconf: the name of the object or variable that will contain the replica set configuration.

_id: sets the name of the replica set. This needs to match the –replSet option that you set when executing the mongod client. In our case: rs0.
members: sets all the elements that will be conforming our replica set. Elements should be contained within and array following this format:
member’s _id: the element’s identifier
host: the ip address of the server that is hosting the element. It has to be followed by the port where the mongod client is listening to.

Now, apply the configuration to the replica set. In the Mongo shell, type the following:

If everything went OK, you should see this:

You can see additional information by executing rs.isMaster().

At this point, your replica set is up and running!

So you might be thinking now, “so far so good, but I haven’t seen any real advantages yet…”

Let us explain a bit more. What makes MongoDB Replication a good strategy to increase your app’s reliability is the fact that your database will be always available for your web application. You might be wondering ‘How is that possible?’ As you might be guessing, having your database replicated N times gives you the opportunity to prevent data loss and sustain a persistent connection between your database and your web app.

MongoDB Replication consists on having a database replicated N times, where one element is the Master/Primary (this is the only one interacting directly with your app) and the rest of them are Slaves/Secondaries (these are interacting with the master).

This is how it works: when the master element dies, MongoDB automatically chooses a slave element and turns it into a master that will be accepting all IO operations. Your database will be always ‘alive’.

Browsing Secondary elements

You can see your replica set slave elements by logging into a mongo shell for each one of them and typing rs.isMaster(). Don’t forget to set the –port option like this:

Once you are in the mongo shell, the first indicator that you will see is:

Then, after you executed rs.isMaster(), you should see something like:

Where “ismaster” and “secondary” will tell you that that specific client is an slave element. Also, you can see that “primary” option is indicating the IP address of the element that is the master.

And that’s it! You are done configuring MongoDB! Continue next to see how to configure your Rails app to make it work with the replica set.

Configuring the Rails application

If you do not have a Rails application created go ahead and create a new one. Since we are not going to use ActiveRecord, you can append –skip-active-record to the “rails new” command to skip ActiveRecord’s installation and configuration.

Once you have the new Rails app, you will need to include Mongoid in your Gemfile and execute “bundle install”.
Execute “rails g mongoid:config”. This will create config/mongoid.yml where you will have to add the configuration for your app’s db. The configuration of this file is a little bit different than usual, so here you have an example of how it would look for our exercise:

As you can notice, the three hosts are the MongoDB clients we created at the beginning; make sure they are still up and running.

After configuring mongoid.yml, go ahead and create a controller, a model and some views. You can use “rails g scaffold” to speed up the process ;). In this case, we are going to create the basic CRUD for users.

Now go ahead and start the Rails server and add some information in your app.

Checking that MongoDB Replication is working.

Now we are ready to test our replica set, we need to log into the three mongod clients:

Inside the client with port 27018 (the master) check that the database you set inside mongoid.yml was created:

You should see the database you used in your mongoid.yml, in our case we have this:

Then select your database, in our case:

And check that the information you captured in your web app is there:

You should see the data you entered through your web application, in our case:

Now go to the slave elements and repeat the process; you should see the same info that is persisting in your master element. Note: when trying to execute “db.users.find({})” you will receive a restriction error message, that’s normal, go ahead and execute:

NOTE: By default MongoDB protects all the info from being read, so with rs.slaveOk() you are telling MongoDB it is OK for an external user to have access to stored data in the replica set.

Testing database reliability: kill the master

Let’s put al of this to a test. Go to your mongod clients and identify the master. If you don’t remember which one is your master, login into a mongo shell of any of the mongod clients and execute “rs.isMaster();”. In the output, look for the field “primary” to get the IP address of the member that is currently the master/primary.

Our master element is the client running in the port 27018, so we will kill it by executing the following:

Now, go back to a Mongo shell and execute rs.isMaster(). In the output, you should see under “primary” that a new member replaced the one that is dead.

Is my application still working?

Go to your browser and enter new information. Everything should be smooth and quiet as usual, your web app is working correctly.

Reviving the dead member

Once you know your web app is still up and running, then go back and execute the mongod client you just killed in the previous step, in our case:

and log into a Mongo shell for that client:

As you may notice, this member is not longer the master; another member has taken its place and has let it with a secondary role. If you want to know which is the new primary you can use rs.isMaster() again and identify who is the new master.

But what happened with the info save while this member was gone?
Let’s look for it. Connect to the member you just revived and execute next:

You will see all the information that was added through your web app when this replica member was down:

How cool is this?!

Reconfiguring MongoDB Replication

You might have noticed that even if you kill all mongod clients, the next time you execute them the replica set will still be there. Why is this? Do you remember the output when we executed the command rs.initiate(rsconf)? It read something like:

Yes, the configuration was saved! Therefore, every time you execute mongod clients using the –replSet and the same replica set you used before, then that configuration will be applied for that replica set.

Changing, removing, or adding a replica set member is not too hard. Connect to a mongo shell for the primary mongod client, and set a new newConfig object containing all new members, and then apply those changes with:

That’s it! Now you know all the basics of increasing the reliability of your web application by setting up MongoDB Replication!

Conclusion

A downside to MongoDB replication is that it does not matter if your database is replicated N times on N servers, it is still being centralized. Which means that you could bump into networking performance issues if you do not provide your server with enough resources (RAM, processor, hard drive space) to handle your data.

But! Even in spite of the issue mentioned above, we still believe that MongoDB Replication is a good option when you need a server that is reliable, easy to configure, and fast to set up. If you want to learn more about this technique and squeeze all the power of MongoDB Replication, we recommend you read the official documentation to see all the configuration options you can use. For instance, you could set a hidden replica member to create a backup file of your database and upload it to an AWS S3 bucket!

Thanks for reading! Let us know what you think in the comments.

M.

0 5 Continue Reading →