We build your Software

We are currently working on a big data project that deals with a lot of structured and unstructured data. Part of the application design is to work with different database systems. MySQL/Aurora is the chosen platform for the structured data. Everything is nicely split up in different tables and linked together to make sense. MongoDB serves as the first layer for sightly less structured data. Well, to be fair: All data is structured somehow, but here the structure is not as firm as it is in MySQL. Our MongoDB database summarizes and captures data from the MySQl so that we don’t have to run complicated queries to get what we need. The latest layer consists of ElasticSearch. The idea here is to not only save all this fun data but to be able to analyze it and use it in clients (websites & apps) while having the ability to perform complicated and fast searches. The primary purpose here is performance and caching - no data is solely saved in ES.

The way the data is layered involves it being passed through these database systems. From outside sources into the SQL database, from there into MongoDB and from there and from the SQL into ElasticSearch. This project could have been implemented with any of the popular web technologies: PHP, Ruby, Python, Go or Swift. We had decided to use the Vapor framework in Swift for this project. We wanted to use a programming language that is easy and fun to write while allowing for complex data structures to be handled rather easily. The new and upcoming Vapor 3 framework has been structured entirely around asynchronous behavior.

We are using one micro service that is importing the data into ElasticSearch. To gather all the data we are sending multiple requests to the core micro service (MySQL-backed) and the summary service (MongoDB-backed). Vapor 3 allows us to send all necessary requests at once and return, once finished, the unified result. The resulting structure looks roughly like this:

func getResults(_ request: Request) throws -> Future<String> { 
    var futures:[Future<Result>] = []
    future.append(self.getMongoDBResults())
    future.append(self.getSQLResults())
    // ...
    return future.flatten(on: request).map(to: String.self) { results in 
        for res in results {
            // “res” is now a result from the futures above
        }
        return “just a demo”
    }
}

While this code may not look very interesting, what makes it so powerful is that all requests in the _Future_ array are send out at the same time. So if we have 100 requests that are in the array, they are all sent out at the same time (or just about). In essence this means that it will take us only the duration of one request to get them all in comparison to doing this sequentially. That makes 100x faster than a sequential way of gathering this data.

Now, it should be noted that the same approach works with the other languages/frameworks as well (even PHP as Hack e.g.) but at the same time it needs to be said as well that none of the other languages and their frameworks, with the exception of Go and some Java frameworks, have been designed to operate this way. Vapor 3 was designed specifically for and with this in mind (that is Promises and Futures).

Due to the native nature of Swift and the asynchronous behavior we are able to build an extremely fast and powerful application that does not just save our time but also server resources. It will be exciting to perform extensive benchmark tests once the application is ready and actually handles millions of data entries. We will keep you posted!