A duel with data: optimizing flat-file validation

How a software company transformed a client’s request into a functional commercial solution.

For our first article on Medium, we’d like to give a special “thank you” to all readers for their support and attention. In the following paragraphs, we’ll share an anecdote with hopes to inspire and guide others facing similar dilemmas.

Image for post
Image for post

The advent of digitalization in the business world has been the catalyst for great progress and achievement. One of the biggest problems surrounding digitalization, though, is the need to manage and validate the large amounts of data that revolve around these newly digitized business processes. As such, it is crucial for businesses to adopt tools that can quickly load, control, and validate the often heterogeneous and unstructured data.

Our story begins here: when an important client in the insurance sector asked us to develop a tool for (positional) flat-file data validation. Most general-purpose tools focus solely on relational data which is structured and therefore much simpler to validate. However, flat-file data has little structure, meaning more complexity and thus, a higher degree of difficulty with verification.

In the early stages of development, it was brought to our attention that no commercially available tool matching these specifics existed yet. At first, we were in disbelief. There was no way that something like this hadn’t already been created and optimized. Yet, after several rounds of thorough research, we confirmed that there was no such tool commercially available. The few that could get the job done were either obsolete or packed within much larger (and expensive) ETL packages, leaving us with one solution: custom development of the tool.

For us, it was a no-brainer. This was an incredible business opportunity staring us right in the face and we had to act on it. So, in agreement with our customer, we decided to create a solution commercially available to other businesses as well. Herein lies the birth of our product: Sherlock, the lightweight and highly configurable flat-file validator. Why the name Sherlock, one may ask? Well, just as accuracy, scrupulousness, and infallibility define the famed detective, they also serve to describe the tool’s core strengths.

The product reviews speak for themselves as Sherlock has yet to disappoint. It has proved to be both effective and versatile, extending across various types of businesses to successfully validate their (positional) flat-file data. Headlined by its high configurability and lightning-fast runtime performances, our customers have been raving about this product:

“It is a very efficient and performing tool. I used to run all those controls by hand, wasting a lot of time that I can now employ more productively”.

“My work is so much easier now. Working in the banking sector I have the daily necessity to validate millions of flat-file records that are critical for many of our business’ needs. With Sherlock, I only need to schedule and configure the validation, and the tool takes care of the rest”.

“Even though the tool has no graphical interface, its batch execution makes it perfect for the job”.

We enjoyed sharing our experience with you and hope that our insight can help you navigate the difficulties associated with data validation. If you’re interested in optimizing such processes within your business, check out sherlock-validator.com to learn more about our amazing product!

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store