21 Mar 2019
In the past few weeks I have been participating in competitions on kaggle.com. Kaggle is an awesome platform where people solve data science problems and compete to provide the most accurate solutions. This has led me to think more deeply about the increasing use of machine learning in software and its prevalence as a new business buzzword.
What strikes me as startling about neural networks is how effective they are at solving problems in general. When I compare two solutions I have submitted for Kaggle the differences are really negligible. All that really changes are the details of the problem at hand: what portions of the data I am concerned with and the format of my answer. The core of the solution often stays the same. As a software developer, this is a complete paradigm shift and a very interesting way to approach to problem solving.
What this means for real-world scenarios is that we can now start to solve a wide catalog problems with very little actual code. Furthermore, code that predicts the movements of the stock market can be adapted to predict the population growth of fresh-water dolphins with relatively little effort.
Granted, the people who are winning Kaggle competitions are using sophisticated metrics to decide how to architect their machine learning solutions. To make a winning solution, you still need to have a deep insight into mechanics of the learning process. However, I think the massive breakthrough of modern machine learning is that we can get a decent answer very quickly with almost no domain knowledge. At the moment my understanding of machine learning theory is still largely on the surface, but I have still been able to submit solutions that are surprisingly accurate.
After some thought, I believe that the hype behind machine learning is at least partially warranted. It has the potential to be an immensely productive software system. I think that all software engineers should investigate libraries like keras just to have it in their toolbelt. My expectation is that machine learning will become increasingly important in solving modern problems with technology. Watch this space!