# Introduction

Here I'm collecting some of my thoughts around how to get better at building machine learning based products and services.

I think this is still a somewhat new area and we're collectively still figuring out how to do this. By collecting my thoughts here I am hoping to help others who are facing similar challenges.

## My background

I've personally worked on many of the aspects I think are relevant for doing machine learning "in practice".

* Originally, I started computer science and some of my favorite topics are programming languages and distributed system design.
* I hold a Ph.D. in machine learning and worked in applied and basic machine learning research for more than 10 years.
* I led teams building machine learning projects for several years.
* I have worked in a individual contributor role (a.k.a. principal or staff engineer) with machine learning teams. I did this from a department level up to some topics on a company level.
* I've recently started to consult companies around the topic.

For some reason, I'm seldom content with learning one thing well, but once that happens I'm always asking myself "but what now." This has taken me from fundamental research to the industry. I also have a hard time keeping myself to look beyond whatever scope I am in to figure out what the real solution to a problem is. I think both of these habits have driven me to look into the big picture when it comes to ML in practice and I hope that helps you, too.

I currently expect that this will rather become a collection of essays than a self-contained book.

## How to get in touch with me

Always happy to hear from you, you can either find me on [Twitter](https://twitter.com/mikiobraun) or [Linkedin](https://www.linkedin.com/in/mikiobraun/).


# Data Science, Machine Learning, AI?

The terms data science, machine learning, and artificial intelligence are sometimes used pretty interchangeably. What really is the difference? Here is how I see it.

**Artificial Intelligence (AI)** is a field of research that tries to create intelligent machines. It is almost as long as the idea of computers. Alan Turing is one of the pioneers to think about the topic and proposing the [Turing test](https://en.wikipedia.org/wiki/Turing_test) as a way to define AI.

**Machine Learning (ML)** is an area of research that has been around since at least the 1980s, that aims at creating programs that can automatically learn from data (instead of a human having to specify and write the program explicitly). ML is one possible approach to getting AI, but it is not the only one. In fact, early attempts at creating AI used a more classical software approach, for example by using logic and symbolic reasoning.

**Data Science** originated in the late 2000s, famously being clarified by [Drew Conway's Venn diagram](http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram) as a new kind of job profile for people who are good at business, statistics and ML, and programming.

So given all that I'd say that data scientists can use ML (but don't have to) to create AIs (but could also work on other kinds of "smart" solutions).

In recent years people have started to use AI when they want to imply that the methods are somehow more powerful than just using statistics, but in fact I think most of what we call AI today is still a far way off.


# Introduction

People & Processes

Delivering machine learning based projects is often a team effort. In this part I'll be writing about some of my thinking how to best organize people and projects.

My main insight is that data science based work is different from "normal" engineering work because there is much higher amount of uncertainty that can require a lot of experimentation to figure out.

Other questions are how to best structure teams, and organizations. Introducing data science into an organization is a change process that is not just about hiring data scientists (which in itself is already a big challenge). You need to gain experience with data science, gain trust as an organization that there is some tangible return on investment, and gradually being to scale out data science across the whole organization.


# Data Science Projects

How do they differ from usual engineering projects?

In my experience, data science projects are different from "ordinary" engineering projects (if such a thing exists). The main difference is that data science projects have to deal with a higher degree of uncertainty and risk. 

Can we get the data? How is the data quality? Will we actually find a way to achieve the level of accuracy in our ML models that is required to make this viable? And so on.

Practically this means that the work of a data scientists is often more open ended and [research-y](https://en.wiktionary.org/wiki/researchy). It is often hard to say how long it is going to take. Sometimes it is even hard to say whether it will work or not. That's not an easy thing to say or hear when you depend on a team to do some work.

The work of a data scientists is highly iterative, like in the picture below. 

![](/files/-MSxBkoi88uSo57uDxWL)

There are several different aspects that you need to figure out how to solve. What is the best way to formulate the task as an ML problem? What data do you use? Which method? After each iteration you need to see whether the results look good, or otherwise think of another thing to try and reiterate.

## Managing Risks and Time Boxing

So what's the best way to organize the work of a data scientist into a data science project? I think the two key ingredients are *managing risk* and *time boxing*.

Organize a data science project in a way that you are aware of the biggest risks, and order the work in a way to reduce the risk the most. The idea is to figure out what would endanger the project first, and then think about ways to test in the most efficient manner whether that's the case. And of course, at some point you will start working towards the solution.

Typical examples for risk in data science projects are:

* Do we have the data?
* Can we get the data?
* Does the data have the right quality?
* Does the data has the kind of information we're looking for?
* Is there an ML method that can give us a solution that's as good as we need it to be?
* Is that method fast enough to be trained with the data we have?
* Is the method fast enough/use acceptable amounts of resources when predicting for the use cases we have?
* Is our infrastructure able to support the method or will we have to build something from scratch?
* Do we know how to use the methods and technologies needed?
* Does the approach make sense for the product we're using?

And so on. Any of these can kill the project. If the data is not available, you don't have to think about how to make prediction fast. If there is no method that can solve the problem, you don't have to worry to get the data.

Once you have your risks defined, you can start asking yourself what you could do about the risks. 

If you're not sure whether there is data, you could make a list of people to ask for the data, or you can look into the data lake to check whether the data is already there. 

If you are not sure whether the data has the information you need, you can take a sample of the data set and use a method that has worked elsewhere to try a small scale experiment. 

Or if you are unsure whether the method can be solved by existing ML methods, you can do some literature research to see whether other people have solve such a problem already.

Once you know this, you time box your next step (for example to two weeks), and ask what you could do that would help to get more clarity on one of the biggest risks. The time box is important because many of these questions are open ended and you could spend the next year on trying out ML methods, but the goal should be to get enough information to decide whether to move forward or not.

## Managing Engineering Projects

Now let's compare this with ways to run bigger engineering projects. I think it is important to look at this, because this is what people not experienced with ML will probably assume.

Engineering projects also have to deal with complexity, of course, but I think these tend to be more about complexity and can often be solved by "thinking it through" (which can be hard, though). How exactly are we going to interface to that other service? How can we deal with all that traffic? I'm not saying these questions are easy solve. Sometimes they are very hard, especially if the organization is large and the systems are complex. But for data science (and sometimes engineering, too), thinking it through is not enough. You need to do some work to figure out how to solve it.

There is a kind of uncertainty in engineering, and that's "building the wrong thing." Iterative approaches like agile software development have been designed to make short iterations in order to deal with the risk of spending a lot of time on something that doesn't solve the original problem.

For bigger projects, companies often have a process that begins more experienced people getting together to do the "thinking through" part. For example, you start with an idea what needs to happen. Amazon-style ["working backwards"](https://www.quora.com/What-is-Amazons-approach-to-product-development-and-product-management) process describes the end result from a customer perspective. This helps to focus on the why, and also to not be distracted by thinking too much about the solution. The next step is often to have other, more technical people do the "thinking through" part to come up with a plan that could work, and if that looks good, you finally begin the actual work.

This sounds a lot like the dreaded waterfall model, and I think there are two reasons why it is not exactly the same as waterfall. First of all, instead of doing a lot of detailed planning and then only "execute", you should take the original plan just as a version of what could work, but still take a more agile approach to create the solution, still focussing on delivering increments of work quickly.

Sometimes you need to take such a plan even if you are aware it will take a long time to build potentially the wrong thing, because of the complexity of the work that needs to be done.

## So, What are the Differences?

The main difference is that data science project might take much more time doing exploratory work. It is not uncommon to spend even the majority of your time trying out ideas and experimenting.

![](/files/-MSxDbFzjHp1UzxYwPyp)

Many people will find this quite surprising. How can it be that even highly paid experts cannot guarantee that it will work? Do they really know what they are doing? I think they do, but knowing what you're doing means something different. It means having the experience to know what to focus on first, and how to deal with new information and continually react and improve.

Data science projects are not the only kind of work that has a high research aspect. Developing a new product can also be essentially a research activity. You cannot upfront explain all the steps that need to be done. There is a lot of experimentation required, and in some cases, if you're doing something completely new, there is also the possibility that it might be a total failure.

In my view, seeing these differences clearly, and also communicating it to your colleagues is important to manage expectations and collaborate better.

We've only covered the very high level topic of how to decide what to do in which order. There are many more topics, like how to create data driven products, how to move technology from experimentation to production, and so on, which will be covered elsewhere.

(c) 2021 by Mikio L. Braun


# \[WIP] Writing Software At Scale

Some personal thoughts on writing software at scale

Outline:

* Why are we even writing software at scale?
* Software teams, and "standard agility", five dysfunctions, etc.
* OKRs, radical focus, etc.
* Bigger processes, working backwards, design documents, etc.
* Theory of Constraints
* DevOps


# How Python became the Language of Choice for Data Science

A slightly updated version of my 2013 post.

*Originally posted Nov 20, 2013 on* [*blog.mikiobraun.de*](http://blog.mikiobraun.de/2013/11/how-python-became-the-language-of-choice-for-data-science.html)*. Slightly edited.*

Nowadays Python is probably the programming language of choice (besides R) for data scientists for prototyping, visualization, and running data analyses on small and medium sized data sets. And rightly so, I think, given the large number of available tools (just look at the list at the top of [this article](http://www.talyarkoni.org/blog/2013/11/18/the-homogenization-of-scientific-computing-or-why-python-is-steadily-eating-other-languages-lunch/)).

However, it wasn’t always like this. In fact, when I started working on my Ph.D. back in 2000 virtually everyone was using [MATLAB](http://www.mathworks.de/products/matlab/) for this. And again, rightly so. MATLAB was very well suited to quickly prototype linear algebra and matrix stuff, came with a nice set of visualizations, and even allowed to do some text mining and file parsing if you really needed it to do so.

The problem was, however, that MATLAB was and is actually very expensive. A single license costs a few thousand Euros, and each toolbox costs another few thousand Euros. However, MATLAB was always very cheap for universities, which made perfect sense: That way, students could be trained in MATLAB so that they already knew how to use it to solve problems and companies would then be willing to pay for the licenses.

All of this changed significantly in 2005 or so. At that time I was working at the Fraunhofer Institute FIRST, which belongs to a group of German publicly funded research institutes focused on applied research. Originally, Fraunhofer institutes could get the same cheap university licenses, but then Mathworks changed their policies to the effect that you could only get the university rate if you are an institution which hands out degrees.

This did not hold for most publicly funded research institutes all over the world, like the Max-Planck-Institutes (like the one in Tübingen where Bernhard Schölkopf is), or the NICTA in Australia where Alex Smola and others were working at the time. So we decided something had to change and we started looking for alternatives.

Python was clearly one of the possible choices, but at the time other opportunities seemed possible as well. For example, [octave](http://www.gnu.org/software/octave/) had been around for a long time and people wondered whether one should not just help them to make octave as good as matlab and fix all remaining compatibility issues. Together with [Stefan Harmeling](https://www.cs.hhu.de/en/research-groups/machine-learning/our-team/team/harmeling) I started phantasizing about a new programming language dubbed *rhabarber* (the [repo](https://github.com/mikiobraun/rhabarber), originally hosted on Google Code, still exists) which would allow to extend even the syntax dynamically to be able to have true matrix literals (or even other things). Later I would play around with [JRuby](http://jruby.org/) as a basis because it allowed better integration with Java to write high performance code where necessary (instead of doing painful low-level stuff with C and [swig](http://www.swig.org/)).

If I remember correctly, the general consensus was already back then that Python would the language of choice. I think early versions of [numpy](http://numpy.org/) already existed, as well as early versions of [matplotlib](http://matplotlib.org/). [Shogun](http://shogun-toolbox.org/), which had been developed and used extensively in our lab, had already begun to provide Python bindings, and so on.

I personally always felt (and still feel, even in 2021) that there are things where MATLAB is still superior to Python. MATLAB was always a quite dynamic environment because you could edit files and it would reload the files automatically. Python is also somewhat restrictive with what you can say on a single line. In MATLAB you would often load some data, start editing the functions and build you data analysis step by step, while in Python you tend to have files which you start from the command line (or at least that’s how I tend to do it).

In any case, early on there was also the understanding that we should focus our efforts on a single project and not have the work scattered over several independent projects, so we planned a workshop at NIPS 2005 on this, but unfortunately the workshop was rejected. However, engagement was so high, that we just rented a seminar room in the same hotel where NIPS was going to be held on the Sunday before the conference, notified all people we thought would be relevant and had the [Machine Learning Tools Satellite Workshop](http://www.raetschlab.org/workshops/MLPython) the day before the NIPS conference.

The hot contender back then was the Elefant toolbox designed by Alex Smola and collaborators, which was a pretty ambituous project. The idea was to use [PETSc](https://www.mcs.anl.gov/petsc/index.html) as the numerical back end. PETSc was developed in the area of large scale numerical simulations and had a number of pretty advanced features like distributed matrices and similar things. I think ultimately, it might have been a bit too advanced. Simple things like creating a matrix were already quite complicated.

I also gave a talk together with Stefan on rhabarber, but most people were skeptical whether a new language was really the right way to go, as Python seemed good enough. In any case, things really started to get going around that time and people were starting to build stuff based on Python. Humans are always hungry for social proof and having that one day meeting with a bunch of people from the same community gave everyone the confidence that he wouldn’t be left alone with Python.

A year later, we finally had our first [Machine Learning Open Source Workshop](https://mloss.org/workshop/nips06/) which eventually led to the creation of the [MLOSS track over at JMLR](http://jmlr.org/mloss) in an attempt to give scientists a better incentive to publish their software. We had [several iterations](https://mloss.org/workshop) of our workshop, had Travis Oliphant give an intro to numpy, invited John Hunter, the main author of matplotlib who sadly passed away in 2012, as well as John W. Eaton, main author of octave, and also had more workshops (although without me). Somehow, the big, open, interoperable framework didn’t emerge, but we’re still trying. Instead there exist many framework which are wrapping the same basic algorithms and tools again and again.

Eventually, Elefant didn’t make the race, but other toolboxes like [scikit- learn](http://scikit-learn.org/stable/) became common place, and nowadays we luckily have a large body of powerful tools to work with data, without having to pay horrenduous licensing fees. Other tools like [Pandas](http://pandas.pydata.org/) were created in other communities and everything came together nicely. I think it’s quite a success story and having been minor part of it is nice, although I didn’t directly contribute in terms of software.

Now in 2021, we have seen so many more additions, and Python is really the de facto standard platform for doing data science. [Tensorflow](https://tensorflow.org/) and [pytorch](https://pytorch.org/) are not written in Python, but that is the main interface that people use. Python also became one of the main languages for serverless services. New frameworks like [Ray](https://ray-project.github.io/) are using Python as their main interface. Python has even added more operators to the language to make it easier to express matrix computations.

Interestingly, I never became that much of a Python enthusiast. I wrote my own stuff in JRuby, which lead to the development of [jblas](http://jblas.org/), but at some point started working on real-time analysis stuff where I just needed better control over my data structures, and used Java and Scala for that. In 2020 I started working more with Python, and I'm still not exited, but the word I associate most with it is **solid**. Depending on how you use it, it is quite fast. It did a good job at being open and extendable. Some newer features like type hints are nice. I still like Scala's collection API, though. 

If you have stories to share (or corrections) on the “early years of Data Science”, I’d love to hear from you.

(c) 2013, 2021 by Mikio L. Braun