Intent Media is now Intent
We use cookies, which include essential cookies and cookies to analyze our traffic. For more information please visit our privacy policy
  • Posted by Intent Media 24 Feb

In Search of Diversity: Part 2

This is the second post in a two part series.  See the first post here.

Creating diversity

In neither machine learning nor in startups do we need to take things as they come. When we find a lack of diversity, we can act to preserve the diversity that we do find or even induce the emergence of diversity.

To do this for ML models all we really need is some sort of metric to evaluate the amount of diversity in a given set of models. Some examples might include simple percentage agreement, variance, separation, or any one of several common distance metrics for comparing vectors of predictions. From there, we can try to construct the set of models with the maximum amount of diversity through things like random selection of training instances, varying fitness function parameters, etc. Depending on how you approach this problem, you might run into computation explosion, but there are definitely “good enough” solutions possible that can consistently produce diverse ensembles of models.

So then, what are the corollaries for a startup or a tech team? If preserving and inducing diversity is a tractable problem for ML models, how about something more complex, like people? How can we encourage diversity in our tech teams?

We know that we have a problem with achieving meaningful diversity in technology company employment. Imbalances occur in gender, race, and other harder to characterize elements of diversity. It seems like this every week, there’s some new discouraging report about just how insidious this problem is. Diversity in technology is a big, hairy problem that some of the richest corporations in the world haven’t cracked. I won’t be solving it in this blog post, but I’d like to propose one strategy to make things better.

Islands unto themselves

Being a former biology guy, I’m inclined to look to evolution for the answer to this problem. Fun fact: evolution loves diversity. In fact, it can’t happen in the absence of diversity. Random mutations are critical for the emergence of life from ooze.

Getting a bit closer to the mechanics, diversity in the population as a whole can often come from protected sub-populations (sometimes called demes). The classic example comes from Darwin’s observations of the various species on the Galápagos Islands that led him to develop his theory of natural selection.

“Darwin’s finches by Gould” by John Gould (14.Sep.1804–3.Feb.1881) — From “Voyage of the Beagle” as found on]; also online through Biodiversity Heritage Library at Licensed under Public Domain via Wikimedia Commons —

“Darwin’s finches by Gould” by John Gould (14.Sep.1804–3.Feb.1881) — From “Voyage of the Beagle” as found on]; also online through Biodiversity Heritage Library at Licensed under Public Domain via Wikimedia Commons —

Small populations, which have often gone through some sort of population bottleneck, are less diverse than the population as a whole. As they continue to evolve they can evolve differently than the main population, adding to the diversity of the entire population. By only having to solve the distinct problems of their habitat, they create potentially novel, unique solutions that the main population might never have found.

Evolving tech teams

Taking the lead from evolutionary biology, I would propose that a powerful strategy for encouraging diversity in teams is to break them down into smaller, more autonomous units with local decision-making power.

Recently at Intent Media, we started making some big changes to our application and organizational structure along these lines. Part of the motivation was Conway’s law. In its original formulation, Conway’s law is:

…organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

So, if our application architecture is going to reflect our org chart, and we don’t want a non-scalable monolithic architecture, then we don’t want a non-scalable monolithic org chart. We want small, agile, autonomous teams free to take on responsibilities and decisions at the squad level.

It’s definitely early days for us, but I’m optimistic that this will be the right approach for taking this startup to the next level (i.e. total world domination). It takes advantage of some of the benefits of the shared worldview that Levchin wistfully describes, allowing teams to build trust more effectively. I also hope that it will create a more welcoming environment for a more diverse range of team members.

For example, the data engineering squad that I’m on has historically shared a love of beer, beards, and well-tested functional code. However, as our team grows, we may have to adapt our micro-culture to accommodate people who don’t love beer or beards.  As we search beyond South of Houston to as far as South America, we will find individuals who will make great team members but are different than the people currently on the team. That will mean that our squad cohesion may now need to be based around a shared love of falafels, lap dogs, and well-tested functional code. This is a totally feasible adaptation that will allow us to survive and grow, made all the easier by our relatively small size.

Small teams like ours also mean that another squad can bond over something like biking around Brooklyn and that doesn’t prevent a non-cyclist from finding a social group he or she can engage in given the diversity of the larger technology organization. In technical terms, this allows different squads to choose Clojure, Ruby, Java, or whatever other tool they think makes sense. Other big ambitions are to enable a micro-services architecture and support a true DevOps mindset, where squads have more locally defined ownership of their applications’ operations.

Moreover, small teams can make more democratic and egalitarian approaches to decision making easier. I’ve talked about this in some detail in another post, but the short version is that I firmly believe that in teams and in societies everyone should have a voice. Small teams, unconferences, and other egalitarian organizational techniques can create fairer, more productive groups of people, who solve their problems effectively as a group.

When you’re in a fast moving startup, getting the right people into the company is crucial. People with the abilities to build a successful company can be hard to find. So you simply can’t afford to not be a welcoming, open place for people of all races, genders, and backgrounds. The people who thrive in fast moving startup environments are the ones who want to connect with their peers and build something collaboratively. They deeply appreciate the opportunity to work in diverse teams that help them grow as a professional and as a person. When recruiting great people like this,  it can be crucial to show them small, agile teams where they will get to have a real impact, regardless of their background, because that’s the sort of opportunity that they want and deserve.

Big, hairy problems

I love working in startups and on machine learning problems. In both of the contexts, you’ll find smart, motivated people who are optimistic about their chances taking on what are, frankly, big, hairy problems. In machine learning, we’ve developed approaches that preserve and induce diversity in our models. This has led to the tremendous success of techniques like random forests.

In technology organizations, I think that we still have a lot of work to do to really understand and apply the insights of research into the power of diversity. I believe that small, autonomous teams are a great tactic for structuring organizations in ways that encourage more diversity. Beyond that, I’m optimistic that we can continue to iterate and find new approaches to building more diverse tech teams. Moreover, we should. Diversity is important and worth working for.

Jeff Smith is a data engineer at Intent Media working on large scale machine learning systems.  He has a background in AI and bioinformatics.  Intent Media is the fifth startup he’s worked at, and it’s easily the most fun one.  You can find him tweeting, blogging, doing more blogging, and drawing comics all over the internet.

Post Comments 0