Ultimate by the Numbers: Designing a Data Driven Program with Alex Davis

What gets measured gets managed. But how can you be sure what you're measuring at tryouts will reflect performance on the field?  At a time when fitness combines are becoming more popular as part of the tryout process, we're lucky to have Alex Davis, one of our favorite data scientists and URCA Contributors to answer this question.

Davis is the coach of Ultimate Canada's U24 programme. As such, he knows the special difficulties that come with selecting a small group of players from hundreds of candidates, using a limited amount of observation time.  

We've all seen fast players who can't time a cut properly. So how much does measuring a 40 meter sprint tell us about a player's potential on the field? For a player who does not make the cut, how do we know when to advise them to work on their speed or agility versus their throwing? Can we give more targeted advice to make a real difference in player development? 

With Alex's characteristic intellectual rigor, he breaks down things we intuitively know abut the tryout process into logical components.  This is one of 130+ presentations by coaches around the world.  To get access to ALL of our UAP presentations, join our global community of ultimate coaches in the UAP Classroom.

 In his UAP Classroom presentation, Alex Davis will share some of his experience with leading tryouts for Team Canada to teach lessons that team leaders can apply to their tryouts, regardless of the level of competition. The nature of national team tryouts exacerbate many common challenges for tryouts—how do you ensure that every player is fairly evaluated? How do you compare players from different tryouts? How do you give constructive feedback to players, so that they can know what to work on to improve their chances next time?

While you may not be faced with the challenge of selecting your national team, here are a few things all coaches can use to enhance the tryout process.

Alex’s talk will cover these challenges in depth, and provide some creative, practical solutions to all of them. Below are a few pieces of advice that are sure to make your tryout process more efficient, rewarding, and satisfying for the players competing.

Step 1: Get Help

If you are one person trying to evaluate 30-50 different players, it is going to be almost impossible for you to effectively assess every single player and give actionable feedback. If your team has captains or other senior returning members, ask them to participate in the tryout as organizers and evaluators, rather than players. You can ask other coaches in your youth league to help. You can even recruit experienced community members. As long as you trust that this person understands good ultimate, they are going to help you pick the best team possible.

Step 2: Systematize The Process

Since you’re going to have all of this help, you will need to take steps to make sure that everyone’s evaluations are consistent. This isn’t an issue when taking objective measurements like 40s or vertical jumps, but how can you compare two players’ ability in throwing breaks if two different evaluators describe them as “solid?” Alex’s answer to this issue is beautifully simple: assign each evaluator specific sets of skills to evaluate, and measure players’ ability on a relative scale to each other. This is illustrated in the image below.

For a given skill (say, break throws), a player is placed on a scale ranging from -2 to 2. A score of 0 means that a player’s break throws were on par with the rest of his or her group. A score of 1 represents a player with above-average breaks, while a player earning a 2 will have dominant throws compared to the rest of the group, likely breaking the mark at will. This evaluation method should be used to measure as many skills as possible, to develop the most complete picture of each player.

A key to making this method effective is to make sure that each evaluator understands that 0 is the default score for any player, and they may be moved up or down depending on their performance. You should expect to see a majority of players earn a 0, with a few earning 1s and -1s, and almost nobody (perhaps no one at all) earning a 2 or -2.

Step 3: Compare Measurements and Fill Roles

Now that you can objectively compare these subjective evaluations of players, you may notice a few players with consistent 1s and 2s across the board. These players will clearly be shoe-ins for the team. The difficulty lies in selecting the rest of the roster, when you have to weigh the strengths and weaknesses of players.

Discuss with your team leadership to determine what roles you need to fill, and how many players you want to select to fit each role. Then, using the measurements, identify which players look to have good potential to fill that role. At this stage, you will doubtless have to rely on truly subjective opinions to pick players. However, by taking steps to make this process as objective as possible, you can limit the uncertainty between different players’ abilities and hopefully make these decisions somewhat easier.

Alex’s method to identify who is a good fit to fill each role involves a good bit of data science and machine learning. It is too much to discuss in a brief blog post, but it is fascinating to hear him describe his process and illustrate his results, so you should definitely check out the video!

Step 4: Give Feedback

Not every player will make the team. This harsh reality is the reason that tryouts are necessary in the first place. When you cut a player, many will seek feedback about where they could improve to have a better chance of making the team in the future.

Because you are armed with a list of measurements relative to other players, you can easily show a player what areas of their game are lacking, and where they are strong. Coupled with some subjective notes from whoever was responsible for evaluating this player, you will be able to give each player an actionable list of skills to work on before next tryout. In addition, if the player tried out for a highly competitive role, and you feel that they would be able to compete for a different, less populated role by shifting their focus, you can relay that information as well.

Feedback doesn’t stop at the players you cut though! The players you do select will also benefit from similar feedback. You can use your measurements from tryouts to let players know where to improve so that they will best be able to fill the role you have chosen for them to play. The list of relative strengths and weaknesses can help you communicate to these players why you have selected them for each role, and why they need to focus on certain items.

Alex’s talk goes into great detail on each of the above points, and explores other areas related to hosting tryouts, be sure to check it out if you have ever wanted to run your tryouts more effectively.

Join our UAP Classroom so you can...

• Watch this presentation at your convenience!
• Access our whole back-catalogue of UAP Conference talks - 10 years and counting!
• Get trained and certified as a UAP Coach
• Talk with other like-minded ultimate coaches!
• Take our mini-courses about Film Analysis, the Physics of Disc Flight, and Mental Toughness Training