The Programmatic Assessment Puzzle: Why Your Training Program Needs a Dashboard That Thinks Like You Do
- Tess
- Apr 17
- 9 min read
Updated: Apr 20
Programmatic assessment promises to transform how we judge trainee competence. But without the right lens, it can just as easily drown educators in data. Here's how configurable analytics dashboards turn scattered data points into a coherent picture of learner progress — and why one-size-fits-all reporting will never get you there.
The Promise and the Problem
If you've been involved in medical education leadership over the past decade, you've watched programmatic assessment move from a theoretical proposition in the literature to a design principle that specialist colleges are actively embedding into their training programs. The logic is compelling: rather than relying on a handful of high-stakes examinations to determine whether a trainee is competent, we collect many low-stakes data points across different contexts, from different assessors, over time. No single observation is pass/fail. Instead, competence decisions emerge from the pattern.
In practice, this means a modern training program generates a considerable volume of assessment data for each learner. Mini-CEX encounters, direct observations of procedural skills, multi-source feedback, case-based discussions, supervisor reports, reflective entries, logbook data, examination results... the list varies by college and specialty, but the direction is universal: more data points, from more sources, more often.
This is good for the validity of our competence judgements. But it creates a genuinely new operational problem for the people who have to use this data; Directors of Training, supervisors, clinical competency committee members, and the trainees themselves. When a training program might collect dozens of data points per trainee per term, the question stops being "do we have enough information?" and becomes "how do we see what this information is telling us?"
The core tension: Programmatic assessment works precisely because it distributes evidence across many small observations. But that same distribution makes it difficult for any single person to hold the whole picture in their head. The assessment framework is only as good as the tools that synthesise it.
Why Spreadsheets and Static Reports Fall Short
Most Directors of Training will recognise the experience of preparing for a mid-term or end-of-term review by pulling together data from multiple sources — an LMS here, a logbook there, a supervisor report filed by email, a mini-CEX completed on paper and later entered into a system. Even when all of this data lives within a single platform, the default reporting view is often a flat list or a summary table: 14 of 20 workplace-based assessments completed, average entrustment rating 3.2, supervisor report received.
These reports answer the question "what has been done?" They rarely answer the more important questions: "what does this trainee's trajectory look like?", "where is the pattern of concern?", or "which trainees in this cohort need my attention this week?"
The gap between raw data and educational insight is precisely where a well-designed analytics dashboard earns its place. But "well-designed" does not mean "comprehensive", it means relevant to the people using it. And this is where the one-size-fits-all approach breaks down.
Every Program Sees Through a Different Lens
Consider the difference between what matters to a Director of Training in a surgical program and their counterpart in psychiatry. The surgical program may prioritise procedural logbook completion, operative entrustment ratings, and whether trainees have achieved minimum case numbers for specific procedures. The psychiatry program may care far more about longitudinal multi-source feedback trends, communication skill assessments, and reflective practice entries. A general practice training program has different priorities again; breadth of clinical exposure across rural and urban rotations, progress across multiple curriculum domains simultaneously, and supervisor continuity.
All three programs are practising programmatic assessment. All three are collecting rich, longitudinal data. But the dashboard view that makes sense for a surgical DoT would be largely useless for their colleague in psychiatry, and vice versa.
The case for configurability: A dashboard should not force every training program into the same reporting template. It should let each program surface the metrics, visualisations, and alerts that reflect how that program thinks about trainee progress. The goal is a view that matches the educational logic of the curriculum, not the technical structure of the database.
What configurability looks like in practice
In MyProgress, configurability means that a Director of Training or program administrator can shape their dashboard around the questions they actually need to answer. This includes choices such as:
Which assessment types to foreground
A surgical program might place procedural entrustment trends front and centre, while a physician training program might lead with workplace-based assessment completion rates across competency domains.
How to visualise trajectory
Some programs want to see entrustment ratings plotted over time to reveal growth curves; others prefer heatmaps that show competency coverage across the curriculum at a glance.
What triggers an alert
Configurable thresholds let programs define what counts as "falling behind" in their context, whether that's fewer than three WBAs in a four-week block, an entrustment plateau, or a missing supervisor report.
Who sees what
A clinical competency committee reviewing all trainees in a cohort needs an aggregate view; an individual supervisor needs a focused view of their own trainees; the trainee themselves needs a self-directed progress view that supports learning conversations.
Connecting the Dots: From Data Points to Competence Narratives
The deeper value of a configurable dashboard is not just operational efficiency, it's epistemological. Programmatic assessment rests on the idea that competence is not visible in any single data point but emerges from the pattern across many. The dashboard is the tool that makes patterns visible.
When a clinical competency committee sits down to make a summative recommendation about a trainee, they should be looking at a synthesised picture, not flipping through individual assessment forms. A dashboard configured to their program's assessment blueprint can show them, at a glance, how a trainee's entrustment ratings have trended across the term, which competency domains are well-evidenced and which have gaps, how this trainee's profile compares to the cohort trajectory, and whether there are qualitative themes emerging from narrative comments.
Crucially, that picture needs to include more than just workplace-based assessments. A genuinely complete competence narrative draws on evidence from across the training program, and much of that evidence originates in other systems. MyProgress is designed to pull in data from external sources so that the dashboard reflects the full breadth of a program’s assessment strategy, not just the components that happen to live natively in the ePortfolio.
Bringing the whole picture together: MyProgress can integrate results and feedback from continuous self-assessment of knowledge platforms, giving educators visibility of how trainees are tracking against the curriculum’s knowledge base throughout the year, not just at examination points. It can also pull through results from high-stakes OSCEs and other summative clinical examinations, so that when a competency committee reviews a trainee’s dashboard, they see workplace-based assessments, knowledge assessment trends, and examination outcomes side by side in a single view.
This matters because programmatic assessment only works when the synthesis is genuinely comprehensive. If a trainee’s OSCE results sit in one system, their knowledge self-assessment data in another, and their WBAs in a third, no single dashboard can tell the full story, and the committee is back to piecing things together manually. Integration is what makes the difference between a platform that stores data and one that supports decision-making.
This is the difference between a committee spending its limited time finding information and spending it interpreting information. The educational decision-making improves because the cognitive load shifts from data retrieval to professional judgement, which is where experienced educators add the most value.
The Monday Morning View
Not every interaction with a dashboard is a high-stakes committee meeting. In many ways, the most valuable dashboard use is the routine glance, the Monday morning check that lets a Director of Training see where things stand without having to ask anyone or run a report.
A well-configured operational view might surface information such as:
Which trainees have not had a workplace-based assessment recorded in the past three weeks
Which supervisor reports are overdue for the current rotation block
Which trainees have an upcoming progress review and whether their evidence portfolio is on track
Any trainees who have triggered an early-warning threshold
Tip: The most effective dashboards don't try to show everything. They surface what needs attention now and let the user drill down when they need more detail. Start by configuring your view around the three or four questions you find yourself asking most often, then refine from there.
This kind of proactive visibility is particularly important in the Australian context, where Directors of Training often carry the role alongside a full clinical load. The dashboard should reduce the administrative burden of staying across the program, not add to it. If a DoT has to spend 30 minutes generating and reading a report to find out that one trainee hasn't been assessed in a month, the system is not working hard enough.
Cohort Views: Improving Teaching, Not Just Tracking Learners
One of the less discussed but most powerful applications of configurable analytics is the cohort-level view. When a Director of Training can see aggregated, de-identified data across all trainees in a rotation or training site, patterns emerge that are invisible at the individual level.
For example, if entrustment ratings for a particular procedure are consistently lower at one training site compared to others, that is unlikely to be a coincidence of trainee ability. It may point to differences in supervision quality, case mix, or teaching culture at that site. Similarly, if multi-source feedback scores for communication dip for most trainees during a particular rotation, the issue may be structural rather than individual.
These insights shift the conversation from "this trainee is underperforming" to "this part of our program may need attention." That is a fundamentally different, and more productive, use of assessment data, and it is only possible when the analytics layer can aggregate and compare across a cohort in a way that each program defines for itself.
Accreditation as a Byproduct, Not a Scramble
Australian specialist colleges operate within accreditation frameworks set by the Australian Medical Council, and individual training sites are accredited by their relevant college. In both cases, accreditation visits require evidence that the program is functioning well: that trainees are being assessed regularly, that progress decisions are being made on a sound evidentiary basis, and that the program has mechanisms to identify and support trainees in difficulty.
When a dashboard is configured to reflect the program's assessment blueprint and accreditation standards, the evidence that accreditation panels ask for becomes a natural output of day-to-day operations rather than a retrospective exercise in data archaeology. Completion rates, assessment distributions, progression decisions, early intervention records, all of this is visible in the dashboard because it is what the program uses to manage itself.
Key principle: If the dashboard genuinely supports how you run your program day to day, accreditation evidence should be a byproduct of normal operations, not a separate project. The best time to prepare for an accreditation visit is every Monday morning, not the month before the panel arrives.
Empowering Trainees to See Their Own Progress
The conversation so far has focused on educator-facing views, but the trainee-facing dashboard is equally important, and arguably more aligned with the principles of adult learning that underpin postgraduate medical education.
When trainees have access to a clear visual representation of their own progress, they are better equipped to identify gaps in their evidence portfolio, prepare meaningfully for supervisor meetings and progress reviews, take ownership of seeking out specific learning opportunities, and self-monitor their development trajectory against expected milestones.
This is not about surveillance, it is about shared visibility. A trainee who can see that they have strong evidence in clinical management but a thin portfolio in the professional and ethical domain can raise that in a supervision conversation themselves, rather than waiting for someone else to notice the gap. The progress view becomes a tool for self-regulated learning, not just institutional oversight.
Getting Started: A Practical Path Forward
If your program is currently relying on flat reports or manual data collation, the prospect of a fully configured analytics dashboard might feel like a significant leap. It needn't be. A practical starting point looks something like this:
Identify your three most-asked questions. What are the things you find yourself checking, chasing, or worrying about most often? Overdue assessments? Trainees without recent supervisor contact? Procedural logbook progress? Start there.
Map your assessment blueprint. What data points does your program collect, and which of them are the strongest signals for the competence decisions you need to make? Your dashboard should foreground these.
Define your alert thresholds. What counts as "falling behind" in your program? Make it explicit and build it into the system so that the dashboard tells you when to act, not just what has happened.
Think about your audiences. The DoT, the supervisor, the competency committee, and the trainee all need different views of the same underlying data. Configure each view for its purpose.
MyProgress is designed to support exactly this kind of graduated, program-specific configuration. You don't need to build the perfect dashboard on day one. You need a platform that lets you start with what matters most and evolve your view as your program's approach to programmatic assessment matures.
Want to see how your program's data could look? We're always happy to walk through a configuration session using your own program structure. Get in touch with the MyProgress team to arrange a conversation.
