Assessment 2: Critical Evaluation Report¶
Overview¶
| Element | Details |
|---|---|
| Weighting | 35% |
| Due Date | End of Week 8 |
| Word Count | 2,500 words |
| Learning Outcomes | LO1, LO2 |
| Format | Governance committee briefing paper |
Task Description¶
Critically evaluate a healthcare AI tool for potential deployment in an Australian aeromedical or critical care context. Produce a recommendation report structured for a clinical governance committee audience.
This assessment builds on your Technical Foundations Analysis (Assessment 1), adding critical evaluation of safety, ethics, and regulatory considerations.
Requirements¶
Your report must address the following five components:
1. System Description (approximately 400 words)¶
Describe the AI system and its intended use case: - What does the system do? - How would it be used in the clinical workflow? - Who would use it and when? - What decisions or actions would it inform?
Draw on your LO1 knowledge but focus on application context.
2. Safety Risk Analysis (approximately 600 words)¶
Analyse the safety risks of deploying this system: - What failure modes could occur? - What is the potential severity of failures? - How might algorithmic bias affect patients? - What human-AI interaction risks exist (automation bias, alert fatigue, etc.)? - How might performance degrade over time? - What additional risks arise in aeromedical/critical care contexts?
Consider both technical and sociotechnical failure modes.
3. Ethical Analysis (approximately 500 words)¶
Evaluate the ethical implications of deployment: - How does deployment align with bioethical principles (autonomy, beneficence, non-maleficence, justice)? - What transparency and explainability considerations apply? - What consent requirements should apply? - What equity implications exist? - Are there professional ethical considerations?
Apply ethical frameworks, not just list concerns.
4. Regulatory and Legal Assessment (approximately 400 words)¶
Assess compliance with Australian regulatory requirements: - What TGA classification would apply? - What evidence requirements must be met? - What post-market obligations exist? - What professional/legal considerations apply to users? - What organisational obligations arise?
5. Recommendation (approximately 600 words)¶
Provide a justified recommendation to the governance committee: - Should deployment proceed? (Yes/No/Conditional) - If conditional, what conditions must be met? - What mitigations are required? - What governance and monitoring should be established? - What would change your recommendation?
Your recommendation should follow logically from your analysis.
Scenario Options¶
Select one of the following scenarios, or propose an alternative for approval:
Option A: Deterioration Prediction in Retrieval¶
A vendor proposes an AI system that monitors patient vital signs during aeromedical transport and predicts deterioration risk. The system would alert crew to increased risk and suggest interventions.
Option B: Dispatch Triage AI¶
A dispatch AI system that analyses incoming case information and recommends resource allocation (air vs. road, team composition, priority level) for retrieval coordination.
Option C: Point-of-Care Ultrasound AI¶
An AI-assisted ultrasound interpretation system for use by retrieval teams, providing automated detection of key findings (e.g., pneumothorax, pericardial effusion, abdominal free fluid).
Option D: Clinical Documentation AI¶
An AI system that generates clinical documentation from audio recordings of patient encounters and crew handovers during retrieval missions.
Proposal for alternative scenario due Week 5.
Submission Format¶
- Governance committee briefing paper format
- 2,500 words (+/- 10%)
- Executive summary (not included in word count)
- Clear headings and professional presentation
- Reference list using consistent citation style (not included in word count)
- Submit via LMS by 11:59 PM on the due date
Briefing Paper Structure¶
Governance committees expect: - Executive summary with clear recommendation - Structured analysis under clear headings - Evidence-based assessment - Explicit statement of risks and mitigations - Clear, actionable recommendation
Marking Criteria¶
| Criterion | Weighting | Description |
|---|---|---|
| Safety Analysis | 25% | Depth and accuracy of safety risk identification and analysis |
| Ethical Reasoning | 20% | Sophistication of ethical framework application |
| Regulatory Accuracy | 15% | Accuracy of regulatory and legal assessment |
| Recommendation Quality | 25% | Logic, justification, and practicality of recommendation |
| Communication | 15% | Appropriate tone, structure, and clarity for governance audience |
Grade Descriptors¶
High Distinction (85-100%) Demonstrates comprehensive safety analysis with sophisticated risk identification across technical and sociotechnical dimensions. Ethical analysis applies frameworks with nuance and depth. Regulatory assessment is accurate and complete. Recommendation is well-justified, practical, and follows logically from analysis. Communication is professional and appropriate for governance audience.
Distinction (75-84%) Demonstrates thorough safety analysis with good risk identification. Ethical analysis applies relevant frameworks effectively. Regulatory assessment is largely accurate. Recommendation is justified and practical. Communication is professional and clear.
Credit (65-74%) Demonstrates adequate safety analysis with main risks identified. Ethical analysis addresses key considerations. Regulatory assessment is mostly accurate. Recommendation is reasonable though justification could be stronger. Communication is appropriate.
Pass (50-64%) Demonstrates basic safety analysis with some risks identified. Ethical analysis is superficial. Regulatory assessment has gaps. Recommendation lacks full justification. Communication needs improvement.
Resources¶
- Week 4-6 course materials
- Week 6 position paper outline (formative feedback)
- Practical exercise 5 (fairness metrics)
- TGA SaMD guidance
- Australian AI Ethics Framework
Academic Integrity¶
This is an individual assessment. You may discuss concepts with peers, but your submission must be your own work. Acknowledge any AI writing assistance used.
Support¶
- Tutor feedback on Week 6 position paper outline
- Discussion forum for clarifying questions
- Writing support services