Assessment 1: Technical Foundations Analysis¶
Overview¶
| Element | Details |
|---|---|
| Weighting | 25% |
| Due Date | End of Week 4 |
| Word Count | 1,500 words |
| Learning Outcomes | LO1 |
| Format | Written report |
Task Description¶
Select a healthcare AI system currently deployed or under evaluation in Australian healthcare (from the provided list or an approved alternative). Produce a technical analysis that demonstrates your understanding of AI architecture, data foundations, and generalisability considerations.
Requirements¶
Your analysis must address the following four components:
1. AI Architecture Explanation (approximately 400 words)¶
Explain the AI architecture underpinning the system: - What type of learning does it use? (supervised, unsupervised, reinforcement, etc.) - What model structure is employed? (decision tree, neural network, ensemble, etc.) - What are the inputs to the system? - What are the outputs? - How does the system generate its predictions/recommendations?
Note: Explain at a level appropriate for an informed clinical colleague, not a data scientist.
2. Training Data Analysis (approximately 400 words)¶
Analyse the characteristics of the training data: - What data sources were used for training? - What population(s) does the training data represent? - What are the known limitations of the training data? - What biases might be present in the data? - What is the quality and completeness of the training data?
If training data details are not publicly available, note this limitation and describe what information would be needed and why.
3. Generalisability Evaluation (approximately 400 words)¶
Evaluate the system's generalisability to Australian healthcare contexts: - How well does the training population match Australian populations? - What differences might affect performance? - What local validation has been conducted (if any)? - What additional validation would be appropriate before deployment? - Consider specific application to your clinical domain or aeromedical context
4. Assumptions and Failure Conditions (approximately 300 words)¶
Identify the assumptions the system makes and conditions under which it may fail: - What assumptions about inputs does the system make? - Under what conditions might the system perform poorly? - What scenarios are likely outside the training distribution? - What failure modes should users be aware of?
AI System Options¶
Select one of the following systems, or propose an alternative for approval:
- Imaging AI: A radiology AI system for chest X-ray interpretation
- Deterioration Prediction: An early warning score/sepsis prediction system
- Clinical Decision Support: A diagnostic support tool
- Documentation AI: A clinical documentation or coding assistance system
- Triage AI: An emergency department or dispatch triage system
If you wish to analyse a different system, submit your proposal for approval by Week 2.
Submission Format¶
- Written report (Word or PDF)
- 1,500 words (+/- 10%)
- Include diagrams if they aid explanation (not included in word count)
- Reference list using consistent citation style (not included in word count)
- Submit via LMS by 11:59 PM on the due date
Marking Criteria¶
| Criterion | Weighting | Description |
|---|---|---|
| Technical Accuracy | 30% | Accuracy and appropriateness of technical explanations |
| Data Analysis | 25% | Depth and insight in training data analysis |
| Generalisability Assessment | 25% | Quality of evaluation for Australian/local context |
| Communication | 20% | Clarity, structure, and appropriateness for clinical audience |
Grade Descriptors¶
High Distinction (85-100%) Demonstrates sophisticated understanding of AI architecture with accurate, nuanced explanation. Training data analysis shows insight into bias sources and quality issues. Generalisability assessment is thorough and considers multiple dimensions. Assumptions and failure conditions are comprehensively identified. Communication is clear, well-structured, and appropriate for audience.
Distinction (75-84%) Demonstrates solid understanding of AI architecture with accurate explanation. Training data analysis identifies key issues. Generalisability assessment considers relevant factors. Assumptions and failure conditions are well-identified. Communication is clear and appropriate.
Credit (65-74%) Demonstrates adequate understanding of AI architecture with mostly accurate explanation. Training data analysis addresses main issues. Generalisability assessment considers some relevant factors. Assumptions and failure conditions are partially identified. Communication is generally clear.
Pass (50-64%) Demonstrates basic understanding of AI architecture with some inaccuracies. Training data analysis is superficial. Generalisability assessment is limited. Assumptions and failure conditions are minimally addressed. Communication needs improvement.
Resources¶
- Week 1-3 course materials
- Provided AI system documentation
- Practical exercises 1-3 (Colab)
- Indicative readings from Weeks 1-3
Academic Integrity¶
This is an individual assessment. While you may discuss concepts with peers, your submission must be your own work. Acknowledge any AI writing assistance used and ensure you can explain all content in your submission.
Support¶
- Tutor consultation hours: [TBC]
- Discussion forum for clarifying questions
- Week 3 data audit exercise provides foundation