For those who don’t know what a contextual bandit is, here is the layman’s version: you are in a situation, you have a few options, and the best option depends on the current context, the time, the mood, and perhaps, in this case, the setting. You pick one option, and you only get feedback on the one you picked. So, no reply. Not even a counterfactual like ”what if I had chosen differently? ” At every single turn, you get a chance! Just context, action, reward! However, over time, the algorithm learns what tends to work in what kind of moment. And that’s where it starts to feel familiar, because life is basically that. We make choices with partial feedback, and we learn patterns from whatever results shout the loudest — you get model weights (wink wink, ML folks).

Surprisingly, the part I can’t shake off is that, based on what we know about the bandit algorithms, a bandit will optimise the reward signal it’s given. So, if the signal is shallow, it can become extremely good at shallow outcomes, and it won’t even know it is missing anything. Thus, if my reward vis-à-vis gratification is quick relief, being noticed, staying busy, or avoiding silence, then my habits train toward those things with frightening efficiency. Truth is, I wouldn’t need a villain, just hard, cold repetition. By extension, I have come to the realisation to start to think about reward like something I would rather choose on purpose, not something I inherit by default. Because, just like model inference, some rewards are noisy and immediate, quick and dirty, and some are quiet and slow, the quiet ones tend to be the ones that make a life feel like it’s actually there.

I hope that you make good choices every day. Truly.

SAO

Disclaimer (read first): This article is informational only and not legal advice. The EU AI Act is a legal text; how it applies depends on your role (provider/deployer/importer/distributor), your system’s classification (high-risk, transparency-risk, prohibited, etc.), and your deployment context. Always consult qualified legal counsel for compliance decisions.

The EU AI Act (formally Regulation (EU) 2024/1689) is Europe’s landmark AI regulation (official text). It uses a risk-based framework: obligations scale with the potential impact of the system, from prohibited practices (Article 5) to high-risk systems (Article 6, Annex I, Annex III) and transparency obligations (Article 50).

In practice, teams often think in four buckets:

• Prohibited practices — not allowed to deploy (see Article 5).

• High-risk systems — allowed, but subject to requirements such as technical documentation (Article 11 and Annex IV), record-keeping (Article 12), and post-market monitoring (Article 72).

• Transparency obligations — certain systems must disclose AI use or label synthetic content (see Article 50).

• Everything else — many AI systems fall outside the high-risk obligations, but customers, auditors, and procurement may still expect clear evidence of intent, evaluation, and monitoring.

Evidence stays coupled to the deployed version — not scattered across tools.

Critically, the Act doesn’t just set one-time rules; it imposes ongoing obligations across the AI system’s lifecycle. This includes maintaining up-to-date technical documentation, keeping detailed logs, and actively monitoring post-market performance for issues. These evidence obligations are where many teams struggle – and that’s exactly the gap this post will address.

Why this post exists: “evidence” is the new delivery artefact

Most teams don’t fail governance because they lack intent. They fail because evidence is not treated like a first-class release artefact.

• Docs live in a wiki.

• Metrics live in an experiment tracker.

• Logs live in a monitoring tool.

• Risk notes live in someone’s head (or a one-off PDF).

Evidence stays coupled to the deployed version — not scattered across tools.

Then procurement, auditors, or regulators ask: “Show me what changed between v1.3 and v1.4, and why that change is safe.”

That’s the gap the EU AI Act amplifies: it pushes organizations to maintain technical documentation, logs, and post-market monitoring as ongoing lifecycle obligations, not a one-time “paperwork sprint.” In other words, if you can’t produce up-to-date evidence for how your AI system was built, how it operates, and how it’s being overseen, you won’t meet the bar.

The practical take is simple:

Ship an Assurance Pack for every release.
A versioned, shareable evidence bundle that stays coupled to the system version you actually deployed.

This article explains:

1. the EU AI Act timeline (what kicks in when),

2. the evidence obligations that matter most for engineering teams, and

3. how to operationalize them as a release artefact (Assurance Packs) so evidence stays current.

EU AI Act in 5 minutes: the phased timeline that matters

High-level timeline of key enforcement milestones.

The EU AI Act entered into force in 2024 and applies progressively over the next few years. The Commission’s Service Desk timeline is the simplest way to track what takes effect when:

High-level timeline of key enforcement milestones.

• 02 Feb 2025 — General provisions start to apply, including definitions (Article 3), AI literacy (Article 4), and prohibited practices (Article 5) (timeline).

• 02 Aug 2025 — Governance structures and general-purpose AI (GPAI) obligations begin to apply (timeline).

• 02 Aug 2026 — The majority of rules apply; this is “day one” of enforcement for many high-risk requirements (notably systems in Annex III) and transparency obligations (Article 50). Member States should have at least one AI regulatory sandbox operational by this date (timeline).

• 02 Aug 2027 — High-risk obligations extend to certain AI systems embedded in regulated products (Annex I) (timeline).

If you remember nothing else: you don’t have “years” to think about evidence. The discipline needs to be built into delivery now. By August 2026, any high-risk AI system you offer in the EU must have compliance evidence ready on demand – and if you’re selling into regulated industries or public sector, the expectation (even before 2026) is to show this mindset today.

First decision: are you even in scope, and at what risk level?

The Act is risk-based

As noted, the AI Act is risk-based: it defines prohibited practices (Article 5), high-risk systems (Article 6 with lists in Annex I and Annex III), transparency obligations (Article 50), and the core definitions you’ll need to interpret all of that (Article 3).

• Prohibited AI – Don’t go there. These are use-cases banned outright (e.g. exploiting vulnerabilities of specific groups, social scoring, certain types of biometric surveillance) (Article 5). No compliance regime exists because you’re not allowed to deploy them at all.

• High-risk AI – This is the most consequential category for compliance. High-risk systems are the only ones subject to the full brunt of the AI Act’s requirements (technical documentation, logging, monitoring, human oversight, etc.).

• Transparency-risk AI – These include systems like conversational AI or deepfake generators that aren’t high-risk but require user disclosures or other transparency measures (Article 50).

• Minimal-risk AI – Everything else. No specific obligations (beyond existing laws), though voluntary best practices are encouraged.

High-risk classification (the engineering trigger)

So what makes an AI system “high-risk”? The classification rules are in Article 6. In simple terms, an AI system is high-risk if either:

1. it’s intended to be used as a safety component of a regulated product (or is itself such a product) listed in Annex I (think: brakes controlled by AI, AI in medical devices); or

2. it’s one of the use-cases listed in Annex III (areas like education, employment, critical infrastructure, law enforcement, etc.).

There is nuance: some Annex III systems can fall out of scope if they do not pose a significant risk — but those exceptions are narrow and must be justified and documented (see Article 6).

In short, “high-risk” is the default if your use-case is in Annex III. The burden is on you to prove otherwise. When in doubt, assume high-risk and prepare the evidence accordingly.

“AI model” vs “AI system”

Another scope question: many teams use pretrained models or APIs (think GPT-4, vision APIs). Are you a provider of an AI system, or just a user of an AI model? The definitions you need (including “AI system” and related actor roles) are in Article 3. In practice, a model becomes part of an AI system when you add the surrounding components (data inputs, a user interface, decision logic, and operating context) that define its purpose.

Why does this matter? Because obligations differ depending on whether you are:

• a provider placing an AI system on the EU market or putting it into service,

• a deployer using an AI system internally,

• a provider of a general-purpose AI model (e.g. offering a foundation model), or

• part of the distribution chain (importer, distributor).

For example, if you fine-tune a large language model and offer it as a SaaS product for hiring, you’re the provider of a high-risk AI system (employment = Annex III). If you just consume someone else’s API for internal use, you’re a deployer and have a lighter (but not zero) load. As an engineering leader, identify which role fits you now (see Article 3): it determines how you approach evidence and ongoing obligations.

The “evidence triangle”: what the EU AI Act forces you to produce and maintain

Strip away the legalese and you can boil the Act’s compliance down to three continuous evidence streams (for high-risk systems):

Evidence triangle: technical documentation + logs + post-market monitoring.

1. Technical documentation – Required under Article 11, with the detailed contents in Annex IV. It covers what the system is, intended purpose, how it works, how it was validated, and how it evolves between versions.

2. Record-keeping (logging) – Automatic logs that capture the system’s operation and outcomes, sufficient to trace issues or decisions (see Article 12). Essentially, an audit trail for the AI’s functioning.

3. Post-market monitoring – A proactive plan and system for monitoring the AI after deployment, to detect problems or degradation and take action (see Article 72). This includes reporting serious incidents under Article 73.

These three obligations reinforce each other. Think of it as an evidence triangle: documentation, logging, and monitoring. If one side is weak, the whole assurance collapses. For example, logs without documentation are just data with no context; documentation without monitoring becomes stale fiction; monitoring without logs means you can’t investigate incidents.

The minimum evidence set (practical mapping)

If you want a practical way to think about evidence, treat these as three versioned outputs that ship with every release:

Mapping obligations to concrete evidence artefacts.

• Article 11 + Annex IV → a technical documentation bundle for the specific version you shipped.

• Article 12 → logging and record-keeping capability (structured event logs, retained as required).

• Article 72 → a post-market monitoring plan plus a running monitoring process/system.

1) Technical documentation: Annex IV is the spine

For high-risk systems, Annex IV of the Act sets out what technical documentation must include. In short, it’s a comprehensive “tech spec + compliance report” for your AI system. The Commission’s service desk provides a view of Annex IV and a link to the official text:

• Annex IV (Service Desk summary) – a readable breakdown of the requirements.

• Official act text (EUR-Lex) – the legal text, if you enjoy that.

A practical (non-exhaustive) reading of Annex IV looks like this:

• Intended purpose + versioning – What is the system meant to do (and not do), who built it, and the exact version (including how it relates to previous versions).

• System architecture + integration – How the AI model and other components work together (e.g. data pipeline, UI, APIs) and the context it operates in.

• Data – The characteristics and provenance of training and test data: where it came from, how it was collected or annotated, any biases or limitations.

• Development process – How you built the model (did you fine-tune a pretrained model? use AutoML? what steps in training?).

• Model performance and limits – The AI’s capabilities and accuracy (overall and on specific relevant groups), and its known limitations or foreseeable unintended outcomes.

• Validation and testing – How you validated the model: which metrics, test datasets, robustness checks, and the results (including test logs and reports).

• Risk management – The known risks (e.g. failure modes, potential for bias, cybersecurity vulnerabilities) and the measures in place to mitigate them (which ties to the required risk management system from Article 9).

• Changes and versions – If you update the AI, what changed and why, and evidence the new version still meets the documented intent, tests, and controls.

• Post-market plan – Annex IV expects the post-market monitoring plan (from Article 72) to be part of your technical documentation bundle for each version.

Key point: Annex IV makes “versioned evidence” unavoidable. It explicitly ties documentation to the AI system’s version and lifecycle. This is why treating evidence as a continuous release artefact (not a one-time PDF) is the sane way forward. Every time you ship a new model or update, the documentation needs to be updated and re-issued.

2) Record-keeping: logs as a compliance primitive (Article 12)

High-risk AI systems must “technically allow for the automatic recording of events (logs) over the lifetime of the system” (see Article 12). In other words, you need to build in logging from the start – it’s not optional. The logs should be detailed enough to:

• Trace decisions and outcomes – If something goes wrong, the log should help identify where and why (think of it as a flight recorder for the AI).

• Facilitate post-market monitoring – Your monitoring system (Article 72) will rely on logs to detect issues. No logs = no meaningful monitoring.

• Capture relevant data – For some systems, the law spells out additional details to log (like timestamp of each use, data checked, who verified the results).

Article 12 is short but potent. This isn’t just “keep some logs because it’s good practice.” It effectively means if you can’t reconstruct what happened in your AI’s operation, you can’t defend your system’s position in an audit or investigation. Engineering teams should treat logging as part of the model’s design: decide what events are critical (inputs? outputs? decisions? model version used?), define a schema, and ensure logs are stored securely and retained as long as required.

One practical tip: the logs required here feed directly into both incident response and monitoring. So design your logging with those use cases in mind (e.g. log the model version and configuration for each prediction, so later you can pinpoint which version was running when an incident occurred).

3) Post-market monitoring: a plan + continuous system (Article 72)

Signals → thresholds → actions → evidence update loop.

Article 72 requires providers of high-risk AI systems to implement a post-market monitoring system and have a post-market monitoring plan to guide it. Think of this as setting up an ongoing process to watch your AI in the wild and catch problems early.

A few details matter for engineering teams:

• The monitoring plan must be documented as part of your technical documentation (see Article 72 and Annex IV). That means by the time you go to market, you need a written plan for how you will monitor the AI, what data you’ll collect, and how you’ll evaluate performance and risks over time.

• The European Commission is tasked with providing a template and required elements for this plan by 2 February 2026 (Article 72(3); see Article 72). Expect a standardized format for what your monitoring plan should include (likely metrics to track, frequency of checks, and processes for handling incidents).

What does post-market monitoring look like in practice? It means you can’t just “set and forget” your model after deployment. You need to actively collect data on how it’s performing – accuracy, drift, error rates, potentially bias metrics, uptime, etc. – and also capture any incidents or near-misses. If performance degrades or new risks emerge (say, the data input distribution shifts), you’re expected to notice and do something (retrain, update the model, or even pull the model from service in extreme cases).

One way to think of it: Model monitoring is like CI/CD for risk and performance. Just as you wouldn’t deploy software without monitoring uptime and errors, the AI Act pushes you not to deploy AI without monitoring its real-world behavior and impact.

Bonus: serious incident reporting (Article 73)

Post-market monitoring feeds into another obligation: serious incident reporting. Under Article 73, providers of high-risk AI systems must be able to detect serious incidents and report them to authorities.

A “serious incident” basically means something went really wrong – e.g. the AI caused someone harm or a major near-miss. The law sets timelines for reporting, including accelerated timelines in some cases. The idea is to give regulators a heads-up on emerging risks.

For engineering teams, this is an extension of incident response. You’ll need to define what constitutes a serious incident for your system (some of this is defined in the law, but you should operationalize it) and have the mechanisms to: (a) detect that it occurred (likely via your logs and monitoring signals), and (b) have a process to investigate and report on it quickly. It’s another reason why logging and version traceability are so important – if you can’t connect an incident to a specific model version and situation, you’ll have a hard time reporting the “causal link” as the law requires.

What this means in practice (start-ups, SMEs, and enterprises)

If you’re a new company (start-up / SME)

Your advantage: you can design evidence into your delivery pipeline from day one. You’re not stuck with legacy systems or habits. And you’ll need that advantage, because compliance is a big ask for a small team.

The Act includes measures intended to support SMEs and start-ups, including regulatory sandboxes (Article 57) and support measures (Article 62) in the official text (ai-act-eurlex). In short, regulators recognise this can be hard for smaller teams and include mechanisms to reduce friction.

Founder playbook

• Pick a compliance-friendly beachhead. If you’re a founder, consider targeting a domain where evidence is already valued (healthcare, finance, public sector). Compliance can become a competitive advantage if you bake it in early.

• Treat evidence as part of the product. Shift your mindset from “we deliver a model or service” to “we deliver a model with an assurance packet.” When selling to enterprise or government, being able to hand over an Assurance Pack (see below) with all the documentation and logs can compress procurement and due diligence. It answers questions like: “What is it for and not for?”, “How was it evaluated?”, “What are the known risks and mitigations?”, and “How will you detect and handle issues?”

• Leverage sandboxes and resources. If a regulatory sandbox is available in your country, use it to pressure-test your evidence and monitoring approach. Any outputs can go into your assurance package — but validate the sandbox terms with counsel and the competent authority.

Even if your AI system is currently minimal risk, consider documenting it and monitoring it as if you had obligations. It’s easier to start with good habits than to retrofit them. Plus, enterprise customers are increasingly asking for evidence (e.g. model cards, test results, risk assessments) even for non-high-risk AI. Be ahead of the curve.

If you’re an existing organization (enterprise)

You likely have some governance structure (responsible AI committees, model documentation, etc.). The typical failure mode is fragmentation and drift:

• The Confluence wiki says one thing about the model, the production code says another.

• You have a model card that was made at launch, but the model has been updated 5 times since then.

• Risk assessments were done in a workshop, but the mitigations were never implemented or tracked.

• Different teams use different tools, and evidence is all over the place (or lost when people leave).

The AI Act’s evidence requirements essentially force alignment and single source of truth. For example, you can’t let the model drift away from its documentation – by law that documentation must be kept in sync with the current system version (see Annex IV). This is as much an organizational challenge as a technical one.

Fastest win for an enterprise: Stop treating documentation and compliance as a separate, downstream process. Integrate it into your ML workflow. For instance, require that every model that goes to production has an “assurance package” generated (even if lightweight at first) that includes all key documentation and evidence artifacts, and store it versioned in a central repository. This way, if someone asks “what changed between last version and this?”, you not only have a git diff of the code, but a diff of the assurance package (data, metrics, documentation updates, risk logs, etc.).

Also, start mapping existing tools to these needs. Maybe your experiment tracking can output evaluation reports, your DataOps can provide data provenance, etc. You don’t necessarily need one giant new system; you can script pulling pieces together. The point is to bundle and version it.

Legacy systems

What about AI systems you already have in the field? Transitional rules and what counts as a material change matter — treat it as a governance decision and check the official text and the Commission timeline (ai-act-eurlex, timeline).

A pragmatic approach: start generating evidence packs for new releases now, and then backfill legacy systems based on priority. Identify which existing AI systems would be considered high-risk (an internal audit can map your systems to Annex III categories). For the most critical ones, it’s worth doing a post hoc documentation and risk assessment exercise to have something on file. For others, plan an update and treat that update as the point where you bring it into compliance (i.e. use the update process to generate the necessary documentation and monitoring).

The key is: don’t assume “old = exempt.” Check the rules and document your rationale for how you handle legacy systems and updates.

The Assurance Pack idea: operationalizing Annex IV + Articles 12 & 72

In practice, I keep coming back to one principle:

Evidence must be coupled to the version you deploy.

This principle flows directly from the Act’s evidence obligations: technical documentation (Article 11 and Annex IV), record-keeping (Article 12), and post-market monitoring (Article 72). You can’t satisfy those by doing a big Word document at launch and forgetting about it. It’s a continuous process.

So, the Assurance Pack is a pragmatic response: make the evidence package a versioned artefact just like your code or model.

Imagine for each AI release, alongside your model artifact and API, you produce a folder (and a downloadable bundle) that contains all the key evidence for that version. That’s an Assurance Pack. It would include things like:

• The intended use and limitations of the AI (what you’d put in Section 1 of Annex IV).

• The architecture and integration description.

• Data sheet: info on training data, when the data was collected, any updates.

• Evaluation results: metrics, validation reports, bias analyses for this version.

• Change log: how this version differs from the last (e.g. “added 10k more training samples from X, fixed bug in preprocessing Y, improved accuracy by 2% on minority class”).

• Risk assessment: an updated risk log or checklist (e.g. “still residual risk Z remains, but acceptable with mitigation Q”).

• Logging specification: what events the system logs, and where logs are kept.

• Post-market monitoring plan: what we’re tracking in production for this version, any new signals or thresholds added.

• Approvals: who reviewed and approved this release for compliance (e.g. the responsible AI lead signs off).

All of the above corresponds to Annex IV or related obligations. By packaging it, you ensure it travels with the software.

What an Assurance Pack looks like (example)

Here’s a conceptual structure of an Assurance Pack:

Assurance Pack structure + manifest.

assurance-pack/
├── manifest.yaml
├── system/
│   ├── intended_use.md
│   ├── system_overview.md
│   ├── architecture.md
│   ├── integration.md
│   ├── limitations.md
│   └── change_log.md
├── data/
│   ├── provenance.md
│   ├── preprocessing.md
│   └── labeling.md
├── evaluation/
│   ├── eval_plan.md
│   ├── metrics_summary.json
│   ├── slice_results.csv
│   └── robustness_tests.md
├── test_logs/
│   └── ... (raw logs or links to test run outputs)
├── risk/
│   ├── risk_assessment.md
│   ├── risk_controls.md
│   └── human_oversight.md
├── logging/
│   ├── logging_spec.md
│   ├── event_schema.json
│   └── retention_policy.md
├── monitoring/
│   ├── post_market_monitoring_plan.md
│   ├── signals_and_thresholds.yaml
│   ├── incident_playbook.md
│   └── escalation_contacts.md
├── governance/
│   ├── roles_and_accountability.md
│   └── approvals.md
└── attestations/
    ├── hashes.txt
    └── signature.sig

And a minimal manifest.yaml inside might contain metadata and pointers:

pack_version: "0.1"

system:
  name: "ExampleAI"
  release_version: "1.4.0"
  previous_release: "1.3.2"

provider: "YourCompany"
deployment_context: ["EU"]

classification:
  high_risk_candidate: true
  rationale: "Annex III use-case (employment) – likely high-risk"

evidence:
  technical_documentation:
    annex_iv_mapping: "system/system_overview.md"
  record_keeping:
    article_12_logging_spec: "logging/logging_spec.md"
  post_market_monitoring:
    article_72_plan: "monitoring/post_market_monitoring_plan.md"

quality_gates:
  min_accuracy: 0.92
  regression_tolerance: 0.01

approvals:
  - role: "Responsible AI Lead"
    name: "Dr. Jane Doe"
    date: "2026-01-24"

It’s intentionally not fancy. The wow factor isn’t the YAML or the folder tree – it’s the process change: making this evidence packet a required output of every release and treating it with the same importance as the model artifact or source code. That enables automation (e.g. you can diff two manifests to see what changed, or automatically check if certain fields are present or within policy thresholds).

How it fits into delivery (conceptual)

How evidence becomes a release artefact in the delivery pipeline.

How do you actually produce these without killing your team’s velocity? The trick is to automate as much as possible and make it part of CI/CD:

1. Develop & train as usual (not much change here).

2. Evaluate as usual – but ensure your evaluation spits out artifacts (metrics, graphs, maybe a PDF report or JSON summary).

3. Assemble the pack: Have a script or CI job that gathers all the pieces (the model card from your training pipeline, the evaluation metrics, the data schema, etc.) and puts them in the right folder structure. Some pieces might be written by humans (e.g. “limitations.md” might be manually maintained), but many can be generated or templated.

4. Validate the pack: Just like you wouldn’t deploy if tests fail, set a rule that you don’t deploy if the assurance pack is incomplete. For example, check that metrics_summary.json shows performance above your required threshold, that risk_assessment.md has been updated for this version (maybe require a specific commit message or checklist).

5. Publish/store the pack: Store it in artifact storage or attach it to the release in your repository. Treat it as deliverable. If an auditor or client wants it later, it should be readily accessible.

6. Deploy the AI system (if all gates pass, including evidence checks).

If you’ve heard of software bills of materials (SBOM) in cybersecurity, this is a similar concept – an evidence BOM for AI. It ensures that whenever your AI ships, the evidence of its trustworthiness ships with it.

What auditors and procurement actually ask (and how packs answer)

When facing an audit or customer security review, you’ll often get a spreadsheet or questionnaire. Assurance Packs, if done well, let you answer many questions by just handing them the pack (or extracting the relevant parts quickly).

Typical questions and where the pack addresses them:

1. “What is this AI system for, and what are its limitations?” – Check intended_use.md and limitations.md in the pack. This should clearly state the purpose, scope, and contexts where the AI should (or should not) be used. It’s basically your Annex IV Section 1(a) & (c) info.

2. “Show me how you evaluated it for that use.” – The pack’s evaluation/ folder should have an evaluation plan (eval_plan.md) and results (metrics, slices, robustness tests). This demonstrates that you tested the system on relevant data and know its performance (see Annex IV).

3. “What are the risks and how do you mitigate them?” – The risk/ section of the pack holds your risk assessment and any mitigations (like human oversight measures, fail-safes, bias mitigations). This aligns with the Act’s risk management expectations (Article 9 and Annex IV sections on risk). If you identified, say, a risk of bias against a subgroup, you’d note it and perhaps point to a control (maybe a periodic bias audit in monitoring).

4. “How do you detect and handle issues in production?” – The monitoring/ folder includes the post-market monitoring plan (signals, thresholds, actions) and an incident response playbook. This shows that you’re not just throwing the model over the wall – you have a process to watch it and respond.

5. “Can you trace decisions if something goes wrong?” – The logging/ folder holds the logging specification and schema, which shows you’ve built traceability (and you can even provide sample log data if appropriate, or show an incident report tracing through logs).

6. “Who takes responsibility?” – The governance/roles_and_accountability.md and approvals.md should list the accountable roles (e.g. who the provider is, who the product owner is, etc.) and who signed off on the release. This aligns with the Act’s emphasis on defined responsibilities and quality management (Article 17 even requires a Quality Management System with assigned responsibilities).

By structuring evidence this way, you turn a potential 2-week back-and-forth Q&A into a quick lookup. It also impresses auditors: it shows maturity if you can immediately pull up the exact document or data they ask for, versioned for the release in question. No scrambling through folders named “final_final_really.pdf” – you have it organized.

Common failure modes (and how to design around them)

Through working on these issues, we’ve seen patterns of how evidence efforts can falter. Here are some common pitfalls and how a release-focused approach addresses them:

Version-to-version evidence diffs make audits/procurement faster.

• Stale documentation: The classic “doc written once and never updated.” Six months later, it’s out of sync with the actual system.
  Solution: Make documentation part of the release checklist. If the pack for version 1.4.0 isn’t present or is missing sections, the release doesn’t get tagged. By tying docs to each version, you force updates as part of the dev process, not as an afterthought.

• Metrics without context: Teams often throw a few performance numbers into a report. But without context (what was the test set? what slices or edge cases were checked? what’s the target performance?), numbers mean little.
  Solution: Require an evaluation plan and sliced results in the pack. This means before training, you define how you’ll validate (which forces thinking about relevant scenarios). And after training, you include not just “overall accuracy 94%” but, for example, “accuracy per subgroup, worst-case = 88% on subgroup X” plus any stress tests. That gives a fuller picture.

• Monitoring that doesn’t match risks: It’s common to see a monitoring dashboard that tracks CPU usage and maybe overall prediction count – but nothing about whether the model is drifting or making more errors.
  Solution: In the monitoring plan, explicitly tie metrics to risks. If your risk assessment says “may have lower accuracy on older patients,” then your monitoring should include a check on performance by age (if you can get that data) or at least a proxy. If bias is a risk, include some bias indicators. Essentially, close the loop: each major risk should have a corresponding monitoring signal or periodic check.

• No traceability across versions: When something goes wrong, the team isn’t sure which model version was running or what data was used. This is a nightmare for accountability (and regulators won’t accept “we’re not sure which model was live”).
  Solution: Always log the model version (and ideally a unique identifier for the model build) on every prediction or decision. And in your pack’s manifest, maybe include a hash of the model file and code used. That way, you can always match a log entry to the exact version and evidence pack. Also, store packs in an immutable store (even just a versioned S3 bucket) so you can pull up the exact docs that correspond to a version.

• No incident playbook: When an issue happens, it’s chaos – people aren’t sure who leads the investigation, or how to decide if something is serious enough to report to regulators.
  Solution: Include an incident response plan in the pack (and of course, internally train on it). It should say: if X type of incident occurs, here’s how to assess it, here’s who convenes to decide next steps, here’s how to file a report under Article 73 if needed. This way, every release reminds the team “we have a process if something goes wrong.” It’s much easier to follow a plan than to ad-lib under pressure.

A starter outline for an Article 72 post-market monitoring plan

The Commission is expected to adopt a template for the post-market monitoring plan by 2 February 2026 (Article 72(3); see Article 72). But you don’t have to (and shouldn’t) wait for that to start planning. Here’s a practical outline you can start filling in now, which should align well with Article 72’s intent:

1) Scope and context: Define which system this plan covers (name, version, etc.), its intended use, and deployment context. Note who the provider is and who the deployers are (if different). Essentially, set the scene: “This plan monitors System X version Y in Z environment for intended purpose P.”

2) Risks and assumptions: Summarize the key risks from your risk assessment. “We are particularly watching for performance degradation on dataset drift, or an uptick in false positives causing potential harm, etc.” List hypotheses like “if metric M goes above threshold T, it could indicate risk R is materializing.”

3) Signals to monitor: For each risk or important aspect, what metrics or signals will you track? For example: input data drift (monitor distribution of inputs over time), output quality (monitor error rates against a validation dataset or human feedback), bias (monitor outcomes by demographic if available), system uptime, etc. Also external signals like user complaints or support tickets could be included.

4) Thresholds and triggers: Decide what levels of those signals warrant action. Maybe you set a warning threshold and an incident threshold. E.g. “If weekly accuracy drops by more than 5 points, data science team investigates within 1 week. If drops by 10 points or more, halt model and failover to manual process.” This is essentially your SLA for model performance and risk.

5) Logging and traceability link: State how you will use logs in monitoring. “All predictions are logged with timestamp and key attributes; monitoring jobs run daily to aggregate these logs and check for anomalies. In case of incident, logs will be analyzed to trace root cause.” (This connects to the Article 12 obligation – your plan should reference that you have the data to support it.)

6) Review cadence: How often will you formally review the monitoring data and overall system performance? Perhaps you have a monthly governance review where you look at trends, and a yearly full audit. Mention that. The Act will expect you to continuously update the technical documentation with new findings, so tie that in: “Results of monitoring will feed into periodic updates of technical documentation and risk assessment.”

7) Continuous improvement: Explain how you will update the system or process when issues are found. For instance, “If drift is detected, we will retrain the model on latest data within X weeks,” or “If new risks are identified, we will update the risk log and implement mitigation.” This shows you have a learning loop, not just monitoring in name only.

8) Reporting workflow: Outline how internal escalation works. “On detecting a serious incident, the on-call ML engineer notifies the AI Risk Officer; a root cause analysis is done; if criteria for Article 73 reporting are met, we will file a report to authorities within the required timeline.” Basically, tie your plan into the legal reporting duty so it’s clear you won’t drop the ball.

Documenting this now not only prepares you for compliance, it actually helps your team. It’s much easier to sleep at night knowing you have a sensor on the system and a plan for what to do if it blips.

“Isn’t this something platforms can just add?”

You might wonder: won’t the AWS/GCP/Azure/ModelOps platforms of the world just solve this with a new feature? They will certainly help on parts – for example, logging and monitoring tools are out there, and they can add compliance checklists. But a key differentiator here is portability and version-coupling of evidence.

Many platform solutions focus on dashboards or documents in situ. The Assurance Pack concept is about a portable bundle that you can hand over to an auditor or customer, or move to another platform, and it still makes sense. It’s decoupled from any specific tooling UI. It’s also verifiable in the sense that you can sign it, hash it, and show it wasn’t tampered with.

Think of the security world: cloud providers offer great security centers, but companies still produce their own artifacts for audits (architecture diagrams, access reviews, etc.) that are portable. Similarly, you as AI providers will want a portable evidence artefact that you own and control, which can be shared externally as needed (under NDA or via secure channels likely).

Platform vendors can assist by generating pieces of the evidence (like auto-generating parts of the technical documentation, or providing one-click model cards), and they might even allow exporting a “compliance bundle”. But until that’s common, rolling your own lightweight process as described can give you a head start (and more control).

What you should do this quarter

A call to action for engineering and product teams:

1. Determine if you’re in scope (and at what level). Map your AI use-cases to the Act’s categories. Are you possibly high-risk (see Article 6)? If yes, is it an Annex I regulated-product scenario or an Annex III use-case? Use the definitions in Article 3 to clarify your role (provider/deployer/etc.). If you’re not high-risk, great – but remember that enterprise customers might still ask for “AI governance” evidence, so it’s worth adopting a light version of these practices anyway.

2. Start treating evidence as a deliverable. Kick off an initiative to define what an “Assurance Pack” would be for your main AI system. Pick a recent release and try to compile the key items. This will highlight gaps (e.g. “oh, we never actually wrote down our intended use and limitations clearly”). It’s fine if it’s messy at first. Create a template and iterate.

3. Build the monitoring plan now. Don’t wait until 2026 when the official template drops. Begin drafting a post-market monitoring plan for your system using the outline above. Even if it’s rough, it will surface questions (What should we monitor? Can we get that data? Who would be on the hook if X happens?) that you’re better off answering sooner rather than later. By the time the Commission’s template arrives, you’ll have a version to align with it, rather than starting from scratch under time pressure.

By focusing on these steps, you’ll not only de-risk compliance, but you’ll likely improve your AI practice overall. It’s the whole “sunlight is the best disinfectant” idea – making yourself document and monitor forces you to build better, more reliable AI.

A pragmatic implementation approach (mapping to the evidence triangle)

You don’t need a mega-platform to start. Whether you build this in-house or buy tooling later, aim for three capabilities that map cleanly to the evidence triangle:

Wrap an AI system with evidence capture, review, and release packaging.

• Evidence Packs (technical documentation): automate the assembly of a versioned evidence bundle aligned to Annex IV. Start with templates + a CI job that gathers evaluation artefacts, data provenance notes, and a human-maintained “limitations / intended use” section. The key is that the pack stays tied to the deployed version (and is easy to diff between releases).

• Monitoring by default (post-market): make the Article 72 monitoring plan executable. Define your core signals (performance, drift, safety/bias proxies, incident triggers), wire them into your inference pipeline, and make the monitoring outputs part of the pack for that release (so you can show not only what you planned but what you observed).

• Provenance & traceability (logging): treat Article 12 logging as a product primitive. Ensure every decision/prediction is traceable to model version + config + relevant input/output metadata (within privacy constraints). If you want stronger integrity guarantees, hash/sign the pack artefacts so you can later prove “this is the exact evidence for the exact model that ran.”

If you want a starter template for an Assurance Pack (folder tree + manifest), feel free to message me on LinkedIn.

Primary sources

• Regulation (EU) 2024/1689 (EUR-Lex, official text)

• Commission Service Desk timeline

• Article 3 — Definitions

• Article 4 — AI literacy

• Article 5 — Prohibited practices

• Article 6 — High-risk classification

• Article 11 — Technical documentation

• Article 12 — Record-keeping

• Article 50 — Transparency obligations

• Article 72 — Post-market monitoring

• Article 73 — Serious incident reporting

• Annex I — Regulated products

• Annex III — High-risk use-cases

• Annex IV — Technical documentation content

Further reading

• Commission Service Desk FAQ

Perhaps they mean well, envisioning a world free of suffering and class divides, where historical injustices are remedied by enforced parity. Yet someone in full possession of full human DNA should recognise that this ideal without a doubt, is myopic and will only soil the cause of genuine fairness by conflating uniformity with equity.

… geez! A world where everyone ends up the same would be dreadfully monotonous and boring, free from the very chaos that makes our world unique, wouldn't you agree? The vibrancy of human experience, our triumphs, our struggles, and our creative flourishes, all emerging from the interplay of difference, free will, freedom, and responsibilities and not from enforced sameness.

To address this, we proposed a hybrid approach that leverages the strengths of both 2D and 3D convolutional neural networks. In our approach, we engineered the 2D CNN component to capture fine-grained texture details, while the 3D CNN extracts essential volumetric context. A key innovation in our methodology was the creation of virtual augmented slices to account for the additional channel in 3D to encode a series of uncertainties directly within our model. This approach enables us to better represent the ambiguities inherent in the data and to perhaps catch uncertainties that may not be present now in the model but at inference. This led us to achieve better results compared to existing methods.

You can read the preprint version of our paper here: https://arxiv.org/pdf/2410.02714
Code available on request.

Our current open question is: how can we best leverage multi-modal data to enhance our predictive models further? For instance, integrating additional imaging modalities such as PET scans or incorporating non-imaging biomarkers, like genetic and clinical data, could offer a more holistic view of disease progression. This fusion of diverse data sources might refine early detection strategies and pave the way for more personalised interventions.

This past summer, I had the privilege of teaching a 3-day course on Deep Learning for Computer Vision.

The course materials are now open source and available on GitHub: DL4CV

In the course, we began with a recap of Python and an introduction to PyTorch, then explored image computation techniques. We progressed from coding a simple neural network from the ground up to building basic architectures and advancing to deep neural networks such as VGG, ResNet, and YOLO. We also performed inference for emotion recognition, traffic tracking, and simple object tracking.

The repository contains:

• Lecture slides

• Practical code samples (in PyTorch)

• Datasets for hands-on projects

• Step-by-step notebook tutorials

Feel free to explore, fork, and contribute. Your feedback is always welcome. #DeepLearning #ComputerVision #OpenSource

QUB-PHEO (Queen’s University Belfast, Perception of Human Engagement in Assembly Operations) is designed to support advanced research in human intention inference and engagement analysis. Our dataset provides a foundation for developing and testing innovative models and applications in this domain.

Project Highlights:
Data Volume: Over 4.5 million 4K frames and 40+ hours of video data
Tasks: 9 tasks and 36 subtasks
Participants: 70 individuals contributing diverse perspectives

We have thoroughly documented our methodology, approach, and core principles in our recently published paper in IEEE Access. You can access the paper here

Our datasets and preprocessing code are open source and available on GitHub

To gain access to the QUB-PHEO dataset, please visit: Dataset Access

While most lecturers and tutors approach this topic as abstract and uninteresting, computational complexity is more than just an abstract theory; it's the distinction between an algorithm that executes in seconds and one that could take hours or days. In this brief post, I will look into the essence of computational complexity, aiming to integrate fundamental concepts with real-world examples to help clarify the concepts for me.

What is computational complexity, really?

Let’s start with an analogy. imagine you are at a theme park with a queue to ride a rollercoaster; you are trying to figure out how long it will take to get on the ride. Oftentimes, this depends on:

1. The number of people in the queue is significant.

2. How fast does the rollercoaster take people on board?

This is a good analogy for computational complexity, as it is all about how long it takes an algorithm (the rollercoaster) to process the input (the people in the queue). Hence, from this, we can deduce that:

• if the rollercoaster can handle one person at a time, it will take as long as the number of people in the queue. This is called the linear time and it is written as O(n). More on this later.

• But if the rollercoaster takes multiple people at once, maybe halves the queue each time, it is much faster. This could be something like logarithmic time, written as O(log n).

Thus, succintly put:

To measure this, we need to take into account, two technical details:

1. Time complexity refers to how the runtime of an algorithm increases as n increases.

2. Space complexity: How the memory usage of an algorithm grows with n.

For example, an algorithm that requires storing every item in a list in memory will have a higher space complexity that one that processes items one at a time. These two factors often trade off against each other- the most efficient algorithms balance both.

Big-O Notation

In computer science, we often like to complicate things - but only because it is fun and helps us think in more abstractly, after all, where is the joy in understanding something complex without appreciating the beauty of its intrinsic patterns?

On the more technical side of things, the O in Big-O stands for “order”, reflecting the asymptotic behaviour of a function as its input size approaches infinity. Hence, in plain English, it tells us how the runtime or space requirements of algorithm grow relative to the size of the input.

Big-O does not care about the exactruntime in seconds (this often depends on the hardware and software); it instead focuses on the growth trend,for example, does the runtime double when the input doubles? or does the runtime increase slightly when the input doubles?

Big-O Notation Classifications

So, we can say Big-O notation is more than just mathematical formality; it is a powerful tool for understanding the efficiency of algorithms across a spectrum of possibilities. Fom lightning-fast constant time operations to painfully slow crawl of exponential growth, Big-O gives us a common language to classify and compare algorithms. Here, we explore the most common classification of Big-O notation, looking at what they mean and how they can shape the way we approach problem-solving in CS.

1. Constant Time - O(1)

   In this case, an O(1) algorithm does not care about the input size. It always takes the same amount of time. A classic example is accessing an element in an array by its index. No matter how large the array is, this operation is instantaneous because you don’t need to traverse the entire array or perform any calculations on the other elements. The index acts like a direct pointer to the location in memory, giving you immediate access—hence the constant “1” in O(1). This is what makes O(1) operations so desirable: their runtime doesn’t grow with the size of the input. Here is how it looks in the code:

    def get_element(input_array, index_position):
        return input_array[index_position]
   

   In the above example, no matter how large arr is, the operation of retrieving an element at a specific index will always take the same amount of time. Truth is an O(1) operation is a dream, no matter how large the input is, the runtime remains the same.

2. Linear Time - O(n)

   An algorithm with O(n) time complexity grows linearly with the size of the input. This means that if the input doubles, the runtime will also double - although not as fast O(1), but it is predictable and manageable. Thus, when it comes to O(n), things start to scale proportionally. A common example of O(n) is finding the maximum value in a list, to find the maximum, you would have to look at every single element in the list, from the very start to the last element in the list. This means the time taken will increase linearly as the size of the list grows.

   Here is a code implementation of maximum number search:

    def maximum_num_search(input_array):
        maximum_number = input_array[0]
        for number in input_array:
            if number > maximum_number: 
                maximum_number = number
        return maximum_number
   

   In this example, the loop iterates through all n elements of arr. Such that if you have 10 elements, it will take 10 steps; while 10,000 elements will take 10,000 steps. This is the very essence of linear time- growing in lockstep with the input size. A real life example in would be in ML where you calculate the loss for a batch of data during training- the time required scales linearly with the batch size since each sample is processed independently. For example computing mean squred error loss over n predictions:

    loss = sum((y_prediction - y_groundtruth) ** 2) / n
   

3. Quadratic Time - O(n²)

   Things start to get less efficient in quadratic time. Algorithms with O(n²) involve nested loops (a loop in a loop), meaning for every element in the input, the algo will process every other element, by extension the runtime grows exponentially as the input size increases. A good example would be checking all pairs of elements in an array, where for each element the algorithm compares it with every other element, leadingto n x n = n² iterations.

    def check_pairs(input_array):
        for i in range(len(input_arry)):
            for j in range(len(input_array)):
                print(input_array[i],  input_arry[j])
   

   likewise, if input_array has 10 elements, the code will run 100 iterations. for 1000 elements, it will run 1,000, 000 iterations. As you can imagine, this quickly becomes impractical for large datasets. For example, calculating the pairwise distance matrix in clustering or dimensionality reduction techniques like t-SNE. Such as given n data points, you compute distances for all n² pairs.

    from sklearn.metrics.pairwise import euclidean_distances
    distances = euclidean_distances(X, X)
   

   Although O(n²) algorithms can sometimes be unavoidable, you should generally be considered a red flag for scalability.

4. Logarithmic Time - O(log n)

   Let’s get a little clever. This time complexity represents algorithms that grow very slowly, even as the input size increases significantly. This is often achieved by repeatedly dividing the problem into smaller chunks, rather than processing the entire input. Imagine you are searching for a word in a dictionary., you don’t start from the first page and go one by one, instead, you flip to the middle, check if the word is earlier or later alphabetically, and eliminate half of the book in one step. Repeat until you find the work.

    def binary_search(input_array, target):
        left, right = 0, len(input_array)-1
        while left <= right:
            mid = (left + right) // 2
            if input_array[mid] == target:
                return mid
            elif input_array[mid] < target:
                left = mid + 1
            else:
                right = mid - 1
   

   This is a binary search in action. Binary search is an ideal example- here the input (often a sorted list) is halved at each step until you get the target value. this approach reduces the number of steps required to get to the final solution as it takes the divide-and-conquer strategy. in this case, if input_array has 1000 elements, binary search might only take about 10 steps to find the target. For 1,000,000 elements, it might take just 20 steps - this right here is the power of logarithmic time - ti scales gracefully, making it a favourite for algorithms dealing with large datasets. in Data Science, for instance, we can see logarithmic time at play, such as in searching for a specific value in a sorted dataset using binary search:

    import bisect
    index = bisect.bisect_left(sorted_list, target)
   

5. Exponential Time: O(2^n)

   Exponential time is the real villain of algorithm efficiency, and trust me you don’t want to mess with it, especially when dealing with streams of data! These algorithms grows so quickly that even for a small increase, the input size can lead to astronomically high runtimes. They are usually a last resort for problems one can’t solve efficiently. A good example would trying to solve the Tower of Hanoi problem recursively - it is like trying to compound your efforts: for each additional disk, the work required doubles, leading to unnecessary exponential growth. The solution might feel elegant in theory, but in practice, it can quickly become computationally infeasible.

   Tower of Hanoi: The Exponential Monster

   The Tower of Hanoi problem involves moving nnn disks from one rod to another, following these rules:

   1. Only one disk can be moved at a time.

   2. A larger disk cannot be placed on top of a smaller disk.

   3. An auxiliary rod can be used as a temporary holding space.

For nnn disks, the minimum number of moves required is 2n−12^n - 12n−1. Here’s how it grows:

• For 3 disks: 2³− 1 = 7 moves.

• For 10 disks 2^10 - 1 = 1023 moves

• For 20 disks: 2²0 - 1 = 1048575 moves

    def tower_of_hanoi(n, source, target, auxiliary):
        if n == 1:
            print(f"Move disk 1 from {source} to {target}")
            return
        tower_of_hanoi(n - 1, source, auxiliary, target)
        print(f"Move disk {n} from {source} to {target}")
        tower_of_hanoi(n - 1, auxiliary, target, source)  # O(2^n)

With each additional disk, the runtime doubles, making this algorithm impractical for anything but small inputs. A good real life scenario would be trying to generate all possible transformations of an object in augmented reality, considering every possible rotation and scale for matching against a template in a scene. If you allow 10 possible rotations, 10 scales, and 10 translations, the total combinations explode exponentially, making this an exponentially crazy thing to do! 🙃

Exponential time algorithms are like pouring petrol on a fire: the problem size grows out of control with each added element. While they may be unavoidable in some theoretical or NP-hard problems, again, they’re computationally expensive and typically unsuitable for real-world applications.

6. Linearithmic Time - O(n log n)

   Finally, 𝑂 ( 𝑛 log ⁡ 𝑛 ) is a happy medium between linear and logarithmic time, striking a balance of efficiency for more complex problems. It’s commonly seen in divide-and-conquer algorithms, like merge sort and quick sort. A classic example is merge sort, where the list is repeatedly split in half (logarithmic) and then recombined in sorted order (linear).

    def merge_sort(arr):
        if len(arr) > 1:
            mid = len(arr) // 2
            left = arr[:mid]
            right = arr[mid:]
   
            merge_sort(left)
            merge_sort(right)
   
            i = j = k = 0
            while i < len(left) and j < len(right):
                if left[i] < right[j]:
                    arr[k] = left[i]
                    i += 1
                else:
                    arr[k] = right[j]
                    j += 1
                k += 1
   
            while i < len(left):
                arr[k] = left[i]
                i += 1
                k += 1
   
            while j < len(right):
                arr[k] = right[j]
                j += 1
                k += 1
   
        return arr  # O(n log n)
   

   For a dataset of size 𝑛 n, merge sort splits it into log ⁡ 𝑛 logn levels, with each level requiring 𝑛 n operations to merge. This makes 𝑂 ( 𝑛 log ⁡ 𝑛 ) algorithms highly efficient for sorting and other large-scale tasks.

Although we love fancy and complicated terms in CS, we embrace them for a reason—they embody powerful concepts that tend to guide us in building efficiency and scalable systems. These terms are not academic jargon, they are the very backbone of smart problem-solving.

So, the next time you’re writing code—whether it’s something you plan to push to production, scale to millions of users, showcase as a proof of concept, or highlight in a research paper—pause and think about its computational complexity. Will your algorithm gracefully handle a growing workload? Does it scale efficiently? As your understanding of these intricacies can be the difference between something that works and something that thrives - there is a big difference!

Write smart, scale boldly, and let the science behind your algorithms shine!

In the ever-evolving landscape of computer vision, the transition from static imagery to the dynamic world of videos marks a significant leap to understanding the dynamicity of our world. As someone who has spent years unravelling the mysteries hidden in static images using 2D Convolutional Neural Networks, I find myself at an exciting juncture in my PhD journey - diving into the spatio-temporal context. The shift from analyzing still frames to understanding the intricate sequences of video data is not just a step forward in complexity, but a step towards the realm brimming with untapped potential and unexplored challenges. My exploration into this domain is driven by a simple yet profound realization- our world is not static. It is a dynamic tapestry where each moment is a continuation of the last, and a stroy unfolding in time.

In my previous work, 2DCNNs served as a powerful tool, adept at capturing spatial hierarchies and patterns within images, exploring the intricate relationship between pixels, and encoding subtle patterns via edges and corners. However, as I delve into video data, I find myself in need of a more sophisticated ally - one capable of understanding not just the spatial but also the temporal nuances of visual data. This is exactly where 3D Convolutional Neural Networks (3D CNNs) enter the picture.

My shift to 3D CNNs is more than just an academic interest; it is a journey towards a deeper understanding of how we can enable machines to perceive and interpret the world in its full dynamism, our stochasticity, and uncertainties even in seconds of actions- much like we do. Every video clip is a symphony of motions, emotions, and interactions, with multilayers of subtle meanings- and of course 3D CNNs promise to be the key to deciphering these complex sequences. As I embark on this journey, I am not just to expand the boundaries of my knowledge, but also to contribute to the broader field of computer vision, pushing towards systems that can understand and interact with the world in richer, more meaningful ways.

In subsequent blog posts, I invite you with me through the explorations of 3DCNNs - from the core concepts that distinguish them from their 2D counterparts to the intricate challenges and learning curves I have encountered while applying them to video data. Whether you are a seasoned expert in the field, a beginner, a grad student, or a curious onlooker, I hope to offer insights and experiences that resonate with this domain.

Background and Core Concepts

The Evolution from 2D to 3D CNNs

Understanding CNNs: Convolutional Neural Networks (CNNs) have been the cornerstone of image analysis in computer vision for years. Traditional 2D CNNs are adept at processing static images—learning spatial hierarchies and patterns by applying filters that capture various aspects of the image, such as edges, textures, and shapes. If you would like to find out more about 2D CNN, please refer to my slides and labs here

Limitation in Capturing Temporal Information: While 2D CNNs excel in spatial understanding, they fall short in comprehending temporal dynamics, which is crucial when dealing with video data. Videos are essentially sequences of frames, where each frame is tied to its predecessor and successor, creating a temporal continuity that 2D CNNs cannot capture.

The Emergence of 3D CNNs

Introduction to 3D CNNs: This is where 3D Convolutional Neural Networks change the game. Unlike their 2D counterparts, 3D CNNs are designed to understand both spatial and temporal features. They achieve this by adding an additional dimension—time—to the convolutional process.

How 3D CNNs Work: In a 3D CNN, the convolutional filters extend along three dimensions—height, width, and depth (time). This allows the network to not only learn from the spatial content of each frame but also gain insights into the motion and changes occurring across frames. As a result, 3D CNNs can unravel the complex tapestry of actions and interactions in video sequences.

Applications of 3D CNNs

Beyond Static Frames: The ability of 3D CNNs to interpret time makes them incredibly powerful for a range of applications. This includes action recognition in videos, where understanding the sequence of movements is key, and medical imaging, where temporal changes in 3D scans can indicate crucial health information. In each of these areas, 3D CNNs offer a more comprehensive understanding by considering the evolution of visual data over time.

Challenges and Opportunities: The shift to 3D CNNs, however, is not without its challenges. The addition of the temporal dimension increases the computational complexity significantly. Additionally, training 3D CNNs requires not only larger datasets but also datasets that accurately represent temporal variations.

My Research Odyssey with 3D CNNs

Transitioning to Spatio-Temporal Analysis

Initial Exploration: My journey into 3D CNNs began as an extension of my work with 2D CNNs, where I had focused on spatial feature extraction from static images. The transition to 3D CNNs marked a significant shift towards integrating the temporal dimension. My initial challenge lay in comprehending the intricacies of 3D convolutional layers – understanding how they extend the spatial interpretation of 2D CNNs to include temporal relationships.

The architectural nuances of 3D CNNs, such as the incorporation of time as a third dimension in convolutional operations, presented both a conceptual and practical learning curve. This was not merely about adapting to a new technique but rethinking the approach to data representation and processing.

Navigating Data Complexity

Data Preprocessing and Management: One of the most formidable challenges I faced was the preprocessing of video data. Unlike static images, video data comes with additional complexities like variable frame rates, diverse resolutions, and most crucially, a substantial increase in data volume. Developing an efficient preprocessing pipeline that could handle such diversity and volume was paramount. This involved not only frame extraction and resizing but also temporal sampling strategies to capture relevant motion information without overburdening the computational process.

Architectural Design and Computational Considerations: Designing the architecture of a 3D CNN requires a delicate balance. The model had to be sophisticated enough to capture intricate temporal patterns without becoming computationally infeasible. This entailed an iterative process of model design, where each layer's parameters were carefully calibrated to maximize learning while minimizing computational costs. The extended training durations and heightened resource demands of 3D CNNs necessitated a more strategic approach, leveraging distributed computing and optimizing algorithms for efficiency.

Gleaning Insights and Developing Solutions

Performance Optimization: In pursuit of optimal performance, I explored a variety of architectural tweaks and parameter adjustments. Strategies such as modifying stride and kernel size in convolutional layers, and incorporating advanced techniques like transfer learning, played a crucial role in surmounting the limitations imposed by the sheer scale of video data.

Combating Overfitting: The increased parameter count in 3D CNNs heightened the risk of overfitting. To mitigate this, I implemented a combination of regularization strategies, data augmentation techniques, and dropout layers. These measures were critical in ensuring that the model generalized well, rather than merely memorizing the training data.

Reflecting on the Journey

Working with 3D CNNs reinforced the virtue of patience. The field of 3D convolutional analysis is still burgeoning, with much left to explore and understand. Navigating this terrain often required an iterative, trial-and-error approach, underscoring the importance of resilience in research.

Charting Future Pathways

Advancing 3D CNN Research

Harnessing Technological Growth: As computational capabilities continue to advance and datasets grow both in size and complexity, the potential applications of 3D CNNs are set to broaden significantly. I am particularly intrigued by the prospects in domains like augmented reality, where interpreting both spatial and temporal information is key to creating immersive experiences.

Ongoing Exploration: My foray into 3D CNNs is an ongoing chapter in my academic journey. I'm keen to delve deeper into novel architectures and apply these models across a wider spectrum of applications. The ultimate goal is to push the frontiers of computer vision and contribute to the development of systems that can interact with our dynamic world more intelligently and intuitively.

Sneak Peek into the Next Blog Post

Intuition, Mathematics, Code: A Technical Deep Dive into 3D CNNs

In my upcoming blog post, we'll take a technical deep dive into the world of 3D Convolutional Neural Networks. I'll unravel the intuition behind these sophisticated models, illuminating how they interpret not just the visual cues in static images but also the temporal dynamics in videos. We'll delve into the mathematics that underpins these networks, demystifying how they learn and process information across both space and time. Expect to see detailed discussions on model architecture, accompanied by snippets of code that bring these concepts to life. Whether you're keen on understanding the nuts and bolts of 3D convolution operations or interested in the practical aspects of implementing these models in PyTorch, the next post promises to be a treasure trove of insights.

From discussing the nuances of kernel size and stride in 3D convolutions to exploring strategies for optimizing network performance, we will cover a spectrum of topics that will cater to both beginners and seasoned practitioners in the field. The goal is to provide you with a comprehensive understanding of 3D CNNs that balance theoretical depth with practical applicability. So, stay tuned for an enriching journey into the technical heart of 3D CNNs!

To take a sneek peak into an experimental 3D CNN Architecture please check here

In Part 1 of this series on camera calibration, we laid the groundwork by exploring the fundamental principles that govern how cameras translate the 3D world into a 2D image. We delved into camera models and the intrinsic and extrinsic parameters that play a vital role in this transformation. But that was merely scratching the surface.

In this second instalment, I'm going to broaden the scope significantly. We'll venture into the critical importance of camera calibration across various real-world applications—from robotics to autonomous vehicles and even the arts. We'll also uncover the lens distortions that could potentially mar your images and then look at the mathematical equations behind them.

So if you've ever wondered how self-driving cars make sense of their environment, how augmented reality applications manage to superimpose digital elements so naturally, or even questioned the mechanics behind your DSLR's crisp photos, you're in for a treat.

Reasons for Calibration

The importance of camera calibration extends far beyond academic interest—it plays a critical role in various real-world applications. I'll investigate why camera calibration is indispensable in key areas in this section.

1. Robotics and Automation

In robotics, precision is not just a nice-to-have - it is the name of the game. Whether the robots are on bustling factory floors or those designed to help people in their homes- these machines will have to 'know' what is around them and where exactly it is located. This is even more true for robots rocking machine perception tech, which allows them to interpret and make sense of their surrounding. Getting the camera calibration right in settings like these is often a big deal.

Take a factory assembly line, for example. Robots are often kitted out with cameras and machine perception algorithms to identify parts or objects. Mess up the camera calibration, and you're in for a world of trouble. Imagine a robot misjudging the position of a piece it's supposed to pick. That's the kind of error that can start a chain reaction of problems. This is not just about assembly lines or specific tasks, either. Suppose a robot is to pick up an item and place it somewhere specific - a well-calibrated camera ensures that the robot's actions are spot-on with what it is seeing. This is not limited to the task at hand but to the robot's ability to navigate more complex situations. Think about it: a finely calibrated camera can act like a robot's "sixth sense", allowing for on-the-fly adjustments during the job.

To sum it up, nailing camera calibration in robotics and automation isn't just a good practice; it is a must. Whether for aiding complex tasks or helping a robot safely navigate an unstructured environment, getting the camera settings right can either make or break the whole operation.

2. Autonomous Vehicles

We are on the brink of a game-changer - self-driving cars are about to become a common sight on our roads. But let us not forget, the tech making this possible is anything but simple. At the core, we have advanced vision systems that let these vehicles 'see the world around them. However, is not always enough; these systems must also be spot-on when interpreting this visual data for real-time decision-making. This is precisely where camera calibration comes in and becomes a critical piece of the puzzle.

For a minute, think about the challenges of driving autonomously. Cars must navigate a world filled with other vehicles, pedestrians, and other unpredictable elements. Oh, get the camera calibration wrong, and you are asking for trouble. Results of miscalibration? We are discussing potentially misjudging the distance to the car in front, which could translate to insufficient time to brake or even a full-on collision.

Here is the kicker: autonomous cars rely on many machine vision tasks - such as detecting obstacles, understanding road signs, or even interpreting road markings. Many of these cars would require more than one camera, each serving a specific purpose. Hence, calibrating each camera is not a one-off job- it is about ensuring all these cameras work harmoniously.

3. Augmented and Virtual Reality

Okay, let's talk AR and VR. These are realms where the line between the digital and the real world gets blurry. Whether overlaying virtual furniture in your real living room or immersing yourself in a completely digital world, the experience has to feel real. That's why camera calibration is a big deal in AR and VR tech.

Think about it. You put on a VR headset and step into a virtual world. You move your head, and the perspective changes perfectly in sync. That's not magic—it's precise calibration. If the camera's off even by a little, you might start to feel motion sickness or have a subpar experience. That's the last thing you want when battling space pirates or exploring a virtual museum.

Now, switch gears to AR. Imagine using an app on your smartphone to visualize how a new sofa would look in your living room. The app has to blend digital objects with the real world smoothly. If the camera calibration is off, that sofa might look like it's floating in mid-air or sinking into the floor. Not the best way to make a buying decision, right?

And let's not forget about more advanced applications. For example, getting the camera calibration wrong could be life and death in medical AR. Surgeons often use AR tech for guided procedures. In scenarios like this, the calibration needs to be absolutely spot-on for accurate guidance and successful outcomes.

So, all in all, whether you're gaming, shopping, or even performing surgery, camera calibration in AR and VR isn't just about enhancing the experience—it's about making it possible in the first place.

4. Film and Photography

Let's get into film and photography, where camera calibration isn't just about the tech—it's also about the art. In settings that demand a heavy dose of scientific rigor, like wildlife documentaries or high-speed sports action, getting your camera settings right is non-negotiable. Picture this: you're shooting a documentary on migratory birds. A well-calibrated camera lets you capture beautiful shots and accurate data on how fast and high these birds fly. That's adding a layer of scientific credibility to your storytelling.

But hey, it's not all about the numbers. Camera calibration also plays a starring role in the artistic side of things. Take landscape photography, for instance. You want those mountain ranges and valleys to look as majestic in the photo as they do in real life. A calibrated camera ensures that the proportions and spatial relationships within the frame are just right, enhancing your shots' emotional impact and narrative quality.

And let's not forget the controlled chaos of a studio setting. Calibration is your best friend, whether you're doing product photography, snapping high-fashion looks, or capturing fine art reproductions. In essence, camera calibration in film and photography is more than a behind-the-scenes technicality; it's a linchpin that can elevate your work from good to great. It's not just about getting the colour balance or the focus right; it's about capturing the subject's soul, be it a fast-paced sporting event or a still life. When your camera is finely tuned, your work speaks volumes—conveying scientific facts or evoking deep emotions.

Types of Distortion

Regarding camera calibration, addressing distortions is not just a side quest - it is the main objective. Distortions are discrepancies between the captured image and the real-world scene, affecting the accuracy of the camera's representations. Distortions can be characterised as deviations from the ideal imaging model, where rays from a single point in three-dimensional space converge at a single point on the imaging sensor. Numerous factors contribute to distortions, including lens shape, refractive index variations, and manufacturing imperfections. These distortions have various types, each with a mathematical model and correction method. Understanding these distortions is pivotal for calibrating the camera to achieve high accuracy in multiple applications.

A. Radial Distortions

1. Barrel Distortions: Barrel distortion is a sub-type of radial distortion, where straight lines appear to curve outward from the centre of the image.

The image magnification decreases with distance from the optical axis. This causes straight lines near the edge of the field to bend inward, resembling the shape of a barrel. This type of distortion is common in wide-angle lenses.

2. Pincushion Distortion: Conversely, image magnification increases with the distance from the optical axis in pincushion distortion. The result is that straight lines bend outward from the centre, akin to a pincushion

Mathematically, a unified model can represent both barrel and pincushion distortions, often employing higher-order polynomials, which is particularly useful when working with more complicated lens systems. The general formula is:

$$\begin{align*} x' &= x\left(1 + k_1 r^2 + k_2 r^4 + \ldots\right) \\ y' &= y\left(1 + k_1 r^2 + k_2 r^4 + \ldots\right) \end{align*}$$

Here, (x, y) are the original coordinates, (x', y') are the distorted coordinates, k₁, k₂, ... are the distortion coefficients, and r is the radial distance from the centre of the image, calculated as:

$$r = \sqrt{x^2 + y^2}$$

In this general model:

• A positive k₁ will produce pincushion distortion, as lines will curve outward.

• A negative ***k₁***​ will produce barrel distortion, where lines curve inward towards the centre.

• Higher-order terms like ***k₂***​ allow for more complex distortion patterns, which might be observed in higher-end or more flawed lens systems.

The model is extendable to as many terms as necessary, but in practice, most systems are sufficiently modelled using just k₁ and sometimes k₂.

B. Tangential Distortions

These distortions occur when the lens and the imaging plane are not parallel. Tangential distortions usually shift the image in a direction orthogonal to the radial distortions and can make the image look tilted or skewed. While radial distortions affect the image in a radially outward direction from the centre, tangential distortions act orthogonally to them. This means that they can make the image appear tilted or skewed, effectively moving the distorted image points horizontally and vertically in a way unrelated to their distance from the optical axis.

Mathematically, tangential distortion can be expressed as:

$$\begin{align*} x' &= x + \left(2p_1 xy + p_2 (r^2 + 2x^2)\right) \\ y' &= y + \left(p_1 (r^2 + 2y^2) + 2p_2 xy\right) \end{align*}$$

Here, x' and y' are the distorted coordinates. x and y are the original coordinates, and r is the radial distance from the origin, calculated as r = √(x² + y²).

The coefficients p₁ and p₂ are the tangential distortion coefficients. These terms aim to correct the tilt in the lens and bring the captured image closer to what would be charged if the lens were perfectly aligned. By adjusting the coefficients during the camera calibration process, one can minimize the effects of tangential distortions.

Conclusion

In this second instalment, we've delved deeper into the reasons for camera calibration across various industries, touched on different types of distortions, and hinted at the mathematics involved. However, we've only scratched the surface. In Part 3, we'll dive into the heart of the mathematics that makes accurate camera calibration possible. From optimization problems to factoring in distortions, we'll explore how all these elements combine to create a robust camera model. Stay tuned!

References

For those looking to delve deeper into the topics covered in this blog post, the following resources are highly recommended:

[1] Codes for distortion plots

[2] Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman

[2] Stanford CS231A: Camera Models

Imagine you're taking a photo of a building with your smartphone. You might notice that the lines of the building don't appear as straight as they do in real life, or the proportions seem slightly off. These are distortions, and discrepancies between the real-world objects and their representations in the image. Such distortions often occur due to the inherent limitations of camera lenses and sensors as they attempt to map a 3D world onto a 2D plane.

Camera calibration is the technique used to understand and correct these distortions. It's a fundamental process for achieving more accurate visual representations, especially in applications like augmented reality, robotics, and 3D reconstruction. In this first part of our series on camera calibration, we'll explore the foundational concepts and models that serve as the backbone of this technique. We'll delve into the intrinsic and extrinsic parameters that influence how a camera captures an image and discuss how these parameters can be determined to correct distortions. By the end of this post, you'll have a solid understanding of the principles behind camera calibration and its importance in various domains.

Camera Models

To understand the intricacies of camera imaging, it's useful to connect the dots with real-world applications. Take the example of a self-driving car, which relies on its camera to accurately gauge the dimensions and distances of surrounding elements like pedestrians, other vehicles, and road signs. Just as understanding the human eye's perception aids in comprehending our interaction with the 3D world, grasping the mechanics of a camera model enhances the precision of such measurements in automated systems. To unpack this further, let's engage in a thought experiment: envision a simple setup (See figure 1) where a small barrier with a pinhole is placed between a 3D object and a film. Light rays from the object pass through the pinhole to create an image on the film. This basic mechanism serves as the cornerstone for what is known as the pinhole camera model, a foundational concept that allows us to fine-tune the way cameras, like the one in a self-driving car, interpret the world.

Pinhole Camera Model

The setup

In the pinhole model, consider a 3D coordinate system defined by unit vectors i, j, k. Place an object point P with coordinates (X, Y, Z) in this world. The camera's aperture is at the origin O, and the image plane (or film) is parallel to the i-j plane at a distance f along the k-axis. The film has a centre C', and the projection of P onto the film is P' with 2D coordinates (x, y) - (Refer to Figure 1 and 2)

The Mathematics

To relate P and P', we draw a line from P through the aperture O, intersecting the film at P'. The triangles POP' and OCP' are similar, which gives us:

$$\frac{x}{f} = \frac{X}{Z} \quad \text{and} \quad \frac{y}{f} = \frac{Y}{Z}$$

Solving for x and y, we get:

$$\begin{align*} x &= f \left( \frac{X}{Z} \right), \\ y &= f \left( \frac{Y}{Z} \right). \end{align*}$$

Here, f represents the focal length of the Camera.

Lens Models

While the pinhole model gives us an idealized perspective of image formation, real-world cameras use lenses to focus light. Lenses introduce additional complexities due to their shape, material, and how they bend light rays. These models account for additional factors like focal length, aperture, and lens distortions. Let's explore lens models to understand these intricacies.

The Setup

Like the pinhole model, lens models use a 3D coordinate system defined by i, j, k. However, instead of a pinhole at O, we have a lens. The image plane is still at a distance f along the k-axis, we denote the centre of this plane as C' (Refer to Figure 3 and 4)

The Mathematics

In lens models, we need to account for distortions introduced by the lens. These distortions are typically represented by d_x and d_y, affecting the x and y coordinates, respectively. The equations for x and y in lens models are:

$$\begin{align*} x &= f \left( \frac{X}{Z} \right) + d_x, \\ y &= f \left( \frac{Y}{Z} \right) + d_y. \end{align*}$$

In these equations, dx and dy are functions of X, Y, and Z and they represent the distortions introduced by the lens.

Intrinsic and Extrinsic Parameters

So far, we've discussed the basic models that describe how cameras work and how they capture the 3D world onto a 2D plane. These models give us a high-level view but are generalized and often idealized. In practice, each camera has its unique characteristics that influence how it captures an image. These characteristics are captured by what are known as intrinsic and extrinsic parameters. While intrinsic parameters deal with the camera's own 'personality' or 'DNA' extrinsic parameters describe how the camera is positioned in space. Together, they offer a complete picture of a camera's behaviour, which is crucial for applications like 3D reconstruction, augmented reality, and robotics.

Intrinsic Parameters

After Understanding the broad overview of intrinsic and extrinsic parameters, let's zoom in on the intrinsic parameters first. These parameters are unique to each camera and provide insights into how it captures images. While these parameters are generally considered constants for a specific camera, it is important to note that they can sometimes change. For instance, in cameras with variable focal lengths or adjustable sensors, intrinsic parameters can vary.

1. Optical Axis

   The optical axis is essentially the line along which light travels into the camera to hit the sensor. In the idealized pinhole and lens models, it's the line that passes through the aperture (or lens centre) and intersects the image plane. It serves as a reference line for other measurements and parameters.

2. Focal Length (f ): This is the distance between the lens and the image sensor. Knowing the focal length is crucial for estimating the distances and sizes of objects in images. It's also a key factor in determining the field of view and is usually represented in pixels.

$$f = \alpha \times \text{sensor size} ,$$

$$\text{Here}, \alpha \space \text{is a constant that relates the physical sensor size to the size in pixels}$$

1. Principal Point ((c_x, c_y)): This is the point on the image plane where the optical axis intersects, it often lies near the centre of the image. it is crucial for tasks like image alignment and panorama stitching.

$$\begin{align*} c_x &= \frac{Image Width}{2},\\ \\ c_y &= \frac{Image Height}{2}. \end{align*}$$

1. Skew Coefficient (s): This parameter is responsible for any angle between the x and y pixel axes of the image plane. It is rarely encountered in modern-day cameras.

$$s = 0 \quad \text{(usually)}$$

The intrinsic matrix denoted by K consolidates these parameters:

$$K = \begin{pmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}$$

Note on Constancy

Although intrinsic parameters like the focal length and principal point are often treated as constants, especially in fixed or pre-calibrated camera setups, they can often change based on specific hardware configurations. For example, the focal length will vary in cameras with zoom capabilities. Therefore, in such special cases, recalibration may be necessary.

Camera with Zoom Capabilities

Cameras with zoom capabilities introduce an additional layer of complexity to the calibration process. While Zooming allows for better framing or focus on specific areas, it also changes intrinsic parameters like the focal length. This section will explore how to handle calibration in scenarios involving zoom.

Calibration at Specific Zoom Levels

When you calibrate a camera at a particular zoom level, the resulting intrinsic parameters are only accurate for that setting. If you continue to record or capture images at the same zoom level, these calibration parameters will remain valid.

$$K_{\text{zoom}} = \begin{pmatrix} f_{\text{zoom}} & s & c_x \\ 0 & f_{\text{zoom}} & c_y \\ 0 & 0 & 1 \end{pmatrix}$$

Here, K_zoom and f_zoom represent the camera matrix and focal length at the specific zoom level, respectively.

Handling Zoom Changes

If you adjust the zoom after calibration, you have two main options:

• Dynamic Calibration: Recalibrate the camera every time you change the zoom. This approach provides the highest accuracy but may be impractical for real-time applications due to computational costs.

• .Parameter Interpolation: If you've calibrated the camera at multiple zoom levels, you can interpolate the intrinsic parameters for new zoom settings. This is computationally efficient but might sacrifice some accuracy.

Understanding intrinsic parameters is key for various computer vision tasks. For instance, in augmented reality, an accurate intrinsic matrix can drastically improve the realism and alignment of virtual objects in real-world scenes.

Extrinsic Parameters

While intrinsic parameters define a camera's 'personality' by capturing its internal characteristics, extrinsic parameters tell the 'story' of the camera's interaction with the external world. These parameters, specifically the rotation matrix R and the translation vector T, are indispensable for mapping points from the camera's 2D image plane back to their original 3D coordinates in the world. This becomes particularly vital in scenarios involving multiple cameras or moving cameras, such as in robotics or autonomous vehicles. By accurately determining these extrinsic parameters, one can achieve high-precision tasks like 3D reconstruction and multi-camera scene analysis.

1. Rotation Matrix (R): This 3x3 matrix gives us the orientation of the camera in the world coordinate system. Specifically, it transforms coordinates from the world frame to the camera frame. For instance, if a drone equipped with a camera needs to align itself to capture a specific scene, the rotation matrix helps in determining the orientation the drone must assume.

   The rotation matrix is usually denoted as R and takes the form:

$$R = \begin{pmatrix} r_{11} & r_{12} & r_{13} \\ r_{21} & r_{22} & r_{23} \\ r_{31} & r_{32} & r_{33} \end{pmatrix}$$

The elements r₁₁, r₁₂, and r₁₃, .... r₃₃ define the camera's orientation relative to the world's coordinate system. Each column of R essentially represents the unit vectors along the camera's local x, y, and z axes, but expressed in terms of the world coordinate system. For example, r₁₁, r₂₁, and r₃₁ describe how much the world's x-component aligns with the camera's local x-axis.

1. Translation Vector (T): This 3x1 vector represents the position of the camera's optical centre in the world coordinate system. The translation vector is generally represented as:

$$T = \begin{pmatrix} t_x \\ t_y \\ t_z \end{pmatrix}$$

The elements t_x, t_y, and t_z in the translation vector represents the position of the camera's optical centre in the world coordinate system. For instance, t_x is the distance from the world origin to the camera's optical centre along the world's x-axis, while t_y and t_z serve the same purpose along the y and z axes, respectively.

Computing R and T gives you a complete picture of the camera's pose in the world, including both orientation and position.

Together, the rotation matrix and the translation vector can be combined into a single 3x4 matrix, often represented as [R|T]:

$$[R|T] = \begin{pmatrix} r_{11} & r_{12} & r_{13} & t_x \\ r_{21} & r_{22} & r_{23} & t_y \\ r_{31} & r_{32} & r_{33} & t_z \end{pmatrix}$$

Conclusion

We've covered a lot of ground in this first instalment of our series on camera calibration, unravelling the complexities behind camera models and the intrinsic and extrinsic parameters that define them. These foundational concepts are the building blocks for more advanced topics like distortion correction, 3D reconstruction, and multi-camera setups. In the next part of this series, we'll go beyond the basics to explore the practical reasons for camera calibration, the types of distortions you might encounter, and the mathematical and technical approaches to correct them. So, stay tuned for more insights into the fascinating world of camera calibration!

References

For those looking to delve deeper into the topics covered in this blog post, the following resources are highly recommended:

[1] Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman

[2] Stanford CS231A: Camera Models

Further Reading

For those looking to delve deeper into the topics covered in this blog post, the following resources are highly recommended:

1. Books:

   • Digital Image Warping by George Wolberg

   • Multiple View Geometry in Computer Vision by Richard Hartley and Andrew Zisserman

   • Computer Vision: Algorithms and Applications by Richard Szeliski

   • 3D Computer Vision: Efficient Methods and Applications by Christian Wöhler

2. Papers:

   • A Four-step Camera Calibration Procedure with Implicit Image Correction by Jean-Yves Bouguet

   • Flexible Camera Calibration By Viewing a Plane From Unknown Orientations by Zhengyou Zhang

By exploring these resources, you'll gain a more comprehensive understanding of camera calibration, enabling you to tackle more complex problems and applications.

Imagine you're a wine connoisseur with a penchant for data. You've collected a vast dataset that includes variables like acidity, sugar content, and alcohol level for hundreds of wine samples. You're interested in distinguishing wines based on these characteristics, but you soon realize that visualizing and analyzing multi-dimensional data is like trying to taste a wine from a sealed bottle—near impossible.

This is where the magic of Principal Component Analysis, or PCA for short, kicks in. Think of PCA as your data's personal stylist, helping your dataset shed unnecessary dimensions while keeping its essence intact. Whether you're dissecting the nuances of wine characteristics or diving into the depths of machine learning algorithms, PCA is your go-to for simplifying things without losing the crux of the data.

A Deep Dive into the Mathematics of PCA

Step 1 : The Covariance Matrix

Let's assume you given a 2D dataset X of size (n×2) (where ( n ) is the number of samples. Each row in X represents a data point in 2D space, with the first column representing the x-coordinates and the second coordinates representing the y-coordinates. The first step in PCA is to calculate its covariance matrix ∑:

$$\Sigma = \frac{1}{n} \sum_{i=1}^{n} (x_i - \mu)(x_i - \mu)^T$$

Here x_ij represents the i^th row in X (a 2D point), and μ is the mean vector of the dataset. The term (x_i - μ) represents the deviation of each point from the mean, and (x_i - μ)^T is its transpose.

Step 2: Eigen Decomposition

After calculating ∑ Next, we perform eigen-decomposition of the covariance matrix. This allows for finding its eigenvalues and eigenvectors. The eigen decomposition of ∑ can be represented as:

$$\Sigma = Q \Lambda Q^{-1}$$

Here Q is a matrix where each column is an eigenvector of ∑ and Λ is a diagonal matrix containing the eigenvalues λ_1, λ_2, ..... λ_d in descending order.

Step 3: Principal Components and Dimensionality Reduction

Let's say you have now found k eigenvectors (principal components) that you would like to use for dimensionality reduction. These k eigenvectors form a 2 x k matrix P.

The projected data Y , in the new k-dimensional space can be calculated as:

$$Y = X \cdot P$$

In this equation, X is the original n x 2 dataset, and P is the 2 x k matrix of principal components. The resulting Y will be of size n x k, effectively reducing the dimensionality of each data point from 2D to k-D.

Implementing PCA with Python

Here's a Python code snippet to get you started with PCA:

1. Data Generation: First let's generate some synthetic data with 100 samples in a 2D feature space between x and y coordinates. This sort of mimics the real-world data where features are often correlated.

    import numpy as np
    import matplotlib.pyplot as plt
   
    # Generate synthetic 2D data
    np.random.seed(0)
    x = np.random.normal(0, 10, 100)  # x-coordinates
    y = 2 * x + np.random.normal(0, 5, 100)  # y-coordinates
    data = np.column_stack((x, y))
   

2. Data Visualisation: Let's visualise what our generated data looks like.

# Plot the synthetic data
plt.figure(figsize=(8, 6))
plt.scatter(data[:, 0], data[:, 1], label='Original Data')
plt.xlabel('X')
plt.ylabel('Y')
plt.title('Synthetic 2D Data')
plt.grid(True)
plt.legend()
plt.show()

1. Data Centering: Before you can apply PCA, it is essential to center the data around the origin. This ensures that the first principal component describes the direction for maximum variance.

# Calculate the mean of the data
mean_data = np.mean(data, axis=0)

# Center the data by subtracting the mean
centered_data = data - mean_data

1. The covariance Matrix calculation: The covariance matrix captures the internal structure of the data. It is the basis for identifying the principal components.

# Calculate the covariance matrix
cov_matrix = np.cov(centered_data, rowvar=False)

print('Covariance of Data',cov_matrix)

$$\text{Covariance Matrix} = \begin{pmatrix} 102.61 & 211.10 \\ 211.10 & 461.01 \end{pmatrix}$$

Code output:

Covariance of Data [[102.60874942 211.10203024]
                     [211.10203024 461.00685553]]

1. Eigen Decomposition: Here we calculate the eigenvalues and eigenvectors of the covariance matrix. The eigenvectors point in the direction of maximum variance, and the eigenvalues indicate the magnitude of this variance - since the first principal component is the eigenvector associated with the largest eigenvalue of the data's covariance matrix. This eigenvector identifies the direction along which the dataset varies the most.

# Calculate the eigenvalues and eigenvectors of the covariance matrix
eig_values, eig_vectors = np.linalg.eig(cov_matrix)

print('Eigenvalues:', eig_values, '\n', 'Eigenvectors: ', eig_vectors)

$$\text{Eigenvalues} = \left[ 4.90, 558.71 \right]$$

$$\text{Eigenvectors} = \begin{pmatrix} -0.91 & -0.42 \\ 0.42 & -0.91 \end{pmatrix}$$

1. Projection and Visualization: Our data is then projected onto the principal component. The original data with the principal component, and the projected data are then plotted together to further emphasize the dimensionality reduction.

# Choose the eigenvector corresponding to the largest eigenvalue (Principal Component)
principal_component = eig_vectors[:, np.argmax(eig_values)]

# Project data onto the principal component
projected_data = np.dot(centered_data, principal_component)

# Re-plot the original data and its projection with the principal component as a red arrow

# Plot the original data and its projection
plt.figure(figsize=(10, 8))
plt.scatter(data[:, 0], data[:, 1], alpha=0.5, label='Original Data')

# Draw the principal component as a red arrow
plt.arrow(mean_data[0], mean_data[1], principal_component[0]*20, principal_component[1]*20,
          head_width=2, head_length=2, fc='r', ec='r', label='Principal Component')

# Plot the projected data as green points
plt.scatter(mean_data[0] + projected_data * principal_component[0],
            mean_data[1] + projected_data * principal_component[1],
            alpha=0.5, color='g', label='Projected Data')

plt.xlabel('X')
plt.ylabel('Y')
plt.title('Data and Principal Component')
plt.grid(True)
plt.legend()
plt.show()

Output:

There we go—the red arrow representing the principal component is now visible in the plot, along with the original data points and their projections (in green). The arrow points in the direction of the highest variance in the dataset, capturing the essence of the data in fewer dimensions.

Why does the PCA point in Downwards left?

You might have noticed that the red arrow, our principal component, points towards the bottom left. IS this supposed to happen? Absolutely, and here is why:

The direction of the principal component is calculated mathematically to capture the maximum variance in the synthetic dataset. This direction is defined by the eigenvector corresponding to the largest eigenvalue of the covariance.

Simply put, the principal components serve as a "line of best fit" for the multidimensional data It doesn't necessarily mean an alignment with the x and y axis but it captures the correlation between these dimensions. In this specific synthetic dataset, the principal component points towards the bottom left, indicating that as one variable decreases, the other tends to decrease as well, and vice-versa.

This is a crucial insight because it tells us not just about the spread of each variable but also about their relationship with each other. So, yes, the direction of the principal component is both intentional and informative.

In case you would like to run the full code use the replit window below:

Real-world Applications

In the Glass: Wine Quality Estimation

Let's circle back to our wine example. You could use PCA to distinguish wines based on key characteristics. By reducing the dimensions, you can visualize clusters of similar wines and maybe even discover the perfect bottle for your next dinner party!

Beyond the Bottle: Other Fields

1. Data Visualization: High-dimensional biological data, stock market trends, etc.

2. Noise Reduction: Image processing and audio signal processing.

3. Natural Language Processing: Feature extraction from text data.

Future Directions

1. Kernel PCA: For when linear PCA isn't enough.

2. Sparse PCA: When you need a sparse representation.

3. Integrating with Deep Learning: Using PCA for better initialization of neural networks.

Further Reading

For those who wish to delve deeper into PCA, here are some textbook references:

1. "Pattern Recognition and Machine Learning" by Christopher M. Bishop

2. "The Elements of Statistical Learning" by Trevor Hastie, Robert Tibshirani, and Jerome Friedman

3. "Machine Learning: A Probabilistic Perspective" by Kevin P. Murphy

Conclusion

In the realm of data science, PCA ages like a well-kept Bordeaux—it only gets richer and more valuable as you delve deeper. This versatile approach is more than just a mathematical trick; it's a lens that brings clarity to your analytical endeavors. So whether you're a wine lover seeking the perfect blend, a data scientist sifting through gigabytes, or a machine learning guru, mastering PCA is like adding a Swiss Army knife to your data analysis toolkit.

This verse of the scripture reminds me of the inherent grace and beauty found in our relationship with the Creator. The imagery of being crowned amidst majestic mountains and vast seas brings forth feelings of awe, wonder, and profound gratitude. We are a unique creation, positioned just a little lower than the heavenly beings, yet bestowed with honour and glory. I'm thankful for this reminder of our divine connection, where the natural world stands as a testament to the Creator's grand design.

The mountains and seas serve not merely as a scenic backdrop but as a profound metaphor for our existence, filled with purpose and meaning. They inspire a sense of humility, yet simultaneously elevate our understanding of our special place in this magnificent creation. I am deeply grateful for the realization that I am part of this intricate and purposeful design, created with intention and love. It's a thought that fills me with thankfulness and inspires me to live a life that reflects this connection. How majestic indeed is His name in all the earth, and how profound is our connection to it all!

Thank you Ahayah!

2. "Mao: The Unknown Story" by Jung Chang - A historical exploration of Mao Zedong's life and impact on China.

3. "The Irish Difference: A Tumultuous History of Irish breakup with Britain" by Fergal Tobin - An in-depth examination of Irish culture, including its historical background and unique characteristics.

4. "Multiple View Geometry in Computer Vision" by Richard Hartley and Andrew Zisserman - I have read papers from both authors, fascinated by their works.

5. "Bayesian Reasoning and Machine Learning" by David Barber - the future is plagued with uncertainty and so is our physical world. Building an interactive machine for our physical world requires understanding uncertainties and mitigating their ripple effects. An exploration of the integration of Bayesian reasoning and machine learning for modeling uncertain systems and mitigating their potential impact.

For the last few months, I have been bothered by the ideology peddled by most employers of labour. This very concern has led me to ask a not-so-popular question – am I being approached for employment or opportunities because of the colour of my skin? Perhaps, this question is ‘popular’ howbeit in the minds of the most concerned few. This rather daunting question, even led to a more stomach-turning one – is this equity or equality If my question turns out to be true? Of course, if False, am I being headhunted because of my intelligence, skills, and ‘diversity’ of my uniqueness? Or is it rather because of prejudice? If True, does this mean I am privileged and profiting from an undeserved opportunity?

As a researcher, when I am faced with a challenging technical problem, especially the ones that leave me tasking for days, I am led to examine the base class. In object-oriented programming, a base class is a fundamental template or blueprint on which other classes are built. These newly created classes inherit functionalities, methods, and principles from the base class. It is also to be noted that new methods, principles, and ‘ideology’ can be created which can override the inherited methods. Please hold this thought as this will make more sense soon. Back to my original ponder, the questions I have asked myself for weeks have led me to this one question. Which is best Equality or Equity? Or succinctly put – Which is the more noble, just, and fair goal- Equity or Equality? While both have inherited the ideas of social justice, fairness of rights and opportunities, – one more than the other, is overriding the very fundamental truth of the base class while claiming to belong to the base class.

As a society, we constantly debate over whether equality or equity is the more desirable goal. On the surface, the two concepts may appear to be interchangeable, but upon deeper examination, it becomes clear that they represent fundamentally different ways of thinking about the world.

Equality is the absolute ideal that everyone should be treated equally, regardless of background or characteristics. This is a noble goal and one that is deeply ingrained in our culture. The idea that all people should be treated with dignity and respect is a fundamental principle of democracy. However, the problem with this approach is that it assumes that everyone starts from the same place and that the same opportunities are available to everyone.

This is a fallacy. In fact, there is no one equal person and in truth, people have different starting points and challenges to overcome. Some individuals may have had a privileged upbringing, while others may have struggled with poverty or discrimination. Treating everyone the same, without taking these differences into account, can perpetuate inequality, defeating the main purpose of fairness, diversity, and inclusion.

Equity, however, is the idea that everyone should have and be provided/presented with an equal opportunity to succeed. This means that individuals and groups who have been traditionally marginalized may require additional resources or support to achieve the same level of success as those who have not faced such barriers.

To achieve equity, we must be willing to acknowledge and address the ways in which structural inequalities exist in our society. This requires us to take a step back and examine the systems and institutions that shape our lives. We must ask ourselves: Are the playing field and opportunities equal for all individuals? Are certain groups or individuals facing barriers or discrimination that make it harder for them to succeed? It is only by acknowledging these difficult truths and taking steps to address them, that we can truly achieve a society that is fair and just for all. Equality may be a nice idea, but it is not enough. We must strive for equity if we are to create a society in which everyone can reach their full potential.

It is however important to note that achieving equity does not mean that everyone will have the same outcome, but rather that everyone will have the same opportunity to succeed. This means that some individuals may still achieve more success than others, but it will not be due to systemic barriers or discrimination that has constantly plagued our society. More importantly, equity is not about granting preferential treatment to certain groups or individuals, but rather about levelling the playing field and providing the necessary resources and support to overcome barriers.

Additionally, equity must be seen as an ongoing-continuous process, as society is ever-changing and dynamic- opportunities to address inequalities, challenges, and discrimination will always arise. In practice, achieving equity may involve a variety of actions, such as implementing policies and practices that promote diversity and inclusion, creating more accessible educational and job training programs, and addressing biases in hiring and promotion practices.

Ultimately, the goal of equity is to create a society in which everyone can reach their full potential, regardless of their background or characteristics. It's not only morally right but also beneficial for society, as a diverse and inclusive society is more productive and innovative.

In conclusion, while equality is a very noble goal, it is not enough to achieve a truly just and fair society. Equity is the more desirable goal, as it acknowledges and addresses the structural inequalities that exist in our society and ensures that everyone has an equal opportunity to succeed. This requires us to be willing to look beyond the surface and examine the systems and institutions that shape our lives. Only by achieving equity can we create a society in which everyone can thrive.