Real-time hand tracking and finger counting is one of the most engaging ways to introduce computer vision concepts. In this post, we will explore how to build a simple yet powerful system that detects hands and counts fingers using MediaPipe and OpenCV in Python.

This project demonstrates how modern AI libraries allow us to build interactive applications with minimal code while still understanding the underlying logic.

🚀 What This Project Does

The system uses your webcam (or IP camera) to:

• Detect a human hand in real time
• Track key landmarks (finger joints)
• Count how many fingers are raised
• Display the result live on the screen

This can be extended into applications such as:

• Gesture-based control systems
• Touchless interfaces
• Robotics control using hand gestures
• Interactive installations

🧠 Key Concepts

This project introduces several important ideas:

1. Computer Vision

Using cameras to extract meaningful information from images.

2. Landmark Detection

MediaPipe identifies 21 key points on the hand, including fingertips and joints.

3. Real-Time Processing

Each frame from the camera is processed continuously to give instant feedback.

4. Logic-Based Finger Counting

Finger states are determined by comparing landmark positions.

🛠️ System Requirements

Software

• Python 3.10 or 3.11
• Visual Studio Code (recommended)
• Git (optional but useful)

⚠️ MediaPipe may not work properly with Python 3.12+

Hardware

• Webcam (built-in or external)
• OR IP camera stream

📂 Project Structure

Finger_Detection_Assignment/

├── Finger_count.py # Main program
├── README.md # This file

Create a virtual environment

Windows

python3 -m venv myenv

macOS / Linux

python3 -m venv myenv

Verify Installation

Run in the terminal

python3 --version 

You should see

Python 3.11.9

Activate the virtual environment

.\myenv\Script\activate

Install Required Python Libraries

Ensure the virtual environment is activated before installing packages.

    pip install mediapipe==0.10.11 opencv-python

Run the program

The code for the finger Detection is below:

import cv2
import mediapipe as mp
import time

mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils


def count_fingers(hand_landmarks, handedness):
    """
    Return how many fingers are up (0–5).
    handedness: 'Left' or 'Right'
    """
    lm = hand_landmarks.landmark
    fingers_up = 0

    # ---- Thumb ----
    thumb_tip = lm[mp_hands.HandLandmark.THUMB_TIP]
    thumb_ip  = lm[mp_hands.HandLandmark.THUMB_IP]

    if handedness == "Right": 
        if thumb_tip.x < thumb_ip.x:
            fingers_up += 1
    else:  # Left hand
        if thumb_tip.x > thumb_ip.x:
            fingers_up += 1

    # ---- Other fingers ----
    finger_tips = [
        mp_hands.HandLandmark.INDEX_FINGER_TIP,
        mp_hands.HandLandmark.MIDDLE_FINGER_TIP,
        mp_hands.HandLandmark.RING_FINGER_TIP,
        mp_hands.HandLandmark.PINKY_TIP,
    ]
    finger_pips = [
        mp_hands.HandLandmark.INDEX_FINGER_PIP,
        mp_hands.HandLandmark.MIDDLE_FINGER_PIP,
        mp_hands.HandLandmark.RING_FINGER_PIP,
        mp_hands.HandLandmark.PINKY_PIP,
    ]

    for tip_id, pip_id in zip(finger_tips, finger_pips):
        if lm[tip_id].y < lm[pip_id].y:
            fingers_up += 1

    return fingers_up


def main():
    cap = cv2.VideoCapture(0)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)


    with mp_hands.Hands(
        max_num_hands=2,
        model_complexity=1,
        min_detection_confidence=0.5,
        min_tracking_confidence=0.5
    ) as hands:

        while True:
            ret, frame = cap.read()
            if not ret:
                break

            frame = cv2.flip(frame, 1)
            rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            results = hands.process(rgb)

            if results.multi_hand_landmarks:
                for hand_landmarks, handedness in zip(
                        results.multi_hand_landmarks,
                        results.multi_handedness):

                    mp_drawing.draw_landmarks(
                        frame,
                        hand_landmarks,
                        mp_hands.HAND_CONNECTIONS
                    )

                    label = handedness.classification[0].label
                    num_fingers = count_fingers(hand_landmarks, label)

                    print(f"Hand: {label}, Fingers up: {num_fingers}")

                    cv2.putText(
                        frame,
                        f"{label}: {num_fingers}",
                        (10, 60 if label == "Right" else 120),
                        cv2.FONT_HERSHEY_SIMPLEX,
                        1.5,
                        (0, 255, 0),
                        3
                    )

            cv2.imshow("Finger Count (0–5)", frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

    cap.release()
    cv2.destroyAllWindows()


if __name__ == "__main__":
    main()

Go to the terminal(path to your code) and run:

python3 Finger_count.py

And you will see the camera open with the finger-count as below

YOLOv8 Red & Green Object Detection

📌 Project Overview

This project uses YOLOv8 to detect and classify two object classes:

• 🟩 greenbox
• 🟥 redbox

⚙️ Training Configuration

After running

yolo detect train model=yolov8n.pt data=data.yaml imgsz=320 epochs=10 batch=16 device=0

📁 Important Output Files

Inside the training folder (runs/detect/train/), the most important files are:

• results.png ✅ (main training graph)
• confusion_matrix.png ✅
• PR_curve.png ✅
• F1_curve.png ✅
• weights/best.pt ✅

📊 results.png (Main Graph)

Shows:

• Training & validation loss
• Precision and recall
• mAP50 and mAP50-95

✔ What to check:

• Loss decreases over time → model is learning
• Validation follows training → no overfitting
• mAP increases → performance improves

Throught the Graph we can analyze

1. TRAINING LOSS ANALYSIS

🔹 train/box_loss

• Decreases from ~0.75 → ~0.53

• Smooth and consistent

  Interpretation:

• Model is improving bounding box localization

  🔹 train/cls_loss

• Drops rapidly from ~1.4 → ~0.28
  Interpretation:
• Model quickly learns to classify redbox vs greenbox
• Task is relatively easy (distinct colors)

🔹 train/dfl_loss

• Gradual decrease

Interpretation:

• Model is refining bounding box precision

2. Validation Loss Analysis

🔹 val/box_loss

• Smooth decrease

Interpretation:

• Good generalization to unseen validation data

🔹 val/cls_loss

• Spike at early epoch (~3)
• Then decreases steadily

Interpretation:

• Early training instability (normal)
• Model stabilizes afterward

🔹 val/dfl_loss

• Smooth decreasing trend

Interpretation:

• Bounding box quality improves on validation data

🚨 Key Concept: Overfitting Check

Conclusion:

• No overfitting observed
• Model generalizes well

3. Precision Analysis

• Starts ~0.94
• Drops briefly
• Converges to ~1.0

Interpretation:

• Very few false positives
• Temporary fluctuation is normal

4. Recall Analysis

• Similar behavior to precision
• Ends near 1.0

Interpretation:

• Model detects almost all objects
• Very few missed detections

📊 5. mAP Analysis (Most Important Metric)

🔹 mAP50

• Final value ≈ 0.995

Interpretation:

• Nearly perfect detection at IoU = 0.5

🔹 mAP50-95

• Improves from ~0.81 → ~0.93

Interpretation:

• Strong performance under stricter evaluation
• Indicates robust bounding box quality

⚠️ 6. Early Epoch Instability

Observed at epoch ~3:

• Drop in precision, recall, and mAP

Reason:

• Random weight initialization
• Learning adjustment phase

Important:

• This is normal behavior in training
• Focus on overall trend, not individual fluctuations

📉 7. Trend vs Raw Values

• Blue line: actual values
• Orange line: smoothed trend

Interpretation:

• Smoothed curve shows true learning behavior
• Model trend is stable and improving

📉 confusion_matrix.png

Shows classification performance:

What to check:

• Strong diagonal → good model
• Off-diagonal → classification errors

Below is the confusing matrix we get after the training

✅ Correct Predictions (Diagonal)

• greenbox → greenbox = 573
• redbox → redbox = 427

🎯 Interpretation:

• Model correctly classifies almost all objects
• Very strong diagonal → excellent performance

❌ Errors (Off-Diagonal)

🔹 False Positives (FP)

• 1 background detected as redbox

👉 Meaning:

• Model detected an object where there is none

🔹 False Negatives (FN)

• 1 greenbox missed (predicted as background)
• 1 redbox missed (predicted as background)

👉 Meaning:

• Model failed to detect some objects

📊 Error Summary

👉 Total errors = 3 (very low)

📈 Performance Interpretation

✅ Strengths

• Nearly perfect classification between red and green
• Almost no confusion between classes
• Extremely high accuracy

📈 PR_curve.png (Precision–Recall Curve)

Shows tradeoff between precision and recall.

What to check:

• Curve near top-right → excellent model
• Large area under curve → high performance

✅ 1. Precision–Recall (PR) Curve

Purpose:

• Shows overall model performance
• Combines:
  • Precision (accuracy)
  • Recall (completeness)

Why important:

• Used to compute mAP (main YOLO metric)
• Best indicator of model quality

Result:

• Curve near top-right
• mAP ≈ 0.995

👉 Conclusion: Excellent model

📈 2. Precision–Confidence Curve (P Curve)

Purpose:

• Shows how precision changes with confidence threshold

Interpretation:

• Higher confidence → fewer false positives
• Model becomes more strict

Result:

• Precision quickly reaches ~1.0
• Very stable

👉 Conclusion: Predictions are highly accurate

📉 3. Recall–Confidence Curve (R Curve)

Purpose:

• Shows how many objects are detected as threshold changes

Interpretation:

• Higher confidence → more missed detections

Result:

• Recall ≈ 1.0 at low confidence
• Drops sharply after ~0.9

👉 Conclusion: High confidence may miss objects

📊 F1_curve.png

Shows best balance between precision and recall.

What to check:

• Peak value → best performance
• Confidence at peak → optimal threshold

👉 It represents the best balance between Precision and Recall

🔹 Shape:

• F1 is very high (~1.0) across a wide range
• Drops sharply after ~0.9 confidence

🔹 Key point:

• Best F1 ≈ 1.00 at confidence ≈ 0.799

✅ 1. Excellent Performance

• F1 ≈ 1.0 → almost perfect balance
• Means:
  • Very few false positives
  • Very few missed detections

👉 Model is extremely strong

⚖️ 2. Optimal Threshold

• Best confidence ≈ 0.8

👉 This is the ideal operating point

At this point:

• Precision is high
• Recall is high
• Overall performance is maximized

🧠 weights/best.pt

The final trained model.

Use it for inference:

yolo detect predict model=weights/best.pt source=your_image.jpg conf=0.8

1. Detection Output

For example:

 yolo detect predict model=best.pt source=test/images/greenbox_0_jpg.rf.b7606581a957e3d3d3b8a36e5a4d82cb.jpg conf=0.8

Model detected:

• 1 object
• Class = greenbox
• Inference time = 12.8 ms

YOLOv8 Red & Green Object Detection

Raspberry Pi Deployment Guide

This project demonstrates how to train, convert, and deploy a YOLOv8 model for red and green color object detection using a dataset from Roboflow.

⚙️ Configuration Overview

1. Download Dataset from Roboflow

You will train the model using a dataset hosted on Roboflow.

1. Open your dataset page on Roboflow: 👉 YOLOv8 Red-Green Detection Dataset

2. Click Download Dataset → YOLOv8 format
   Then extract it inside your project folder.

3. The folder structure should look like this:

   ├── train/

   ├── valid/

   ├── data.yaml

4. Inside your data.yaml, confirm that it contains something like this:

   train: ../train/images
   val: ../valid/images
   nc: 2
   names: ['redbox', 'greenbox']
   

2. Set Up Training Environment (on Windows)

This section explains how to prepare your environment on Windows for training YOLOv8.

1. Open PowerShell in your project folder.

2. Create a new Python virtual environment: ```bash python -m venv yolo

3. Activate the environment: ```bash .\yolo\Scripts\activate

4. PyTorch CUDA build (current default index provides CUDA-enabled wheels) ``` pip install –upgrade pip pip install torch torchvision torchaudio –index-url https://download.pytorch.org/whl/cu121

   # Ultralytics (YOLOv8) pip install ultralytics

5. Verify installation:

   yolo version
   ## 3. Train the YOLOv8 Model
   Train your YOLOv8 model using your Roboflow dataset.
   

   yolo detect train model=yolov8n.pt data=data.yaml imgsz=320 epochs=10 batch=16 device=0 ``` we use the device=0 to train using GPU. If your laptop doesn’t have the gpu, you can train on google colab

After training, your weights will be located at:

runs\detect\train\weights\best.pt

4. Testing the model on Laptop

Below code is used for testing model and detection with best.pt on laptop

from ultralytics import YOLO
import cv2

# --- Load your trained YOLOv8 model ---
model = YOLO("best.pt")   # make sure best.pt is in the same folder as this script

# --- Open webcam (0 = default cam, or replace with video path or image folder) ---
cap = cv2.VideoCapture(0)   # use 0, 1, or a file path like 'video.mp4'

if not cap.isOpened():
    print("Cannot open webcam")
    exit()

print("✅ Webcam opened successfully. Press 'q' to quit.")

while True:
    ret, frame = cap.read()
    if not ret:
        print("Frame grab failed.")
        break

    # --- Run YOLOv8 inference on the frame ---
    results = model(frame, stream=True)

    # --- Process detections ---
    for r in results:
        boxes = r.boxes
        for box in boxes:
            cls_id = int(box.cls[0])
            conf = float(box.conf[0])
            label = model.names[cls_id]

            # Get coordinates
            x1, y1, x2, y2 = map(int, box.xyxy[0])

            # Draw box and label
            color = (0, 255, 0) if label == "greenbox" else (0, 0, 255)
            cv2.rectangle(frame, (x1, y1), (x2, y2), color, 2)
            cv2.putText(frame, f"{label} {conf:.2f}", (x1, y1 - 10),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.7, color, 2)

            print(f"Detected: {label} (conf {conf:.2f})")

    # --- Show the result ---
    cv2.imshow("YOLOv8 Detection", frame)

    # Exit on 'q'
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

5. Export Model to ONNX (For Raspberry pi)

yolo export model=best.pt format=onnx imgsz=320 opset=12 dynamic=False simplify=False nms=True

6. Transfer Model to Raspberry pi

scp best.onnx aupp@pi-ip:/home/aupp/Documents/

7. Create and activate virtual environment

python3 -m venv ~/yolo && source ~/yolo/bin/activate

Install Python Package

pip install --upgrade pip
pip install "numpy==1.23.5" onnxruntime==1.16.3 flask==3.0.0 opencv-python-headless==4.9.0.80

8. Testing ONNX on The webstream with Raspberry pi

The code below is used for testing the model of the red-green detection with rapberry pi

#!/usr/bin/env python3
import os, time, cv2
from threading import Thread, Lock
from flask import Flask, Response, jsonify, make_response
from ultralytics import YOLO

# -------- config --------
MODEL_PATH = "onnx_path"   # put best.onnx next to this app.py
CAM_INDEX  = 0
IMG_SIZE   = 320

CONF       = 0.5

# -------- model --------
model = YOLO(MODEL_PATH)

# -------- camera thread --------
class Camera:
    def __init__(self, index=0, width=None, height=None):
        self.cap = cv2.VideoCapture(index)
        if width:  self.cap.set(cv2.CAP_PROP_FRAME_WIDTH,  width)
        if height: self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)
        self.ok, self.frame = self.cap.read()
        self.lock = Lock()
        self.running = True
        self.t = Thread(target=self.update, daemon=True)
        self.t.start()

    def update(self):
        while self.running:
            ok, f = self.cap.read()
            if ok:
                with self.lock:
                    self.ok, self.frame = ok, f
            else:
                time.sleep(0.01)

    def read(self):
        with self.lock:
            return self.ok, None if self.frame is None else self.frame.copy()

    def release(self):
        self.running = False
        time.sleep(0.05)
        self.cap.release()

cam = Camera(CAM_INDEX)

# -------- flask --------
app = Flask(__name__)

INDEX_HTML = """<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>YOLOv8 ONNX - Raspberry Pi Stream</title>
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <style>
    :root { color-scheme: light dark; }
    body { margin:0; min-height:100vh; display:grid; place-items:center;
           background:#0b0c10; color:#eaf0f6; font-family:system-ui,Segoe UI,Roboto,sans-serif; }
    .card { width:min(96vw,900px); background:#111417; border-radius:16px; padding:14px;
            border:1px solid rgba(255,255,255,0.08); box-shadow:0 10px 40px rgba(0,0,0,.35); }
    h1 { margin:6px 0 10px; font-size:1.05rem; }
    .row { display:flex; gap:10px; justify-content:space-between; align-items:center; }
    .btn { border:1px solid rgba(255,255,255,.12); background:#1b2229; color:#eaf0f6;
           padding:6px 12px; border-radius:10px; cursor:pointer; font-weight:600; }
    .btn:hover { background:#222b33; }
    .frame { width:100%; aspect-ratio:16/9; background:#0d1117; border-radius:12px; overflow:hidden;
             border:1px solid rgba(255,255,255,0.08); display:grid; place-items:center; }
    img { width:100%; height:100%; object-fit:contain; }
    small { opacity:.65; }
  </style>
</head>
<body>
  <div class="card">
    <div class="row">
      <h1>Raspberry Pi • YOLOv8 (ONNX) Live</h1>
      <button class="btn" onclick="reloadStream()">Reload</button>
    </div>
    <div class="frame">
      <img id="stream" src="/stream" alt="Stream">
    </div>
    <div class="row" style="margin-top:8px;">
      <small>Status: <span id="health">checking…</span></small>
      <small>URL: <code id="url"></code></small>
    </div>
  </div>
<script>
  async function checkHealth() {
    try {
      const r = await fetch('/health', {cache:'no-store'});
      const j = await r.json();
      document.getElementById('health').textContent = j.camera_ok ? 'camera OK' : 'no camera';
    } catch (e) {
      document.getElementById('health').textContent = 'server offline';
    }
  }
  function reloadStream() {
    const img = document.getElementById('stream');
    img.src = '/stream?ts=' + Date.now();
  }
  document.getElementById('url').textContent = location.href;
  checkHealth(); setInterval(checkHealth, 4000);
</script>
</body></html>
"""

@app.route("/")
def index():
    return make_response(INDEX_HTML, 200)

@app.route("/health")
def health():
    ok, _ = cam.read()
    return jsonify({"camera_ok": bool(ok)})

def gen_mjpeg():
    while True:
        ok, frame = cam.read()
        if not ok or frame is None:
            time.sleep(0.02)
            continue
        results = model.predict(frame, imgsz=IMG_SIZE, conf=CONF, verbose=False)
        annotated = results[0].plot()
        # optional resize to cut bandwidth/CPU:
        # annotated = cv2.resize(annotated, (640, 360))
        ok, jpg = cv2.imencode(".jpg", annotated, [cv2.IMWRITE_JPEG_QUALITY, 80])
        if not ok:
            continue
        b = jpg.tobytes()
        yield (b"--frame\r\n"
               b"Content-Type: image/jpeg\r\n"
               b"Cache-Control: no-cache\r\n"
               b"Content-Length: " + str(len(b)).encode() + b"\r\n\r\n" + b + b"\r\n")

@app.route("/stream")
def stream():
    return Response(gen_mjpeg(), mimetype="multipart/x-mixed-replace; boundary=frame")

if __name__ == "__main__":
    try:
        app.run(host="0.0.0.0", port=5000, threaded=True)
    finally:
        cam.release()

After that You will see the output as shown in the image below

-> Red Detection

-> Green Detection