One of the largest components of EcoDRIFTER is the floating plastic detection using ML.

I had trained my model by using annotating software called Roboflow’s Autodistill which trained the Yolo V8 algorithm and generated files for consumption in Raspberry Pi. I am sharing a step by step guide of this exercise. The last step of connecting with Raspberry Pi will be covered in the next article as the article is getting very long.

What are we doing here

To use autodistill, you input unlabeled data into a Base Model ( Grounding Dino)which uses an Ontology to label a Dataset that is used to train a Target Model( YoloV8 )which outputs a Distilled Model fine-tuned to perform a specific Task.

Autodistiil is a library that allows you to distill Knowledge from large foundational models and transfer it into smaller highly optimized ones; it consists of six core elements

 1. Task 2. base model 3. Ontology 4. data set 5. Target Model and the 7. Using it on a device like Raspberry Pi  . Currently it supports only one base model “Grounded SAM”. 

You do not have to bother about the code , the things that needs to be done are

1. Jupyter & python installation
2. Autodistill Python file installation
3. Creating home/videos and home/Images directories
4. Image/Video uploads
5. Editing Captiontology

How to do it

Installation of the appropriate environment for training YOLO

Set up of Jupyter , Python and Autodistill environments

Steps to reproduce my results:

1. Open PowerShell (enhanced command prompt in Windows):
   1. Click on the Windows menu icon, type “Power” and you should see PowerShell
      
      
   2. Look towards the right side and you will see an option to “Run as Administrator”. If Windows requests you to confirm, click on Yes
      
2. Before we execute the Jupyter notebook, we must download the input roboflow autodistill notebook from here https://github.com/roboflow/notebooks – as of 5th May 2025 this was the direct link. Search word: “Autodistill”.
   
   Please move this Jupyter notebook out of your default download folder into a separate code folder where you will create the necessary subfolders as well.
   For me, this was C:\Users\hitan\Coding\ED-v03 – you just need to setup your directory correctly the first time. Open this link in github , I would not recommend Google colab, as I have explained in the FAQ
3. If you are absolutely new to Python and Jupyter, I recommend acquainting yourself with Jupyter introductory videos like this.You can learn how to install python and jupyter notebook.
4. Navigate to the code directory which contains the Jupyter Notebook (.ipynb):
   
5. Now,launching your Jupyter notebook is super easy – simply type “jupyter notebook” and press Enter. Jupyter will open a browser window with the directory structure of the current folder. You can see the pynb files have a distinct icon.
   
6. If it is your first run, you must trust the Python notebook. So, please click on File -> Trust Notebook
7. First, check if your Jupyter notebook has GPU access. Go to http://localhost:8888/notebooks/how-to-auto-train-yolov8-model-with-autodistill.ipynb” since I have made a local copy http://localhost:8888/notebooks/how-to-auto-train-yolov8-model-with-autodistill.ipynbf it instead of using Google colab notebook
8. Navigate to the first code cell and press Shift-Enter to execute it.

Code to be run before every model re-run

1. I have aggregated all initialization section code in one code snippet. Simply search for “INITI” in the open PYNB Jupyter notebook file and you should reach it.
   
2. Why:
   1. The original code from Roboflow was designed to be easier to explain, building various variable definitions at each point that they were required.
   2. This was a problem – it meant that every re-run of the code, whether to train a new model, or run test inference on a new video, I would have to remember which 5-6 setup snippets needed to be run.
      

Image dataset preparation

4. Create home/ videos and home/images directory for the images and videos 
5. Upload images & videos in the given path
6. Display image sample
7. Convert video into images

Divide videos into two parts, 2 training videos and 1 test video atleast, you can put more. Every 6th frame from each video is converted into an image . You can change this value

8. This searches VIDEO_DIR_PATH for files ending in .mov or .mp4.It returns a list of video file paths and stores it in video_paths.Split videos into test and train sets, The first two videos (video_paths[:2]) go to TEST_VIDEO_PATHS for testing. The rest (video_paths[2:]) go to TRAIN_VIDEO_PATHS for training.

Display Image sample

This shows that autodistill has started annotating the images

Autolabel Dataset

1. Using ontology autodistill starts annotating the images

Ontology – an Ontology defines how your Base Model is prompted, what your Dataset will describe, and what your Target Model will predict. A simple Ontology is the CaptionOntology which prompts a Base Model with text captions and maps them to class names.  I edited the existing code of Milk bottle for my use

From this

To this

2. Initiate base model and Autolabel

Base Model – A Base Model is a large foundation model that knows a lot about a lot. Base models are often multimodal and can perform many tasks. They’re large, slow, and expensive. Examples of Base Models are GroundedSAM and GPT-4’s upcoming multimodal variant. We use a Base Model (along with unlabeled input data and an Ontology) to create a Dataset. Here GROUNDEDSAM is being called

Display dataset sample

Dataset – a Dataset is a set of auto-labeled data that can be used to train a Target Model. It is the output generated by a Base Model.

Here is a dataset sample, notice how it has detected all plastic bottles and objects in the device and annotated it

Training YOLOv8

Now that we have the dataset we are ready to train YOLOv8. Target Model – a Target Model is a supervised model that consumes a Dataset and outputs a distilled model that is ready for deployment. Target Models are usually small, fast, and fine-tuned to perform a specific task very well (but they don’t generalize well beyond the information described in their Dataset). Examples of Target Models are YOLOv8 and DETR.

Evaluate target model

Now we need to use our autoannotated dataset to evaluate the model.

These are two inferences that the model ran on the videos

Export Section

Now the dataset needs to be exported in the format readable by Raspberry Pi. The ONXX or NCNN are better performance

Connecting to Raspberry Pi & loading the code there

This article is getting very long hence I will explain the connection 

Frequently Asked Questions

Q1. Why did you install Python locally, instead of continuing to work in Google COLAB?

A1. Much longer setup time, and negligible speed improvement over a desktop gaming setup with a decent GPU. 

My desktop has more power available than the shared Google COLAB (Paid)
Local system:

COLAB

Google COLAB is excellent, but for a complex project like EcoDrifter – there are multiple package dependencies. In Google COLAB, your code runs in a virtual environment. However, this environment is closed when the page is inactive for a little while. This meant that every new run of the model required re-installation of the dependencies, which would add 2-3 min to every iteration. Further, all the training videos and images were in Google Drive, which required remounting requiring a confirmation prompt + few second delay every single time. Running the jupyter model locally can be nearly unsupervised in comparison.