Solar Panels and Logistics Features
One of our clients specializes in the technical maintenance and analytics of the condition of solar panels during their resale.
The panel is complex to manufacture and very fragile. During operation, it often incurs various defects that affect its performance. When reselling a solar panel, the buyer wants confirmation that the product will be of proper quality. For this, they request images from the seller in a special spectrum where all defects are visible. Moreover, if a broken panel reaches its destination and its defects are discovered there, it also has to be shipped back to the repair shop. Logistics are overloaded, and the buyer incurs losses.
To avoid transporting a defective product across the world and back, it is necessary to identify defects as early as possible and not to purchase such panels. To accomplish this, panels are photographed with serial numbers in a special spectrum and sent to the buyer for inspection.
Bottlenecks
Studying each panel is a time-consuming process. Authorized specialists are very rare as qualified personnel, so sometimes their role is performed by anyone who is more or less “in the know,” which impacts the quality. And if we add human factors to this — carelessness, or simple fatigue, when the eye gets “blurry” and it becomes difficult to discern minor defects, you end up with a false negative result. On top of that, there is still a need to fill out a report, documenting every scratch. It took up to 20 minutes to process a single panel, which significantly limited maneuverability and created “bottlenecks.”
Needless to say, the technician’s qualification in quality is something that is hard to scale and hard to replace. In the case of illness or absence from work, logistical “clots” form in the terminals from panels awaiting inspection. This already undermines the expectations of end customers who are eagerly waiting for the equipment within a specific timeframe.
Could ML (Machine Learning) help us?
The client decided that this task urgently needed to be automated and put onto the tracks of machine vision. For this, a team from India was hired, which attempted to assemble a solution based on OpenCV by applying heuristics for crack detection. In this approach, an image of the panel in a special black-and-white spectrum was inputted, a grid of cells was located on the snapshot, and then various crack patterns were searched for within the grid by filtering out noise.
Processing one panel took up to 30 (!) minutes (compared to 20 minutes for human processing), and the quality left much to be desired. Several hundred snapshots needed to be processed each day, and at this speed, it simply wasn’t feasible.
Coming to the Rescue
The client was disheartened and almost lost faith in machine vision, but then the project fell into our capable hands.
In the very first iteration, we abandoned CV heuristics and decided to immediately deploy neural networks, specifically our favorite YOLO architecture. The processing time was reduced to 10 minutes per panel, but the quality was not much better than the solution from the team in India. This made us realize that the principle of “garbage in, garbage out” (poor input data leads to poor analysis results) was not just a truism but a hard-earned maxim — there were significant problems in our dataset.
All is Not Simple with the Data
A significant limitation for us was the original dataset: we had images where defects were outlined in blue and labeled in the picture. Moreover, the defects were circled with ovals instead of the usual square boxes in specialized software and were very poorly localized (a small crack was circled with a huge oval). In such a situation, you can’t separate the data from the labeling, and these artifacts significantly made our life difficult.
Overall, the quality of the original dataset left much to be desired, and we came to understand that the dataset needed refinement. We tried to label the data ourselves but quickly realized that this was a bad idea — ML engineers are not suitable for this task, and distracting others was also too cumbersome. So we decided to go with re-labeling through crowdsourcing solutions.
Wisdom of the Crowd
On the Yandex.Toloka platform, we created a project. We tried to do everything professionally: educational materials, online exams, assigning skills to labelers, and manual (and later automated) quality control of performers.
Thanks to the well-structured process and the availability of a pool of qualified labelers, we could now scale the labeling: adding new classes of defects, labeling new batches of data, and receiving the finished result within a couple of hours (compared to two weeks of in-house labeling) all within a reasonable budget.
Moving Forward with Implementation
After uploading the re-annotated data, the model showed progress: the IoU (Intersection over Union — the intersection between the actual defect box and the predicted box) increased from 40% to 75%, which already served as a good confidence threshold. Further tuning of the YOLO architecture reduced the processing time from 10 minutes to just 1 minute. The model was mature and ready to be deployed into the real world.
As part of the implementation efforts, we developed a desktop application for logistics center specialists, where they could upload gigabytes of images. As a result, they received a report with analytics on the most problematic panels and could also view automatically annotated panels online through a browser. The overall processing time at route nodes was halved.
For the manufacturer, we developed a back-office interface through which they could view the uploaded data from logistic nodes, see the panels themselves with highlighted defects, generate detailed reports, and draw conclusions based on them.
Up next: new challenges.
Despite the project’s successful launch, we understand that there is no limit to perfection and are tackling tasks for further optimization, helping the client save money intelligently. We are transitioning to less expensive CPU instances for processing instead of costly GPU machines, combating fraud (yes, sometimes images are Photoshopped to hide defects), and continually improving the IoU metrics of our model.
