Fine-Grained Image Recognition

Due to the non-disclosure agreement (NDA), I cannot describe too many technical details about the project.

Fine-Grained Image Recognition for Dishes

My job focuses on applying deep learning (DL) for image processing. I worked as a project owner for a project that aims at providing fine-grained dish recognition service. The challenges in this project are preprocessing crawled data labeled with ambiguous tags, e.g., the same dish can be named differently and the same tag may refer to different dishes, and distinguishing the dishes that have very similar visual features. E.g., “Curry potato beef” looks very similar to “Curry potato pork.” To address these challenges, I built a hierarchical dish tag system (DTS) that has the largest coverage of dishes in the world and collected 10 million food images that are well labeled according to this tag system. By leveraging hierarchy information during the training process, the subtle difference among dishes can be identified. As a result, the dish recognition services can identify more than 8,500 categories of dishes and 250,000 subcategories; and provide rich information of dishes like ingredients, calories, and recipes.

This service achieves the best performance among peer companies and has been widely adopted by Baidu Nuomi and Baidu App and lead to several patents. Moreover, since it is possible to customize the food library according to the food industry’s needs, this service has been widely used in the scenario of self-checkout in the food industry, which reaches the goals of a rapid cash register and low workforce cost.

Fine-Grained Image Recognition for Damaged Cars

Moreover, I was also engaged in another project called CAR damage detection that aims to identify the damaged part with give more detailed information about damage types, damage area, and damage severity. This technology is beneficial for the car insurance industry. The challenge in this project is the fusion of multiple models such as part segmentation, damage segmentation and damage classification, and key-point detection. By applying a bi-linear approach, our system achieved a good result after the model ensemble.

The car damage service has been ready for launch to the market. And my proposed approach has been patented. This technology enables an insurance company to save cost for car damage grading，and speed up the procedure of damage claim.