DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment

BackgroundDietary assessment is a fundamental component of nutrition research and plays a pivotal role in managing chronic diseases. Traditional dietary assessment methods, particularly in the context of Thai cuisine, often require extensive training and may lead to estimation errors. ObjectivesTo address these challenges, Institute of Nutrition, Mahidol University (INMU) iFood, an innovative artificial intelligence–based Thai food dietary assessment system, allows for estimating the nutritive values of dishes from food images. MethodsINMU iFood leverages state-of-the-art technology and integrates a validated automated Thai food analysis system. Users can use 3 distinct input methods: food image recognition, manual input, and a convenient barcode scanner. This versatility simplifies the tracking of dietary intake while maximizing data quality at the individual level. The core improvement in INMU iFood can be attributed to 2 key factors, namely, the replacement of Yolov4-tiny with Yolov7 and the expansion of noncarbohydrate source foods in the training image data set. ResultsThis combination significantly enhances the system’s ability to identify food items, especially in scenarios with closely packed food images, thus improving accuracy. Validation results showcase the superior performance of the INMU iFood integrated V7-based system over its predecessor, V4-based, with notable improvements in protein and fat estimation. Furthermore, INMU iFood addresses limitations by offering users the option to import additional food products via a barcode scanner, thus providing access to a vast database of nutritional information through Open Food Facts. This integration ensures users can track their dietary intake effectively, with expanded access to over 3000 food items added to or updated in the Open Food Facts database covering a wide variety of dietary choices. ConclusionsINMU iFood is a promising tool for researchers, health care professionals, and individuals seeking to monitor their dietary intake within the context of Thai cuisine and for ultimately promoting better health outcomes and facilitating nutrition-related research.

Food image recognition is a key enabler for many smart home applications such as smart kitchen and smart personal nutrition log. In order to improve living experience and life quality, smart home systems collect valuable insights of users’ preferences, nutrition intake and health conditions via accurate and robust food image recognition. In addition, efficiency is also a major concern since many smart home applications are deployed on mobile devices where high-end GPUs are not available. In this paper, we investigate compact and efficient food image recognition methods, namely low-level and mid-level approaches. Considering the real application scenario where only limited and noisy data are available, we first proposed a superpixel based Linear Distance Coding (LDC) framework where distinctive low-level food image features are extracted to improve performance. On a challenging small food image dataset where only 12 training images are available per category, our framework has shown superior performance in both accuracy and robustness. In addition, to better model deformable food part distribution, we extend LDC’s feature-to-class distance idea and propose a mid-level superpixel food parts-to-class distance mining framework. The proposed framework show superior performance on a benchmark food image datasets compared to other low-level and mid-level approaches in the literature.

In recent years, food image recognition has emerged as a research hotspot in the field of computer vision. This paper proposes a dish recognition system based on image recognition and deep learning image processing and recognition, which can replace manual dish recognition. Because the food image is complex and dynamic, the system employs a Convolutional Neural Network (CNN). The dish detection algorithm has been completed using CNN. Multiple sets of comparative experiments were designed for the data before and after the improvement, and the ionfluence of the improvement points on the test data results was analyzed.

Food image recognition is very useful in nowadays applications. It has been shown recently that the recognition accuracy depends on the ratio between background and foreground of the food image. In this work, the food image recognition using ResNet with FOOD-101 dataset is studied. The food area in the image is determined using ENet. The effect of the increase of the ratio between background and foreground of food images in terms of recognition accuracy is determined. It is found that increasing such ratio from the inner food area of the image can help increasing the accuracy. An average of 2.5% for increasing the accuracy is obtained when increasing the ratio of 100% from the inner ratio. Additionally, a higher accuracy can also be achieved compared to the overall full image recognition accuracy, for the case of inner food area ratio with 50% increasing ratio. These results show that with a different food image input, it is possible to increase the food image recognition accuracy.

Food image recognition is a key research area in food computing, with widespread applications in dietary assessment, menu analysis, and nutrition monitoring. However, due to the influence of imaging devices and environmental factors, this technology is often subject to varying degrees of noise interference during application, which limits classification performance. To address this issue, we proposed a food image recognition method based on anti-noise learning and covariance feature enhancement. We designed a Noise Adaptive Recognition Module (NARM), which inputs additional noisy images during model training, treating denoising as an auxiliary task to progressively enhance the model's noise invariance and improve recognition accuracy. To mitigate the potential negative effects caused by the additional noise and to enhance the representation power of small eigenvalues, we innovatively proposed the Eigenvalue-Enhanced Global Covariance Pooling (EGCP) and integrated it into NARM. Simultaneously, we designed a new Weighted Multi-Granularity Fusion (WMF) method to improve feature extraction capability progressively. Additionally, by combining the Progressive Temperature-Aware Feature Distillation (PTAFD) method, we further optimized model efficiency without adding extra overhead to the original backbone network. The experimental outcomes show that the model we have developed delivers top-tier performance on the ETH Food-101 and Vireo Food-172 food image datasets. Specifically, our method achieves a Top-1 accuracy of 92.57 on ETH Food-101, surpassing other existing methods, and also performs well on the Top-5 accuracy on ETH Food-101 and the Top-1 and Top-5 accuracy on Vireo Food-172.

The recognition of food images is of great significance for nutrition monitoring, food retrieval and food recommendation. However, the accuracy of recognition had not been high enough due to the complex background of food images and the characteristics of small inter-class differences and large intra-class differences. To solve these problems, this paper proposed a food image recognition method based on transfer learning and ensemble learning. Firstly, generic image features were extracted by using the convolutional neural network models (VGG19, ResNet50, MobileNet V2, AlexNet) pre-trained on the ImageNet dataset. Secondly, the 4 pre-trained models were transferred to the food image dataset for model fine-tuning. Finally, different basic learner combination strategies were adopted to establish the ensemble model and classify feature information. In this paper, several kinds of experiments were performed to compare the results of food image recognition between single models and ensemble models on food-11 dataset. The experimental results demonstrated that the accuracy of the ensemble model was the highest, reaching 96.88%, which was superior to any base learner. Therefore, the convolutional neural network model based on transfer learning and ensemble learning has strong learning ability and generalization ability, and it is feasible and practical to apply the method to food image recognition.

With the development of artificial intelligence technology, food image recognition has become an important research direction in the field of computer vision. The region of Northern Shaanxi is famous for its rich food culture. This paper aims to propose a food image recognition method based on an improved ResNet network to enhance the recognition rate of characteristic foods in Northern Shaanxi. Firstly, the principles and structure of basic convolutional neural networks (CNNs) were introduced, with a focus on the application and optimization design of CNNs in food image recognition. This mainly included AC blocks fused with asymmetric convolutions, attention modules based on improving food image recognition performance, and residual structure design for enhancing learning effectiveness. Secondly, the FoodResNet18 model was constructed with a specially designed enhancement block and a deep, shallow shared attention residual module to enhance the feature extraction ability and perception of visual information by the model. To improve the generalization ability of the model, this paper comprehensively preprocessed the self-built Northern Shaanxi Food-300 dataset, covering the sources of data, processing methods, and data augmentation strategies used for training. The model training and comparative analysis show that the food image recognition method based on improved ResNet outperforms traditional CNN models in multiple experiments. In the ablation experiment, the specific contribution of the design module to the final recognition performance was analyzed, and the advantages of the deep shallow shared attention residual module in feature extraction and preservation were verified.

Malnutrition is a common and severe problem in patients with cancer that directly increases the incidence of complications and significantly deteriorates quality of life. Nutritional risk screening and dietary assessment are critical because they are the basis for providing personalized nutritional support. No digital smartphone-based self-administered tool for nutritional risk screening and dietary assessment among hospitalized patients with cancer has been developed and evaluated. This study aims to develop a digital smartphone-based self-administered mini program for nutritional risk screening and dietary assessment for hospitalized patients with cancer and to evaluate the validity of the mini program. We have developed the R+ Dietitian mini program, which consists of 3 parts: (1) collection of basic information of patients, (2) nutritional risk screening, and (3) dietary energy and protein assessment. The face-to-face paper-based Nutritional Risk Screening (NRS-2002), the Patient-Generated Subjective Global Assessment Short Form (PG-SGA-SF), and 3 days of 24-hour dietary recall (3d-24HRs) questionnaires were administered according to standard procedure by 2 trained dietitians as the reference methods. Sensitivity, specificity, positive predictive value, negative predictive value, κ value, and correlation coefficients (CCs) of nutritional risk screened in R+ Dietitian against the reference methods, as well as the difference and CCs of estimated dietary energy and protein intakes between R+ Dietitian and 3d-24HRs were calculated to evaluate the validity of R+ Dietitian. A total of 244 hospitalized patients with cancer were recruited to evaluate the validity of R+ Dietitian. The NRS-2002 and PG-SGA-SF tools in R+ Dietitian showed high accuracy, sensitivity, and specificity (77.5%, 81.0%, and 76.7% and 69.3%, 84.5%, and 64.5%, respectively), and fair agreement (κ=0.42 and 0.37, respectively; CC 0.62 and 0.56, respectively) with the NRS-2002 and PG-SGA-SF tools administered by dietitians. The estimated intakes of dietary energy and protein were significantly higher (P<.001 for both) in R+ Dietitian (mean difference of energy intake: 144.2 kcal, SD 454.8; median difference of protein intake: 10.7 g, IQR 9.5-39.8), and showed fair agreement (CC 0.59 and 0.47, respectively), compared with 3d-24HRs performed by dietitians. The identified nutritional risk and assessment of dietary intakes of energy and protein in R+ Dietitian displayed a fair agreement with the screening and assessment conducted by dietitians. R+ Dietitian has the potential to be a tool for nutritional risk screening and dietary intake assessment among hospitalized patients with cancer. Chinese Clinical Trial Registry ChiCTR1900026324; https://www.chictr.org.cn/showprojen.aspx?proj=41528.

The National Nutrition Survey (NNS) is the Philippines' official source of data on nutritional status, diet, and lifestyle-related risk factors for non-communicable diseases. A food consumption survey is one of its major components that collects individual dietary data through 24-h food recall interviews and household weighing of food intake using a dietetic scale. Conventional manual surveys are costly and tedious to conduct, which can also compound the inconsistencies and biases in the data gathered. Technological advances in mobile devices and networking have made it possible to gather data more economically and more efficiently with emerging technologies such as machine learning (ML) and increased use of automation open vast opportunities. In this paper, we explored the digitalization of food component forms of the NNS and developed a fully digital dietary survey collection system from collection to transmission of data. Google AutoML Vision, an AutoML platform based on NAS was used to create a Filipino food image recognition model as an added feature of the developed system. A Filipino food image dataset comprises 32 food classes and 3,295 images were created for this process. The developed food image recognition model scored a modest 76% on the precision metric in correctly identifying foods, which performed at par with other custom food recognition models. Nutrient approximation based on the Philippine Food Composition Table database also proved to be useful in displaying nutrition facts of recognized foods. This study showed the potential of a digital dietary survey system, combined with a unique and specific food image recognition tool, is a very encouraging proposition. Expansion of the food image dataset, exploration of other ML techniques, automated food weighta approximation, and continuous enhancement of the developed system are advocated.

The food segmentation technology based on artificial intelligence has witnessed rapid development in recent years. Influenced by artificial intelligence (especially the rapidly evolving deep learning), it has provided possible conditions for the emergence of this new interdisciplinary research field in food computing. This thesis comprehensively summarizes and discusses the current research progress in food image recognition, introduces the current mainstream food image recognition datasets, and conducts performance comparisons and superiority/desirability distinctions of different recognition algorithms' recognition results on these datasets. Moreover, in future research work, from the perspective of food image recognition based on the characteristics of food images, which parts should be emphasized in food image recognition, it summarizes that the food segmentation technology is upgrading from a single recognition task to a multimodal collaborative decision-making system, and will undoubtedly drive the value leap of the food industry chain in the future.

BackgroundAccurate and feasible assessment of dietary intake remains challenging for research and healthcare. Experience Sampling Methodology (ESM) is a real-time real-life data capturing method with low burden and good feasibility not yet fully explored as alternative dietary assessment method.MethodsThis scoping review is the first to explore the implementation of ESM as an alternative to traditional dietary assessment methods by mapping the methodological considerations to apply ESM and formulating recommendations to develop an Experience Sampling-based Dietary Assessment Method (ESDAM). The scoping review methodology framework was followed by searching PubMed (including OVID) and Web of Science from 2012 until 2024.ResultsScreening of 646 articles resulted in 39 included articles describing 24 studies. ESM was mostly applied for qualitative dietary assessment (i.e. type of consumed foods) (n = 12), next to semi-quantitative dietary assessment (i.e. frequency of consumption, no portion size) (n = 7), and quantitative dietary assessment (i.e. type and portion size of consumed foods) (n = 5). Most studies used ESM to assess the intake of selected foods. Two studies applied ESM as an alternative to traditional dietary assessment methods assessing total dietary intake quantitatively (i.e. all food groups). ESM duration ranged from 4 to 30 days and most studies applied ESM for 7 days (n = 15). Sampling schedules were mostly semi-random (n = 12) or fixed (n = 9) with prompts starting at 8–10 AM and ending at 8–12 PM. ESM questionnaires were adapted from existing questionnaires, based on food consumption data or focus group discussions, and respond options were mostly presented as multiple-choice. Recall period to report dietary intake in ESM prompts varied from 15 min to 3.5 h.ConclusionsMost studies used ESM for 7 days with fixed or semi-random sampling during waking hours and 2-h recall periods. An ESDAM can be developed starting from a food record approach (actual intake) or a validated food frequency questionnaire (long-term or habitual intake). Actual dietary intake can be measured by ESM through short intensive fixed sampling schedules while habitual dietary intake measurement by ESM allows for longer less frequent semi-random sampling schedules. ESM sampling protocols should be developed carefully to optimize feasibility and accuracy of dietary data.

Monitoring and estimation of food intake is of great significance to health-related research, such as obesity management. Traditional dietary records are performed in manual way. These methods are of low efficiency and a waste of labor, which are highly dependent on human interaction. In recent years, some researches have made progress in the estimation of food intake by using the computer vision technology. However, the recognition results of these researches are usually for the whole food object in the image, and the accuracy is not high. In terms of this problem, we provide a method to the food smart recognition and automatic dietary assessment on the mobile device. First, the food image is processed by MASK R-CNN which is more efficient than traditional methods. And more accurate recognition, classification and segmentation results of the multiple food items are output. Second, the OpenCV is used to display the food category and the corresponding food information of unit volume on the recognition page. Finally, in order to facilitate daily use, TensorFlow Lite is used to process the model to transplant to the mobile device, which can help to monitor people’s dietary intake.

Background: Accurate estimation of dietary intake is challenging. However, whilst some progress has been made in high-income countries, low- and middle-income countries (LMICs) remain behind, contributing to critical nutritional data gaps. This study aimed to validate an objective, passive image-based dietary intake assessment method against weighed food records in London, UK, for onward deployment to LMICs. Methods: Wearable camera devices were used to capture food intake on eating occasions in 18 adults and 17 children of Ghanaian and Kenyan origin living in London. Participants were provided pre-weighed meals of Ghanaian and Kenyan cuisine and camera devices to automatically capture images of the eating occasions. Food images were assessed for portion size, energy, nutrient intake, and the relative validity of the method compared to the weighed food records. Results: The Pearson and Intraclass correlation coefficients of estimates of intakes of food, energy, and 19 nutrients ranged from 0.60 to 0.95 and 0.67 to 0.90, respectively. Bland–Altman analysis showed good agreement between the image-based method and the weighed food record. Under-estimation of dietary intake by the image-based method ranged from 4 to 23%. Conclusions: Passive food image capture and analysis provides an objective assessment of dietary intake comparable to weighed food records.

Food image recognition tasks are generally addressed by using a closed dataset. In a real-world setting, however, the dataset is updated as the new class of food appears, and it is impossible to train a model that distinguishes all kinds of food in advance. Also, inter-class similarity and intra-class diversity make food image recognition tasks more challenging. Previous works have shown that a personalized classifier using an individual user's food records can deal with some of these problems. On top of that, we propose a personalized classifier that learns from not only the accumulating food records of the individual user but also the growing records of the entire user. To conduct a realistic experiment, we build a new dataset of daily food images using a food-logging application called FoodLog Athl. As a result, our proposed method significantly outperforms prior personalized classification methods for food image recognition in a realistic setting.

In our paper we tried to classify food images using convolutional neural network. Convolutional neural network extracts spatial features from images so it is very efficient to use convolutional neural network for image clasification problem. Recently people are sharing food images in social media and writing review on food. So there is a lot of food image in the social media but some image may not be labeled. It will be very helpful for restaurants if they can advertise their food to those people who is looking similar kind of foods they offer. Food classification system can help social media platform to identify food. Food classification system can enable an opportunity for social media platform to offer advertisement service for restaurants and beverage companies to their targeted users. It will be financially beneficial for both social media platform and beverage companies. Food classification is very difficult task because there is high variance in same category of food images. We developed a convolutional neural network model to classify food images in food-11 dataset. We also used a pre-trained Inception V3 convolutional neural network model to classify food images.