A Deep Learning based Food Recognition System for Lifelog Images

Accurate dietary assessment is essential for enhancing health and managing nutrition-related issues. This paper outlines the creation of a Real-Time Food Recognition and Calorie Estimation System that employs deep learning methodologies, particularly Convolutional Neural Networks (CNN) and the MobileNetV2 framework. Users can upload images of food via a web-based platform, where the model classifies the food and estimates its caloric and macronutrient content. Utilizing transfer learning and data augmentation techniques, the model achieves an 85% accuracy rate in classifying 10 different food categories. With a lightweight and scalable design, the system guarantees real-time predictions and is compatible with various devices. Additionally, features such as dynamic feedback and historical tracking improve user interaction and system flexibility. By automating food recognition and nutritional assessment, this system offers a dependable, efficient, and user-friendly solution for personalized dietary monitoring.

Obesity is becoming a widely concerned health problem of most part of the world. Computer vision based recognition system has great potential to be an efficient tool to monitor food intake and cope with the growing problem of obesity. This paper proposes a food recognition algorithm based on sparse representation. The proposed algorithm learns overcomplete dictionaries from local descriptors including texture and color features that are extracted from food image patches. With the learned two overcomplete dictionaries, a feature vector of the food image can be generated with the sparsely encoded local descriptors. SVM is used for the classification. This research creates a Chinese food image dataset for experiments. Classifying Chinese food is more challenging because they are not as distinguishable visually as western food. The proposed algorithm achieves an average classification accuracy of 97.91% in a dataset of 5309 images that comprises 18 classes. The proposed method can be easily employed to dataset with more classes. Our results demonstrate the feasibility of using the proposed algorithm for food recognition.

The visual analysis of foods on social media by using food recognition algorithm provides valuable insight from the health, cultural and marketing. Food recognition offers a means to automatically recognise foods as well the useful information such as calories and nutritional estimation by using image processing and machine learning technique. The interest points in food image can be detected effectively by using Maximally Stable Extremal Region (MSER). As MSER used global segmentation and many food images have a complex background, there are numerous irrelevant interest points are detected. These interest points are considered as noises that lead to computation burden in the overall recognition process. Therefore, this research proposes an Extremal Region Selection (ERS) algorithm to improve MSER detection by reducing the number of irrelevant extremal regions by using unsupervised learning based on the k-means algorithm. The performance of ERS algorithm is evaluated based on the classification performance metrics by using classification rate (CR), error rate (ERT), precision (Prec.) and recall (rec.) as well as the number of extremal regions produced by ERS. UECFOOD-100 and UNICT-FD1200 are the two food datasets used to benchmark the proposed algorithm. The results of this research have found that the ERS algorithm by using optimum parameters and thresholds, be able to reduce the number of extremal regions with sustained classification performance.

The recent works in cross-modal image-to-recipe retrieval pave a new way to scale up food recognition. By learning the joint space between food images and recipes, food recognition is boiled down as a retrieval problem by evaluating the similarity of embedded features. The major drawback, nevertheless, is the difficulty in applying an already-trained model to recognize different cuisines of dishes unknown to the model. In general, model updating with new training examples, in the form of image-recipe pairs, is required to adapt a model to new cooking styles in a cuisine. Nevertheless, in practice, acquiring sufficient number of image-recipe pairs for model transfer can be time-consuming. This paper addresses the challenge of resource scarcity in the scenario that only partial data instead of a complete view of data is accessible for model transfer. Partial data refers to missing information such as absence of image modality or cooking instructions from an image-recipe pair. To cope with partial data, a novel generic model, equipped with various loss functions including cross-modal metric learning, recipe residual loss, semantic regularization and adversarial learning, is proposed for cross-domain transfer learning. Experiments are conducted on three different cuisines (Chuan, Yue and Washoku) to provide insights on scaling up food recognition across domains with limited training resources.

FVEstimator: A novel food volume estimator Wellness model for calorie measurement and healthy living

Introduction Previous studies indicate that lack of sleep might increase risk for unhealthy eating and obesity. This is particularly important during adolescence, when most youth sleep less than the recommended 8-10 hours/night and dietary habits are developed that extend into adulthood. This study investigated the relationship between multi-night sleep restriction and the appeal of food, including neuroimaging to examine the impact on brain responses to food-related stimuli. Methods Healthy 14-17 year-old adolescents (n=39) completed an experimental sleep manipulation across consecutive five-night periods to compare nightly sleep of approximately 9 hours (healthy sleep duration) vs. about 6.3 hours (mild sleep restriction). At the end of each week, participants underwent functional MRI while performing a visual food appeal task. The task included 42 photos in each of four food categories (sweets, snacks, fast-food, meat/fruit/vegetable) and one non-food category. Photos were presented every 3 seconds in blocks of 7 within each category; 6 interleaved blocks per category. Teens rated the appeal of each block of photos. General linear modeling explained regional brain response according to categorical presentation time-courses. Results The pattern of brain responses to the different food types was similar across the two sleep conditions. However, the sleep manipulation led to significant regional effects when contrasting the totality of food vs non-food images. Specifically, when compared to non-food, food images overall resulted in greater activation in the ventral tegmental area (VTA) and substantia nigra (SN) during sleep restriction. Both regions have been previously shown to be involved in processing reward-related information. Conclusion Our findings suggest neuronal responses in reward circuitry for teens viewing food images are influenced by sleep duration. Sleep restriction may affect reward processing of food in teens by increasing brain activation in VTA/SN network components that underlie dopamine-mediated motivational drive. Support Supported by NIH R01HL120879

The National Nutrition Survey (NNS) is the Philippines' official source of data on nutritional status, diet, and lifestyle-related risk factors for non-communicable diseases. A food consumption survey is one of its major components that collects individual dietary data through 24-h food recall interviews and household weighing of food intake using a dietetic scale. Conventional manual surveys are costly and tedious to conduct, which can also compound the inconsistencies and biases in the data gathered. Technological advances in mobile devices and networking have made it possible to gather data more economically and more efficiently with emerging technologies such as machine learning (ML) and increased use of automation open vast opportunities. In this paper, we explored the digitalization of food component forms of the NNS and developed a fully digital dietary survey collection system from collection to transmission of data. Google AutoML Vision, an AutoML platform based on NAS was used to create a Filipino food image recognition model as an added feature of the developed system. A Filipino food image dataset comprises 32 food classes and 3,295 images were created for this process. The developed food image recognition model scored a modest 76% on the precision metric in correctly identifying foods, which performed at par with other custom food recognition models. Nutrient approximation based on the Philippine Food Composition Table database also proved to be useful in displaying nutrition facts of recognized foods. This study showed the potential of a digital dietary survey system, combined with a unique and specific food image recognition tool, is a very encouraging proposition. Expansion of the food image dataset, exploration of other ML techniques, automated food weighta approximation, and continuous enhancement of the developed system are advocated.

Recent developments in the field of preventive healthcare have received considerable attention due to the effective management of various chronic diseases including diabetes, heart stroke, obesity, and cancer. Various automated systems are being used for activity and food recognition in preventive healthcare. The automated systems lack sophisticated segmentation techniques and contain multiple sensors, which are inconvenient to be worn in real-life settings. To monitor activity and food together, our work presents a novel wearable system that employs the motion sensors in a smartwatch together with a piezoelectric sensor embedded in a necklace. The motion sensor generates distinct patterns for eight different physical activities including eating activity. The piezoelectric sensor generates different signal patterns for six different food types as the ingestion of each food is different from the others owing to their different characteristics: hardness, crunchiness, and tackiness. For effective representation of the signal patterns of the activities and foods, we employ dynamic segmentation. A novel algorithm called event similarity search (ESS) is developed to choose a segment with dynamic length, which represents signal patterns with different complexities equally well. Amplitude-based features and spectrogram-generated images from the segments of activity and food are fed to convolutional neural network (CNN)-based activity and food recognition networks, respectively. Extensive experimentation showed that the proposed system performs better than the state of the art methods for recognizing eight activity types and six food categories with an accuracy of 94.3% and 91.9% using support vector machine (SVM) and CNN, respectively.

Monitoring and estimation of food intake is of great significance to health-related research, such as obesity management. Traditional dietary records are performed in manual way. These methods are of low efficiency and a waste of labor, which are highly dependent on human interaction. In recent years, some researches have made progress in the estimation of food intake by using the computer vision technology. However, the recognition results of these researches are usually for the whole food object in the image, and the accuracy is not high. In terms of this problem, we provide a method to the food smart recognition and automatic dietary assessment on the mobile device. First, the food image is processed by MASK R-CNN which is more efficient than traditional methods. And more accurate recognition, classification and segmentation results of the multiple food items are output. Second, the OpenCV is used to display the food category and the corresponding food information of unit volume on the recognition page. Finally, in order to facilitate daily use, TensorFlow Lite is used to process the model to transplant to the mobile device, which can help to monitor people’s dietary intake.

Recently, food recognition has received more research attention for mHealth applications that use automated visual-based methods to assess dietary intake. The goal is to improve the food diaries by addressing the challenges faced by existing methodologies. In addition to the classical challenge of the absence of rigid food structure and intra-class variations, food diaries employing deep networks trained with pristine images are susceptible to quality variations in real-world conditions of image acquisition and transmission. Similarly, existing progressive classifiers that use visual features via a convolutional neural network (CNN) classify food categories and cannot detect food ingredients. We aim to provide a system that selects the optimal subset of features from quality resilient CNNs and subsequently incorporates the parallel type of classification to tackle such challenges. The first progressive classifier recognizes food categories, and its multilabel extension detects food ingredients. Following this idea, after extracting features from the quality resilient category and ingredient CNN models by fine-tuning it on synthetic images generated using the novel online data augmentation method random iterative mixup. Our feature selection strategy uses the Shapley additive explanation (SHAP) values from the gradient explainer to select the best features. Then, novel progressive kernel extreme learning machine (PKELM) is exploited to cater to domain variations due to quality distortions, intra-class variations, and so forth, by remodeling the network structure based on activity value with the nodes. PKELM extension for multilabel classification detects ingredients by employing a bipolar step function to process test output and then selecting the column labels of the resulting matrix with a value of one. Moreover, during online learning, the PKELM novelty detection mechanism can label unlabeled instances and detect noisy samples. Experimental results showed superior performance on an integrated set of measures for seven publicly available food datasets.

In our paper we tried to classify food images using convolutional neural network. Convolutional neural network extracts spatial features from images so it is very efficient to use convolutional neural network for image clasification problem. Recently people are sharing food images in social media and writing review on food. So there is a lot of food image in the social media but some image may not be labeled. It will be very helpful for restaurants if they can advertise their food to those people who is looking similar kind of foods they offer. Food classification system can help social media platform to identify food. Food classification system can enable an opportunity for social media platform to offer advertisement service for restaurants and beverage companies to their targeted users. It will be financially beneficial for both social media platform and beverage companies. Food classification is very difficult task because there is high variance in same category of food images. We developed a convolutional neural network model to classify food images in food-11 dataset. We also used a pre-trained Inception V3 convolutional neural network model to classify food images.

The analysis of dietary pattern is important in health-care to reduce the risk factors of getting diet-related chronic diseases. An automatic dietary pattern assessment via food recognition algorithm provide an alternative way to improve the overall quality of dietary pattern assessment. However, due to very high variability of food images, the fusion of multiple type of features have become inevitable. The Bag of Features (BoF) model to encode the features are originally designed to consider a single type of features only. Therefore, this paper investigates the methods to fuse multiple features extracted from food images. Specifically, three fusion methods have been evaluated namely descriptor-level fusion, representation-level fusion and score-level fusion. The fusion is performed on the feature detector-level between Different of Hessian (DoH) and MSER detector. Speeded-up Robust Feature (SURF) is employed for feature description. The features are encoded by using k-means clustering and Support Vector Machine with linear kernel has been employed for classification. The evaluation has been performed on UECIOO-Food dataset.

Traditional machine learning algorithms using hand-crafted feature extraction techniques (such as local binary pattern) have limited accuracy because of high variation in images of the same class (or intraclass variation) for food recognition tasks. In recent works, convolutional neural networks (CNNs) have been applied to this task with better results than all previously reported methods. However, they perform best when trained with large amount of annotated (labeled) food images. This is problematic when obtained in large volume, because they are expensive, laborious, and impractical. This article aims at developing an efficient deep CNN learning-based method for food recognition alleviating these limitations by using partially labeled training data on generative adversarial networks (GANs). We make new enhancements to the unsupervised training architecture introduced by Goodfellow et al., which was originally aimed at generating new data by sampling a dataset. In this article, we make modifications to deep convolutional GANs to make them robust and efficient for classifying food images. Experimental results on benchmarking datasets show the superiority of our proposed method, as compared to the current state-of-the-art methodologies, even when trained with partially labeled training data.

With the development of the Internet, more and more people like to share their daily food, which makes the Internet rich in food image data. Many researchers collect food images on the Internet, label them, and combine machine learning and other methods to achieve food recognition systems. These food recognition systems can quickly help users record their daily diet and analyze their eating habits. The advanced food identification system can currently accurately identify hundreds of foods. At the same time, it can explore the food nutrition content and give food reference suggestions by accessing the food nutrition database. In this paper, we collected and evaluated large-scale food image data sets and food recognition systems.

Food recognition is an important pattern recognition problem. It involves identification various food types for certain applications such as food industry, calories counting or restaurant automation. Feature extraction is an essential stage in food recognition. It involves extracting various mathematical or handcrafted features for classification. The numerous types of extracted features create an issue of redundancy and over fitting. This article proposes feature weighting based on the concept of entropy for food recognition. The feature weighting indicates to selecting the top 25% of features after sorting them according to their importance. We evaluate Local Binary Pattern, Gabor features, Histogram of Gradient for features weighting based on entropy. The evaluation is done based on two types of neural networks, namely, extreme learning machine ELM and fast learning machine fast learning network for whole features extraction and for only 25% selection of most important features. The model has shown a superiority of ELM and fast learning network when top 25% of features were selected for both ELM and fast learning network with more superiority of fast learning network.