Cross-domain Cross-modal Food Transfer

Obesity is becoming a widely concerned health problem of most part of the world. Computer vision based recognition system has great potential to be an efficient tool to monitor food intake and cope with the growing problem of obesity. This paper proposes a food recognition algorithm based on sparse representation. The proposed algorithm learns overcomplete dictionaries from local descriptors including texture and color features that are extracted from food image patches. With the learned two overcomplete dictionaries, a feature vector of the food image can be generated with the sparsely encoded local descriptors. SVM is used for the classification. This research creates a Chinese food image dataset for experiments. Classifying Chinese food is more challenging because they are not as distinguishable visually as western food. The proposed algorithm achieves an average classification accuracy of 97.91% in a dataset of 5309 images that comprises 18 classes. The proposed method can be easily employed to dataset with more classes. Our results demonstrate the feasibility of using the proposed algorithm for food recognition.

The visual analysis of foods on social media by using food recognition algorithm provides valuable insight from the health, cultural and marketing. Food recognition offers a means to automatically recognise foods as well the useful information such as calories and nutritional estimation by using image processing and machine learning technique. The interest points in food image can be detected effectively by using Maximally Stable Extremal Region (MSER). As MSER used global segmentation and many food images have a complex background, there are numerous irrelevant interest points are detected. These interest points are considered as noises that lead to computation burden in the overall recognition process. Therefore, this research proposes an Extremal Region Selection (ERS) algorithm to improve MSER detection by reducing the number of irrelevant extremal regions by using unsupervised learning based on the k-means algorithm. The performance of ERS algorithm is evaluated based on the classification performance metrics by using classification rate (CR), error rate (ERT), precision (Prec.) and recall (rec.) as well as the number of extremal regions produced by ERS. UECFOOD-100 and UNICT-FD1200 are the two food datasets used to benchmark the proposed algorithm. The results of this research have found that the ERS algorithm by using optimum parameters and thresholds, be able to reduce the number of extremal regions with sustained classification performance.

In this paper, we propose a deep learning based system for food recognition from personal life archive im- ages. The system first identifies the eating moments based on multi-modal information, then tries to focus and enhance the food images available in these moments, and finally, exploits GoogleNet as the core of the learning process to recognise the food category of the images. Preliminary results, experimenting on the food recognition module of the proposed system, show that the proposed system achieves 95.97% classification accuracy on the food images taken from the personal life archive from several lifeloggers, which potentially can be extended and applied in broader scenarios and for different types of food categories.

The National Nutrition Survey (NNS) is the Philippines' official source of data on nutritional status, diet, and lifestyle-related risk factors for non-communicable diseases. A food consumption survey is one of its major components that collects individual dietary data through 24-h food recall interviews and household weighing of food intake using a dietetic scale. Conventional manual surveys are costly and tedious to conduct, which can also compound the inconsistencies and biases in the data gathered. Technological advances in mobile devices and networking have made it possible to gather data more economically and more efficiently with emerging technologies such as machine learning (ML) and increased use of automation open vast opportunities. In this paper, we explored the digitalization of food component forms of the NNS and developed a fully digital dietary survey collection system from collection to transmission of data. Google AutoML Vision, an AutoML platform based on NAS was used to create a Filipino food image recognition model as an added feature of the developed system. A Filipino food image dataset comprises 32 food classes and 3,295 images were created for this process. The developed food image recognition model scored a modest 76% on the precision metric in correctly identifying foods, which performed at par with other custom food recognition models. Nutrient approximation based on the Philippine Food Composition Table database also proved to be useful in displaying nutrition facts of recognized foods. This study showed the potential of a digital dietary survey system, combined with a unique and specific food image recognition tool, is a very encouraging proposition. Expansion of the food image dataset, exploration of other ML techniques, automated food weighta approximation, and continuous enhancement of the developed system are advocated.

The analysis of dietary pattern is important in health-care to reduce the risk factors of getting diet-related chronic diseases. An automatic dietary pattern assessment via food recognition algorithm provide an alternative way to improve the overall quality of dietary pattern assessment. However, due to very high variability of food images, the fusion of multiple type of features have become inevitable. The Bag of Features (BoF) model to encode the features are originally designed to consider a single type of features only. Therefore, this paper investigates the methods to fuse multiple features extracted from food images. Specifically, three fusion methods have been evaluated namely descriptor-level fusion, representation-level fusion and score-level fusion. The fusion is performed on the feature detector-level between Different of Hessian (DoH) and MSER detector. Speeded-up Robust Feature (SURF) is employed for feature description. The features are encoded by using k-means clustering and Support Vector Machine with linear kernel has been employed for classification. The evaluation has been performed on UECIOO-Food dataset.

Traditional machine learning algorithms using hand-crafted feature extraction techniques (such as local binary pattern) have limited accuracy because of high variation in images of the same class (or intraclass variation) for food recognition tasks. In recent works, convolutional neural networks (CNNs) have been applied to this task with better results than all previously reported methods. However, they perform best when trained with large amount of annotated (labeled) food images. This is problematic when obtained in large volume, because they are expensive, laborious, and impractical. This article aims at developing an efficient deep CNN learning-based method for food recognition alleviating these limitations by using partially labeled training data on generative adversarial networks (GANs). We make new enhancements to the unsupervised training architecture introduced by Goodfellow et al., which was originally aimed at generating new data by sampling a dataset. In this article, we make modifications to deep convolutional GANs to make them robust and efficient for classifying food images. Experimental results on benchmarking datasets show the superiority of our proposed method, as compared to the current state-of-the-art methodologies, even when trained with partially labeled training data.

Monitoring and estimation of food intake is of great significance to health-related research, such as obesity management. Traditional dietary records are performed in manual way. These methods are of low efficiency and a waste of labor, which are highly dependent on human interaction. In recent years, some researches have made progress in the estimation of food intake by using the computer vision technology. However, the recognition results of these researches are usually for the whole food object in the image, and the accuracy is not high. In terms of this problem, we provide a method to the food smart recognition and automatic dietary assessment on the mobile device. First, the food image is processed by MASK R-CNN which is more efficient than traditional methods. And more accurate recognition, classification and segmentation results of the multiple food items are output. Second, the OpenCV is used to display the food category and the corresponding food information of unit volume on the recognition page. Finally, in order to facilitate daily use, TensorFlow Lite is used to process the model to transplant to the mobile device, which can help to monitor people’s dietary intake.

In recent years, people are facing with billions of food images, videos and recipes on social medias. An appropriate technology is highly desired to retrieve accurate contents across food images and cooking recipes, like cross-modal retrieval framework. Based on our observations, the order of sequential sentences in recipes and the noises in food images will affect retrieval results. We take into account the sentence-level sequential orders of instructions and ingredients in recipes, and noise portion in food images to propose a new framework for cross-retrieval. In our framework, we propose three new strategies to improve the retrieval accuracy. (1) We encode recipe titles, ingredients, instructions in sentence level, and adopt three attention networks on multi-layer hidden state features separately to capture more semantic information. (2) We apply attention mechanism to select effective features from food images incorporating with recipe embeddings, and adopt an adversarial learning strategy to enhance modality alignment. (3) We design a new triplet loss scheme with an effective sampling strategy to reduce the noise impact on retrieval results. The experimental results show that our framework clearly outperforms the state-of-art methods in terms of median rank and recall rate at top k on the Recipe 1M dataset.

The "Recipe Generation from Food Images" project introduces an innovative approach to culinary technology by leveraging deep learning techniques to automatically analyze food images and generate comprehensive cooking recipes. The system's key components include crafting catchy titles, compiling detailed ingredient lists, and outlining step-by-step cooking instructions. Targeting individuals intrigued by enticing food imagery but lacking the expertise or inspiration to recreate dishes, the project offers a user-friendly solution to bridge this gap. Ultimately, it aims to contribute to culinary technology by providing a valuable tool for cooking enthusiasts and researchers, catering to the evolving needs of contemporary culinary experiences. Through its utilization of advanced deep learning methodologies, the project seeks to democratize access to culinary knowledge and inspire creativity in the realm of home cooking.

Accurate dietary assessment is essential for enhancing health and managing nutrition-related issues. This paper outlines the creation of a Real-Time Food Recognition and Calorie Estimation System that employs deep learning methodologies, particularly Convolutional Neural Networks (CNN) and the MobileNetV2 framework. Users can upload images of food via a web-based platform, where the model classifies the food and estimates its caloric and macronutrient content. Utilizing transfer learning and data augmentation techniques, the model achieves an 85% accuracy rate in classifying 10 different food categories. With a lightweight and scalable design, the system guarantees real-time predictions and is compatible with various devices. Additionally, features such as dynamic feedback and historical tracking improve user interaction and system flexibility. By automating food recognition and nutritional assessment, this system offers a dependable, efficient, and user-friendly solution for personalized dietary monitoring.

Food recognition plays a much critical role in various health-care applications. However, it poses many challenges to current approaches due to the diverse appearances of food dishes and the non-uniform composition of ingredients for the foods in the same category. Current methods primarily focus on the appearance of foods without considering their semantic information, easily finding the wrong attention areas of food images. Second, these methods lack the dynamic weighting of multiple semantic features in the modeling process. Thus this paper proposes a novel Multi-View Attention Network within the multi-task learning framework that incorporates multiple semantic features into the food recognition task from both ingredient recognition and recipe modeling. It also utilizes the multi-view attention mechanism to automatically adjust the weights of different semantic features and enables different tasks to interact with each other so as to obtain a more comprehensive feature representation. The experiments conducted on both ChineseFoodNet and VIREO Food-172 benchmark databases validate the proposed method with the obvious improvement of the performance and the lower parameter size.

Food recognition technology remains a challenging task to the computer vision community due to the diverse nature of food. Oriental food, with similar features such as colour and texture, makes the recognition process less effective and challenging with a convolutional neural network (CNN). More importantly, there are no literature reports on the use of Local Interpretable Model-agnostic Explanations (LIME) to increase the transparency of oriental food recognition. Hence, this paper investigates oriental food recognition using two different CNN models and implements LIME to interpret the model. The testing accuracy obtained by the proposed CNN models for oriental food recognition with optimum hyper-parameter setting is about 85.7% coupled with the utilization of the LIME model to increase the transparency of the deep learning models.

Adversarial contrastive learning (ACL) is considered an effective way to improve the robustness of pre-trained models. In contrastive learning, a projector which consists of multilayer perceptron (MLP) will project high dimension 3D point cloud feature into low dimension for calculating contrastive loss during contrastive pretraining.We propose a novel method for generating high-quality 3D adversarial examples for adversarial training, which leverages the virtual adversarial loss with the feature representations prior to projection in a contrastive learning framework. To train the self-supervised contrastive learning framework adversarially, we introduce our robust aware loss function. Additionally, we show that incorporating high difference points using the Difference of Normal (DoN) operator as an additional input for adversarial self-supervised contrastive learning can significantly enhance the adversarial robustness of the pre-trained model. Our proposed method, POINTACL, is evaluated on several downstream tasks, including 3D classification and 3D segmentation using multiple datasets. Our experimental results demonstrate that POINTACL achieves state-of-the-art performance in terms of robust accuracy when compared to other contrastive adversarial learning methods.

Global eating habits cause health issues leading people to mindful eating. This has directed attention to applying deep learning to food-related data. The proposed work develops a new framework integrating neural network and natural language processing for classification of food images and automated recipe extraction. It address the challenges of intra-class variability and inter-class similarity in food images that have received shallow attention in the literature. Firstly, a customized lightweight deep convolution neural network model, MResNet-50 for classifying food images is proposed. Secondly, automated ingredient processing and recipe extraction is done using natural language processing algorithms: Word2Vec and Transformers in conjunction. Thirdly, a representational semi-structured domain ontology is built to store the relationship between cuisine, food item, and ingredients. The accuracy of the proposed framework on the Food-101 and UECFOOD256 datasets is increased by 2.4% and 7.5%, respectively, outperforming existing models in literature such as DeepFood, CNN-Food, Wiser, and other pre-trained neural networks.

Dense cloud radio access networks (cloud-RANs) provide a promising way to enable scalable connectivity and handle diversified service requirements for massive mobile devices. To fully exploit the performance gains of dense cloud-RANs, channel state information of both the signal link and interference links is required. However, with limited radio resources for training, the channel estimation problem in dense cloud-RANs becomes a high-dimensional estimation problem, i.e., the number of measurements will be typically smaller than the dimension of the channel. In this paper, we shall develop a generic high-dimensional structured channel estimation framework for dense cloud-RANs, which is based on a convex structured regularizing formulation. Observing that the wireless channel possesses ample exploitable statistical characteristics, we propose to convert the available spatial and temporal prior information into appropriate convex regularizers. Simulation results demonstrate that exploiting the spatial and temporal dynamics can achieve good estimation performance even with limited training resources. The alternating direction method of multipliers algorithm is further adopted to solve the resultant large-scale high-dimensional channel estimation problems. The proposed framework thus enjoys modeling flexibility, low training overhead, and computation cost scalability.