Design and Implementation of a Data Preprocessing Automatic Assessment Module in Jupyter Notebook

Programmatic processing, analysis, and visualization of scientific research data necessitate computational skills that scientists do not consistently acquire during their education. Python is a programming language currently in the ascendant among the scientific community. In early 2022, 25 marine scientists and marine biology graduate students were introduced to the fundamentals of Python programming in a 10.5-hour workshop series offered over seven weeks. A functional, web-based Python environment using Jupyter Notebooks, JupyterLab, Binder, and GitHub was paired with scaffolding and active learning pedagogy, to ensure an engaging learning experience. We describe the course design, adaptations, and outcomes, and offer recommendations for developing a similar introductory programming workshop.

This repository contains data analysis for a series of DFT calculations of metal intercalation into (F/OH)-substituted anatase TiO2. The repository consists of - A series of .yaml files, containing data extracted from VASP calculations. - Jupyter notebooks containing code for data analysis and figure plotting: - Comparison of the relative energies for groups of F_O or OH_O defects as a function of their positions relative to the V_Ti defect. - Calculation of intercalation energies for X=(Li, Na, Mg, Ca, Al) into (F/OH)-substituted anatase TiO2.

Application of Google earth engine python API and NAIP imagery for land use and land cover classification: A case study in Florida, USA

JupyterLab is a web-based interactive development platform that is widely used in the Earth science community. Using Jupyter Notebooks, it is possible to perform data analysis tasks, annotate and visualize results in a way that is easy to reproduce, present and share with others. JupyterLab allows the use of “extensions”, which add functionality to the platform. One of these is Jupyter-AI [1], which allows the use of Large Language Models (LLMs), such as ChatGPT, Claude Sonnet and Ollama, within the JupyterLab environment, through  a chat interface or directly within notebooks. By integrating LLMs into JupyterLab, it is possible to leverage their code generation capabilities to assist a user to translate their analysis tasks from an idea to actual executable code in an efficient manner. One drawback of using these LLMs in tasks involving spatio–temporal data is that the models typically do not have access to the data necessary for the analysis task and will often resort to generating fictional data or using placeholders in the code that they create. This requires the user to adapt the provided code to their data, which removes some of the utility provided by the LLM.In this context we make use of FrevaGPT, an approach for using LLMs in climate data analysis that allows for quick, complex and reproducible analyses of data sets, such as decadal climate model forecasts. Leveraging LLM’s capability to write code and using few-shot prompting (in-context learning) allows the LLM to utilize Freva [2,3] (Free Evaluation Framework), a data search and analysis platform, which provides a standardised interface to spatio-temporal datasets hosted on an HPC cluster [4].  FrevaGPT integrates seamlessly into Jupyter-AI and, by making use of the Freva library, combines the code-generating capabilities of LLMs with contextual understanding of how to access relevant datasets on the HPC cluster. This in addition with FrevaGPT’s ability to execute generated code in an isolated environment on an HPC node, annotating and explaining any intermediate results, as well as automatically correcting errors encountered along the way, could serve as a starting ramp for researchers to efficiently produce new analysis products based on spatio-temporal climate data. This PICO will include examples of using FrevaGPT within JupyterLab to analyse spatio-temporal datasets from the climate of the past, as well seasonal to decadal climate predictions.  

Spreadsheet is one of the tools that can be used to learn data analysis. Data analysis in spreadsheet can be done using formula. Spreadsheet tools can also be used for exams. For the assessment, there is a problem when the number of answers that need to be checked is large, that is it takes a long time to check all the answers. For this reason, an automatic grading system (autograder) that can evaluate formula in spreadsheet is needed. The method used in developing the autograder system is matching the answer key formula with the student's answer formula. The autograder system assesses the answer by calculating the similarity of the student's answer formula with the answer key formula. This paper explains how to build an autograder system that can evaluate the formula. At the end, an autograder system has been built successfully. It has been tested with 43 testcases and all of them are passed.

Iron-Folic Acid (IFA) is a crucial process indicator, that helps to halt the burden of iron deficiency anemia worldwide, especially in low and middle income countries like Ethiopia. Iron-folic acid supplementation is a crucial method in Ethiopia to reduce maternal deaths from anemia and poor birth outcomes. Despite these initiatives, there is still extremely little adherence to IFA consumption for the 90-day or more WHO-recommended timeframe. Therefore, the aim of this study was to use the DHS 2016 information to create a predictive model for IFA consumption and figure out the key factors influencing it among Ethiopian pregnant women. A secondary data analysis of Ethiopian DHS from 2016 was performed. A total weight of sample of 3109 women (15–49) was included in this study. Data were extracted using STATA version 17 software, and imported to a Jupyter notebook for further analysis. Machin learning algorithm using different classification models were implemented. All analysis and calculation were performed using Python 3 programming language in Jupyter Notebook using imblearn, sklearn, xgboost, and shap packages. Random Forest demonstrated the best performance with accuracy of 92.93%, precision of 91.32%, recall of 94.77%, F1-score of 93.99% and AUC 94%. Region, distance to health facility, age at first marriage, household head, number of ANC visit, media exposure, community literacy, husband occupation, gravidity, family size were top predicting factors of IFA utilization among women in Ethiopia. Random Forest were best predictive models with improved performance. Highly predicting factors of IFA utilization were identified using machine learning algorithms particularly random forest. These findings can guide policymakers and program managers to target interventions more effectively, promote ANC attendance, enhance community literacy and address household-level determinants to improve IFA supplementation rates. However, as the analysis is based on cross-sectional survey data, causal inferences cannot be made.

Perform big data analysis and visualisation on your own computer? Yes, you can! Commodity computers are now very powerful in comparison to only a few years ago. On top of that, the performance of today's software and data development techniques facilitates complex computation with fewer resources. Cloud computing is not always the solution, and reliability or even privacy is regularly a concern. While the Infrastructure as a Service (IaaS) and Software as a Service (SaaS) philosophies are a key part of current scientific endeavours, there is a misleading feeling that we need to have remote computers to do any kind of data analysis. One of the aims of the ATLAS Open Data project is to provide resources — data, software and documents — that can be stored and executed in computers with minimal or non-internet access, and in as many different operating systems as possible. This approach is viewed as complementary to the IaaS/SaaS approach, where local university, students and trainers' resources can be used in an effective and reproducible way — making the High Energy Physics and Computer Sciences fields accessible to more people. We present the latest developments in the production and use of local Virtual Machines and Docker Containers for the development of physics data analysis. We also discuss example software and Jupyter notebooks, which are in constant development for use in classrooms, and students and teachers computers around the world. Also, we showcase how those tools are developed and in use by international efforts (IPPOG, ICTP-PWF and CEVALE2VE.org) in Latin America and Europe to bring knowledge in countries like Mexico, Venezuela, Colombia, Peru, Ecuador, Argentina and Uruguay with support of the ICTP, CERN and many universities in the region.

BioUML (https://www.biouml.org)—is a web-based integrated platform for systems biology and data analysis. It supports visual modelling and construction of hierarchical biological models that allow us to construct the most complex modular models of blood pressure regulation, skeletal muscle metabolism, COVID-19 epidemiology. BioUML has been integrated with git repositories where users can store their models and other data. We have also expanded the capabilities of BioUML for data analysis and visualization of biomedical data: (i) any programs and Jupyter kernels can be plugged into the BioUML platform using Docker technology; (ii) BioUML is integrated with the Galaxy and Galaxy Tool Shed; (iii) BioUML provides two-way integration with R and Python (Jupyter notebooks): scripts can be executed on the BioUML web pages, and BioUML functions can be called from scripts; (iv) using plug-in architecture, specialized viewers and editors can be added. For example, powerful genome browsers as well as viewers for molecular 3D structure are integrated in this way; (v) BioUML supports data analyses using workflows (own format, Galaxy, CWL, BPMN, nextFlow). Using these capabilities, we have initiated a new branch of the BioUML development—u-science—a universal scientific platform that can be configured for specific research requirements.

Digital transformation in the field of education demands continuous innovation in learning processes, particularly in informatics courses that emphasize the mastery of computational thinking and programming skills. One widely developed approach is gamification, which refers to the integration of game elements into learning contexts to enhance students’ motivation and engagement. This study aims to analyze trends, dominant elements, platforms used, impacts, and future development directions of gamification-based informatics learning systems through a Systematic Literature Review (SLR) method. The research methodology follows the PRISMA guidelines, encompassing stages of literature identification, title and abstract screening, full-text eligibility assessment, as well as data analysis and synthesis. A total of 24 articles were identified in the initial search and subsequently selected based on predefined inclusion and exclusion criteria, resulting in 10 articles for in-depth analysis. The PICOS framework was employed to clarify the scope of the review, including informatics learners as the population, gamification as the intervention, conventional learning as the comparison, and outcomes in terms of motivation, engagement, and learning achievement. The findings indicate a significant increase in the implementation of gamification in informatics education over the past five years. The most dominant gamification elements identified are points, badges, levels, and leaderboards, with web-based e-learning platforms serving as the primary medium for implementation. Overall, gamification has consistently demonstrated positive effects on students’ motivation, engagement, and learning outcomes. This study concludes that gamification holds strong potential as an effective and sustainable learning strategy to support digital transformation in informatics education.

Jupyter notebooks are widely used for data analysis across a large number of scientific disciplines. As a result of the COVID-19 pandemic, I developed a series of computational exercises using the Jupyter notebook to replace the laboratory exercises usually undertaken in my course. My students had no prior coding knowledge and therefore these exercises were structured in a "cookbook" format using the susceptible-infected-resistant model for disease, data from the Lenski long-term evolutionary experiment, and a fission yeast transcriptomic data set. Despite limited internet connectivity and on-line instruction, my students completed these computational exercises and then tested their own hypotheses. Because Jupyter notebooks can be annotated with text and images, student notebooks were submitted for assessment in the form of a structured scientific report. An advantage of this approach was that all the computational analyses presented in these reports could be easily replicated. The notebook and complete instructions used in my course are provided for others who want to adopt this approach.

<p>Snow cover is an essential climate variable directly affecting the Earth’s energy balance, therefore estimating the snow parameters play an important role in hydrological, land surface, meteorological and climate models. Remote sensing provides a good understanding of snow cover monitoring thus several satellite snow products have been developed and disseminated so far.  In this study, Jupyter Notebook as an open source interactive satellite snow products retrieval, visualization and analysis tool has been developed by using Python language. Jupyter Notebook allows easy and straightforward data analysis with the possibility of live interaction and requires little programming knowledge.</p><p>The developed tool provides the capabilities of downloading the satellite snow products, georeferencing them and performing spatial analysis like zonal statistics. In this study EUMETSAT HSAF snow products, namely H10 (Snow detection), H13 (Snow Water Equivalent) and H34 (Snow cover) are used. The tool allows user to upload their own region in ESRI shapefile format for spatial and temporal analysis and the uploaded region can be visualized on interactive map via custom interactive widget like ipyleaflet. The cloud percentage for the snow cover product can be selected and daily snow covered area or snow water equivalent change for the uploaded region can be calculated for the selected period. With this tool, it is aimed to retrieve the satellite snow products easily and perform spatial and temporal analysis of snow cover for the area of interest without getting lost in data formats. Therefore, users with little or no knowledge about programming can interact easily with EUMETSAT HSAF snow products. Furthermore, with the high extensibility of Jupyter Notebook, it can also be improved or modified in accordance with the need of the end users.</p>

This accessible and self-contained guide provides a comprehensive introduction to the popular programming language Python, with a focus on applications in chemistry and chemical physics. Ideally suited to students and researchers of chemistry learning to employ Python for problem-solving in their research, this fast-paced primer first builds a solid foundation in the programming language before progressing to advanced concepts and applications in chemistry. The required syntax and data structures are established, and then applied to solve problems computationally. Popular numerical packages are described in detail, including NumPy, SciPy, Matplotlib, SymPy, and pandas. End of chapter problems are included throughout, with worked solutions available within the book. Additional resources, datasets, and Jupyter Notebooks are provided on a companion website, allowing readers to reinforce their understanding and gain confidence applying their knowledge through a hands-on approach.

Java programming language is one of the most popular programming languages among IT developers. There are common problems in the beginning learning process such as the installation and configuration process of java software like JDK (Java Development Kit). During the online class session, it takes a lot of time and a large internet data packet to solve this problem. This research proposes a website-based system called livE (online - java Exercise). Students can practice coding and also do exams in real-time, simply by using a web browser and the internet. If lecturers want to make corrections, lecturers do not need to download student answer files. They can directly run the program code from student answers in the livE system. This indicates that the system implementation provides process efficiency in the student's job sheet exercises and lecturer's corrections. The result from the test that has been carried out shows that livE is running as expected. Usability testing shows that users are "totally agree" with system quality.

Urban has developed an elastic and powerful approach to the analysis of massive datasets using Amazon Web Services’ Elastic MapReduce (EMR) and the Spark framework for distributed memory and processing. The goal of the project is to deliver powerful and elastic Spark clusters to researchers and data analysts with as little setup time and effort possible, and at low cost. To do that, at the Urban Institute, we use two critical components: (1) an Amazon Web Services (AWS) CloudFormation script to launch AWS Elastic MapReduce (EMR) clusters (2) a bootstrap script that runs on the Master node of the new cluster to install statistical programs and development environments (RStudio and Jupyter Notebooks). The Urban Institute’s Spark for Social Science Github page holds code used to setup the cluster and tutorials for learning how to program in R and Python.

Metabolomic analysis of equine urine to determine route of altrenogest administration