Software releases

• Legend labels in scatterplot
• Enable detection of date in both new and old WIP file format
• Intensity calibration axis conversion in case of intensity reference provided in nm

• Plot PCA-SVM coefficients when linear kernel is used

• Fix export of cluster maps
• Fix use of clusters as included labels for clustering
• Fix automated detection of maps for Horiba and Renishaw

• Legend color inconsistency between cluster tree and mean classes plot
• Fixed appearance of controls for HM-HCA in model step
• In regression results table plot all rows when there are more responses than concentrations
• Correct minor typos in helper text

• Plot maps of class predictions similarly to cluster predictions (when spectral mapping information is available)
• Export results without calibrated and preprocessed spectra (useful for large data sets, available for high expertize)
• Add various aggregation options besides averaging (backend only, available via parameters file): 'mean', 'median', 'sd', MAD ('medad'), MAD from min ('minad')
• Import data directly from clouds (available in pyRamanmetrix and Backend APIs

• Improved model interpretability via named legend in coefficients plots of PLS2, PCA-LDA, PCA-LM, etc.
• Import of multiple ZIP files together
• Handle dates in German in WiTec files

• Wavenumber axis detection failed when the spectral resolution does not increase monotonously; fixed by setting wn_check_tol data loader parameter

• Calculate derivative of spectra during preprocessing (at calibration step)
• SNV and Min-Max normalization options
• The results table and mean spectral plots are made more compact for regression tasks; this also prevents frontend from hanging; can be controlled via model_n_classes_max parameter

• In regression test NA values are handled in the results table
• Importing data directly from HTTPS link is implemented in pyRamanmetrix and for backend APIs
• Added import_txt2parameters function to pyRamanmetrix
• Added include_spectra parameter in export_results2zip of pyRamanmetrix

• Ignore excluded data when calculating average for difference plot in model step
• Enable zooming-in shortcut Ctrl++ for German keyboard

• Get dates automatically from new WIP file format

• Ignore excluded data in the despiked plot
• Customized slider intervals for calibration sampling step (e.g. resolution) and SNIP iterations

• Fix reference update at calibration step
• Fix loading of Renishaw files with multiple spectra

• Compatibility with Deflate64 compression (used by Windows for large ZIPs)
• Make it possible to load WIP files from new WiTec software

• SNIP algorithm modified to handle negative intensities (behaviour is unchanged when no negative values are present)

• Fix appearance of the last cluster in the exported cluster maps
• Fix reference update at calibration step

• kMeans clustering
• Despiking using neighbours

• Limit number of outliers displayed in quality check, so PDF reports can be generated faster
• Refactor SpData class as subclass of photonicdata.BaseSpData

• Fixed import of files with multiple spectra

• ω⁴ correction at normalization step

• Removed the scroll at loading logo

• Fixed import of SPC files with multiple spectra
• Improved large numbers apperence in the plots
• Fix appearence of peak positions at standard plot
• Fix calibration error when interferent spectra provided
• Set fixed random seed to make model training reproducable
• Improve logic for quality reference preprocessing

• Color pallets can be adjusted in parameters file
• Axis labels can be adjusted in parameters file
• Automatically detect dates in WIP files
• Automatically detect x_pos and y_pos in 2D maps withing WIP files
• Export cluster maps along with results when x and y positions and clusters are available
• Use cluster predictions (after running clustering) in the 'Included labels' filter at the model step
• Check all/none box in loadings plots

• Update API for model hierarchy to accept other conditions besides 'classes': 'notequal', 'less', 'lessequal', 'greater', 'greaterequal'
• Add 'Preprocessed reference' to quality check plots
• Use different color pallets for classes and clusters

• Fix error regarding missing cluster predictions when performing regression
• Catch errors on import

• Add 'data_loaders' to parameters file

• Model persistence issue on test data import
• Loading files with different formats for the same date/device
• Error display during the calibration step
• Error handling for metadata import
• Ability to run long calculations (>12 hours) in the online version
• Open links externally

• Use clusters (after running clustering) as class labels via 'Customized from metadata' class label 'cluster_predictions'
• Combination of EMSC and SNIP for baseline correction
• Provide support for WIP files with the wavelength axis saved as a polynomial
• Import of test metadata separately from test data
• Display cross-validation progress at the model step

• Add 'model_convert' API for converting v05 models into older v04 model versions
• Introduce 'session_download' and 'session_upload' APIs for exporting/importing sessions at the backend
• Customize 'hierarchy_predict_json' API with a filter, e.g., 'hierarchy_predict_json/predictions'
• Add 'hierarchy_predict_json_predictions' API
• Update 'predict_json' API to return top matches
• Add the option to enable ω⁴ correction in the parameters file
• Limit the number of rows displayed in regression results

• Appearance of test results for a single class in test data
• Fixed appearance of major vote confusion table and parameters in test report when the model is loaded in empty session

• Possibility to refer to all data by setting . or * value in in the path (or file) column of the metadata table
• Possibility to keep the calibration data after calibration in pyRamanmetrix by setting calibrate(..., drop_calib_data=False)
• Import of CSV or TXT files without wavenumber axis (see HT Raman, Data Structure Types, and Metadata Table for details)
• Added automatic date parsing from the filenames in YYYY_MM_DD format

• Improved date parsing from the filenames in timestamp format (YYMMDD_hhmmss)
• Customized error messages when inner cross-validation (CV) fails
• Use N-fold CV with N≤10 instead of 10-fold CV if minimal number of observations per class is less than 10
• Security, performance, and compatibility updates to GUI and installer

• Correct plotting of the raw mean spectra with different wavenumber ranges
• Fixed a bug in report printing with unloaded charts
• Fixed broken chart functionality after exporting chart to PNG
• Corrected traces behavior on saving PNG
• Fixed an issue with chart traces standard deviation

• Support of SP and FSM files
• Support of TXT/CSV files with coma or semicolon separator
• Import/export of models via pyRamanmetrix
• Possibility to include/exclude spectra at the import step by indicating logical values in the include column of metadata
• Add PCA option for EMSC signal interference

• Refactor the software, reducing the size of the setup file and the required disk space for installation by a factor of three
• Best number of features is decided at the saturation point of the fitted curve (not by maximal value)
• Updated CNN model for feature extraction to a more sophisticated but faster version
• Changed the format of PKL files for model import/export
• Website for a license and software downloads has a new domain name - desktop.ramanmetrix.eu

• Metadata parsing with different data types in the same column
• Correct multiple response selection in test

• New buttons in plots for mean classes. The button that makes it possible to check/uncheck all traces and the button that checks/unchecks the standard (in the raw data plot) or excluded (in the model step) data.

• In the dark mode, setting black background is now possible (useful for making screenshots for the black PowerPoint slides)

• Add warnings to calibration step

• Users can add an "include" column to metadata table to mark the data that has to be included (TRUE) or completely excluded (FALSE) from the whole data processing routine

• A minimal HTML report with plots is exported along with the ZIP file

• Speed-up SVM model training

• Smoothing is added at the calibration step

• More loading plots are available now for LDA and PLS-based models

• Added R2 and MPE regression metrics

• Added a table with metrics for all responses for multiple regression

• Added predict_dict and export functions to pyRamanmetrix

• Load files with ".0", ".1", ".2" extensions the same way as TXT

• Introduce interferent signals to EMSC model based preprocessing by labeling those spectra in interferent_sample column of metadata table and specifying their type in type column

• Inverted X-axis option for all plots

• Slider manual input option for all sliders

• Hard modeling (HM) for feature extraction by fitting selected peaks

• Interface for setting fitted peaks for hard modeling

• Analysis of subtracted spectra (e.g. subtract control sample from treated)

• Step Data, metadata refresh after a new dataset upload
• Quality filters title alignment

• Plot standard data ('standard', 'standard_intensity', 'reference_sample', 'interferent_sample', 'dark_bg') in 'Mean spectra (raw data)' plot
• Despiking step, new interval for Despiking threshold slider
• Quality plots, tooltips added

• Import functions into python 3.10.4 by importing server/pyRamanmetrix.py file, e.g. import pyRamanmetrix as rm (see Tools / Python module pyRamanmetrix)

• Multi-response PLS regression (PLS2 and PLSW2A) models

• Support of WITec (*.wip) files with Raman spectra (see Data Input / Data structure Types)

• Difference spectra plot at model step

• Export -> Default parameters in high-level expertise mode

• Appearance of a blank screen at the first start

• Support \ in file path in metadata table

• Fix PLS2-DA decision-making for values outside of [0; 1] range

• Avoid an error if a date cannot be recognized

• Allow unsorted standard peaks in parameters.txt

• Preserve zoom on saving figures as PNG

• Inclusion of wavenumber range boundary values

• added PLS2-DA (classification) and PLS1 (regression) models

• changed design of the license window

• updated labels order in scatterplot legend

• avoid erasing imported raw data when the import of metadata fails

• fixed breaking of the layout on window resize

• improved, fixed bugs and changed view for "included labels" feature in the model construction step

• fixed errors for the sliders on the normalization and quality filters steps, that occurred when the selected calibration degree parameter is a decimal number

• corrected typos in the tooltip blocks

• added possibility to use reference spectra for EMSC and quality check (should be marked using 'reference_sample' column in metadata) instead of the mean over the dataset (see reference_sample in the Metadata Table)

• added SD to means.csv in results

• added major vote predictions (if applicable) to results

• fixed bugs related to float values used as labels

• fixed problem with endless animation on downloads

• optimized default parameters

• added link in the menu to the documentation website

• displayed errors on upload/download data

• updated About page: design and list of packages

• general improvements on stability, updated dependencies

• improvements of data integrity, added md5 sum for models

• support of JDX and Renishaw TXT files, support of Horiba XY and TXY maps (see Supported Devices)

• display warnings when failed to import files

• bugs fixed for test data report and voting after a quality check

• fixed appearance of the scatter plot in the test data prediction

• added plot with calibration functions to the test report

• improved scatterplot: one can choose a type of the scatterplot diagonal plot (line, histogram, distribution)

• added possibility to stop calculations at any step

• aggregation of prediction results by voting

• import metadata tables separately

• updated component for displaying backend errors

• display notification about the new version

• added voting results to the reports

• dark mode

• possibility to link license by HDD instead of MAC-address (no conflict with VPN)

• usage of all available labels to exclude data from model

• some unexpected installation problems solved

• improved quality of the plots saved in PNG format

New name and icon, added an average spectra to the model step, updated user interface, improved plots (see Interactive plots), and fixed bugs.

• Added 2-level cross-validation

• Implemented automated optimization for a number of features

• Add 'shift' option on the Standard spectrum plot.

Other updates

• Improved despiking algorithm