A map of current machine learning frameworks

Marie-Hélène Burle

October 3, 2023


frontlogo

Disclaimers


Keeping up with the ever evolving field of machine learning is daunting. This webinar aims to bring a little help in this navigation and maybe inspire you to explore new tools. It is however flawed in several ways:

  • it is not exhaustive and focuses on the most used frameworks accessible through 3 popular languages,
  • a lot of the content could be accused of being biased and subjective,
  • it will be outdated quickly.

Ultimately, the best frameworks are the ones that work best for you and your needs. The only piece of advice I would give is to always use open source tools.

Traditional machine learning\n(ML)\n→ computational statistics\n→ predictive analytics Traditional machine learning (ML) → computational statistics → predictive analytics Deep learning\n(DL)\n→ artificial neural networks Deep learning (DL) → artificial neural networks Fields Fields Fields--Traditional machine learning\n(ML)\n→ computational statistics\n→ predictive analytics Fields--Deep learning\n(DL)\n→ artificial neural networks

ml Traditional machine learning (ML) → computational statistics → predictive analytics 1 - Clustering - Naives Bayes - Ensemble methods - Model selection - Decision trees ml->1 dl Deep learning (DL)    → artificial neural networks 2 - Architecture building - NN operations - Automatic differentiation (AD) - Optimization functions dl->2 Fields Fields Fields->ml Fields->dl

Python Python Julia Julia R R Languages Languages Languages--Python Languages--Julia Languages--R

Most popular frameworks cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

scikit-learn is a toolbox of ML algorithms

TensorFlow was a leading DL framework for years, but is now fading

PyTorch slowly took over TensorFlow, but is now starting to face competition

Most popular frameworks cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

JAX is its own domain specific language (DSL) of restrictive NumPy-like “pure” functions with automatic differentiation (AD) and accelerated linear algebra (XLA)—a domain-specific compiler

keras is a high-level API for DL

Most popular frameworks cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

A fast evolving field:  - MXNET was just retired a few days ago
          - caffe2 is now part of PyTorch
          - JAX is looking like a replacement for TensorFlow
          - keras has been the high-level frontend for a shifting list of             backends (currently TensorFlow, PyTorch, JAX)

Python cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

Several popular packages written for Python, but not in Python (too slow)
→ written in Fortran, C, C++, or DSL

Julia cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

Classic Python packages available in Julia thanks to wrappers
In addition many packages written in Julia (as the language is fast enough)

Julia cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

Let’s have a look at the Julia specific packages.

A rich ecocystem of packages cluster_ml Traditional machine learning cluster_dl Deep learning AugmentedGaussianProcesses AugmentedGaussianProcesses EvoTrees EvoTrees MachineLearning MachineLearning TSML TSML DecisionTree DecisionTree MLJ MLJ ZigZagBoomerang ZigZagBoomerang LightGBM LightGBM Turing Turing BayesianOptimization BayesianOptimization GLMNet GLMNet LIBSVM LIBSVM MLBase MLBase Julia Julia MLBase--Julia MLKernels MLKernels Clustering Clustering ForneyLab ForneyLab Flux Flux Lux Lux Knet Knet Merlin Merlin MLJFlux MLJFlux FluxTraining FluxTraining GraphNeuralNetworks GraphNeuralNetworks Julia--MLJFlux

Not counting utility packages... cluster_ml Traditional machine learning cluster_dl Deep learning cluster_utilities Utilities: AugmentedGaussianProcesses AugmentedGaussianProcesses EvoTrees EvoTrees MachineLearning MachineLearning TSML TSML DecisionTree DecisionTree MLJ MLJ ZigZagBoomerang ZigZagBoomerang LightGBM LightGBM Turing Turing BayesianOptimization BayesianOptimization GLMNet GLMNet LIBSVM LIBSVM MLBase MLBase Julia Julia MLBase--Julia MLKernels MLKernels Clustering Clustering ForneyLab ForneyLab Flux Flux Lux Lux Knet Knet Merlin Merlin MLJFlux MLJFlux FluxTraining FluxTraining GraphNeuralNetworks GraphNeuralNetworks Metalhead Metalhead DataLoaders DataLoaders MLUtils MLUtils MLDataUtils MLDataUtils MLDatasets MLDatasets TensorBoardLogger TensorBoardLogger ObjectDetector ObjectDetector Julia--MLJFlux

And many AD implementations! cluster_ml Traditional machine learning cluster_dl Deep learning cluster_utilities Utilities: cluster_autodiff Autodiff: AugmentedGaussianProcesses AugmentedGaussianProcesses EvoTrees EvoTrees MachineLearning MachineLearning TSML TSML DecisionTree DecisionTree MLJ MLJ ZigZagBoomerang ZigZagBoomerang LightGBM LightGBM Turing Turing BayesianOptimization BayesianOptimization GLMNet GLMNet LIBSVM LIBSVM MLBase MLBase Julia Julia MLBase--Julia MLKernels MLKernels Clustering Clustering ForneyLab ForneyLab Flux Flux Lux Lux Knet Knet Merlin Merlin MLJFlux MLJFlux FluxTraining FluxTraining GraphNeuralNetworks GraphNeuralNetworks Metalhead Metalhead DataLoaders DataLoaders MLUtils MLUtils MLDataUtils MLDataUtils MLDatasets MLDatasets TensorBoardLogger TensorBoardLogger ObjectDetector ObjectDetector StochasticAD StochasticAD ForwardDiff ForwardDiff ReverseDiff ReverseDiff Zygote Zygote Enzyme Enzyme ChainRules ChainRules TaylorDiff TaylorDiff Julia--MLJFlux

Notable options cluster_ml Traditional machine learning cluster_dl Deep learning cluster_utilities Utilities: cluster_autodiff Autodiff: AugmentedGaussianProcesses AugmentedGaussianProcesses EvoTrees EvoTrees MachineLearning MachineLearning TSML TSML DecisionTree DecisionTree MLJ MLJ ZigZagBoomerang ZigZagBoomerang LightGBM LightGBM Turing Turing BayesianOptimization BayesianOptimization GLMNet GLMNet LIBSVM LIBSVM MLBase MLBase Julia Julia MLBase--Julia MLKernels MLKernels Clustering Clustering ForneyLab ForneyLab Flux Flux Lux Lux Knet Knet Merlin Merlin MLJFlux MLJFlux FluxTraining FluxTraining GraphNeuralNetworks GraphNeuralNetworks Metalhead Metalhead DataLoaders DataLoaders MLUtils MLUtils MLDataUtils MLDataUtils MLDatasets MLDatasets TensorBoardLogger TensorBoardLogger ObjectDetector ObjectDetector StochasticAD StochasticAD ForwardDiff ForwardDiff ReverseDiff ReverseDiff Zygote Zygote Enzyme Enzyme ChainRules ChainRules TaylorDiff TaylorDiff Julia--MLJFlux

Machine learning:

MLJ is a ML toolbox with Julia functions and wrappers to scikit-learn models

Notable options cluster_dl Deep learning cluster_utilities Utilities: cluster_autodiff Autodiff: cluster_ml Traditional machine learning AugmentedGaussianProcesses AugmentedGaussianProcesses EvoTrees EvoTrees MachineLearning MachineLearning TSML TSML DecisionTree DecisionTree MLJ MLJ ZigZagBoomerang ZigZagBoomerang LightGBM LightGBM Turing Turing BayesianOptimization BayesianOptimization GLMNet GLMNet LIBSVM LIBSVM MLBase MLBase Julia Julia MLBase--Julia MLKernels MLKernels Clustering Clustering ForneyLab ForneyLab Flux Flux Lux Lux Knet Knet Merlin Merlin MLJFlux MLJFlux FluxTraining FluxTraining GraphNeuralNetworks GraphNeuralNetworks Metalhead Metalhead DataLoaders DataLoaders MLUtils MLUtils MLDataUtils MLDataUtils MLDatasets MLDatasets TensorBoardLogger TensorBoardLogger ObjectDetector ObjectDetector StochasticAD StochasticAD ForwardDiff ForwardDiff ReverseDiff ReverseDiff Zygote Zygote Enzyme Enzyme ChainRules ChainRules TaylorDiff TaylorDiff Julia--MLJFlux

Neural networks:

Flux is a DL framework similar to PyTorch in pure Julia

Lux is a rewrite of Flux keeping the trainable and non-trainable parameters decoupled

Notable options cluster_ml Traditional machine learning cluster_dl Deep learning cluster_utilities Utilities: cluster_autodiff Autodiff: AugmentedGaussianProcesses AugmentedGaussianProcesses EvoTrees EvoTrees MachineLearning MachineLearning TSML TSML DecisionTree DecisionTree MLJ MLJ ZigZagBoomerang ZigZagBoomerang LightGBM LightGBM Turing Turing BayesianOptimization BayesianOptimization GLMNet GLMNet LIBSVM LIBSVM MLBase MLBase Julia Julia MLBase--Julia MLKernels MLKernels Clustering Clustering ForneyLab ForneyLab Flux Flux Lux Lux Knet Knet Merlin Merlin MLJFlux MLJFlux FluxTraining FluxTraining GraphNeuralNetworks GraphNeuralNetworks Metalhead Metalhead DataLoaders DataLoaders MLUtils MLUtils MLDataUtils MLDataUtils MLDatasets MLDatasets TensorBoardLogger TensorBoardLogger ObjectDetector ObjectDetector StochasticAD StochasticAD ForwardDiff ForwardDiff ReverseDiff ReverseDiff Zygote Zygote Enzyme Enzyme ChainRules ChainRules TaylorDiff TaylorDiff Julia--MLJFlux

Autodiff:

Zygote is the AD engine for Flux

Enzyme is a novel, more powerful AD tool for languages using the LLVM compiler. Now being incorporated in Lux

R cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

Classic Python packages available in R thanks to wrappers
In addition many packages written for R, but not in R (too slow)
→ written in Fortran, C, or C++

R cluster_ml Traditional machine learning cluster_dl Deep learning scikit-learn scikit-learn Python Python scikit-learn--Python Julia Julia scikit-learn--Julia R R scikit-learn--R Several Julia ML packages Several Julia ML packages Several Julia ML packages--Julia Many R ML packages Many R ML packages Many R ML packages--R Torch Torch TensorFlow TensorFlow JAX JAX Keras Keras Several Julia DL packages Several Julia DL packages Several R DL packages Several R DL packages Python--Torch Python--TensorFlow Python--JAX Python--Keras Julia--Torch Julia--TensorFlow Julia--Keras Julia--Several Julia DL packages R--Torch R--TensorFlow R--Keras R--Several R DL packages

Let’s have a look at the R specific packages.

cluster_ml Traditional machine learning cluster_dl Deep learning e1071 e1071 ipred ipred kernlab kernlab rpart rpart party party caret caret gmodels gmodels randomForest randomForest ranger ranger gbm gbm xgboost xgboost C50 C50 cluster cluster R R cluster--R class class mclust mclust naivebayes naivebayes glmnet glmnet RSNNS RSNNS h2o h2o darch darch deepr deepr RCNN4R RCNN4R rcppDL rcppDL rnn rnn neuralnet neuralnet deepnet deepnet R--RCNN4R

A very large field, particularly when it comes to statistical machine learning

Quite a lot of DL packages, despite R’s slow speed

The Machine Learning CRAN Task View is a good reference for many packages

Languages summary

1 - Huge userbase - Mature frameworks - Smooth APIs - Easy to scale 2 Needs other languages for speed 3 - All in Julia - Tools at the edge of research - Great for developpers 4 - Small userbase - Not the smoothest for end users 5 - Countless packages - Very user-friendly - Well documented 6 - Needs C for speed - Slow - Too slow for DL - Hard to scale Strengths Strengths Weaknesses Weaknesses Python Python Julia Julia R R

Languages summary

1 Good 2 Good 3 Good 4 Good 5 Good 6 Very limited Machine learning Machine learning Deep learning Deep learning Python Python Julia Julia R R

Frameworks picks

Machine learning 1 scikit-learn or MLJ 2 Julia or R stats packages 3 Toolboxes of classic algorithms 1->3 4 Advanced and specialized techniques Which ones to use depends on specific problems 2->4

Frameworks picks

Deep learning 1 PyTorch 2 JAX 4 Optional Keras at high-level Optional JIT 1->4 3 Lux with Enzyme 2->4 5 Sophisticated AD Optional packages for ease of use (e.g. Equinox) Domain-specific language 2->5 6 Explicit parameters High performance AD JIT (Julia) 3->6

Frameworks picks

Deep learning 1 PyTorch 2 JAX 4 Optional Keras at high-level Optional JIT 1->4 3 Lux with Enzyme 2->4 5 Sophisticated AD Optional packages for ease of use (e.g. Equinox, Flax) 2->5 6 Explicit parameters High performance AD JIT (Julia) 3->6 7 Easy All in Python Unified framework 4->7 8 Fastest options at the moment Many optional moving parts Requires learning a DSL (JAX) or Julia (Lux) 5->8 6->8

Automatic differentiation (AD) 1 Static graph and XLA 2 Dynamic tape-based 1->2 a TensorFlow 3 Dynamic tape-based and JIT 2->3 b PyTorch 4 Eager graph and XLA 3->4 c PyTorch 5 Non-standard interpretation and XLA 4->5 d TensorFlow2 6 Works at lower level 5->6 e JAX 7 Works on LLVM itself 6->7 f Zygote g Enzyme A Inconvenient B Hard to optimize C Not currently developped D Dynamism doesn't play well with XLA E Only sublanguage F Harder implementation G Very hard to implement

 Back to webinar page