ML Problems

For what kind of problems ML will be better:

  • Repetition: If the task if repetitive with same kind of pattern.
  • Wrong predicition is cheap: The cost of wrong predicitons is cheap.
  • Scale: Cost of integrating ML solution often comes with non-trivial investment. So if the prediction can be made at scale it is a better investment.
  • Changing pattern, no fixed rule: If the problem is constantly changing with time, the pattern that models the problem also changes. Hardcoded rules won’t be sufficient and human knowledge about it might be limited. Let the machine learn over time.

Latency vs throughput

  • Latency: How much time does it take to process one query. Metric for latency calculation is p50 or median ore 50% of the request. Higher percentile means outlier and is important to look at to understand the problem.
  • Throughput: How many queries can be processed in a second.

Four requirements:

  • Reliability: The system continues to perform the correct function at the desired level of performance even in the face of adversity.
  • Scalability: Handing more request either by scaling resources or having models. Having models means monitoring and maintaining scales with it.
  • Maintainability: Same set up, strucutre and proper documentation.
  • Adapatability: Adapt to new data automatically.

Multilabel: When same data point has multiple labels it is called multilabel. The way to work with this is to choose the number of highest probabilites for the number of label the data point ground truth belongs to.

Framing a ML problem matters. If framed a prediction problem as classification. The model needs to be trained for new class. How about it is frame as regression class so that it gives probability and label is chosen based on highest probability.

Cross-entropy for multiclass classification, RMSE or MAE for regression, logistic loss or log loss for binary classification.

When there are multiple objectives, it is good idea to decouple them first because it makes model development and maintenance easier.

Data Engineering

Data Sources

  • User input data (usually requires fast process)
  • System-generated data (logs, system outputs, needed for debugging and potentially improving the application), this does not need to be stored long term if not needed
  • Third-party data: data collected by third party companies on the public who aren’t their direct customers.

Data Formats

  • How to store multiple data types so that it is cheap to store and easy to access? text, image, networks etc.
  • How to store complex models so that they can be loaded and run correctly on different hardware?

Row major vs column-major format:

  • CSV: row major, faster, better for accessing samples, alot of writes
  • Parquet: column major, better for accessing features, best for column based read.

Data Models

  • SQL (declarative language)
  • NoSQL (document or relational model)

Document model: Each document in the form of JSON, XML or binary format like BSON (Binary JSON) is row and collection of document is the table. Graph model: Relation is more important. Better for filtering.

Data warehouse: repository for storing structured data Data lake: repository for storing unstructured data

Databases are optimized for two types of workloads:

  • Transactional processing: Basically CRUD operation, online transaction processing (OLTP). Requires low latency and high availability. ACID (Atomicity: if any transatcion fails, none happens, Consistency: All transactions coming through must follow predefined rules, Isolation: two transactions can happen at same time as if they were isolated, Durability: once transaction is committed, it will remain commited even in the case of system failure.)

  • Analytical processing: Database optimized for aggregations across column across multiple rows. They are efficient with queries called online analytical processing (OLAP).

ETL: Extract, Transform and Load

Modes of Dataflow: Passing data from one process to another.

  • Data passing through databases
  • Data passing through services using requests such as the requests provided by REST and RPC APIs. RPCs are like calling functions in the program to make a request to remote network service.
  • Data passing through a real-time transport like Apache Kafka, amazon kinesis. Two types of real-time tansport: pubsub (publish subscribe) and message queue. In pubsub, any service can publish to different topics in a real-time transport and it does not care who consumes it. Old data are removed. In case of message queue, there is a specific intended consumers example: RocketMQ, RabbitMQ etc.

Batch processing vs stream processing

  • Batch processing: When the historical data is processed in batch jobs. They are processed once a day.
  • Stream processing: Computation in streaming data periodically, within time shorter than priods of batch jobs.

Batch processing is usually used to compute features that change less often, batch features.

Stream processing is used to compute features that change quickly, information that change with time example: number of drivers nearby. These are dynamic features.

Training Data

Sampling

Nonprobability Sampling:

  • Convenience sampling: samples of data are selected based on their availability.
  • Snowball sampling: future samples are selected based on existing samples
  • Judgment sampleing: Experts decide what samples to include
  • Quota sampling: Samples are selected based on quotas for certain slices of data without randomization. Example: in survey, 100 people in age 30-40, 20 people below 19 age etc.

Probability based sampling:

  • Simple Random Sampling: Each sample points get equal probability. Easy to implement but may not include rare class samples.

  • Stratified Sampling: The sample is divided into groups and each groups is separately sampled. This makes sure that rare groups or class are included.

  • Weighted Sampling: Each sample is given a weight, which determines the probability of it being selected.

  • Reservior Sampling: This is useful to deal with streaming data in production. We can’t fit all data for training so we need to sample in the stream.

  • Step 1: select a reservior size to consider or data to consider, example: k = 4

  • Step 2: For each incoming nth element, generate a random number, i between 1 <= i <= n.

  • Step 3: If 1 <= i <= k: replace the ith element in the reservior with the nth element else do nothing. Each incoming nth element has probability of k/n.

  • Importance Sampling: This allows sampling from one distribution when we have only access to another distribution. We need to sample x from P(x) but P(x) is slow, expensive or infeasible to sample from. If we have Q(x) that is easier to sample from. Q(x) if proposal distribution or the importance distribution.

E_p(x) [x] = E_q(x) [x \frac{P(x)}{Q(x)}]

Labeling

  • Hand Labels: Data privacy issue, Expensive if require expert labeler, Slow process
  • Label Multiplicity: Data from different sources can have different labels for same row. Data lineage means keeping track of sources, labels and versions of the data so that we can track the source of error later.
  • Natural Labels
  • Feedback loop length: Time it takes from when a prediction is served until when the feedback on it is provided. Recommender system have short feedback loops. Shorter time means we can evaluate our models faster and find the issue with it. Long feedback is observed in fraud detection. The model performance can be reported in the quaterly report.
  • Handling the Lack of Labels:
    • Weak supervision: In this we label the data based on heuristic function that is created by expert. This might not cover all the labels but a model can be trained on weakly supervised model and made to predict on new data.
    • Semi-supervision: It starts with few hand labeled data, model is trained to predit these labels. The trained model makes prediction on new data points. These new data points are used as input to make prediciton on new, so it is self supervision.
    • Transfer learning: This process uses a general model trained on large amount of generic data to label the data points.
    • Active learning: The model is trained on the examples that we are sure of its label or less uncertainty.

Class imbalance

Class imbalance is especially affect the deep learning models for following reasongs.

  • Lack of examples means insufficient signal for the model, the label might not exist for model.
  • Class imbalance makes it easier for the model to get stuck in a nonoptimal solution by exploiting simple heuristic instead of learning anything useful about the underlying pattern of the data.
  • Asymmetric costs of error, the cost of wrong prediction on a sample of the rare class might be much higher than a wrong prediciton on a sample of the majority class.

Handling class imbalance:

  • Choosing right metric: F1, precision, recall, AUC-ROC, AUC-ROC
  • Data-level methods: Undersampling, oversampling, dynamic sampling (oversample the low-performing classes and undersample the high performing classes during the training process)
  • Algorithm-level methods: Cost sensitive loss where each wrong example is weighted, Weighted class-balanced loss (W_i = N/(number of samples of class i)), Focal Loss $$ FL(p_t) = -(1-p_t)^\gamma log(p_t) $$

Data Augmentation

Simple Label-Preserving Transformations: This means the original data is changed but label is preserved. In case of images, the images are rotated, flipped, cropped, iverted etc. In case of text, words are changed with synomys assuming the replacement wouldn’t change the meaning or the sentiment of the sentence. Embedding of words can be used to.

Perturbation: Adding noisy samples to training data so that model recognize the weak spots in their learned decision boundary and imporve their performance. This is also called ‘adversarial augmentation’.

Data Synthesis: In NLP, the templates are used to generate training data.

Feature Engineering

Learned features vs engineered features: Deep learning is learned features. Engineered features is where domain knowledge is important. Common features engineering operations:

  • Missing data

    • Missing not at random (MNAR): Missing values is missing due to a reason.
    • Missing at random (MAR): Missing value if due to another observed variable
    • Missing completely at random (MCAR): There is no pattern in when the value is missing.

  • Deletion: remove the column itself, row deletion,

  • Imputation: Default value, mean, median or mode, bias or noise or data leakage may be injected when imputation

  • Scaling: $$ x\cap(a) = \frac{x-min(x)}{max(x) - min(x)} $$ this makes in range of [0, 1].

$$ x’ = a + \frac{(x-min(x))(b-a)}{max(x) - min(x)} $$ this makes in range of [a, b].

$$ x’ = \frac{x-mean(x)}{\delta} $$ this is standardization.

  • Discretization: process of turning a continuous features into discrete feature, binning.
  • Encoding: Categories can be numerous. So the way to work with it is to use hash function. For example: if hash space is 18 bits which corresponds to $2^18=262,144$ possible hashed values all the categories even unseen will be encoded. There will be collisions.
  • Feature crossing: combining two or more features to generate new features to model non-linear relations.
  • Discrete and continuous positional embedding: Fourier series can take any discrete and continuous value.

Data Leakage: Some form of labels are leaked to training data. Causes of data leakage:

  • Splitting time-correlated data randomly instead of time:
  • Scaling before splitting: The scaling should happen after the split and mean and std should come from train split.
  • Filling in missing data with statistics from the test split: The missing values should come from summary statistics from the train split.
  • Poor handling of data duplication before splitting: The data duplication should happen after splitting the data into test and train.
  • Group leakage: A group of examples have strongly correlated labels but are divided into splits. Like example: a patient with lung cancer has many samples of test results. If divided into test and train, some samples with common features will end up in both train and test.
  • Leakage from data generation process: If there are different data from different sources, the good performance could be due to the generation process difference itself.

To detect data leakage, check the predicition of each features as well as combination so that we can find if there is high correlation between feature and labels.

Engineering good features:

  • Too many features, the more opportunities for data leakage.
  • Too many features, chances of overfitting
  • Increase in memory requirement.
  • Increase inference latency
  • Useless features becomes technical debts. As there is change in data pipeline, all affected features needs to be adjusted.

Feature importance:

  • Built-in feature importance functions implemented by XGBoost.
  • SHAP (SHapley Additive exPlanations) is model agnostic methods.
  • InterpretML

Feature generalization:

  • More coverage the feature has more generalization. Coverage means the percentage of samples that have the features.

Model development and offline evaluation

Modeling development and training:

  • Avoid the state-of-the-art-trap
  • Start with simplest models
  • Avoid human biases in selecting models
  • Evaluate good performance now vs good performance later: Look into the learning curve and get a sense if the adding data helps.
  • Evaluate trade-offs: false positive vs false negative, GPU vs CPU needed.
  • Understand model’s assumptions: Prediction assumption, IID, smoothness, Tractability, Boundaries, Conditional independence, Normally distributed

Ensembles: Use ensemble of different models to make final predicitons.

Bagging (Bootstrap Aggregating): Random Forest

Boosting: Iterative ensemble algorithms that convert weak learners to strong ones.

Stacking: Results from base models are used to train meta learners to give final predicition.

Experiment tracking:

  • Loss curve corresponding to the train split and each of the eval splits
  • The model performance metrics that you care about on allnontest splits, such as accuracy, F1, perplexity.
  • Log of corresponding sample, prediction, ground truth label.
  • Speed of the model, evaluated by the number of steps per second
  • System performance metrics such as memory usage, CPU/GPU utilization
  • The values over time of any parameter and hyperparameter whose changes can affect the model’s performance.

Versioning: Data versioning system

Debugging:

  • Start simple and gradually add more components
  • Overfit a single batch
  • Set a random seed

Distributed Training:

  • Data parallelism: Difficulty is how to update the gradient collected from different machines. The accumulations will have to wait for all the gradient to be available.
  • Model parallelism: It is actually not parallel if the different layers are in different machines. If the same weight matrix is split into parts, then it can be parallel.
  • Pipeline parallelism:

AutoML

  • soft AutoML: parameter, hyperparameter search
  • hard AutoML: architecture search and learned optimizer
    • Search space
    • Performance estimation strategy
    • Search strategy

Model offline evaluation:

  • Random baseline
  • Simple heuristic like chronological order
  • Zero rule baseline (predicts most common class)
  • Human baseline: if the goal beat prediction done by human
  • Existing solutions: We are replacing existing method, it is useful.

Evaluation methods

  • Perturbation tests: To test how model performs in case of noisy data, we can test by making small changes to the test split to see how these changes affect the model’s performance
  • Invariance tests: Test if removing some features leads to difference in the result. This will help to find biases in the data.
  • Directional expectation tests: If we know if any of the feature change leads to similar direction change in output, we can test if this works or not.
  • Model calibration:
  • Confidence measurement: Usefulness threshold for each individual prediction.
  • Slice-based evaluation: Use metrics in different slices of the data. Simpson’s paradox: a phenomenon in which a trend appears in several groups of data but disappears or reverse when the groups combined.
    • Heuristic-based: Domain knowledge (mobile vs web traffic)
    • Error analysis: Manually go through misclassified examples and find patterns among them
    • Slice finder: Generating slice candidates with algorithms such as beam search, clustering, or decision.

Model deployment and prediciton service

Exporting model means converting the model into a format that can be used by another application. Model definition defines the structure of the model such as how many hidden layers it has, how many units in each layer. The parameter values provide the values for these units and layers.

Batch prediction vs online prediction:

  • Batch prediction: uses only batch features
  • Online prediction: uses both batch features (pre-computed embeddings)
  • Online prediction with batch and streaming features: uses both batch features and streaming features. This is streaming prediction. This serves prediction. The prediction is made on request and also uses fresh, real-time data (like what the user just clicked)

From Batch predicition to online predicton: Converting online to batch is also a way to reduce latency. Unifying batch and streaming prediction: The data pipeline should be same in the training and deploying.

Three ways to reduce inference latency:

  • Make it do inference faster
  • Make the model smaller
  • Make the hardware its deployed on run faster

Model compression: Process to make model smaller.

  • Low-rank optimization: Large parameter model is replaced with smaller model with less parameters.
  • Knowledge distillation: A small model is trained to mimic a larger model or ensemble of models (teacher). DistilBERT
  • Prunning: Making non important parameters zero. Requires less storage space.
  • Quantization: using smaller size of number like 32 bit to 16 bit. It reduces the precision, size, range of values to represents becomes less.

Inference optimization:

Model on cloud vs model on edge: Deployment in device is called model deployment in edge.

Models in cloud means more money for compute and becomes expensive. It requires stable internet connections. People are moving from cloud to edge.

Compiling and optimizing models for edge devices: Different devices have different hardware architecture and needs to be converted into hard warenative code. IRS (Intermediate representation) are middle like compilers that converts the code into manchine understanbalbe code inspite of the hardware type. there is no onetoone mapping. IRs are usually computation graphs of the ML models.

Local optimization techniques:

  • Vectorization: Given a loop, instead of executing one item at a time, execute multiple elements contiguous in memory at the same time to reduce latency caused by data I/O.
  • Parallelization: Divide the input vectors into chuncks and run in different machines
  • loop tiling: Change data accessing order in a loop to leverage hardware’s memory layout and cache.
  • Operator fusion: Instead of multple loops, fuse the operations.

Data distribution shifts and monitoring

Software system failures

  • Dependency failure: dependency package not available
  • Deployment failure: wrong version is deployed
  • Hardware failures: hardware crash
  • Downtime or crashing: Hosted server is crashed

ML-specific failures:

  • Datal collections and processing problems
  • Poor hyperparameters
  • Changes in the training pipelines not correctly replicated in the inference pipeline
  • Data distribution shifts that causes a model’s performance to deteriorate over time, edge cases and degenerate feedback loops

Edge cases are outliers.

Degenerate feedback loops: a degenerate feedback loop is created when a system’s outputs are used to generate the system’s future inputs, which, in turn, influence the system’s future outputs. In ML, a system’s predictions can influence how users interact with the system, and because users’ interactions with the system are sometimes used as training data to the same system, degenerate feedback loops can occur and cause unintended consequences. Degenerate feedback loops are especially common in tasks with natural labels from users, such as recommender systems and ads click-through-rate prediction.

Detecting degenerate feedback loops: Check for degenerate feedback loops by measuring how diverse the recommendations are and how accurate they are across popular and rare items, and watch whether the outputs narrow over time after deployment.

Correcting feedback loops:

  • Randomization: Show users some random items instead of only the top-ranked ones, and use their feedback to learn each item’s true quality.
  • Positional features: User feedback depends on where an item is shown. The top song gets far more clicks than the other four just because it’s on top. So you can’t tell whether the model picked a great song or users simply click whatever is first. Solution: give the model the position as an input feature, so it can separate “this song is good” from “this song was shown first.” The feature can be numerical (positions 1, 2, 3) or Boolean (is it in the first position?).

Data shift:

  • Covariate shift: distribution of the input changes, but the conditional probability of an output given an input remains the same.
  • Label shift: the case when the output distribution changes but, for a given output, the input distribution stays the same.
  • Concept drift: when the input distribution remains the same but the conditional distribution of the output given an input changes.

Detection of data shifts

  • Kolmogorov-Smirnov (KS)
  • Fisher’s exact test
  • Chi-squared test

Three ways to make a model work on a new distribution

  • Train on massive datasets. The hope is that the training distribution is broad enough to cover whatever shows up in production. This dominates research.
  • Adapt without new labels (unsupervised domain adaptation)
  • Retrain with labeled data from the new distribution. shifts are inevitable and retrain on a schedule (monthly, weekly, daily)

Monitoring and observability:

  • Monitoring: tracking, measuring and logging metrics so you can tell when something goes wrong.
  • Observability: setting up the system (instrumentation) so you can investigate what went wrong.

Instrumentation examples: timers on functions, counting NaNs in features, tracking how inputs are transformed through the system, and logging unusual events such as very long inputs.

Operational metrics:

  • Availability: how often the system offers reasonable performance. It’s measured by uptime, the percentage of time the system is up. AWS EC2 (as of October 2021) guaranteed at least 99.99% monthly uptime (“four nines”), with service credits if it fell short.
  • SLO/SLA (service level objectives (SLOs) or service level agreements (SLAs)): define what “up” means. An example SLA: median latency under 200 ms and 99th percentile under 2 s. A provider may also promise an uptime percentage and refund customers if it’s missed.

ML-specific metrics:

  • model’s accuracy-related metrics : Design the system to collect feedback, Log all feedback
  • predictions :

Why predictions are easy to monitor

Each prediction is usually a single number: a continuous value for regression (such as a house price) or a category for classification. Low dimensionality makes them simple to visualize, summarize and interpret.

What to watch for

Distribution shift in predictions: because predictions are low-dimensional, two-sample tests can be used to check whether their distribution has changed. A proxy for input shift: if the model itself hasn’t changed (same weights and biases), a change in the prediction distribution generally means the input distribution changed. Odd behavior: for example, predicting an unusual run of False in a row.

  • Features: 1) Feature validation (scale min-max, regular expression format), 2) Distribution tests. Feature monitoring is useful but costly and noisy, so be selective about what you track and what you alert on, and make sure you can trace problems back through the pipeline.

  • raw inputs: A feature change might come from a processing bug and not a real data change. So monitoring raw inputs before processing could seem cleaner.

Monitoring toolbox

  • Logs
  • Dashboards
  • Alerts: A good alert has a clear trigger condition, goes to the right people through the right channel, explains itself with a runbook, and fires only for things that truly matter.

Observability: Monitoring tells you something is wrong, while observability lets you work out why, down to specific inputs and user groups, from logs and metrics alone.

Continual learning and test in production

The goal of continual learning is to safely and efficiently automate the update.

Continual learning in practice means updating models in small batches, testing a challenger against the current champion before replacing it, and only as often as the data and decay rate justify.

Stateless Retraining vs Stateful Training

  • Stateless retraining: train from scratch each time. This is what most companies do.
  • Stateful training: continue training from the last checkpoint on new data. Also called fine-tuning or incremental learning.

Benefits of stateful training:

  • Needs less data
  • Cheaper and faster
  • Possibly no data storage

Stateful training can’t easily do:

  • Model iteration: adding a feature or changing the architecture. This still requires training from scratch.

Continual learning lets models respond to sudden shifts, rare events and users with little history, and since it includes batch learning, it is worth adopting as tooling gets easier.

Challenges of continual learning

  • Fresh data access
  • Evaluation
  • Algorithm: Neural networks update easily on small batches, but matrix-based and tree-based models and the feature-scaling code need incremental versions to support fast continual learning.

Stages of continual learning

  • Manual stateless retraining
  • Automated retraining
  • Automated, stateful training
  • Continual learning

Test in Production

  • Shadow deployment:

    • Run the candidate mdoel in parallel with the existing one. Every request goes to both, but only the existing model’s predicition served. Log the candidate’s predicitons for analysis.
    • Replace the old model only when the new one looks satisfactory
    • Very safe, but expensive: it roughly doubles inference compute

  • A/B testing:

    • Route a share of traffic to the candidate and the rest to the existing model, then compare metrics. It’s the standard method.
    • Usually both variants run at once. If one model’s predicitons effect the other’s run them alternately.
    • Two key requirements:
      • True randomization.
      • Enough samples for confidence
    • Significance is checked with two-sample tests, the same kind used to detect distribution shift.
    • We can test more than two variants.

  • Canary Release:

    • Roll the candidate out to a small share of traffic, increase it if performance is satisfactory, and abort and route everything back to the old model if key metrics degrade. It ends when the canary serves all traffics or is aborted.
    • Similar setup to A/B testing, but traffic needn’t be randomized. For example, roll out first in a less critical market.

  • Interleaving experiments:

    • Instead of giving each user one model’s recommendations, show each user a mix from both models and see which items they click.
    • It measures user preference, which doesn’t guarantee better core metrics such as retention.
    • Position bias matters so at each position an item must be equally likely to come from A or B.

  • Bandits

    • Define the candidates: The current model and one or more challenger, each able to serve online predictions.
    • Define the reward: A clear, quick signal for a good prediction, such as click, a completed action or an accepted suggestion. Bandits need short feedback loops so avoid rewards that arrive weeks later.
    • Set up the logging. For every request, log which model served it, the request ID, the prediction and a timestamp, so you can match feedback to the right model later.
    • Initialize the scores. Start each model with no evidence. For Thompson Sampling with success/failure rewards, give each model a Beta(1, 1) belief: 1 success and 1 failure as a starting point.
    • Route each request. Draw a random sample from each model’s current belief. Send the request to the model with the highest draw. Serve that model’s prediction.
    • Collect feedback. When the reward arrives (click or no click), match it to the model that served the request.
    • Update the beliefs. Add 1 to that model’s success count if rewarded, or to its failure count if not.
    • Repeat. Over time, better models win the draws more often and get more traffic, while uncertain ones still get tried occasionally.
    • Add guardrails. Keep a minimum traffic share (such as 5%) for every model so you keep learning. Monitor safety metrics like latency and errors, and abort a model that degrades. Compare against a static test set or shadow results before the challenger gets live traffic.
    • Decide when to stop. When one model clearly dominates (for example, it gets nearly all traffic and its credible interval is above the others), promote it and retire the rest.

  • Contextuals bandits

    • A contextual bandit is a bandit that uses information about the current situation (the context) to decide which option to show, instead of picking one best option for everyone.

Infrastructure and tooling for MLOps

Storage and compute

  • Storage of data in cloud
  • Compute resources available. This determines how scalable the system is. AWS Elastic

Memory metric: GB

Operation speed: FLOPs (floating point operations per speed)

Public cloud also provide the compute resource. Pay as you go.

Resource management

  • Tools to schedule and orchestrate the workloads to make the most out of the available compute resources. Example: Airflow, Kubeflow and Metaflow.

ML Platform:

  • tools aid the developement of ML applications such as model stores, feature stores, and monitoring tools. Example: SageMaker, MLflow

Development enviroment This is where code is written and tested. It consists of IDE, versioning and CI/CD. Git for Version control, DVC to version data, weights and biases or comet.ml to track experiments during development, MLflow to track artifacts of model when deploying them.

SageMake studio is IDE in cloud. Having dev and production environment in cloud is better because of standardization. No issue because of package etc.

From Dev to Prod: Containers: Docker image: Receipe Docker container: food Docker Compose: lightweight container orchestrator that can manage containers on a single host

Cron, Schedulers and Orchestrators: Cron is linear directed in nature. It can’t condition.

Schedulers are cron programs that can handle conditions. Slurm is the schedulers. Ochestrators deals with lower-level abstractions like machines, instances, clusters and service-level grouping, replications, etc. Example Kubernetes.

Data Science Workflow Management DAGs: Airflow, Argo, Prefect, Kubeflow, Metaflow

Model deployment

  • Online or Batch production: SageMaker, Azure ML

Model Card Things to be documented for future use:

  • Model definition: Information needed to create the shape of the modle
  • Model parameters: the actual values the parameters of the model
  • Featurize and predict functions: Given the request, how do you extract features, input these features into the model to get back a prediction?
  • Dependencies: Packages, need to run the model are usually packaged together into a container.
  • Data: Data used to train the model might be pointers to the location where the data is stored or the name/version of the data
  • Model generation code: the code that specifies how the model is created
  • Experiment artifacts: Artifacts generated during the model development process. Like: graphs with loss curve
  • Tags: Tags to help with model discovery and filtering such as owner.

Feature Store

  • Feature management
  • Feature computation
  • Feature consistency

Feast is one of the example.

The human side of machine learning

Ensuring user experience consistency Human in the loop AI: Give different machine generated predictions and make the human pick the best predicitons. Smoothing failing: If the main model takes longer than certain seconds, use the prediction from simpler backup model. Team structure: