Feature Extractor

Extensions that are used for training and running inference with the sequence model.

Accessor: feature_extractor

DataFrame Extensions

These can be called on the DataFrame

class animal_soup.FeatureExtractorDataframeExtension(df)

Pandas dataframe extensions for training the feature extractor.

train(mode='fast', batch_size=32, gpu_id=0, initial_lr=0.0001, stop_method='learning_rate', flow_model_in=None, flow_window=11, feature_model_in=None, model_out=None, ethogram_mode='inference')

Train feature extractor model.

The flow generator used to reconstruct the feature extractor will be reconstructed based on the mode ( default ‘fast’) by default unless a user-specified flow generator model checkpoint can be passed to flow_model_in. If you specify your own model paths to the flow generator (perhaps you have re-trained it previously and want to use those model weights instead), in order to have the correct flow generator model reconstructed, the mode argument must match the flow generator model type you are trying to instantiate. For example, if the flow generator model checkpoint is for a TinyMotionNet model, then you should specify mode as ‘slow`.

Additionally, a user may also specify a feature_model_in which can be used to reconstruct the feature model being used for training. For example, you have already re-trained the feature extractor and want to use those model weights instead. However, if you are passing in both a flow_model_in and feature_model_in, the specified mode must match the type of flow generator and feature extractor you are trying to instantiate.

See the table below for details on the correct mode/model pairings.

Parameters:

• mode (str, default 'slow') –

  Argument must be one of [“slow”, “medium”, “fast”]. Determines the model used for training the feature extractor.

  mode	flow model	feature model
  slow	TinyMotionNet	ResNet3D-34
  medium	MotionNet	ResNet50
  fast	TinyMotionNet3D	ResNet18

• batch_size (int, default 32) – Batch size.

• gpu_id (int, default 0) – Specify which gpu to use for training the model.

• initial_lr (float, default 0.0001) – Initial learning rate.

• stop_method (str, default learning_rate) –

  Method for stopping training. Argument must be one of [“learning_rate”, “num_epochs”]

  stop method	description
  learning_rate	Stop training when learning rate drops below a given threshold, means loss has stopped improving
  num_epochs	Stop training after a given number of epochs

• flow_model_in (str or Path, default None) – Location of checkpoint used for flow generator. If None, then will use default checkpoint of flow generator based on the mode.

• flow_window (int, default 11) – Flow window size. Used to infer optic flow features to pass to the feature extractor.

• feature_model_in (str or Path, default None) – If you want to train the model using different model weights than the default. User can provide a location to a different model checkpoint. For example, if you had retrained the feature extractor previously and wanted to use those weights instead. This should be a path to a hidden_two_stream model checkpoint that can be used to reconstruct the spatial and flow classifier. The model being reconstructed should align with the mode argument. See the table above for correct model/mode pairings.

• model_out (str or Path, default None) – User provided location of where to store model output such as model checkpoint with updated weights, hdf5 file with model results/metrics, etc. Should be a directory. By default, the model output will get stored in the same directory as the dataframe.

• ethogram_mode (str, default 'inference') – One of [“inference”, “ground”]. Specifies the location of where to look for ground truth ethograms for training. If “inference”, will try to use ethograms stored from running inference. If “ground”, will look for a column in the dataframe called “ethograms” to use for training.

Series Extensions

These can be called on an individual Series, or rows, of the DataFrame

class animal_soup.FeatureExtractorSeriesExtensions(s)

Pandas series extensions for inference of the feature extractor.

Parameters:

s (Series) –

infer(mode='fast', gpu_id=0, feature_model_in=None, flow_model_in=None, flow_window=11)

Run feature extractor inference on a single trial.

Parameters:

• mode (str, default 'fast') –

  One of [“slow”, “medium”, “fast”]. Indicates what feature extractor to use for inference.

  mode	flow model	feature model	inference speed
  slow	TinyMotionNet	ResNet3D-34	~13 fps
  medium	MotionNet	ResNet50	~80 fps
  fast	TinyMotionNet3D	ResNet18	~150 fps

• gpu_id (int) –

• feature_model_in (str | Path) –

• flow_model_in (str | Path) –

• flow_window (int) –

gpu_id: int, default 0

Specify which gpu to use for training the model.

flow_model_in: str or Path, default None

Location of checkpoint used for flow generator. If None, then will use default checkpoint of flow generator based on the mode. Note: If using a non-default flow generator checkpoint, the flow generator instantiated must match the mode argument. For example, if the flow_model_in checkpoint is to a TinyMotionNet model, then the mode argument MUST be ‘fast’.

flow_window: int, default 11

Flow window size. Used to infer optic flow features to pass to the feature extractor.

feature_model_in: str or Path, default None

If you want to train the model using different model weights than the default. User can provide a location to a different model checkpoint. For example, if you had retrained the feature extractor previously and wanted to use those weights instead. This should be a path to a hidden_two_stream model checkpoint that can be used to reconstruct the spatial and flow classifier. Note: if also passing in a non-default checkpoint to reconstruct a re-trained flow generator, flow_model_in is not None, then the model checkpoints must be for the same mode. See table above for mode/model pairings.