Services

CoreService

rpc GetFeastCoreVersion(GetFeastCoreVersionRequest) returns (GetFeastCoreVersionResponse)

Retrieve version information about this Feast deployment

rpc GetEntity(GetEntityRequest) returns (GetEntityResponse)

Returns a specific entity

rpc ListFeatures(ListFeaturesRequest) returns (ListFeaturesResponse)

Returns all feature references and respective features matching that filter. If none are found an empty map will be returned If no filter is provided in the request, the response will contain all the features currently stored in the default project.

rpc ListStores(ListStoresRequest) returns (ListStoresResponse)

Retrieve store details given a filter.

Returns all stores matching that filter. If none are found, an empty list will be returned. If no filter is provided in the request, the response will contain all the stores currently stored in the registry.

rpc ApplyEntity(ApplyEntityRequest) returns (ApplyEntityResponse)

Create or update and existing entity.

This function is idempotent - it will not create a new entity if schema does not change. Schema changes will update the entity if the changes are valid. Following changes are not valid: - Changes to name - Changes to type

rpc ListEntities(ListEntitiesRequest) returns (ListEntitiesResponse)

Returns all entity references and respective entities matching that filter. If none are found an empty map will be returned If no filter is provided in the request, the response will contain all the entities currently stored in the default project.

rpc UpdateStore(UpdateStoreRequest) returns (UpdateStoreResponse)

Updates core with the configuration of the store.

If the changes are valid, core will return the given store configuration in response, and start or update the necessary feature population jobs for the updated store.

rpc CreateProject(CreateProjectRequest) returns (CreateProjectResponse)

Creates a project. Projects serve as namespaces within which resources like features will be created. Feature table names as must be unique within a project while field (Feature/Entity) names must be unique within a Feature Table. Project names themselves must be globally unique.

rpc ArchiveProject(ArchiveProjectRequest) returns (ArchiveProjectResponse)

Archives a project. Archived projects will continue to exist and function, but won’t be visible through the Core API. Any existing ingestion or serving requests will continue to function, but will result in warning messages being logged. It is not possible to unarchive a project through the Core API

rpc ListProjects(ListProjectsRequest) returns (ListProjectsResponse)

Lists all projects active projects.

rpc ApplyFeatureTable(ApplyFeatureTableRequest) returns (ApplyFeatureTableResponse)

Create or update an existing feature table. This function is idempotent - it will not create a new feature table if the schema does not change. Schema changes will update the feature table if the changes are valid. All changes except the following are valid: - Changes to feature table name. - Changes to entities - Changes to feature name and type

rpc ListFeatureTables(ListFeatureTablesRequest) returns (ListFeatureTablesResponse)

List feature tables that match a given filter. Returns the references of the Feature Tables matching that filter. If none are found, an empty list will be returned. If no filter is provided in the request, the response will match all the feature tables currently stored in the registry.

rpc GetFeatureTable(GetFeatureTableRequest) returns (GetFeatureTableResponse)

Returns a specific feature table

rpc DeleteFeatureTable(DeleteFeatureTableRequest) returns (DeleteFeatureTableResponse)

Delete a specific feature table

JobService

rpc StartOfflineToOnlineIngestionJob(StartOfflineToOnlineIngestionJobRequest) returns (StartOfflineToOnlineIngestionJobResponse)

Start job to ingest data from offline store into online store

rpc GetHistoricalFeatures(GetHistoricalFeaturesRequest) returns (GetHistoricalFeaturesResponse)

Produce a training dataset, return a job id that will provide a file reference

rpc StartStreamToOnlineIngestionJob(StartStreamToOnlineIngestionJobRequest) returns (StartStreamToOnlineIngestionJobResponse)

Start job to ingest data from stream into online store

rpc ListJobs(ListJobsRequest) returns (ListJobsResponse)

List all types of jobs

rpc CancelJob(CancelJobRequest) returns (CancelJobResponse)

Cancel a single job

rpc GetJob(GetJobRequest) returns (GetJobResponse)

Get details of a single job

Types

Entity

Field Type Description Required
spec EntitySpecV2

User-specified specifications of this entity.

No
meta EntityMeta

System-populated metadata for this entity.

No

EntitySpecV2

Field Type Description Required
name string

Name of the entity.

No
valueType Enum

Type of the entity.

No
description string

Description of the entity.

No
labels map<string, string>

User defined metadata

No

EntityMeta

Field Type Description Required
createdTimestamp Timestamp No
lastUpdatedTimestamp Timestamp No

FeatureSpecV2

Field Type Description Required
name string

Name of the feature. Not updatable.

No
valueType Enum

Value type of the feature. Not updatable.

No
labels map<string, string>

Labels for user defined metadata on a feature

No

FileFormat

Defines the file format encoding the features/entity data in files

Field Type Description Required
parquetFormat ParquetFormat (oneof) No

StreamFormat

Defines the data format encoding features/entity data in data streams

Field Type Description Required
avroFormat AvroFormat (oneof) No
protoFormat ProtoFormat (oneof) No

FileFormat.ParquetFormat

Defines options for the Parquet data format

StreamFormat.ProtoFormat

Defines options for the protobuf data format

Field Type Description Required
classPath string

Classpath to the generated Java Protobuf class that can be used to decode Feature data from the obtained stream message

No

StreamFormat.AvroFormat

Defines options for the avro data format

Field Type Description Required
schemaJson string

Optional if used in a File DataSource as schema is embedded in avro file. Specifies the schema of the Avro message as JSON string.

No

DataSource

Defines a Data Source that can be used source Feature data

Field Type Description Required
type SourceType No
fieldMapping map<string, string>

Defines mapping between fields in the sourced data and fields in parent FeatureTable.

No
eventTimestampColumn string

Must specify event timestamp column name

No
datePartitionColumn string

(Optional) Specify partition column useful for file sources

No
createdTimestampColumn string

Must specify creation timestamp column name

No
fileOptions FileOptions (oneof) No
bigqueryOptions BigQueryOptions (oneof) No
kafkaOptions KafkaOptions (oneof) No
kinesisOptions KinesisOptions (oneof) No

DataSource.FileOptions

Defines options for DataSource that sources features from a file

Field Type Description Required
fileFormat FileFormat No
fileUrl string

Target URL of file to retrieve and source features from. s3://path/to/file for AWS S3 storage gs://path/to/file for GCP GCS storage file:///path/to/file for local storage

No

DataSource.BigQueryOptions

Defines options for DataSource that sources features from a BigQuery Query

Field Type Description Required
tableRef string

Full table reference in the form of [project:dataset.table]

No

DataSource.KafkaOptions

Defines options for DataSource that sources features from Kafka messages. Each message should be a Protobuf that can be decoded with the generated Java Protobuf class at the given class path

Field Type Description Required
bootstrapServers string

Comma separated list of Kafka bootstrap servers. Used for feature tables without a defined source host[:port]]

No
topic string

Kafka topic to collect feature data from.

No
messageFormat StreamFormat

Defines the stream data format encoding feature/entity data in Kafka messages.

No

DataSource.KinesisOptions

Defines options for DataSource that sources features from Kinesis records. Each record should be a Protobuf that can be decoded with the generated Java Protobuf class at the given class path

Field Type Description Required
region string

AWS region of the Kinesis stream

No
streamName string

Name of the Kinesis stream to obtain feature data from.

No
recordFormat StreamFormat

Defines the data format encoding the feature/entity data in Kinesis records. Kinesis Data Sources support Avro and Proto as data formats.

No

FeatureTable

Field Type Description Required
spec FeatureTableSpec

User-specified specifications of this feature table.

No
meta FeatureTableMeta

System-populated metadata for this feature table.

No

FeatureTableSpec

Field Type Description Required
name string

Name of the feature table. Must be unique. Not updated.

No
entities string[]

List names of entities to associate with the Features defined in this Feature Table. Not updatable.

No
features FeatureSpecV2[]

List of features specifications for each feature defined with this feature table.

No
labels map<string, string>

User defined metadata

No
maxAge Duration

Features in this feature table can only be retrieved from online serving younger than max age. Age is measured as the duration of time between the feature’s event timestamp and when the feature is retrieved Feature values outside max age will be returned as unset values and indicated to end user

No
batchSource DataSource

Batch/Offline DataSource to source batch/offline feature data. Only batch DataSource can be specified (ie source type should start with ‘BATCH_’)

No
streamSource DataSource

Stream/Online DataSource to source stream/online feature data. Only stream DataSource can be specified (ie source type should start with ‘STREAM_’)

No

FeatureTableMeta

Field Type Description Required
createdTimestamp Timestamp

Time where this Feature Table is created

No
lastUpdatedTimestamp Timestamp

Time where this Feature Table is last updated

No
revision int64

Auto incrementing revision no. of this Feature Table

No
hash string

Hash entities, features, batchsource and streamsource to inform JobService if jobs should be restarted should hash change

No

Store

Store provides a location where Feast reads and writes feature values. Feature values will be written to the Store in the form of FeatureRow elements. The way FeatureRow is encoded and decoded when it is written to and read from the Store depends on the type of the Store.

Field Type Description Required
name string

Name of the store.

No
type StoreType

Type of store.

No
subscriptions Subscription[]

Feature sets to subscribe to.

No
redisConfig RedisConfig (oneof) No
redisClusterConfig RedisClusterConfig (oneof) No

Store.RedisConfig

Field Type Description Required
host string No
port int32 No
initialBackoffMs int32

Optional. The number of milliseconds to wait before retrying failed Redis connection. By default, Feast uses exponential backoff policy and “initialbackoffms” sets the initial wait duration.

No
maxRetries int32

Optional. Maximum total number of retries for connecting to Redis. Default to zero retries.

No
flushFrequencySeconds int32

Optional. How often flush data to redis

No
ssl bool

Optional. Connect over SSL.

No

Store.RedisClusterConfig

Field Type Description Required
connectionString string

List of Redis Uri for all the nodes in Redis Cluster, comma separated. Eg. host1:6379, host2:6379

No
initialBackoffMs int32 No
maxRetries int32 No
flushFrequencySeconds int32

Optional. How often flush data to redis

No
keyPrefix string

Optional. Append a prefix to the Redis Key

No
enableFallback bool

Optional. Enable fallback to another key prefix if the original key is not present. Useful for migrating key prefix without re-ingestion. Disabled by default.

No
fallbackPrefix string

Optional. This would be the fallback prefix to use if enable_fallback is true.

No

Store.Subscription

Field Type Description Required
project string

Name of project that the feature sets belongs to. This can be one of - [project_name] - * If an asterisk is provided, filtering on projects will be disabled. All projects will be matched. It is NOT possible to provide an asterisk with a string in order to do pattern matching.

No
name string

Name of the desired feature set. Asterisks can be used as wildcards in the name. Matching on names is only permitted if a specific project is defined. It is disallowed If the project name is set to “*” e.g. - * can be used to match all feature sets - my-feature-set* can be used to match all features prefixed by “my-feature-set” - my-feature-set-6 can be used to select a single feature set

No
exclude bool

All matches with exclude enabled will be filtered out instead of added

No

GetEntityRequest

Request for a single entity

Field Type Description Required
name string

Name of entity (required).

No
project string

Name of project the entity belongs to. If omitted will default to ‘default’ project.

No

GetEntityResponse

Response containing a single entity

Field Type Description Required
entity Entity No

ListEntitiesRequest

Retrieves details for all versions of a specific entity

Field Type Description Required
filter Filter No

ListEntitiesResponse

Field Type Description Required
entities Entity[] No

ListFeaturesRequest

Field Type Description Required
filter Filter No

ListFeaturesResponse

Field Type Description Required
features map<string, FeatureSpecV2> No

ListStoresRequest

Field Type Description Required
filter Filter No

ListStoresResponse

Field Type Description Required
store Store[] No

ApplyEntityRequest

Field Type Description Required
spec EntitySpecV2

If project is unspecified, will default to ‘default’ project. If project specified does not exist, the project would be automatically created.

No
project string

Name of project that this entity belongs to.

No

ApplyEntityResponse

Field Type Description Required
entity Entity No

GetFeastCoreVersionRequest

GetFeastCoreVersionResponse

Field Type Description Required
version string No

UpdateStoreRequest

Field Type Description Required
store Store No

UpdateStoreResponse

Field Type Description Required
store Store No
status Status No

CreateProjectRequest

Request to create a project

Field Type Description Required
name string

Name of project (required)

No

CreateProjectResponse

Response for creation of a project

ArchiveProjectRequest

Request for the archival of a project

Field Type Description Required
name string

Name of project to be archived

No

ArchiveProjectResponse

Response for archival of a project

ListProjectsRequest

Request for listing of projects

ListProjectsResponse

Response for listing of projects

Field Type Description Required
projects string[]

List of project names (archived projects are filtered out)

No

UpdateFeatureSetStatusResponse

ApplyFeatureTableRequest

Field Type Description Required
project string

Optional. Name of the Project to apply the Feature Table to. If unspecified, will apply FeatureTable to the default project.

No
tableSpec FeatureTableSpec

Feature Table specification to apply

No

ApplyFeatureTableResponse

Field Type Description Required
table FeatureTable No

GetFeatureTableRequest

Field Type Description Required
project string

Optional. Name of the Project to retrieve the Feature Table from. If unspecified, will apply FeatureTable to the default project.

No
name string

Name of the FeatureTable to retrieve.

No

GetFeatureTableResponse

Field Type Description Required
table FeatureTable

The Feature Table retrieved.

No

ListFeatureTablesRequest

Field Type Description Required
filter Filter

Filter used when listing Feature Tables

No

ListFeatureTablesResponse

Field Type Description Required
tables FeatureTable[]

List of matching Feature Tables

No

DeleteFeatureTableRequest

Field Type Description Required
project string

Optional. Name of the Project to delete the Feature Table from. If unspecified, will delete FeatureTable from the default project.

No
name string

Name of the FeatureTable to delete.

No

DeleteFeatureTableResponse

ListEntitiesRequest.Filter

Field Type Description Required
project string

Optional. Specifies the name of the project to list Entities in. It is NOT possible to provide an asterisk with a string in order to do pattern matching. If unspecified, this field will default to the default project ‘default’.

No
labels map<string, string>

Optional. User defined metadata for entity. Entities with all matching labels will be returned.

No

ListFeaturesRequest.Filter

Field Type Description Required
labels map<string, string>

User defined metadata for feature. Features with all matching labels will be returned.

No
entities string[]

List of entities contained within the featureSet that the feature belongs to. Only feature tables with these entities will be searched for features.

No
project string

Name of project that the feature tables belongs to. Filtering on projects is disabled. It is NOT possible to provide an asterisk with a string in order to do pattern matching. If unspecified this field will default to the default project ‘default’.

No

ListStoresRequest.Filter

Field Type Description Required
name string

Name of desired store. Regex is not supported in this query.

No

ListFeatureTablesRequest.Filter

Field Type Description Required
project string

Optional. Specifies the name of the project to list Feature Tables in. If unspecified would list Feature Tables in the default project.

No
labels map<string, string>

Optional. Feature Tables with all matching labels will be returned. If unspecified would list Feature Tables without filtering by labels.

No

Job

Field Type Description Required
id string

Identifier of the Job

No
type JobType

Type of the Job

No
status JobStatus

Current job status

No
hash string

Deterministic hash of the Job

No
retrieval RetrievalJobMeta (oneof) No
batchIngestion OfflineToOnlineMeta (oneof) No
streamIngestion StreamToOnlineMeta (oneof) No

StartOfflineToOnlineIngestionJobRequest

Ingest data from offline store into online store

Field Type Description Required
project string

Feature table to ingest

No
tableName string No
startDate Timestamp

Start of time range for source data from offline store

No
endDate Timestamp

End of time range for source data from offline store

No

StartOfflineToOnlineIngestionJobResponse

Field Type Description Required
id string

Job ID assigned by Feast

No

GetHistoricalFeaturesRequest

Field Type Description Required
featureRefs string[]

List of feature references that are being retrieved

No
entitySource DataSource

Batch DataSource that can be used to obtain entity values for historical retrieval. For each entity value, a feature value will be retrieved for that value/timestamp Only ‘BATCH*’ source types are supported. Currently only BATCHFILE source type is supported.

No
project string

Optional field to specify project name override. If specified, uses the given project for retrieval. Overrides the projects specified in Feature References if both are specified.

No
outputLocation string

Specifies the path in a bucket to write the exported feature data files Export to AWS S3 - s3://path/to/features Export to GCP GCS - gs://path/to/features

No
outputFormat string

Specify format name for output, eg. parquet

No

GetHistoricalFeaturesResponse

Field Type Description Required
id string

Export Job with ID assigned by Feast

No
outputFileUri string No

StartStreamToOnlineIngestionJobRequest

Field Type Description Required
project string

Feature table to ingest

No
tableName string No

StartStreamToOnlineIngestionJobResponse

Field Type Description Required
id string

Job ID assigned by Feast

No

ListJobsRequest

Field Type Description Required
includeTerminated bool No

ListJobsResponse

Field Type Description Required
jobs Job[] No

GetJobRequest

Field Type Description Required
jobId string No

GetJobResponse

Field Type Description Required
job Job No

CancelJobRequest

Field Type Description Required
jobId string No

CancelJobResponse

Job.RetrievalJobMeta

Field Type Description Required
outputLocation string No

Job.OfflineToOnlineMeta

Job.StreamToOnlineMeta

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

DataSource.SourceType

Type of Data Source.

Name Description
INVALID
BATCH_FILE
BATCH_BIGQUERY
STREAM_KAFKA
STREAM_KINESIS

DataSource.SourceType

Type of Data Source.

Name Description
INVALID
BATCH_FILE
BATCH_BIGQUERY
STREAM_KAFKA
STREAM_KINESIS

Store.StoreType

Name Description
INVALID
REDIS

Redis stores a FeatureRow element as a key, value pair.

The Redis data types used (https://redis.io/topics/data-types): - key: STRING - value: STRING

Encodings: - key: byte array of RedisKey (refer to feast.storage.RedisKeyV2) - value: Redis hashmap

REDIS_CLUSTER

Store.StoreType

Name Description
INVALID
REDIS

Redis stores a FeatureRow element as a key, value pair.

The Redis data types used (https://redis.io/topics/data-types): - key: STRING - value: STRING

Encodings: - key: byte array of RedisKey (refer to feast.storage.RedisKeyV2) - value: Redis hashmap

REDIS_CLUSTER

UpdateStoreResponse.Status

Name Description
NO_CHANGE

Existing store config matching the given store id is identical to the given store config.

UPDATED

New store created or existing config updated.

UpdateStoreResponse.Status

Name Description
NO_CHANGE

Existing store config matching the given store id is identical to the given store config.

UPDATED

New store created or existing config updated.

JobType

Name Description
INVALID_JOB
BATCH_INGESTION_JOB
STREAM_INGESTION_JOB
RETRIEVAL_JOB

JobStatus

Name Description
JOB_STATUS_INVALID
JOB_STATUS_PENDING

The Job has be registered and waiting to get scheduled to run

JOB_STATUS_RUNNING

The Job is currently processing its task

JOB_STATUS_DONE

The Job has successfully completed its task

JOB_STATUS_ERROR

The Job has encountered an error while processing its task

JobType

Name Description
INVALID_JOB
BATCH_INGESTION_JOB
STREAM_INGESTION_JOB
RETRIEVAL_JOB

JobStatus

Name Description
JOB_STATUS_INVALID
JOB_STATUS_PENDING

The Job has be registered and waiting to get scheduled to run

JOB_STATUS_RUNNING

The Job is currently processing its task

JOB_STATUS_DONE

The Job has successfully completed its task

JOB_STATUS_ERROR

The Job has encountered an error while processing its task