Services

CoreService

rpc GetFeastCoreVersion(GetFeastCoreVersionRequest) returns (GetFeastCoreVersionResponse)

Retrieve version information about this Feast deployment

rpc GetFeatureSet(GetFeatureSetRequest) returns (GetFeatureSetResponse)

Returns a specific feature set

rpc GetEntity(GetEntityRequest) returns (GetEntityResponse)

Returns a specific entity

rpc ListFeatureSets(ListFeatureSetsRequest) returns (ListFeatureSetsResponse)

Retrieve feature set details given a filter.

Returns all feature sets matching that filter. If none are found, an empty list will be returned. If no filter is provided in the request, the response will contain all the feature sets currently stored in the registry.

rpc ListFeatures(ListFeaturesRequest) returns (ListFeaturesResponse)

Returns all feature references and respective features matching that filter. If none are found an empty map will be returned If no filter is provided in the request, the response will contain all the features currently stored in the default project.

rpc GetFeatureStatistics(GetFeatureStatisticsRequest) returns (GetFeatureStatisticsResponse)

Get feature statistics computed over the data in the batch stores.

Returns a dataset containing TFDV statistics mapped to each valid historical store.

rpc ListStores(ListStoresRequest) returns (ListStoresResponse)

Retrieve store details given a filter.

Returns all stores matching that filter. If none are found, an empty list will be returned. If no filter is provided in the request, the response will contain all the stores currently stored in the registry.

rpc ApplyFeatureSet(ApplyFeatureSetRequest) returns (ApplyFeatureSetResponse)

Create or update and existing feature set.

This function is idempotent - it will not create a new feature set if schema does not change. Schema changes will update the feature set if the changes are valid. All changes except the following are valid: - Changes to feature set id (name, project) - Changes to entities - Changes to feature name and type

rpc ApplyEntity(ApplyEntityRequest) returns (ApplyEntityResponse)

Create or update and existing entity.

This function is idempotent - it will not create a new entity if schema does not change. Schema changes will update the entity if the changes are valid. Following changes are not valid: - Changes to name - Changes to type

rpc ListEntities(ListEntitiesRequest) returns (ListEntitiesResponse)

Returns all entity references and respective entities matching that filter. If none are found an empty map will be returned If no filter is provided in the request, the response will contain all the entities currently stored in the default project.

rpc UpdateStore(UpdateStoreRequest) returns (UpdateStoreResponse)

Updates core with the configuration of the store.

If the changes are valid, core will return the given store configuration in response, and start or update the necessary feature population jobs for the updated store.

rpc CreateProject(CreateProjectRequest) returns (CreateProjectResponse)

Creates a project. Projects serve as namespaces within which resources like features will be created. Feature set names as must be unique within a project while field (Feature/Entity) names must be unique within a Feature Set. Project names themselves must be globally unique.

rpc ArchiveProject(ArchiveProjectRequest) returns (ArchiveProjectResponse)

Archives a project. Archived projects will continue to exist and function, but won’t be visible through the Core API. Any existing ingestion or serving requests will continue to function, but will result in warning messages being logged. It is not possible to unarchive a project through the Core API

rpc ListProjects(ListProjectsRequest) returns (ListProjectsResponse)

Lists all projects active projects.

rpc UpdateFeatureSetStatus(UpdateFeatureSetStatusRequest) returns (UpdateFeatureSetStatusResponse)

Internal API for Job Controller to update featureSet’s status once responsible ingestion job is running

rpc ApplyFeatureTable(ApplyFeatureTableRequest) returns (ApplyFeatureTableResponse)

Create or update an existing feature table. This function is idempotent - it will not create a new feature table if the schema does not change. Schema changes will update the feature table if the changes are valid. All changes except the following are valid: - Changes to feature table name. - Changes to entities - Changes to feature name and type

rpc ListFeatureTables(ListFeatureTablesRequest) returns (ListFeatureTablesResponse)

List feature tables that match a given filter. Returns the references of the Feature Tables matching that filter. If none are found, an empty list will be returned. If no filter is provided in the request, the response will match all the feature tables currently stored in the registry.

rpc GetFeatureTable(GetFeatureTableRequest) returns (GetFeatureTableResponse)

Returns a specific feature table

JobControllerService

rpc ListIngestionJobs(ListIngestionJobsRequest) returns (ListIngestionJobsResponse)

List Ingestion Jobs given an optional filter. Returns allow ingestions matching the given request filter. Returns all ingestion jobs if no filter is provided. Returns an empty list if no ingestion jobs match the filter.

rpc RestartIngestionJob(RestartIngestionJobRequest) returns (RestartIngestionJobResponse)

Restart an Ingestion Job. Restarts the ingestion job with the given job id. NOTE: Data might be lost during the restart for some job runners. Does not support stopping a job in a transitional (ie pending, suspending, aborting), terminal state (ie suspended or aborted) or unknown status

rpc StopIngestionJob(StopIngestionJobRequest) returns (StopIngestionJobResponse)

Stop an Ingestion Job. Stop (Aborts) the ingestion job with the given job id. Does nothing if the target job if already in a terminal state (ie suspended or aborted). Does not support stopping a job in a transitional (ie pending, suspending, aborting) or unknown status

Types

Entity

Field Type Description Required
spec EntitySpecV2

User-specified specifications of this entity.

No
meta EntityMeta

System-populated metadata for this entity.

No

EntitySpecV2

Field Type Description Required
name string

Name of the entity.

No
valueType Enum

Type of the entity.

No
description string

Description of the entity.

No
labels map<string, string>

User defined metadata

No

EntityMeta

Field Type Description Required
createdTimestamp Timestamp No
lastUpdatedTimestamp Timestamp No

Source

Field Type Description Required
type SourceType

The kind of data source Feast should connect to in order to retrieve FeatureRow value

No
kafkaSourceConfig KafkaSourceConfig (oneof) No

KafkaSourceConfig

Field Type Description Required
bootstrapServers string

Comma separated list of Kafka bootstrap servers. Used for feature sets without a defined source host[:port]]

No
topic string

Kafka topic to use for feature sets without user defined topics

No
partitions int32

Number of Kafka partitions to to use for managed feature stream.

No
replicationFactor int32

Defines the number of copies of managed feature stream Kafka.

No

FeatureSet

Field Type Description Required
spec FeatureSetSpec

User-specified specifications of this feature set.

No
meta FeatureSetMeta

System-populated metadata for this feature set.

No

FeatureSetSpec

Field Type Description Required
project string

Name of project that this feature set belongs to.

No
name string

Name of the feature set. Must be unique.

No
entities EntitySpec[]

List of entities contained within this featureSet. This allows the feature to be used during joins between feature sets. If the featureSet is ingested into a store that supports keys, this value will be made a key.

No
features FeatureSpec[]

List of features contained within this featureSet.

No
maxAge Duration

Features in this feature set will only be retrieved if they are found after [time - max_age]. Missing or older feature values will be returned as nulls and indicated to end user

No
source Source

Optional. Source on which feature rows can be found. If not set, source will be set to the default value configured in Feast Core.

No
labels map<string, string>

User defined metadata

No
version int32

Read-only self-incrementing version that increases monotonically when changes are made to a feature set

No

EntitySpec

Field Type Description Required
name string

Name of the entity.

No
valueType Enum

Value type of the entity.

No

FeatureSpec

Field Type Description Required
name string

Name of the feature.

No
valueType Enum

Value type of the feature.

No
labels map<string, string>

Labels for user defined metadata on a feature

No
presence FeaturePresence (oneof)

Constraints on the presence of this feature in the examples.

No
groupPresence FeaturePresenceWithinGroup (oneof)

Only used in the context of a “group” context, e.g., inside a sequence.

No
shape FixedShape (oneof)

The feature has a fixed shape corresponding to a multi-dimensional tensor.

No
valueCount ValueCount (oneof)

The feature doesn’t have a well defined shape. All we know are limits on the minimum and maximum number of values.

No
domain string (oneof)

Reference to a domain defined at the schema level.

No
intDomain IntDomain (oneof)

Inline definitions of domains.

No
floatDomain FloatDomain (oneof) No
stringDomain StringDomain (oneof) No
boolDomain BoolDomain (oneof) No
structDomain StructDomain (oneof) No
naturalLanguageDomain NaturalLanguageDomain (oneof)

Supported semantic domains.

No
imageDomain ImageDomain (oneof) No
midDomain MIDDomain (oneof) No
urlDomain URLDomain (oneof) No
timeDomain TimeDomain (oneof) No
timeOfDayDomain TimeOfDayDomain (oneof) No

FeatureSetMeta

Field Type Description Required
createdTimestamp Timestamp

Created timestamp of this specific feature set.

No
status FeatureSetStatus

Status of the feature set. Used to indicate whether the feature set is ready for consumption or ingestion. Currently supports 2 states: 1) STATUSPENDING - A feature set is in pending state if Feast has not spun up the jobs necessary to push rows for this feature set to stores subscribing to this feature set. 2) STATUSREADY - Feature set is ready for consumption or ingestion

No

tensorflow.metadata.v0.FeaturePresence

Describes constraints on the presence of the feature in the data.

Field Type Description Required
minFraction double

Minimum fraction of examples that have this feature.

No
minCount int64

Minimum number of examples that have this feature.

No

tensorflow.metadata.v0.FeaturePresenceWithinGroup

Records constraints on the presence of a feature inside a “group” context (e.g., .presence inside a group of features that define a sequence).

Field Type Description Required
required bool No

tensorflow.metadata.v0.FixedShape

Specifies a fixed shape for the feature’s values. The immediate implication is that each feature has a fixed number of values. Moreover, these values can be parsed in a multi-dimensional tensor using the specified axis sizes. The FixedShape defines a lexicographical ordering of the data. For instance, if there is a FixedShape { dim {size:3} dim {size:2} } then tensor0=field[0] then tensor0=field[1] then tensor1=field[2] then tensor1=field[3] then tensor2=field[4] then tensor2=field[5]

The FixedShape message is identical with the TensorFlow TensorShape proto message.

Field Type Description Required
dim Dim[]

The dimensions that define the shape. The total number of values in each example is the product of sizes of each dimension.

No

tensorflow.metadata.v0.ValueCount

Limits on maximum and minimum number of values in a single example (when the feature is present). Use this when the minimum value count can be different than the maximum value count. Otherwise prefer FixedShape.

Field Type Description Required
min int64 No
max int64 No

tensorflow.metadata.v0.IntDomain

Encodes information for domains of integer values. Note that FeatureType could be either INT or BYTES.

Field Type Description Required
name string

Id of the domain. Required if the domain is defined at the schema level. If so, then the name must be unique within the schema.

No
min int64

Min and max values for the domain.

No
max int64 No
isCategorical bool

If true then the domain encodes categorical values (i.e., ids) rather than ordinal values.

No

tensorflow.metadata.v0.FloatDomain

Encodes information for domains of float values. Note that FeatureType could be either INT or BYTES.

Field Type Description Required
name string

Id of the domain. Required if the domain is defined at the schema level. If so, then the name must be unique within the schema.

No
min float

Min and max values of the domain.

No
max float No

tensorflow.metadata.v0.StringDomain

Encodes information for domains of string values.

Field Type Description Required
name string

Id of the domain. Required if the domain is defined at the schema level. If so, then the name must be unique within the schema.

No
value string[]

The values appearing in the domain.

No

tensorflow.metadata.v0.BoolDomain

Encodes information about the domain of a boolean attribute that encodes its TRUE/FALSE values as strings, or 0=false, 1=true. Note that FeatureType could be either INT or BYTES.

Field Type Description Required
name string

Id of the domain. Required if the domain is defined at the schema level. If so, then the name must be unique within the schema.

No
trueValue string

Strings values for TRUE/FALSE.

No
falseValue string No

tensorflow.metadata.v0.StructDomain

Domain for a recursive struct. NOTE: If a feature with a StructDomain is deprecated, then all the child features (features and sparse_features of the StructDomain) are also considered to be deprecated. Similarly child features can only be in environments of the parent feature.

Field Type Description Required
feature Feature[] No
sparseFeature SparseFeature[] No

tensorflow.metadata.v0.NaturalLanguageDomain

Natural language text.

tensorflow.metadata.v0.ImageDomain

Image data.

tensorflow.metadata.v0.MIDDomain

Knowledge graph ID, see: https://www.wikidata.org/wiki/Property:P646

tensorflow.metadata.v0.URLDomain

A URL, see: https://en.wikipedia.org/wiki/URL

tensorflow.metadata.v0.TimeDomain

Time or date representation.

Field Type Description Required
stringFormat string (oneof)

Expected format that contains a combination of regular characters and special format specifiers. Format specifiers are a subset of the strptime standard.

No
integerFormat IntegerTimeFormat (oneof)

Expected format of integer times.

No

tensorflow.metadata.v0.TimeOfDayDomain

Time of day, without a particular date.

Field Type Description Required
stringFormat string (oneof)

Expected format that contains a combination of regular characters and special format specifiers. Format specifiers are a subset of the strptime standard.

No
integerFormat IntegerTimeOfDayFormat (oneof)

Expected format of integer times.

No

DataSource

Defines a Data Source that can be used source Feature data

Field Type Description Required
type SourceType No
fieldMapping map<string, string>

Defines mapping between fields in the sourced data and fields in parent FeatureTable.

No
timestampColumn string

Must specify timestamp column name

No
datePartitionColumn string

(Optional) Specify partition column useful for file sources

No
fileOptions FileOptions (oneof) No
bigqueryOptions BigQueryOptions (oneof) No
kafkaOptions KafkaOptions (oneof) No
kinesisOptions KinesisOptions (oneof) No

DataSource.FileOptions

Defines options for DataSource that sources features from a file

Field Type Description Required
fileFormat string

File Format of the file containing the features

No
fileUrl string

Target URL of file to retrieve and source features from. s3://path/to/file for AWS S3 storage gs://path/to/file for GCP GCS storage file:///path/to/file for local storage

No

DataSource.BigQueryOptions

Defines options for DataSource that sources features from a BigQuery Query

Field Type Description Required
tableRef string

Full table reference in the form of [project:dataset.table]

No

DataSource.KafkaOptions

Defines options for DataSource that sources features from Kafka messages. Each message should be a Protobuf that can be decoded with the generated Java Protobuf class at the given class path

Field Type Description Required
bootstrapServers string

Comma separated list of Kafka bootstrap servers. Used for feature tables without a defined source host[:port]]

No
topic string

Kafka topic to collect feature data from.

No
classPath string

Classpath to the generated Java Protobuf class that can be used to decode Feature data from the obtained Kafka message

No

DataSource.KinesisOptions

Defines options for DataSource that sources features from Kinesis records. Each record should be a Protobuf that can be decoded with the generated Java Protobuf class at the given class path

Field Type Description Required
region string

AWS region of the Kinesis stream

No
streamName string

Name of the Kinesis stream to obtain feature data from.

No
classPath string

Classpath to the generated Java Protobuf class that can be used to decode Feature data from the obtained Kinesis record

No

FeatureSpecV2

Field Type Description Required
name string

Name of the feature. Not updatable.

No
valueType Enum

Value type of the feature. Not updatable.

No
labels map<string, string>

Labels for user defined metadata on a feature

No

FeatureTable

Field Type Description Required
spec FeatureTableSpec

User-specified specifications of this feature table.

No
meta FeatureTableMeta

System-populated metadata for this feature table.

No

FeatureTableSpec

Field Type Description Required
name string

Name of the feature set. Must be unique. Not updated.

No
entities string[]

List names of entities to associate with the Features defined in this Feature Table. Not updatable.

No
features FeatureSpecV2[]

List of features specifications for each feature defined with this feature table.

No
labels map<string, string>

User defined metadata

No
maxAge Duration

Features in this feature table can only be retrieved from online serving younger than max age. Age is measured as the duration of time between the feature’s event timestamp and when the feature is retrieved Feature values outside max age will be returned as unset values and indicated to end user

No
batchSource DataSource

Batch/Offline DataSource to source batch/offline feature data. Only batch DataSource can be specified (ie source type should start with ‘BATCH_’)

No
streamSource DataSource

Stream/Online DataSource to source stream/online feature data. Only stream DataSource can be specified (ie source type should start with ‘STREAM_’)

No

FeatureTableMeta

Field Type Description Required
createdTimestamp Timestamp

Time where this Feature Table is created

No
lastUpdatedTimestamp Timestamp

Time where this Feature Table is last updated

No
revision int64

Auto incrementing revision no. of this Feature Table

No

Store

Store provides a location where Feast reads and writes feature values. Feature values will be written to the Store in the form of FeatureRow elements. The way FeatureRow is encoded and decoded when it is written to and read from the Store depends on the type of the Store.

For example, a FeatureRow will materialize as a row in a table in BigQuery but it will materialize as a key, value pair element in Redis.

Field Type Description Required
name string

Name of the store.

No
type StoreType

Type of store.

No
subscriptions Subscription[]

Feature sets to subscribe to.

No
redisConfig RedisConfig (oneof) No
bigqueryConfig BigQueryConfig (oneof) No
cassandraConfig CassandraConfig (oneof) No
redisClusterConfig RedisClusterConfig (oneof) No

Store.RedisConfig

Field Type Description Required
host string No
port int32 No
initialBackoffMs int32

Optional. The number of milliseconds to wait before retrying failed Redis connection. By default, Feast uses exponential backoff policy and “initialbackoffms” sets the initial wait duration.

No
maxRetries int32

Optional. Maximum total number of retries for connecting to Redis. Default to zero retries.

No
flushFrequencySeconds int32

Optional. How often flush data to redis

No

Store.BigQueryConfig

Field Type Description Required
projectId string No
datasetId string No
stagingLocation string No
initialRetryDelaySeconds int32 No
totalTimeoutSeconds int32 No
writeTriggeringFrequencySeconds int32

Required. Frequency of running BQ load job and flushing all collected rows to BQ table

No

Store.CassandraConfig

Field Type Description Required
host string No
port int32 No

Store.RedisClusterConfig

Field Type Description Required
connectionString string

List of Redis Uri for all the nodes in Redis Cluster, comma separated. Eg. host1:6379, host2:6379

No
initialBackoffMs int32 No
maxRetries int32 No
flushFrequencySeconds int32

Optional. How often flush data to redis

No
keyPrefix string

Optional. Append a prefix to the Redis Key

No
enableFallback bool

Optional. Enable fallback to another key prefix if the original key is not present. Useful for migrating key prefix without re-ingestion. Disabled by default.

No
fallbackPrefix string

Optional. This would be the fallback prefix to use if enable_fallback is true.

No

Store.Subscription

Field Type Description Required
project string

Name of project that the feature sets belongs to. This can be one of - [project_name] - * If an asterisk is provided, filtering on projects will be disabled. All projects will be matched. It is NOT possible to provide an asterisk with a string in order to do pattern matching.

No
name string

Name of the desired feature set. Asterisks can be used as wildcards in the name. Matching on names is only permitted if a specific project is defined. It is disallowed If the project name is set to “*” e.g. - * can be used to match all feature sets - my-feature-set* can be used to match all features prefixed by “my-feature-set” - my-feature-set-6 can be used to select a single feature set

No
exclude bool

All matches with exclude enabled will be filtered out instead of added

No

FeatureSetReference

Defines a composite key that refers to a unique FeatureSet

Field Type Description Required
project string

Name of the project

No
name string

Name of the FeatureSet

No

IngestionJob

Represents Feast Injestion Job

Field Type Description Required
id string

Job ID assigned by Feast

No
externalId string

External job ID specific to the runner. For DirectRunner jobs, this is identical to id. For DataflowRunner jobs, this refers to the Dataflow job ID.

No
status IngestionJobStatus No
source Source

Source this job is reading from.

No
stores Store[]

Store this job is writing to.

No
featureSetReferences FeatureSetReference[]

List of Feature Set References

No

SpecsStreamingUpdateConfig

Config for bi-directional communication channel between Core Service and Ingestion Job

Field Type Description Required
source KafkaSourceConfig

out-channel for publishing new FeatureSetSpecs (by Core). IngestionJob use it as source of existing FeatureSetSpecs and new real-time updates

No
ack KafkaSourceConfig

ack-channel for sending acknowledgments when new FeatureSetSpecs is installed in Job

No

FeatureSetSpecAck

Field Type Description Required
featureSetReference string No
featureSetVersion int32 No
jobName string No

GetFeatureSetRequest

Request for a single feature set

Field Type Description Required
project string

Name of project the feature set belongs to. If omitted will default to ‘default’ project.

No
name string

Name of feature set (required).

No

GetFeatureSetResponse

Response containing a single feature set

Field Type Description Required
featureSet FeatureSet No

ListFeatureSetsRequest

Retrieves details for all versions of a specific feature set

Field Type Description Required
filter Filter No

ListFeatureSetsResponse

Field Type Description Required
featureSets FeatureSet[] No

GetEntityRequest

Request for a single entity

Field Type Description Required
name string

Name of entity (required).

No
project string

Name of project the entity belongs to. If omitted will default to ‘default’ project.

No

GetEntityResponse

Response containing a single entity

Field Type Description Required
entity Entity No

ListEntitiesRequest

Retrieves details for all versions of a specific entity

Field Type Description Required
filter Filter No

ListEntitiesResponse

Field Type Description Required
entities Entity[] No

ListFeaturesRequest

Field Type Description Required
filter Filter No

ListFeaturesResponse

Field Type Description Required
features map<string, FeatureSpec> No

ListStoresRequest

Field Type Description Required
filter Filter No

ListStoresResponse

Field Type Description Required
store Store[] No

ApplyEntityRequest

Field Type Description Required
spec EntitySpecV2

If project is unspecified, will default to ‘default’ project. If project specified does not exist, the project would be automatically created.

No
project string

Name of project that this entity belongs to.

No

ApplyEntityResponse

Field Type Description Required
entity Entity No

ApplyFeatureSetRequest

Field Type Description Required
featureSet FeatureSet

Feature set version If project is unspecified, will default to ‘default’ project. If project specified does not exist, the project would be automatically created.

No

ApplyFeatureSetResponse

Field Type Description Required
featureSet FeatureSet No
status Status No

GetFeastCoreVersionRequest

GetFeastCoreVersionResponse

Field Type Description Required
version string No

UpdateStoreRequest

Field Type Description Required
store Store No

UpdateStoreResponse

Field Type Description Required
store Store No
status Status No

CreateProjectRequest

Request to create a project

Field Type Description Required
name string

Name of project (required)

No

CreateProjectResponse

Response for creation of a project

ArchiveProjectRequest

Request for the archival of a project

Field Type Description Required
name string

Name of project to be archived

No

ArchiveProjectResponse

Response for archival of a project

ListProjectsRequest

Request for listing of projects

ListProjectsResponse

Response for listing of projects

Field Type Description Required
projects string[]

List of project names (archived projects are filtered out)

No

ListIngestionJobsRequest

Request for listing ingestion jobs

Field Type Description Required
filter Filter No

ListIngestionJobsResponse

Response from listing ingestion jobs

Field Type Description Required
jobs IngestionJob[] No

RestartIngestionJobRequest

Request to restart ingestion job

Field Type Description Required
id string

Job ID assigned by Feast

No

RestartIngestionJobResponse

Response from restartingan injestion job

StopIngestionJobRequest

Request to stop ingestion job

Field Type Description Required
id string

Job ID assigned by Feast

No

StopIngestionJobResponse

Request from stopping an ingestion job

GetFeatureStatisticsRequest

Field Type Description Required
featureSetId string

Feature set to retrieve the statistics for. A fully qualified feature set id in the format of project/feature_set must be provided.

No
features string[]

Optional filter which filters returned statistics by selected features. These features must be present in the data that is being processed.

No
store string

Optional filter to select store over which the statistics will retrieved. Only historical stores are allowed.

No
startDate Timestamp

Optional start and end dates over which to filter statistical data Start date is inclusive, but end date is not. Only dates are supported, not times. Cannot be used with dataset_ids. If this period spans multiple days, unaggregatable statistics will be dropped.

No
endDate Timestamp No
ingestionIds string[]

Optional list of ingestion Ids by which to filter data before retrieving statistics. Cannot be used with the date ranges If multiple dataset ids are provided, unaggregatable statistics will be dropped.

No
forceRefresh bool

Setting this flag to true will force a recalculation of statistics and overwrite results currently in the cache, if any.

No

GetFeatureStatisticsResponse

Field Type Description Required
datasetFeatureStatisticsList DatasetFeatureStatisticsList

Contains statistics for the requested data. Due to the limitations of TFDV and Facets, only a single dataset can be returned in, despite the message being of list type.

No

UpdateFeatureSetStatusRequest

Field Type Description Required
reference FeatureSetReference

FeatureSetReference of FeatureSet to update

No
status FeatureSetStatus

Target status

No

UpdateFeatureSetStatusResponse

ApplyFeatureTableRequest

Field Type Description Required
project string

Optional. Name of the Project to apply the Feature Table to. If unspecified, will apply FeatureTable to the default project.

No
tableSpec FeatureTableSpec

Feature Table specification to apply

No

ApplyFeatureTableResponse

Field Type Description Required
table FeatureTable No

GetFeatureTableRequest

Field Type Description Required
project string

Optional. Name of the Project to retrieve the Feature Table from. If unspecified, will apply FeatureTable to the default project.

No
name string

Name of the FeatureTable to retrieve.

No

GetFeatureTableResponse

Field Type Description Required
table FeatureTable

The Feature Table retrieved.

No

ListFeatureTablesRequest

Field Type Description Required
filter Filter

Filter used when listing Feature Tables

No

ListFeatureTablesResponse

Field Type Description Required
tables FeatureTable[]

List of matching Feature Tables

No

ListFeatureSetsRequest.Filter

Field Type Description Required
project string

Name of project that the feature sets belongs to. This can be one of - [project_name] - * If an asterisk is provided, filtering on projects will be disabled. All projects will be matched. It is NOT possible to provide an asterisk with a string in order to do pattern matching. If unspecified this field will default to the default project ‘default’.

No
featureSetName string

Name of the desired feature set. Asterisks can be used as wildcards in the name. Matching on names is only permitted if a specific project is defined. It is disallowed If the project name is set to “*” e.g. - * can be used to match all feature sets - my-feature-set* can be used to match all features prefixed by “my-feature-set” - my-feature-set-6 can be used to select a single feature set

No
labels map<string, string>

User defined metadata for feature set. Feature sets with all matching labels will be returned.

No
status FeatureSetStatus

Filter by FeatureSet’s current status Project and Feature Set name still must be specified (could be “*”)

No

ListEntitiesRequest.Filter

Field Type Description Required
project string

Optional. Specifies the name of the project to list Entities in. It is NOT possible to provide an asterisk with a string in order to do pattern matching. If unspecified, this field will default to the default project ‘default’.

No
labels map<string, string>

Optional. User defined metadata for entity. Entities with all matching labels will be returned.

No

ListFeaturesRequest.Filter

Field Type Description Required
labels map<string, string>

User defined metadata for feature. Features with all matching labels will be returned.

No
entities string[]

List of entities contained within the featureSet that the feature belongs to. Only feature sets with these entities will be searched for features.

No
project string

Name of project that the feature sets belongs to. Filtering on projects is disabled. It is NOT possible to provide an asterisk with a string in order to do pattern matching. If unspecified this field will default to the default project ‘default’.

No

ListStoresRequest.Filter

Field Type Description Required
name string

Name of desired store. Regex is not supported in this query.

No

ListIngestionJobsRequest.Filter

Field Type Description Required
id string

Filter by Job ID assigned by Feast

No
featureSetReference FeatureSetReference

Filter by ingestion job target feature set.

No
storeName string

Filter by Name of store

No

ListFeatureTablesRequest.Filter

Field Type Description Required
project string

Optional. Specifies the name of the project to list Feature Tables in. If unspecified would list Feature Tables in the default project.

No
labels map<string, string>

Optional. Feature Tables with all matching labels will be returned. If unspecified would list Feature Tables without filtering by labels.

No

tensorflow.metadata.v0.DatasetFeatureStatisticsList

A list of features statistics for different datasets. If you wish to compare different datasets using this list, then the DatasetFeatureStatistics entries should all contain the same list of features.

Field Type Description Required
datasets DatasetFeatureStatistics[] No

DirectRunnerConfigOptions

Field Type Description Required
targetParallelism int32

* Controls the amount of target parallelism the DirectRunner will use. Defaults to the greater of the number of available processors and 3. Must be a value greater than zero.

No
deadLetterTableSpec string

BigQuery table specification, e.g. PROJECTID:DATASETID.PROJECT_ID

No
tempLocation string

A pipeline level default location for storing temporary files. Support Google Cloud Storage locations or local path

No

DataflowRunnerConfigOptions

Field Type Description Required
project string

Project id to use when launching jobs.

No
region string

The Google Compute Engine region for creating Dataflow jobs.

No
workerZone string

GCP availability zone for operations.

No
serviceAccount string

Run the job as a specific service account, instead of the default GCE robot.

No
network string

GCE network for launching workers.

No
subnetwork string

GCE subnetwork for launching workers. e.g. regions/asia-east1/subnetworks/mysubnetwork

No
workerMachineType string

Machine type to create Dataflow worker VMs as.

No
autoscalingAlgorithm string

The autoscaling algorithm to use for the workerpool.

No
usePublicIps bool

Specifies whether worker pools should be started with public IP addresses.

No
tempLocation string

A pipeline level default location for storing temporary files. Support Google Cloud Storage locations, e.g. gs://bucket/object

No
maxNumWorkers int32

The maximum number of workers to use for the workerpool.

No
deadLetterTableSpec string

BigQuery table specification, e.g. PROJECTID:DATASETID.PROJECT_ID

No
labels map<string, string>

Labels to apply to the dataflow job

No
diskSizeGb int32

Disk size to use on each remote Compute Engine worker instance

No
enableStreamingEngine bool

Run job on Dataflow Streaming Engine instead of creating worker VMs

No
workerDiskType string

Type of persistent disk to be used by workers

No
kafkaConsumerProperties map<string, string>

Kafka consumer configuration properties

No

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

SourceType

Name Description
INVALID
KAFKA

SourceType

Name Description
INVALID
KAFKA

FeatureSetStatus

Name Description
STATUS_INVALID
STATUS_PENDING
STATUS_JOB_STARTING
STATUS_READY

FeatureSetJobDeliveryStatus

Name Description
STATUS_IN_PROGRESS
STATUS_DELIVERED

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

FeatureSetStatus

Name Description
STATUS_INVALID
STATUS_PENDING
STATUS_JOB_STARTING
STATUS_READY

DataSource.SourceType

Type of Data Source.

Name Description
INVALID
BATCH_FILE
BATCH_BIGQUERY
STREAM_KAFKA
STREAM_KINESIS

DataSource.SourceType

Type of Data Source.

Name Description
INVALID
BATCH_FILE
BATCH_BIGQUERY
STREAM_KAFKA
STREAM_KINESIS

feast.types.ValueType.Enum

Name Description
INVALID
BYTES
STRING
INT32
INT64
DOUBLE
FLOAT
BOOL
BYTES_LIST
STRING_LIST
INT32_LIST
INT64_LIST
DOUBLE_LIST
FLOAT_LIST
BOOL_LIST

Store.StoreType

Name Description
INVALID
REDIS

Redis stores a FeatureRow element as a key, value pair.

The Redis data types used (https://redis.io/topics/data-types): - key: STRING - value: STRING

Encodings: - key: byte array of RedisKey (refer to feast.storage.RedisKey) - value: byte array of FeatureRow (refer to feast.types.FeatureRow)

BIGQUERY

BigQuery stores a FeatureRow element as a row in a BigQuery table.

Table name is derived is the same as the feature set name.

The entities and features in a FeatureSetSpec corresponds to the fields in the BigQuery table (these make up the BigQuery schema). The name of the entity spec and feature spec corresponds to the column names, and the value_type of entity spec and feature spec corresponds to BigQuery standard SQL data type of the column.

The following BigQuery fields are reserved for Feast internal use. Ingestion of entity or feature spec with names identical to the following field names will raise an exception during ingestion.

columnname | columndatatype | description ====================|==================|================================ - eventtimestamp | TIMESTAMP | event time of the FeatureRow - createdtimestamp | TIMESTAMP | processing time of the ingestion of the FeatureRow - ingestionid | STRING | unique id identifying groups of rows that have been ingested together - job_id | STRING | identifier for the job that writes the FeatureRow to the corresponding BigQuery table

BigQuery table created will be partitioned by the field “event_timestamp” of the FeatureRow (https://cloud.google.com/bigquery/docs/partitioned-tables).

The following table shows how ValueType in Feast is mapped to BigQuery Standard SQL data types (https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types):

BYTES : BYTES STRING : STRING INT32 : INT64 INT64 : IN64 DOUBLE : FLOAT64 FLOAT : FLOAT64 BOOL : BOOL BYTESLIST : ARRAY STRINGLIST : ARRAY INT32LIST : ARRAY INT64LIST : ARRAY DOUBLELIST : ARRAY FLOATLIST : ARRAY BOOL_LIST : ARRAY

The column mode in BigQuery is set to “Nullable” such that unset Value in a FeatureRow corresponds to NULL value in BigQuery.

CASSANDRA

Unsupported in Feast 0.3

REDIS_CLUSTER

Store.StoreType

Name Description
INVALID
REDIS

Redis stores a FeatureRow element as a key, value pair.

The Redis data types used (https://redis.io/topics/data-types): - key: STRING - value: STRING

Encodings: - key: byte array of RedisKey (refer to feast.storage.RedisKey) - value: byte array of FeatureRow (refer to feast.types.FeatureRow)

BIGQUERY

BigQuery stores a FeatureRow element as a row in a BigQuery table.

Table name is derived is the same as the feature set name.

The entities and features in a FeatureSetSpec corresponds to the fields in the BigQuery table (these make up the BigQuery schema). The name of the entity spec and feature spec corresponds to the column names, and the value_type of entity spec and feature spec corresponds to BigQuery standard SQL data type of the column.

The following BigQuery fields are reserved for Feast internal use. Ingestion of entity or feature spec with names identical to the following field names will raise an exception during ingestion.

columnname | columndatatype | description ====================|==================|================================ - eventtimestamp | TIMESTAMP | event time of the FeatureRow - createdtimestamp | TIMESTAMP | processing time of the ingestion of the FeatureRow - ingestionid | STRING | unique id identifying groups of rows that have been ingested together - job_id | STRING | identifier for the job that writes the FeatureRow to the corresponding BigQuery table

BigQuery table created will be partitioned by the field “event_timestamp” of the FeatureRow (https://cloud.google.com/bigquery/docs/partitioned-tables).

The following table shows how ValueType in Feast is mapped to BigQuery Standard SQL data types (https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types):

BYTES : BYTES STRING : STRING INT32 : INT64 INT64 : IN64 DOUBLE : FLOAT64 FLOAT : FLOAT64 BOOL : BOOL BYTESLIST : ARRAY STRINGLIST : ARRAY INT32LIST : ARRAY INT64LIST : ARRAY DOUBLELIST : ARRAY FLOATLIST : ARRAY BOOL_LIST : ARRAY

The column mode in BigQuery is set to “Nullable” such that unset Value in a FeatureRow corresponds to NULL value in BigQuery.

CASSANDRA

Unsupported in Feast 0.3

REDIS_CLUSTER

IngestionJobStatus

Status of a Feast Ingestion Job

Name Description
UNKNOWN

Job status is not known.

PENDING

Import job is submitted to runner and currently pending for executing

RUNNING

Import job is currently running in the runner

COMPLETED

Runner’s reported the import job has completed (applicable to batch job)

ABORTING

When user sent abort command, but it’s still running

ABORTED

User initiated abort job

ERROR

Runner’s reported that the import job failed to run or there is a failure during job

SUSPENDING

job has been suspended and waiting for cleanup

SUSPENDED

job has been suspended

IngestionJobStatus

Status of a Feast Ingestion Job

Name Description
UNKNOWN

Job status is not known.

PENDING

Import job is submitted to runner and currently pending for executing

RUNNING

Import job is currently running in the runner

COMPLETED

Runner’s reported the import job has completed (applicable to batch job)

ABORTING

When user sent abort command, but it’s still running

ABORTED

User initiated abort job

ERROR

Runner’s reported that the import job failed to run or there is a failure during job

SUSPENDING

job has been suspended and waiting for cleanup

SUSPENDED

job has been suspended

ApplyFeatureSetResponse.Status

TODO: 0 should correspond to invalid rather than NO_CHANGE

Name Description
NO_CHANGE

Latest feature set is consistent with provided feature set

CREATED

New feature set created

ERROR

Error occurred while trying to apply changes

UPDATED

Changes detected and updated successfully

UpdateStoreResponse.Status

Name Description
NO_CHANGE

Existing store config matching the given store id is identical to the given store config.

UPDATED

New store created or existing config updated.

ApplyFeatureSetResponse.Status

TODO: 0 should correspond to invalid rather than NO_CHANGE

Name Description
NO_CHANGE

Latest feature set is consistent with provided feature set

CREATED

New feature set created

ERROR

Error occurred while trying to apply changes

UPDATED

Changes detected and updated successfully

UpdateStoreResponse.Status

Name Description
NO_CHANGE

Existing store config matching the given store id is identical to the given store config.

UPDATED

New store created or existing config updated.

FeatureSetStatus

Name Description
STATUS_INVALID
STATUS_PENDING
STATUS_JOB_STARTING
STATUS_READY

FeatureSetStatus

Name Description
STATUS_INVALID
STATUS_PENDING
STATUS_JOB_STARTING
STATUS_READY