numerapi.base_api module¶
Parts of the API that is shared between Signals and Classic
- class numerapi.base_api.Api(public_id: str | None = None, secret_key: str | None = None, verbosity: str = 'INFO', show_progress_bars: bool = True)[source]¶
Bases:
objectWrapper around the Numerai API
- check_new_round(hours: int = 12) bool[source]¶
Check if a new round has started within the last hours.
- Parameters:
hours (int, optional) – timeframe to consider, defaults to 12
- Returns:
True if a new round has started, False otherwise.
- Return type:
bool
Example
>>> NumerAPI().check_new_round() False
- check_round_open() bool[source]¶
Check if a round is currently open.
- Returns:
True if a round is currently open for submissions, False otherwise.
- Return type:
bool
Example
>>> NumerAPI().check_round_open() False
- diagnostics(model_id: str, diagnostics_id: str | None = None) Dict[source]¶
Fetch results of diagnostics run
- Parameters:
model_id (str) – Target model UUID (required for accounts with multiple models)
diagnostics_id (str, optional) – id returned by “upload_diagnostics”
- Returns:
diagnostic results with the following content:
validationCorrMean (float)
validationCorrSharpe (float)
examplePredsCorrMean (float)
validationMmcStd (float)
validationMmcSharpe (float)
validationCorrPlusMmcSharpeDiff (float)
validationMmcStdRating (float)
validationMmcMeanRating (float)
validationCorrPlusMmcSharpeDiffRating (float)
- perEraDiagnostics (list) each with the following fields:
era (int)
examplePredsCorr (float)
validationAlpha (float)
validationBmc (float)
validationCorr (float)
validationCorrV4 (float)
validationFeatureCorrMax (float)
validationFeatureNeutralCorr (float)
validationFeatureNeutralCorrV3
validationMmc (float)
validationFncV4 (float)
validationIcV2 (float)
validationRic (float)
validationBmc (float)
validationCorrPlusMmcStd (float)
validationMmcMean (float)
validationCorrStdRating (float)
validationCorrPlusMmcSharpe (float)
validationMaxDrawdownRating (float)
validationFeatureNeutralCorrMean (float)
validationCorrPlusMmcMean (float)
validationFeatureCorrMax (float)
status (string),
validationCorrMeanRating (float)
validationFeatureNeutralCorrMeanRating (float)
validationCorrSharpeRating (float)
validationCorrPlusMmcMeanRating (float)
message (string)
validationMmcSharpeRating (float)
updatedAt (datetime)
validationFeatureCorrMaxRating (float)
validationCorrPlusMmcSharpeRating (float)
trainedOnVal (bool)
validationCorrStd (float)
erasAcceptedCount (int)
validationMaxDrawdown (float)
validationCorrPlusMmcStdRating (float)
validationAdjustedSharpe (float)
validationApy (float)
validationAutocorr (float)
validationCorrCorrWExamplePreds (float)
validationCorrMaxDrawdown (float)
validationCorrV4CorrWExamplePreds (float)
validationCorrV4MaxDrawdown (float)
validationCorrV4Mean (float)
validationBmcMean (float)
validationCorrV4Sharpe (float)
validationCorrV4Std (float)
validationFeatureNeutralCorrV3Mean (float)
validationFeatureNeutralCorrV3MeanRating (float)
validationFncV4CorrWExamplePreds (float)
validationFncV4MaxDrawdown (float)
validationFncV4Mean (float)
validationFncV4Sharpe (float)
validationFncV4Std (float)
validationIcV2CorrWExamplePreds (float)
validationIcV2MaxDrawdown (float)
validationIcV2Mean (float)
validationIcV2Sharpe (float)
validationIcV2Std (float)
validationRicCorrWExamplePreds (float)
validationRicMaxDrawdown (float)
validationRicMean (float)
validationRicSharpe (float)
validationRicStd (float)
validationAlphaCorrWExamplePreds (float)
validationAlphaMaxDrawdown (float)
validationAlphaMean (float)
validationAlphaSharpe (float)
validationAlphaStd (float)
- Return type:
dict
Example
>>> napi = NumerAPI(secret_key="..", public_id="..") >>> model_id = napi.get_models()['uuazed'] >>> api.upload_diagnostics("prediction.cvs", model_id=model_id) '93c46857-fed9-4594-981e-82db2b358daf' >>> napi.diagnostic(model_id) {"validationCorrMean": 0.53231, ... }
- download_dataset(filename: str, dest_path: str | None = None, round_num: int | None = None) str[source]¶
Download specified file for the given round.
- Parameters:
filename (str, optional) – file to be downloaded
dest_path (str, optional) – complete path where the file should be stored, defaults to the same name as the source file
round_num (int, optional) – tournament round you are interested in. defaults to the current round
- Returns:
path of the downloaded file
- Return type:
str
Example
>>> filenames = NumerAPI().list_datasets() >>> NumerAPI().download_dataset(filenames[0]}")
- download_submission(submission_id: str | None = None, model_id: str = '', dest_path: str = '') str[source]¶
Download previous submissions from numerai
- Parameters:
submission_id (str, optional) – the submission to be downloaded
model_id (str, optional) – if provided, the latest submission of that model gets downloaded
dest_path (str, optional) – where to save the downloaded file
- Returns:
path to downloaded file
- Return type:
str
Example
>>> # fetch latest submission >>> api = NumerAPI(secret_key="..", public_id="..") >>> model_id = api.get_models()["uuazed"] >>> api.download_submission(model_id=model_id) >>> # fetch older submssion >>> ids = api.submission_ids(model_id) >>> import random; submission_id = random.choice(ids)["id"] >>> api.download_submission(submission_id=submission_id)
- get_account() Dict[source]¶
Get all information about your account!
- Returns:
user information including the fields:
assignedEthAddress (str)
availableNmr (decimal.Decimal)
availableUsd (decimal.Decimal)
email (str)
id (str)
insertedAt (datetime)
mfaEnabled (bool)
status (str)
username (str)
apiTokens (list) each with the following fields:
name (str)
public_id (str)
scopes (list of str)
models * username * id * submissions * v2Stake
status (str)
txHash (str)
- Return type:
dict
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> api.get_account() {'apiTokens': [ {'name': 'tokenname', 'public_id': 'BLABLA', 'scopes': ['upload_submission', 'stake', ..] }, ..], 'assignedEthAddress': '0x0000000000000000000000000001', 'availableNmr': Decimal('99.01'), 'email': 'username@example.com', 'id': '1234-ABC..', 'insertedAt': datetime.datetime(2018, 1, 1, 2, 16, 48), 'mfaEnabled': False, 'status': 'VERIFIED', 'username': 'cool username', }
- get_account_leaderboard(limit: int = 50, offset: int = 0) List[Dict][source]¶
Get the current account leaderboard
- Parameters:
limit (int) – number of items to return (optional, defaults to 50)
offset (int) – number of items to skip (optional, defaults to 0)
- Returns:
list of leaderboard entries
Each dict contains the following items:
username (str)
displayName (str)
rank (int)
nmrStaked (decimal.Decimal)
v2Corr20 (float)
cort20 (float)
corrV4 (float)
fncV4 (float)
icV2 (float)
mmc (float)
ric (float)
return1y (float)
return3m (float)
returnAllTime (float)
return1yNmr (decimal.Decimal)
return3mNmr (decimal.Decimal)
returnAllTimeNmr (decimal.Decimal)
- Return type:
list of dicts
Example
>>> numerapi.NumerAPI().get_account_leaderboard() [{'username': 'leonidas', 'rank': 1, 'nmrStaked': Decimal('3034.00'), ... }]
- get_current_round(tournament: int | None = None) int | None[source]¶
Get number of the current active round.
- Parameters:
tournament (int) – ID of the tournament (optional)
- Returns:
number of the current active round
- Return type:
int
Example
>>> NumerAPI().get_current_round() 104
- get_models(tournament: int | None = None) Dict[source]¶
Get mapping of account model names to model ids for convenience
- Parameters:
tournament (int) – ID of the tournament (optional)
- Returns:
modelname->model_id mapping, string->string
- Return type:
dict
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model = api.get_models() {'uuazed': '9b157d9b-ce61-4ab5-9413-413f13a0c0a6'}
- intra_round_scores(model_id: str)[source]¶
Fetch intra-round scores for your model.
While only the final scores are relevant for payouts, it might be interesting to look how your scores evolve throughout a round.
- Parameters:
model_id (str)
- Returns:
list of intra-round model performance entries
For each entry in the list, there is a dict with the following content:
roundNumber (int)
- intraRoundSubmissionScores (dict)
date (datetime)
day (int)
displayName (str): name of the metric
payoutPending (float)
payoutSettled (float)
percentile (float)
value (float): value of the metric
- Return type:
list of dicts
- list_datasets(round_num: int | None = None) List[str][source]¶
List of available data files
- Parameters:
round_num (int, optional) – tournament round you are interested in. defaults to the current round
- Returns:
filenames
- Return type:
list of str
Example
>>> NumerAPI().list_datasets() [ "numerai_training_data.csv", "numerai_training_data.parquet", "numerai_validation_data.csv", "numerai_validation_data.parquet" ]
- list_rounds(number: int | None = None, target: str | None = None, status: str | None = None, limit: int | None = None) List[Dict][source]¶
List rounds with the filters supported by the round resolver.
- Parameters:
number (int, optional) – round number filter
target (str, optional) – round target filter
status (str, optional) – round status filter. One of upcoming, open, resolving, or resolved
limit (int, optional) – maximum number of rounds to return
- Returns:
round entries matching the provided filters
- Return type:
list of dicts
- model_upload(file_path: str, tournament: int | None = None, model_id: str | None = None, data_version: str | None = None, docker_image: str | None = None) str[source]¶
Upload pickled model to numerai.
- Parameters:
file_path (str) – pickle file, needs to endwith .pkl
tournament (int) – ID of the tournament (optional)
model_id (str) – Target model UUID
data_version (str, optional) – which data version to use. ID or name. Check available options with ‘model_upload_data_versions’
docker_image (str, optional) – which docker image to use. ID or name. Check available options with ‘model_upload_docker_images’
- Returns:
model_upload_id
- Return type:
str
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model_id = api.get_models()['uuazed'] >>> api.model_upload("example.pkl", model_id=model_id) '93c46857-fed9-4594-981e-82db2b358daf'
- model_upload_data_versions() Dict[source]¶
Get available data version for model uploads
- Returns:
name to ID mapping
- Return type:
dict[str, str]
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> api.model_upload_data_versions() {'v4.1': 'a76bafa1-b25a-4f22-9add-65b528a0f3d0'}
- model_upload_docker_images() Dict[source]¶
Get available docker images for model uploads
- Returns:
name to ID mapping
- Return type:
dict[str, str]
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> api.model_upload_docker_images() {'Python 3.10': 'c72ae05e-2831-4c50-b20f-c2fe01c206ef', 'Python 3.9': '5a32b827-cd9a-40a9-a99d-e58401120a0b', ... }
- modelid_to_modelname(model_id: str) str[source]¶
Get model name from a model_id.
- Parameters:
model_id (str)
- Returns:
modelname
- Return type:
str
- models_of_account(account) Dict[str, str][source]¶
Get all models (name and id) of an account
- Parameters:
account (str) – account name
- Returns:
modelname->model_id mapping, string->string
- Return type:
dict
Example
>>> api = NumerAPI() >>> NumerAPI().models_of_account("uuazed") {'uuazed': '9b157d9b-ce61-4ab5-9413-413f13a0c0a6', ...}
- pending_model_payouts(tournament: int | None = None) Dict[source]¶
Fetch actual and pending payouts for the authenticated user’s models.
- Parameters:
tournament (int, optional) – tournament filter, defaults to the API instance tournament
- Returns:
payout groups with actual and pending lists
- Return type:
dict
- pipeline_status(date: str | None = None) Dict[source]¶
Get status of Numerai’s scoring pipeline
- Parameters:
date (str, optional) – date in YYYY-MM-DD format. Defaults to today.
- Returns:
- pipeline status information including the following fields:
dataReadyAt (str)
isScoringDay (bool)
resolvedAt (datetime)
scoredAt (datetime)
startedAt (datetime)
tournament (str)
- Return type:
dict
Example
>>> napi = NumerAPI() >>> napi.pipeline_status()
- raw_query(query: str, variables: Dict | None = None, authorization: bool = False, *, retries: int = 3, delay: int = 5, backoff: int = 2)[source]¶
Send a raw request to the Numerai’s GraphQL API.
This function allows to build your own queries and fetch results from Numerai’s GraphQL API. Checkout https://medium.com/numerai/getting-started-with-numerais-new-tournament-api-77396e895e72 for an introduction and https://api-tournament.numer.ai/ for the documentation.
- Parameters:
query (str) – your query
variables (dict, optional) – dict of variables
authorization (bool, optional) – does the request require authorization, defaults to False
retries (int) – for 5XX errors, how often should numerapi retry
delay (int) – in case of retries, how many seconds to wait between tries
backoff (int) – in case of retries, multiplier to increase the delay between retries
- Returns:
Result of the request
- Return type:
dict
- Raises:
ValueError – if something went wrong with the requests. For example, this could be a wrongly formatted query or a problem at Numerai’s end. Have a look at the error messages, in most cases the problem is obvious.
Example
>>> query = '''query($tournament: Int!) {rounds(tournament: $tournament number: 0) {number}}''' >>> args = {'tournament': 1} >>> NumerAPI().raw_query(query, args) {'data': {'rounds': [{'number': 104}]}}
- round_model_performances(username: str) List[Dict][source]¶
Fetch round model performance of a user.
DEPRECATED - please use round_model_performances_v2 instead
- round_model_performances_v2(model_id: str)[source]¶
Fetch round model performance of a user.
DEPRECATED - please use submission_scores instead when possible.
- Parameters:
model_id (str)
- Returns:
list of round model performance entries
For each entry in the list, there is a dict with the following content:
atRisk (float)
corrMultiplier (float or None)
mmcMultiplier (float or None)
roundPayoutFactor (float or None)
roundNumber (int)
roundOpenTime (datetime)
roundResolveTime (datetime)
roundResolved (bool)
roundTarget (str)
- submissionScores (dict)
date (datetime)
day (int)
displayName (str): name of the metric
payoutPending (float)
payoutSettled (float)
percentile (float)
value (float): value of the metric
- Return type:
list of dicts
- set_bio(model_id: str, bio: str) bool[source]¶
Set bio field for a model id.
- Parameters:
model_id (str) – Target model UUID
bio (str)
- Returns:
if the bio was changed successfully
- Return type:
bool
Example
>>> napi = numerapi.NumerAPI() >>> model_id = napi.get_models()["uuazed"] >>> napi.set_bio(model_id, "This model stinks.") True
- set_global_data_dir(directory: str)[source]¶
Set directory used for downloading files
- Parameters:
directory (str) – directory to be used
- set_link(model_id: str, link_text: str, link: str) bool[source]¶
Set link field for a model id.
- Parameters:
model_id (str) – Target model UUID
link_test (str)
link (str)
- Returns:
if the bio was changed successfully
- Return type:
bool
Example
>>> napi = numerapi.NumerAPI() >>> model_id = napi.get_models()["uuazed"] >>> napi.set_link(model_id, "buy my predictions", "numerbay.ai") True
- set_submission_webhook(model_id: str, webhook: str | None = None) bool[source]¶
Set a model’s submission webhook used in Numerai Compute. Read More: https://docs.numer.ai/tournament/compute
- Parameters:
model_id (str) – Target model UUID
webhook (str) – The compute webhook to trigger this model
- Returns:
confirmation that your webhook has been set
- Return type:
bool
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> api.set_submission_webhook(model_id="..", webhook="..") True
- stake_change(nmr: float | str, action: str = 'decrease', model_id: str = '') Dict[source]¶
Change stake by value NMR.
- Parameters:
nmr (float or str) – amount of NMR you want to increase/decrease
action (str) – increase or decrease
model_id (str) – Target model UUID (required for accounts with multiple models)
- Returns:
stake information with the following content:
dueDate (datetime)
status (str)
requestedAmount (decimal.Decimal)
type (str)
- Return type:
dict
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model = api.get_models()['uuazed'] >>> api.stake_change(10, "decrease", model) {'dueDate': None, 'requestedAmount': decimal.Decimal('10'), 'type': 'decrease', 'status': ''}
- stake_decrease(nmr: float | str, model_id: str) Dict[source]¶
Decrease your stake by value NMR.
- Parameters:
nmr (float or str) – amount of NMR you want to reduce
model_id (str) – Target model UUID (required for accounts with multiple models)
tournament (int) – ID of the tournament (optional, defaults to 8)
- Returns:
stake information with the following content:
dueDate (datetime)
status (str)
requestedAmount (decimal.Decimal)
type (str)
- Return type:
dict
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model = api.get_models()['uuazed'] >>> api.stake_decrease(10, model) {'dueDate': None, 'requestedAmount': decimal.Decimal('10'), 'type': 'decrease', 'status': ''}
- stake_drain(model_id: str | None = None) Dict[source]¶
Completely remove your stake.
- Parameters:
model_id (str) – Target model UUID
- Returns:
stake information with the following content:
dueDate (datetime)
status (str)
requestedAmount (decimal.Decimal)
type (str)
drain (bool)
- Return type:
dict
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model_id = api.get_models()['uuazed'] >>> api.stake_drain(model_id) {'dueDate': None, 'requestedAmount': decimal.Decimal('11000000'), 'type': 'decrease', 'status': '', 'drain": True}
- stake_increase(nmr: float | str, model_id: str) Dict[source]¶
Increase your stake by value NMR.
- Parameters:
nmr (float or str) – amount of additional NMR you want to stake
model_id (str) – Target model UUID (required for accounts with multiple models)
tournament (int) – ID of the tournament (optional, defaults to 8)
- Returns:
stake information with the following content:
dueDate (datetime)
status (str)
requestedAmount (decimal.Decimal)
type (str)
- Return type:
dict
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model = api.get_models()['uuazed'] >>> api.stake_increase(10, model) {'dueDate': None, 'requestedAmount': decimal.Decimal('10'), 'type': 'increase', 'status': ''}
- submission_ids(model_id: str)[source]¶
Get all submission ids from a model
- Parameters:
model_id (str)
- Returns:
list of submissions
For each entry in the list, there is a dict with the following content:
insertedAt (datetime)
filename (str)
id (str)
- Return type:
list of dicts
- Example:
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model_id = napi.get_models()["uuazed"] >>> api.submission_ids(model_id)
- submission_scores(model_id: str, display_name: str | None = None, version: str | None = None, day: int | None = None, resolved: bool | None = None, last_n_rounds: int | None = None, distinct_on_round: bool | None = None) List[Dict][source]¶
Fetch submission score history for a model.
- Parameters:
model_id (str) – target model UUID
display_name (str, optional) – score metric name filter
version (str, optional) – score version filter
day (int, optional) – day filter
resolved (bool, optional) – resolved-state filter
tournament (int, optional) – tournament filter, defaults to the API instance tournament
last_n_rounds (int, optional) – limit by most recent rounds
distinct_on_round (bool, optional) – keep only the latest score per round after applying other filters
- Returns:
list of submission score entries
- Return type:
list of dicts
- upload_diagnostics(file_path: str = 'predictions.csv', tournament: int | None = None, model_id: str = '', df: DataFrame | None = None) str[source]¶
Upload predictions to diagnostics from file.
- Parameters:
file_path (str) – CSV file with predictions that will get uploaded
tournament (int) – ID of the tournament (optional, defaults to None)
model_id (str) – Target model UUID (required for accounts with multiple models)
df (pandas.DataFrame) – pandas DataFrame to upload, if function is given df and file_path, df will be uploaded.
- Returns:
diagnostics_id
- Return type:
str
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model_id = api.get_models()['uuazed'] >>> api.upload_diagnostics("prediction.cvs", model_id=model_id) '93c46857-fed9-4594-981e-82db2b358daf' >>> # upload from pandas DataFrame directly: >>> api.upload_diagnostics(df=predictions_df, model_id=model_id)
- upload_predictions(file_path: str = 'predictions.csv', model_id: str | None = None, df: DataFrame | None = None, data_datestamp: int | None = None, timeout: None | float | Tuple[float, float] = (10, 600)) str[source]¶
Upload predictions from file. Will read TRIGGER_ID from the environment if this model is enabled with a Numerai Compute cluster setup by Numerai CLI.
- Parameters:
file_path (str) – CSV file with predictions that will get uploaded
model_id (str) – Target model UUID (required for accounts with multiple models)
df (pandas.DataFrame) – pandas DataFrame to upload, if function is given df and file_path, df will be uploaded.
data_datestamp (int) – Data lag, in case submission is done using data from the previous day(s).
timeout (float|tuple(float,float)) – waiting time (connection timeout, read timeout)
- Returns:
submission_id
- Return type:
str
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> model_id = api.get_models()['uuazed'] >>> api.upload_predictions("prediction.cvs", model_id=model_id) '93c46857-fed9-4594-981e-82db2b358daf' >>> # upload from pandas DataFrame directly: >>> api.upload_predictions(df=predictions_df, model_id=model_id)
- v3_stake_auth(submission_id: str, staker: str, amount: float | str | None = None, max_amount: float | str | None = None) Dict[source]¶
Issue a staking v3 authorization for a selected submission.
- Parameters:
submission_id (str) – submission id for the selected submission
staker (str) – staker wallet address
amount (float or str, optional) – max stake amount for the authorization. Retained as a backwards-compatible alias for max_amount.
max_amount (float or str, optional) – max stake amount for the authorization.
- Returns:
authorization payload with the following fields:
authorizationSigner (str)
authorizationDigest (str)
chainId (str)
deadline (str)
maxAmount (str)
amount (str) alias for maxAmount
modelId (str)
nmrAddress (str)
nonce (str)
roundId (str)
signature (str)
staker (str)
stakingAddress (str)
submissionId (str)
submissionHash (str)
tournamentId (str)
- Return type:
dict
- v3_stake_claim(round_id: int | str, model_id: str, staker: str) Dict[source]¶
Fetch a staking v3 claim proof for a model and staker.
- Parameters:
round_id (int or str) – round id
model_id (str) – model id
staker (str) – staker wallet address
- Returns:
claim payload with the following fields:
apiModelId (str)
burnAmountWei (str)
merkleRoot (str)
modelId (str)
payoutAmountWei (str)
proof (list of str)
roundId (str)
staker (str)
submissionId (str)
tournamentId (str)
- Return type:
dict
- v3_stake_config() Dict[source]¶
Fetch staking v3 contract configuration.
- Returns:
staking v3 configuration with the following fields:
address (str)
authorizationSigner (str)
nmrAddress (str)
owner (str)
paused (bool)
pendingOwner (str)
serviceWallet (str)
- Return type:
dict
- v3_stake_round(round_id: int | str) Dict[source]¶
Fetch staking v3 round status by round id.
- Parameters:
round_id (int or str) – round id
- Returns:
staking v3 round data with the following fields:
closeTime (str)
merkleRoot (str)
openTime (str)
payoutFactor (str)
remainingBurn (str)
remainingPayout (str)
resolveTime (str)
resolved (bool)
roundId (str)
stakeCap (str)
stakeThreshold (str)
state (str)
totalPayout (str)
totalStaked (str)
tournamentId (str)
- Return type:
dict
- wallet_transactions() List[source]¶
Get all transactions in your wallet.
- Returns:
List of dicts with the following structure:
from (str)
posted (bool)
status (str)
to (str)
txHash (str)
amount (decimal.Decimal)
time (datetime)
tournament (int)
- Return type:
list
Example
>>> api = NumerAPI(secret_key="..", public_id="..") >>> api.wallet_transactions() [{'amount': Decimal('1.000000000000000000'), 'from': '0x000000000000000000000000000000000000313bc', 'status': 'confirmed', 'time': datetime.datetime(2023, 4, 19, 13, 28, 45), 'to': '0x000000000000000000000000000000000006621', 'tournament': None, 'txHash': '0xeasdfkjaskljf314451234', 'type': 'withdrawal'}, ... ]