A comprehensive guide to competing at Numerai
Introduction
Nice to meet you. My name is tit_BTCQASH. In this article, I will carefully explain how to read the Numerai homepage and how to submit predictions using Google Colab.
If you have any questions or feedback, contact me on Twitter!
Numerai, founded in 2015, runs a weekly data science tournament at https://numer.ai. Data scientists from around the world can download clean, obfuscated data for free, build a model and submit to predict the stock market. Because it’s obfuscated data, you don’t have to know anything about finance to participate. Participants have skin in the game — a stake applied to their predictions. Good predictions are rewarded with more crypto (Numeraire) and bad predictions are burned (the crypto stake is destroyed forever).
Numerai takes the best contributions and builds a meta model which is used to inform the trades of their global equity hedge fund. Participants can submit with or without a stake — but Numerai only uses staked predictions.
Participants don’t have to prepare the data themselves, but are required to optimize the data given by the team and submit our own predictions.
blog_UKI’s article: Numerai Tournament: Blending Traditional Quantitative Approach & Modern Machine Learning is recommended to read once you grasp the basics of Numerai.
Table of contents
- How to read Numerai’s homepage and glossary
- How to optimize the features and obfuscated stock price data given by the Numerai team and submit predictions
- How to read Model Diagnostics
- Conclusion
- Convenient links related to Numerai and Community
1. How to read Numerai’s homepage and glossary
The URL of the Numerai homepage is https://numer.ai/tournament.
Numeraire (NMR) Token: An ERC-20-based token which is used to stake on Numerai. NMR can be earned by competing in the Numerai tournament and by making technical contributions to Numerai.
NMR tokens can be used across the Erasure Protocol, on which Numerai is built. Right now, the apps on the Erasure protocol are Numerai, Numerai Signals and Erasure Bay, but the protocol is open for anyone to build on.
Corr: Correlation coefficient between the submitted stock price prediction result and the target answer.
MMC: Meta model contribution (MMC) indicates how valuable your model’s predictions are to Numerai’s meta model. Predictions that are very original and moderately accurate will have higher MMC than very accurate but unoriginal predictions.Learn more about MMC here.
FNC: Feature neutral correlation (FNC) is your model’s correlation with the target, after its predictions have been neutralized to all of Numerai’s features.Learn more about FNC here.
Corr / MMC / FNC Rep: Rep (reputation) is the weighted average of that metric over the past 20 rounds, and is what’s used to rank users on the leaderboard. Learn more about reputation here.
Stake: People stake NMR to indicate how confident they are in their model’s predictions. Competitors who make good predictions are rewarded, and the NMR staked on poorly performing models gets destroyed (burned). The minimum stake value is 0.01NMR. You can participate without staking to learn more about how your model performs before you decide to stake.
Payouts: Rewards received according to the NMR staked. You can earn or burn up to 25% of your stake each week (each round is four weeks). Learn more about staking and payouts.
Upper homepage components
DOCS: Links to Numerai’s documentation which includes rules, FAQ, and a new users section.
CHAT: A chat space that includes channels for announcements, general discussion, data science, and more. The Numerai team is very available on RocketChat and their profiles have the `team` tag. Team members are primarily on Pacific Time. Join here.
FORUM: A long-form space for discussions related to Numerai, data science, stake strategy and more. Mainly information sharing within the community. Read here.
LEADERBOARD: Ranking of prediction results submitted to Numerai. Can be sorted by stake size, Rep, MMC and more.
ACCOUNT: There are four links: WALLET, MODELS, SETTINGS, LOGOUT.
● Wallet: This page is about depositing and withdrawing NMR tokens. You can deposit by sending NMR tokens to the address in your wallet. You cannot send other tokens to this address.
The withdraw tab allows you to withdraw your NMR tokens to addresses outside of your Numerai wallet. Be sure before requesting a withdrawal that the external address you are withdrawing to is suited to receive the NMR ERC-20 token. Lost NMR cannot be recovered.
● Models: This page adds / deletes models to be submitted to Numerai. You can add / remove models by pressing ADD NEW MODEL / ABSORB EXISTING ACCOUNT.
Learn more about multi-model accounts and account absorption.
● SETTINGS: Email, Password, 2-step authentication settings (2FA) and API key settings.
2FA is advised because the account can hold NMR. If you use 2FA, you must save your recovery codes. Numerai will not reset your account if you lose access to your 2FA device.
Lower homepage components:
Model Information: This page is about ranking, Reputation, and MMC Rep of the model (TIT_BTCQASH this time). You can switch models by pressing the ↓ button.
Data information: Data download link for the latest round and upload link for forecast results.
Stake: Settings to adjust the NMR stake values. There are different types of stake methods available, such as Corr and Corr + MMC. For example, Corr allows you to stake NMR only on correlation, and Corr + MMC allows you to stake NMR on correlation and MMC.
Pending Payouts: A table of forecast payouts for each round.
2. How to optimize the features and obfuscated stock price data given by the Numerai team
If you want to submit to Numerai right away, Kaggle Notebooks from tournament participants katsu1110 and Carlo Lepelaars are both very helpful.
In this article, I will explain a step further (where to improve with the code, etc.) from the above articles and the official example model. I hope this article makes the process easy to understand, and raises the number of competitors in the tournament. (And if you find a good way to raise Corr, please tell me secretly.)
The code introduced this time can be run on Google Colab. If you press the Run button, you can create a submission file, so try using it.
Basics before explaining the code
i) Numerai dataset structure
The dataset can be downloaded from the latest round data download link. There is a detailed explanation in UKI’s article mentioned above: so I will briefly describe the contents.
numerai_training_data.csv is a csv file that contains training data. numerai_tournament_data.csv is a csv file that contains data for validation.
id: Label for the encrypted stock.
era: A label about how long the data was collected. If the era is the same, it means that the data was collected during the same period.
data_type: There are four values: train, validation, test, live. train is the data for training, validation is the data for verification, test is the data for Numerai to test, and live is the data for the current round.
feature: Binned feature quantity. Features are binned into 5 levels: 0,0.25,0.5,0.75,1. Features are in groups labeled as: “feature_intelligence”, “feature_wisdom”, “feature_charisma”, “feature_dexterity”, “feature_strength”, “feature_constitution”.
target: Binned teacher data. The target is also binned into 5 levels: 0,0.25,0.5,0.75,1. The target data is given in numerai_training_data.csv, but it is NAN in the test and live data of numerai_tournament_data.csv.
ii) Flow until data submission
- Data reading
- Feature engineering
- Machine learning
- About the strength of the model
- Preparation of csv file in which the prediction result is written
- Neutralization method
2 A. Data reading
I will quote (partially edit) the data reading part from the article by Carlo Lepelaars. Call download_current_data (DIR) to download the latest round of data to the directory specified by DIR. Calling train, val, test = load_data (DIR, reduce_memory = True) will store the data separately for train, val, test data.
2 B. Feature engineering
The features of the Numerai dataset have low correlation with each other, and some results can be obtained without feature engineering. Also, reducing features using techniques such as PCA tends to lower Corr, which is not very good.
(*This is just the result of my verification. I do not deny the possibility that Corr could improve.)
I think that what is effective in Numerai is to reduce the correlation between features while increasing the features. Officially, the number of features will be increased from 310 to 3100 (https://twitter.com/numerai/status/1347361350205415425). It seems that even feature engineering may not be necessary from now on, but I will briefly introduce how to handle features.
First of all, if you look at the train data, you can see that it is roughly divided into 6 types: “feature_intelligence”, “feature_wisdom”, “feature_charisma”, “feature_dexterity”, “feature_strength”, and “feature_constitution”.
I quote the code from Carlo Lepelaars’s article, but the mean, deviation, skewness, etc. of these features are useful features.Therefore, call train = get_group_stats (train) to add these features to the train data and so on.
If you have a PC with plenty of memory, it will be a good feature amount if you include feature difference data, interaction feature amount, etc. (Corr will increase by about 20%). When I run on Google Colab, it crashes, so I will post only the code. There are discussions in the forum about how to reduce this.
Since feature engineering such as that used in Kaggle can be used as it is in Numerai, good Corr and Sharpe ratio can be obtained by playing with train, val, and test data. One of the tasks required to obtain good results with Numerai is feature engineering, so this is one of the elements of continued experimentation.
2 C. Machine learning
What you need to consider when applying machine learning to the Numerai dataset is:
i) What machine learning method to use (LightGBM, XGBoost, neural networks, etc.)
ii) What hyperparameters to use
iii) Whether to stack the prediction results etc.
This time, I will use LightGBM considering the calculation time. Except for train data, id, era, and data_type are not necessary for machine learning. Train the remaining feature_ ○○ as an explanatory variable and target as teacher data. Using the trained data, predictive data is also created for the Validation data and Live data included in the val.
If you consider i) to iii), the values of Corr etc. will improve, so this part is also one of the elements of replay.
2 D. About the strength of the model
Calculate spearman, payout, numerai_sharpe, mae to estimate the strength of the model in the Validation data. The larger the spearman, payout and numerai_sharpe, the better.
Among these, first of all, you can make a good model by finding the condition that the value of spearman is large (0.025 or more is a guide).
(* If you focus only on Corr, various problems may occur. I think there is some disagreement with those who are familiar with Numerai, but since this is an article for people who submit prediction results for the first time, let me express it like this.)
The explanation of terms is as follows:
spearman: The average value of Correlation. The higher, the better (reference is 0.022 to 0.04.)
payout: The ratio of the average return
numerai_sharpe: The ratio of the average return divided by the standard deviation. The higher the better (1 or more as a guide)
mae: mean absolute error
2 E. Preparation of csv file in which the prediction result is written
Write the file for neutrize to submission_file.csv. This file requires id and prediction columns, and id is required to be in the order of Validation data and test data (+ Live data). Please note that if the order is different, it will be rejected on the Numerai side.
2 F. Neutralization method
By linearly regressing Example_model (a sample model officially distributed by Numerai) and your own model, you can improve the Sharpe ratio while reducing the correlation between a single feature and the prediction result. However, if you overdo it, Corr will drop significantly, so I think about 0.3 to 0.5 is good. One of the elements of the game is what kind of model to neutralize and how much to neutralize.
Submit the obtained neutralized_submission_file.csv from Upload predictions on the Numerai homepage and you’re done.
3. How to Read Model Diagnostics
Validation Sharpe: Sharpe ratio in Validation data should be 1 or more
Validation Mean: Corr average value in Validation data should be around 0.025 ~
Feature Neutral Mean: Corr mean value when neutralizing all features (not very helpful)
Validation SD: Standard deviation of correlation between validation data and predicted value for each Era (not very helpful)
Feature Exposure: An index showing how well the feature amount and the prediction result are balanced. The smaller the better.
Max Drawdown: Maximum drawdown -0.05 or less is a guide
Corr + MMC Sharpe: Sharpe ratio of Corr and MMC combined
MMC Mean: Mean of MMC Corr with Example Preds Correlation with sample model 0.5 ~ 0.8 is a guide
4. Conclusion
Currently I’m around 470th on the leaderboard, which might not be considered a high rank, but it is a positive return.
Although it is quite different from the code posted this time, I am using a model with seven types of neutralization while maintaining Correlation to some extent.
If you enjoyed reading this article, for tips: NMR:0x000000000000000000
0000000000000000021d96
5. Convenient links related to Numerai and Community
Numerai has a very active community that shares information and builds tools for each other, for example:
● Website that makes it easy to see how your model’s performance compares to others:
● Mobile app to track your payouts from the tournament, for iOS and Android by Ceunen
● Numerai Advent Calendar 2020 by Kunigaku:
● Numerati, a dashboard for the tournament by Jo-Fai Chow:
● Easy guide with Google Colab by community member SurajP:
● Walkthrough of basic tables, graphs and visualizations a new participant may want to investigate, by OmniAnalytics:
