We show the importance of correlation in creating an investment portfolio and how to generate an efficient frontier based on two assets. The same problem is scalable to multiple assets.
kaggle command API¶
We explore what commands can be executed using Kaggle.
!kaggle -h
kaggle competitions¶
We can see that there are many competitions available.
ref : Name of the competition.deadline: When the competition is due for its submission.category: Competitions have different categories as followsfeatured: These competitions usually have prize money (i.e., 25k-100k) split between the top 3 teams.playground: These datasets are just to play and explore.gettingStarted: Beginners should try exploring these datasets to get new skillsmasters: Machine learning experts can try these datasets and win prize money >100k.research: These are datasets for research purposes.recruitment: Firms are using kaggle to identify new hires so you can try these datasets to build up your profile.reward: Total prize money for the top x3 teams entering the competition. teamCount: Number of teams that are entering the competition.userHasEntered: Have you entered the specified competition?!kaggle c list
gettingStarted competitions are shown below
!kaggle c list --category gettingStarted
masters competitions are shown below
!kaggle c list --category masters
playground competitions are shown below
!kaggle c list --category playground
featured competitions are shown below
!kaggle competitions list --category featured
titanic dataset leaderboard¶
We can explore the best scores of each team/user for a specific competition. We can see that many people have a top score of 1.0 which means they have perfect prediction!
!kaggle competitions leaderboard titanic --show
titanic dataset¶
We will explore one of the most well-known datasets, that is the titanic dataset. Always list all the files associated to the competition of interest before downloading as some of the requied files can be >100MB. In the titanic dataset, the files are small since they are < 1MB.
!kaggle competitions files titanic
We can easily download the files into our selected directory
!kaggle competitions download titanic
We can also evaluate our machine learning submission scores in Kaggle by competition. In the below example, the competition was the home-credit-default-risk competition
!kaggle competitions submissions -c home-credit-default-risk
We have learnt how to use the kaggle API to explore kaggle competitions and download datasets. We also learnt how to obtain our submitted machine learning model performance scores based on our competition submissions. For more details see the Kaggle API Github or see the documentation on the Kaggle website.
The steps required to build a machine learning model are given as follows:
Exploratory Data Analysis
Feature Engineering
Model Selection
Model Hyperparameter Training
I will describe the major actions that need to be performed in the above and include Python modules that I have written to assist me through building a machine learning model.
Contents
This summarizes my Python setup for data science/finance/economics research. It includes some of the useful Python packages, JupyterLab extensions, and programs I find useful in my development.
Natural Language Processing (NLP) is a field of artificial intelligence that focuses on extracting and understanding important information from text. Some applications are as follows:
Specifically for finance, NLP can be used in the following applications * Extracting the buy/sell sentiment on a corpus of sell-side analysts reports on a specific stock. * Understanding economic sentiment from conference calls, online job postings, inflation chatter, e-invoicing, etc. * Extracting the amount of media attention to political events * Extracing a long-term view of the global economic sentiment from Warran Buffet's annual reports.
There are several popular NLP libraries as shown below and I found the nice summary below from a ActiveWizards:
Important packages to install using conda for Natural Language Processing are:
In this blog series, we will go through the following packages where [x] is available and [ ] is in the pipeline:
re
nltk
nltk
gensim
nltk
spacy
polyglot
sklearn
BeautifulSoup:Timeseries forecasting can be generally split into two categories
1) Signal processing. Signal processing is typically what is used in engineering and econometrics. ARIMA/GARCH models attempt to filter out the 'signals' from the noise and extrapolate the signals into the future. Famous models for interest rate pricing are 2-factor models (i.e., Vasicek models, Cox-Ingersoll-Ross) models. CIR models allow for mean-reversion,
THis code shows how to fit a trivariate t-copula and simulate it