Robinhood recently launched a credit card offering 3% cash back on all purchases. While the app is fantastic, it has one major drawback: I can’t download my transactions as a CSV or Quicken file. If this were a website, I could easily use Chrome DevTools to inspect the HTTP requests and replay them in a Python script. Unfortunately there is no web portal for the card - all I have is the app. So how can we inspect the network traffic?

Turns out one way to do this is to set up a web proxy on a separate machine. This computer will then record all of the traffic between the iOS app and Robinhood servers. Once we have the requests, we can replay similar requests in our script to obtain the transaction data. Essentially, we man-in-the-middle attack ourselves.

This process is easy to accomplish with mitmproxy. mitmproxy is an open-source proxy tool that allows users to intercept, inspect, and modify HTTP and HTTPS traffic, making it ideal for tasks like this. Simply install the tool and run the proxy:

brew install mitmproxy
...
🍺  mitmproxy was successfully installed!

mitmproxy

You should see a blank screen like this:

mitmproxy running with no requests

Now run ifconfig to determine the machine’s IP address. This is usually listed under en0 or eth0. With this information, we can point our iPhone at the proxy to start capturing requests. On the iOS device, go to Settings > WiFi and click the blue info circle next to the current WiFi connection. Scroll all the way to the bottom and click “Configure Proxy”. Select “Manual” and then input the IP address with port 8080.

iOS manual proxy configuration

The traffic we’re interested in is likely HTTPS, so we need to complete one more step by trusting the mitmproxy CA. Navigate to mitm.it on your iPhone and you should see the following page:

iOS web page with mitm cert for download

After downloading the file, we must install it and enable full trust. Go back to Settings > General > VPN & Device Management > mitmproxy. Click “install” in the upper right-hand corner. Now navigate to Settings > About > Certificate Trust Settings all the way at the bottom. Enable full trust for the mitmproxy certificate.

If everything is working correctly, mitmproxy will start to show network requests from your iPhone. Here is what my console looks like from installing the mitmproxy CA and opening the credit card app:

mitmproxy running with requests

At this point, we can see we are working with a GraphQL API. Explore this interface for a bit, and you will eventually find the request responsible for fetching transactions. You can immediately filter down to these by pressing F and typing ~b transaction. This returns all requests with “transaction” in the body. Press enter to see the details on a given request. There are some headers that appear important:

content-type:     application/json
authorization:    Bearer eyJhb...
user-agent:       rhcardapp/1.35.0 CFNetwork/1498.700.2 Darwin/23.6.0
x-x1-client:      mobile-app-rh@1.35.0

And the GraphQL query:

{
  "query": "...",
  "operationName": "TransactionListQuery",
  "variables": {
    "q": {
      "creditCustomerId": "aSecretIDontWantToShareWithYou :)",
      "filters": {
        "values": []
      },
      "sortDetails": {
        "field": "TIME",
        "ascending": false
      },
      "limit": 40
    }
  }
}
---
query TransactionListQuery(
  $q: TransactionSearchRequest!
) {
  transactionSearch(q: $q) {
    items {
      id
      amountMicro
      transactionAt
      merchantDetails {
        merchantName
      }
    }
  }
}

With all this information, we can finally start to write our Python script. Let’s replay the same request to confirm things are working:

import requests

GRAPHQL_ENDPOINT = "https://api.robinhood.com/creditcard/graphql"
query = """
    query TransactionListQuery(
        $q: TransactionSearchRequest!
    ) {
        transactionSearch(q: $q) {
            items {
                id
                amountMicro
                transactionAt
                merchantDetails {
                    merchantName
                }
            }
        }
    }
"""
headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {auth_token}",
    "User-Agent": "rhcardapp/1.35.0 CFNetwork/1498.700.2 Darwin/23.6.0",
    "x-x1-client": "mobile-app-rh@1.35.0",
}
body = {
    "query": query,
    "variables": {
        "q": {
            "creditCustomerId": customer_id,
            "filters": {"values": []},
            "sortDetails": {"field": "TIME", "ascending": False},
            "limit": 40,
        }
    },
    "operationName": "TransactionListQuery",
}
response = requests.post(GRAPHQL_ENDPOINT, json=body, headers=headers)
results = response.json()

Now that we have our transactions, we can parse the data into a Transaction dataclass. This class can then be used to output any file format. Here, I used the Quiffen package to generate a QIF file:

from dataclasses import dataclass
from datetime import datetime
from decimal import Decimal

import quiffen

@dataclass
class Transaction:
    timestamp: datetime
    amount: Decimal
    merchant: str

qif = quiffen.Qif()
acc = quiffen.Account(name="RH Gold", desc="RH Gold credit card")
qif.add_account(acc)

for transaction in transactions:
    qif_transaction = quiffen.Transaction(
        date=transaction.timestamp,
        amount=transaction.amount,
        payee=transaction.merchant,
    )
    acc.add_transaction(qif_transaction, header=quiffen.AccountType.CREDIT_CARD)

qif.to_qif("./rh-cc-transactions.qif")

To turn this into a general tool anyone can use, we repeat the same process to obtain the authorization token and the customer id. That’s it! If you would like to use this tool yourself, the Github repo contains complete instructions.

Resources:

• mitmproxy
• robin_stocks - Unofficial Robinhood library for Python

Just toying around with my own personal project, I’ve found that the key to testing web apps in Go is to use dependency injection with high-order functions.

Dependency Injection

Dependency injection means that we want to supply everything our function requires to accomplish its goal. We don’t want to rely on any global state or outside services.

However, its unclear how to do this at first. Most tutorials start out having you write applications like this:

package main

import (
    "fmt"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

Source: Writing Web Applications - Golang Wiki

This isn’t a bad approach, but what happens when we want to add a database? Or an outside package to handle our sessions, such as Gorilla Session? Oftentimes people will instantiate a database manager or a session handler globally and call it a day. But that will cause you difficulty when you try to test your handler. It is better to write a function which will generate your handler for you.

package main

import (
    "fmt"
    "net/http"
    "github.com/markberger/database"
)

type AppDatabase interface {
    GetBacon() string
}

func homeHandler(db AppDatabase) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "Hi there, I love %s!", db.GetBacon())
    })
}

func main() {
    db := database.NewDatabase()
    http.HandleFunc("/", homeHandler(db))
    http.ListenAndServe(":8080", nil)
}

This is great, because we no longer have to rely on global variables. Also, this makes mocking a breeze. We simply have to create a mock database that fulfills the interface AppDatabase, which means we only have to implement one function, GetBacon.

Testing

Now that we have a program, we can start writing tests. The key to testing http.Handler is to use a httptest.ResponseRecorder found in the net/http/httptest package.

package main

import(
    "net/http"
    "net/http/httptest"
    "testing"
)

type MockDd struct {}

function (db MockDb) GetBacon() {
    return "bacon"
}

function TestHome(t *testing.T) {
    mockDb := MockDb{}
    homeHandle := homeHandler(mockDb)
    req, _ := http.NewRequest("GET", "", nil)
    w := httptest.NewRecorder()
    homeHandle.ServeHTTP(w, req)
    if w.Code != http.StatusOK {
        t.Errorf("Home page didn't return %v", http.StatusOK)
    }
}

Of course, all that boilerplate can get pretty redundant, especially when testing POST requests. Therefore, I’ve been using these functions to test my handlers.

package main

import (
    "net/http"
    "net/http/httptest"
    "net/url"
    "testing"
)

type HandleTester func(
    method string,
    params url.Values,
) *httptest.ResponseRecorder

// Given the current test runner and an http.Handler, generate a
// HandleTester which will test its given input against the
// handler.

func GenerateHandleTester(
    t *testing.T,
    handleFunc http.Handler,
) HandleTester {

    // Given a method type ("GET", "POST", etc) and
    // parameters, serve the response against the handler and
    // return the ResponseRecorder.

    return func(
        method string,
        params url.Values,
    ) *httptest.ResponseRecorder {

        req, err := http.NewRequest(
            method,
            "",
            strings.NewReader(params.Encode()),
        )
        if err != nil {
            t.Errorf("%v", err)
        }
        req.Header.Set(
            "Content-Type",
            "application/x-www-form-urlencoded; param=value",
        )
        w := httptest.NewRecorder()
        handleFunc.ServeHTTP(w, req)
        return w
    }
}

function TestHome(t *testing.T) {
    mockDb := MockDb{}
    homeHandle := homeHandler(mockDb)
    test := GenerateHandleTester(t, homeHandle)
    w := test("GET", url.Values{})
    if w.Code != http.StatusOK {
        t.Errorf("Home page didn't return %v", http.StatusOK)
    }
}

For a more thorough example of these methods in action, check out my toy project’s tests here.

If you have any better methods of testing web apps written in the standard library, feel free to leave a comment or send me email.

Why should I use Chef?

I don’t know any reasons for using Chef specifically, but using a configuration management tool can save you a lot of headaches. If you every break your virtual private server, you can always create a new one and redeploy your software in minutes without going through all of the motions by hand. While it might seem trivial now, you’ll be happy you invested the time six months later when you’ve completely forgotten what you did to get nginx working just right.

Assumptions

I’m assuming you have the following:

• A recent version of Ruby
• A recent version of RubyGems
• A linux server

If you are looking for a hosting service, I personally recommend Digital Ocean (referral link).

Setup

To start, install knife-solo and berkshelf.

gem install knife-solo
gem install berkshelf

Knife-solo is a tool which adds a few extensions to Chef’s command line tool, knife, to make life easier. Berkshelf is a dependency management tool for Chef cookbooks.

What’s a cookbook?

A cookbook is a collection of scripts which focus on configuring a certain aspect of the machine or installing a certain application. So, for example, there are nginx cookbooks which focus on installing and configuring nginx. Each one of the scripts inside a given cookbook is referred to as a recipe in order to further the kitchen analogy.

Let’s get cooking

Now that we have knife-solo, create a new directory and run the following command:

knife solo init chef_dir/

This will give you the following:

chef_dir/
    Berksfile
    cookbooks/
    data_bags/
    environments/
    nodes/
    roles/
    site-cookbooks/

You’ll notice that knife-solo has created two cookbook directories: cookbooks and site-cookbooks. Site cookbooks are meant for specific server configurations, like our virtual private server, while cookbooks are meant for general use cases, such as installing nginx.

Let’s create a new cookbook for our server.

knife cookbook create -o site-cookbooks/ vps

Now you’ll have this:

vps/
    attributes/
    definitions/
    files/
        default/
    libraries/
    providers/
    recipies/
        default.rb
    resources/
    templates/
        default/
    metadata.rb
    README.md
    CHANGELOG.md

In this tutorial we are just going to install some packages, configure ssh, and create a new user, so you can ignore most of these directories.

We’re going to be using the user cookbook and the openssh cookbook, so add the following lines to your metadata.rb file.

depends "user"
depends "openssh"

This is Chef’s way of knowing which cookbooks to have before attempting to run any of the recipes inside our new cookbook.

Now let’s open up recipies/default.rb and write our first recipe.

node.packages.each do |pkg|
    package pkg
end

# provision user account
include_recipe 'user::data_bag'

# provision ssh
include_recipe 'openssh'

First, we iterate over a list of packages and install each one. Then we execute the data_bag recipe found in the user cookbook followed by the default recipe for openssh.

Right now we are using the default settings found in each individual cookbook, and Chef has no idea what packages we want to install. To resolve our unknown variables and change the default settings we need to use the attributes folder.

Inside site-cookbooks/vps create attributes/default.rb and add the following:

default.packages = %w(vim git)

default.users = ['ourUser']
# Don't create an ssh key for us
default.user.ssh_keygen = false

default.openssh.server.permit_root_login = 'no'
default.openssh.server.password_authentication = 'no'
default.openssh.server.allow_groups = 'sudo'
default.openssh.server.port = '27984'
default.openssh.server.login_grace_time = '30'
default.openssh.server.use_p_a_m = 'no'
default.openssh.server.print_motd = 'no'

Great! Now Chef will install vim and git, create the ourUser account, and lock down ssh for us.

Now we need to tell user what we want the ourUser account to look like.

Go to chef_dir/data_bags/ and create a users directory. Create ourUser.json inside the users directory and add the following:

{
    "id": "ourUser",
    "password": "your_password",
    "groups": [
        "sudo"
    ],
    "ssh_keys": [
        "your_public_ssh_key"
    ]
}

Note that you need to generate the "your_password" string using mkpasswd -m sha-512.

Now we need to create a configuration file for Chef to use. Create a vps.json inside the nodes directory and place the following:

{
    "run_list": [
        "recipe[vps]"
    ]
}

Finally, we need to add our dependencies to the Berksfile so that Chef will install them on our linux box before executing our recipe. Your Berksfile should look like this:

source "https://supermarket.getchef.com"

cookbook 'user'
cookbook 'openssh'

That’s it! Now we can finally run Chef on our server.

knife solo bootstrap root@our_server nodes/vps.json

This will install Chef on the server, download the cookbooks in the Berksfile, and execute our default recipe in the vps cookbook.

Finally, we need to clean up the machine. However, we must login as ourUser since we’ve prevented root login in our Chef recipe.

knife solo clean ourUser@our_server

Hopefully this helped you set up Chef with your linux machine. Chef seems to be easy enough to use from this point once you’re able to somewhat see how all the pieces fit together. This is my first time using Chef so if I’ve made an error or committed some bad practices, please let me know in the comments or feel free to email me.

Also if you’re looking for a Chef cookbook to install Ghost, checkout the one I made to deploy this blog using sqlite3.

First make sure that your vps has the latest updates:

sudo apt-get update
sudo apt-get upgrade

Now install python and virtualenv:

sudo apt-get install build-essential python-dev python-pip
sudo pip install virtualenv

Make a folder for your website:

sudo mkdir -p /var/www/mysite
sudo chown -R <your user id> /var/www/mysite
cd /var/www/mysite

Setup virtualenv and install flask:

virtualenv .env --no-site-packages
source .env/bin/activate
pip install flask

Place your flask app in this folder. Make sure that your host is set to 0.0.0.0 and that your app is under if __name__ == '__main__':. If your app is in a function, uwsgi will not be able to call it.

Now is a good time to test your app with the flask development server to see if everything is working so far. If everything runs smoothly, install nginx and uwsgi:

deactivate
sudo apt-get install nginx uwsgi uwsgi-plugin-python

Next we must create a socket file for nginx to communicate with uwsgi:

cd /tmp/
touch mysite.sock
sudo chown www-data mysite.sock

By changing the owner of mysite.sock to www-data, nginx will be able to write to the socket. Now all we have to do is add our configuration files for nginx and uwsgi. First delete the default configuration for nginx:

cd /etc/nginx/sites-available
sudo rm default

Create a new configuration file mysite and add the following:

server {
    listen 80;
    server_tokens off;
    server_name www.mysite.com mysite.com;

     location / {
         include uwsgi_params;
         uwsgi_pass unix:/tmp/mysite.sock;
     }

     location /static {
         alias /var/www/mysite/static;
     }

     ## Only requests to our Host are allowed
     if ($host !~ ^(mysite.com|www.mysite.com)$ ) {
        return 444;
     }
}

In order to enable the site, we must link our configuration file to /etc/nginx/sites-enabled/:

sudo ln -s /etc/nginx/sites-available/mysite /etc/nginx/sites-enabled/mysite

The process is similar for uwsgi. Create the file /etc/uwsgi/apps-available/mysite.ini and add the following:

[uwsgi]
vhost = true
socket = /tmp/mysite.sock
venv = /var/www/mysite/.env
chdir = /var/www/mysite
module = app
callable = app

Module is the name of your python script and callable is the name of your flask instance. So if your flask site was in a file called mysite.py that looked like this:

from flask import Flask
my_app = Flask(__name__)

@my_app.route('/')
def hello_world():
    return 'Hello World!'

if __name__ == '__main__':
    my_app.run(host='0.0.0.0')

Your mysite.ini file would be:

module = mysite
callable = my_app

Link the configuration file to the enabled-apps folder:

sudo ln -s /etc/uwsgi/apps-available/mysite.ini /etc/uwsgi/apps-enabled/mysite.ini

Finally, restart nginx and uwsgi:

sudo service nginx restart
sudo service uwsgi restart

Thats it. If you notice any errors in my guide, please feel free to email me. Here are some tips in case you get stuck:

• Check that your flask site runs under virtualenv without any errors.
• Ensure that you can run your site with just uwsgi from the command line.
• If you run uwsgi with sudo you will change the owner of mysite.sock to root and this will create errors for nginx. Make sure that you change the owner back to www-data.
• If uwsgi cannot find your app, you probably have an issue with file permissions. In order to serve the site uwsgi must have executable permissions for your python script and your .env folder.
• The logs for nginx and uwsgi are /var/log/nginx/error.log and /var/log/uwsgi/app/mysite.log respectively. If nginx is working properly, you will want to look at /var/log/nginx/access.log.

As with any learning experience, there were plenty of frustrations along the way. Sadly, most of these frustration were caused by git and not the problems I was trying to solve. Before Summer of Code, I had only used git for personal projects and my knowledge was limited. The learning curve for git is extremely high, and it can be discouraging to spend half of your time fighting with git instead of hacking on your project. Additionally, there are some conventions beginners are not aware of when they start working on an open source project. Here are some of the things I wish I knew before I started hacking on Tahoe-LAFS.

Clone with SSH

If you fork a repository on Github, clone the repository with the ssh link. Otherwise, you will have to authenticate with Github each time you push a branch. For information on authenticating with Github via ssh keys, check out Github’s detailed guide.

Never Work on Master

One of the most important rules to remember is: never commit to the master branch. Master is pristine and can only be touched after multiple people have signed off on the code. Since we cannot commit code to master, all work must be done on a branch, which can be thought of as another copy of the code base. In order to create a branch based on master, use:

git branch <your_branch_name> master

Or if you are currently on master:

git branch <your_branch_name>

Or if you want to create a branch and check it out in the same command:

git checkout -b <your_branch_name>

Updating Master to Reflect Upstream

Since git is a decentralized version control system, there is no obvious way to sync your master branch with the project’s master branch. To do this, we first need to add a new remote to the project’s main repository.

git remote add upstream <link_to_main_repo>

Now we need to retrieve information about this new remote repository.

git fetch upstream

Finally, we can update the local master branch to reflect upstream’s master branch.

git checkout master
git merge upstream/master

Now our master branch is identical to the master branch on upstream.

Edit: In general, you shouldn’t have to worry about writing your patch on the latest version of upstream. Anyone who can commit to the project’s master branch should be able to merge your branch if it is based on a recent version of upstream. Thanks to Jed Brown for suggesting this change.

Fixing Commits

Oh no! After you committed that awesome patch, you noticed that you made a spelling mistake in your code. Even worse, there is also a spelling mistake in your commit message. Thankfully, both of these errors can easily be fixed. To fix the commit message, use:

git commit --amend

This will bring up the previous commit message. Fix the spelling mistake, save, and exit to recommit the patch.

Now to fix the spelling error in the patch. Fix the spelling error and make an additional commit. Don’t worry about the commit message. Then type git rebase -i HEAD~2:

pick 40e4369 My sweet bug-busting commit
pick ec48bcf My spelling fix
...
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#  x, exec = run command (the rest of the line) using shell

HEAD~2 means we want to rebase the two commits before our current position. Rebase is a powerful tool, and a proper explanation is outside the scope of this post. But, for our fix we want to use fixup on ec48bcf.

pick 40e4369 My sweet bug-busting commit
f    ec48bcf My spelling fix

Save and exit. Now a simple mistake is not breaking an otherwise useful commit. It’s important to note, that if the branch has already been pushed to Github, you will have to do a forced push to sync the branch.

git push -f origin <my_branch>

As a rule of thumb, if someone else is using your branch in any capacity, you should not force push the branch. Instead of rebasing to combine commits, leave the separate commit as is.

Check for Trailing Whitespace

This doesn’t have to do with git, but it is still worth knowing. Before making any commits, it’s good practice to run git diff to ensure you aren’t accidently committing any debug code. Additionally, check for any newlines, tabs, or spaces you may have added by accident, especially trailing whitespace. While trailing whitespace isn’t the end of the world, it annoys a lot of people. If you use Sublime Text, I recommend the Trailing Spaces plugin.

Thanks for reading!

9/10/13: The initial version of this post refered to upstream as trunk.