Project Serum - Decentralized Exchange on Solana

The Solana project Serum is a model for decentralized exchanges with on-chain limit orderbooks and matching engines. The primary selling point of a DEX is permissionless, non-custodial Defi and the Solana blockchain brings speed and low transaction cost to the table. With block times of 400ms, it can currently process ~50,000 transactions per second. This allows Serum to handle hundreds of orders per second per market on-chain.

The success of Serum can be measured in the number of projects it has spawned since inception. In this post we’re going through the basic setup of the Serum DEX and it’s concepts. I found online material to be scattered and outdated at times and hope to bring some clarity and a tutorial to this space. Let me know if you have any questions. What we’re setting up looks like the following screenshot. Everything discussed here is open source. The end result shall look like the following screenshot.

If you have not yet set up Rust and Solana, you can follow my post on the Solana development environment setup, start a local test validator and drop yourself some test SOL:

solana-test-validator --no-bpf-jit
solana airdrop 100

Build & Deploy the Serum DEX

Git clone the serum-dex repository, build and deploy the rust program to the local test validator:

git clone https://github.com/project-serum/serum-dex.git
cd serum-dex/dex
cargo build-bpf
solana deploy target/bpfel-unknown-unknown/release/serum_dex.so

After deployment, we get the program id that is relevant in addressing the program account going forward:

Program Id: HEKdDjSvXxxfDud8puPxgLYdWFfnVe8A4jENDxHzVR16

Note that the program is ~340K in size and will cost you ~2.42 SOL to deploy if this were mainnet.

ls -l target/bpfel-unknown-unknown/release/serum_dex.so
-rwxr-xr-x  2 noxim users 348024 Jan 26 10:13 target/bpfel-unknown-unknown/release/serum_dex.so

We can estimate the cost for rent:

solana rent 348024
Rent per byte-year: 0.00000348 SOL
Rent per epoch: 0.006634331 SOL
Rent-exempt minimum: 2.42313792 SOL

Create a sample Serum market with the crank

We now want to deploy a market and can use the crank program that the serum-dex ships. We can deploy a sample exchange market with the whole-shebang crank program that setups up a coin mint, a pc mint, corresponding vaults and all other accounts required by a trading pair:

cd crank
cargo run -- localnet whole-shebang ~/.config/solana/id.json HEKdDjSvXxxfDud8puPxgLYdWFfnVe8A4jENDxHzVR16

It takes about 15 seconds to run whole shebang and relevant output for us are these bits:

Coin mint: 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS
Pc mint: 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY
Vault signer nonce: 0
Creating coin vault...
Created account: HQzahqbSP2EUX99ZucekJxmTKQ6JpcrLkMsiYPsu1b9r ...
Creating pc vault...
Created account: H5ymt1Hp25BDd8h8b5XjtmJff2KJP6EVbyqZuUBKEHJJ ...
...
Listing 6saerH11Lqky4QfLUG3Papjsb1eesuEeMcyvMEMBcAWN ...
Market keys: MarketPubkeys {
    market: 6saerH11Lqky4QfLUG3Papjsb1eesuEeMcyvMEMBcAWN,
    req_q: DWL23Yf4R2pycobpLfNCr2bYsocT3cbMY2eMSJ6SZpJR,
    event_q: AcK9jke263Hh9vxKYKu4YHHwwYHmgN3qDCFB25EMM6K7,
    bids: 7djc71dGdLRh4gyfp1hfsxeftzB6wyxQN6UQ1tm9nTTu,
    asks: DRitKgVT18C5yprofY3Qcy9xZpp4YByHotRsybxSAvc2,
    coin_vault: HQzahqbSP2EUX99ZucekJxmTKQ6JpcrLkMsiYPsu1b9r,
    pc_vault: H5ymt1Hp25BDd8h8b5XjtmJff2KJP6EVbyqZuUBKEHJJ,
    vault_signer_key: 8xuXac585gbD7qzk1wkRRhX6qAuhhe748fW47eXUHayy,
}

Serum DEX Market Accounts

The coin vault and price currency vault are PDA accounts that hold the base and quote currency balances. With the vault signer key being a PDA with access to these accounts. From the crank code, the default sizes are as follows:

MarketAccountSize     = 376
ReqQueueAccountSize   = 640
EventQueueAccountSize = 1 << 20
BidsAccountSize       = 1 << 16
AsksAccountSize       = 1 << 16

The market account holds metadata such as tick and lot sizes and is 376 bytes in size. The request queue maintains order placement and cancellation requests and is 640 bytes in size. The event queue reports the outputs from matched or filled orders and is 1 MiB in size. The bid and ask accounts created here are 65 KiB in size. Note that the size of these accounts makes creating a serum market expensive.

Resizing Serum DEX Accounts

The event queue and the bid and asks slabs are holding an array of items. When the event queue is full, the DEX will no longer work as fills can no longer be processed. When bid and ask slabs are full, non competitive orders are getting booted from the radix tree. In the Serum code this is called critbit. Crit-Bit (or radix) trees are data structures that are space-optimized and fits the on-chain constraints. Each node that is the only child is merged with its parent.

The minimum size of entries in the event queue is 128, plus the variables in the struct. The minimum size for the bid and ask slabs is 256. You have to size these accounts to trade off attack surface to what you want to pay for rent.

Visualize the Orderbook with the Serum Dex UI

To visualize what we just deployed onto our local chain, we can use the serum-dex-ui from Github:

git clone https://github.com/project-serum/serum-dex-ui.git
cd serum-dex-ui

Edit src/pages/TradePage.tsx and uncomment the following stub for quick setup without the TradingView private dependency:

// import { TVChartContainer } from '../components/TradingView';
// Use following stub for quick setup without the TradingView private dependency
function TVChartContainer() {
   return <></>
}

If you’re running a local test validator that isn’t on your local machine, for example on a raspberry pi validator, you can make it available via src/utils/connection.tsx and add a connection url:

   // ...
   { name: 'localnet', endpoint: 'http://127.0.0.1:8899', custom: false },
   { name: 'raspberry pi', endpoint: 'http://10.3.141.135:8899', custom: false },
 ];

And run the Serum DEX exchange UI locally, you will need npm and yarn:

yarn
yarn start
# OR
yarn build

If you want to deploy it to your test server where you may share a test validator with your team, update the package.json homepage field.

"homepage": "http://10.3.141.135/dex-ui/",

Then copy the contents of the build directory to your server root under /dex-ui.

When visiting the UI on localhost or remote, add a custom market with the + sign with the listing address above, name the listing and your coin and pc token for the trading pair. I’m creating a hypothetical NOX token against USDC on an arbitrarily named market nox.im. In the real world it’d be something like BTC/USD.

Which will redirect you to the market http://localhost:3000/#/market/6saerH11Lqky4QfLUG3Papjsb1eesuEeMcyvMEMBcAWN.

For development, we’re using the same SPL token wallet that I used with the raspberry pi before. In the wallet connector pick “Sollet Extension”.

Mint market SPL tokens

In order to trade we need to create ourselves associated token accounts for the coin and pc mint and mint ourselves some tokens. We can use the Solana toolchain to create the two local token accounts for testing as follows:

spl-token create-account 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS
Creating account 6SCLkaqrTJB56VirYYeMMP5SjBuerJqnEbh4tfxDpZoV
spl-token create-account 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY
Creating account 8dFBGd4E253yXeFoSs6aHcVP4r1x5Q3rCexuhiZ6wWvb

And mint the two tokens:

spl-token mint 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS 1000000
Minting 1000000 tokens
Token: 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS
Recipient: 6SCLkaqrTJB56VirYYeMMP5SjBuerJqnEbh4tfxDpZoV

spl-token mint 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY 1000000
Minting 1000000 tokens
Token: 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY
Recipient: 8dFBGd4E253yXeFoSs6aHcVP4r1x5Q3rCexuhiZ6wWvb

Create the token in your Sollet browser extension with the token mint address

And transfer some tokens to your browser extension wallet, currently everything belongs to your ~/.config/solana/cli/config.yml identity:

spl-token transfer 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS 10000 BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn
Transfer 10000 tokens
Sender: 6SCLkaqrTJB56VirYYeMMP5SjBuerJqnEbh4tfxDpZoV
Recipient: BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn
Recipient associated token account: 8usSUzEioSQ7MNAiD6EeKY7PDrG7g7esFrPQcin3CXna

spl-token transfer 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY 10000 BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn
Transfer 10000 tokens
Sender: 8dFBGd4E253yXeFoSs6aHcVP4r1x5Q3rCexuhiZ6wWvb
Recipient: BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn
Recipient associated token account: GZc7y3aPg6VRHozmGbwpiZc2JeRk8cjJEK457hNgbNjq

You can find these in the Solana explorer by their wallet or transaction hash.

And on our main account we can check our token balances.

You should now be able to connect the Serum DEX to your local test validator, the market we created with the whole-shebang crank and your browser extension wallet on the same cluster. If that works we should be able to make orders with the minted tokens we transferred to ourselves that belong to the token mints of the serum market.

Serum market decimals and lot sizes

The project Serum DEX operates its orderbook in integers. This means that you have to specify lot sizes for markets, offsetting the decimal places of the asset traded. The increment of both tokens cannot be below 1 unit of each. This can be found in dex/src/instruction.rs:

pub struct InitializeMarketInstruction {
    // In the matching engine, all prices and balances are integers.
    // This only works if the smallest representable quantity of the coin
    // is at least a few orders of magnitude larger than the smallest representable
    // quantity of the price currency. The internal representation also relies on
    // on the assumption that every order will have a (quantity x price) value that
    // fits into a u64.
    // ...
    pub coin_lot_size: u64,
    pub pc_lot_size: u64,
    // ...
    // ...
    pub pc_dust_threshold: u64,
}

For each Serum market, we have to specify the coin lot size (base token), the pc (price currency) lot size (quote token) and the price currency dust threshold.

EUR/USD market with 3 decimals for price and quantity

For example, 2.3300 lot of the EURUSD with a quote of 1.1400 means that you will pay 2.6562 USD to buy 1EUR. Lot sizes in Serum work by offsetting the token mints decimal places. We can understand the base lot size offsetting the mint decimals, the quote lot size as offsetting the base decimals.

mint / base lot size
1000000 / 1000 = 1000 -> 3 decimal places for lots
base lot size / quote lot size
1000 / 1 = 1000 -> 3 decimal places for the price

To summarize

• mint 6 decimals, e.g. USDT
• base lot size 1000
• quote lot size 1

will result in

• lot decimals $1/1000 = 0.001$
• price decimals $1000/10^6 = 0.001$

Inversely we can also compute the lot sizes from the decimal places for size, price and quantity. For the same example, we desire 3 decimal places for size and quantity and 3 decimals for the price.

• base lot size = $10^6*0.001 = 1000$
• quote lot size = $10^60.0010.001 = 1$

If the compute any of these as a fraction of 1 (e.g. 0.1), it means our mint decimals doesn’t accommodate the desired decimals in the market. Remember that every order will have a (quantity x price) value required that fits into a u64.

Scaling orders

Understanding base and quote lot sizes, we can not trade into Serum. Assuming we have a token with a 6 decimal mint, e.g. USDC, 1 USD will equal 1000000 base units. To programmatically find out the right denomination we can verify as follows:

• price: 10^(base mint decimals - price decimals) = 10^3
• size: 10^(size decimals) = 10^3
• quantity = price * size, $1.1410^3 * 2.3310^3 = 2656200$ = 2.6562 USD

Serum Errors

As usual with Solana programs, DEX errors are in hexadecimal. Do check the file serum-dex/blob/master/dex/src/error.rs#L78 when you encounter errors. For example 0x34 is InsufficientFunds and indicates you don’t have sufficient tokens for a trade to be made.

Custom program errors when dealing with serum, can either be looked up in an enum, or for assertion errors indicate the file and line number as per this comment in dex/src/error.rs.

#[derive(Debug, IntoPrimitive, FromPrimitive, Clone, Copy, PartialEq, Eq)]
#[repr(u32)]
pub enum DexErrorCode {
    InvalidMarketFlags = 0,
    // ...

    CoinVaultProgramId = 10,
    // ...
    
    Unknown = 1000,

    // This contains the line number in the lower 16 bits,
    // and the source file id in the upper 8 bits
    #[num_enum(default)]
    AssertionError,
}

As a side note to this code comment above, bit orders are ordered like <upper><lower>.

For example, custom program error 0x100067e hints at line 1662 in state.rs, meaning the assertion on that line is failing. In this case it indicates that one of the dex accounts isn’t properly sized or padded when creating a market.

check_assert_eq!(data.len() % 8, 4)?;

Serum Client Accounts

When you’re subscribing to web sockets on serum accounts, make sure you use the memcmp filter at offset 0 for the byte string “serum”. All serum related accounts, when initialized, start with this padding. It avoid you having to filter later on in your code.

Open Orders Account

In order to interact with a Serum DEX deployment, clients have to create an open orders account. It stores amounts deposited into serum, tracks open orders with client IDs and serum internal IDs.

It should be noted that the open orders account tracks fills with a delay. There is a serum crank to process the event queue. When the outputs of the event queue have been processed, the total and free amounts are updated. The total amounts track the amounts free and outstanding in orders. The free amounts track what can be settled to the clients wallet token accounts.

When settling funds in the dex, we are withdrawing the free amounts from the open orders account to our wallets token account. The settle funds instruction zeroes out the free amounts and subtracts from the open orders total amounts and the total market deposits.