Private indexer

The tools presented in this module so far are fully public: everyone can use them. However, they are meant for public networks, and the user is dependent on the infrastructure and configuration of these services. Some organizations are more interested in deploying their private Mavryk network, either for private testing before going public or for entirely private use. Refer to the Private blockchain module for more info. Therefore, it is possible to use an indexer on private networks.

Fortunately, many public developments on Mavryk are open-source and can easily be adapted for a private network, i.e. an indexer can be deployed and configured to watch a private network.

In this chapter, we will see how the source code of MvIndex and Mavryk Explorer can be used to make a private indexer, either for use on a private network or a public network.

This chapter is divided into three parts:

how to quickly set up a private network.
how to install and configure these tools
how to use them

Prerequisites

We will simulate a private network that will be indexed by MvIndex. Therefore, this private network should hold the basic expected elements on any network:

a few funded accounts
a smart contract
some operations

Let's start by setting up a private network. The process is quite difficult as it requires starting and configuring Mavryk nodes (please refer to the Private Blockchain module for more info). Instead, let's choose an easier way by only simulating a private network using Ganache, a node module used for tests and simulations of Mavryk and Ethereum blockchains. Ganache launches the Flexmasa docker image, which runs Mavryk nodes and bakers in a sandbox environment. It also provides ten funded accounts from the start.

Let's use the Raffle smart contract from the LIGO module and migrate it onto our private network using the Truffle configuration from the Build a Dapp module.

Installing the prerequisites

The complete source code of the raffle contract can be found here.

It contains a ganache configuration (with predefined accounts), three smart contracts and their associated migrations:

the first contract holds a big map and does not do anything. This dummy contract is used to bypass a Mavryk Explorer bug regarding big maps: the first big map (whose index is 0) is not fetched by the frontend.
a raffle contract using a map
a raffle using a big map

The second and third smart contracts will highlight the difference between the way maps and big maps are handled by indexers.

$ cd examples/explorer
$ npm install -g truffle@tezos
$ npm install 

MvIndex is written in Go, a statically typed compiled programming language developed by Google. Install it using the instructions at golang.org/doc/install.

The next action is to install the docker images of MvIndex. Install Docker using the instructions at docs.docker.com/get-docker/.

Launching Ganache

In package.json, one script is defined:

  "scripts": {
"start-sandbox": "ganache-cli --flavor tezos --seed alice"
}

The npm repository of ganache can be found here: There are several versions of `ganache-cli`. Only the versions suffixed by `-tezos` support the Mavryk blockchain (you can find them under the versions tab). The specific version used in the project can be found in the `devDependencies` section in `package.json`. When this article was written, the version was 6.12.1-tezos.0

This script starts a simulated private Mavryk blockchain with ganache:

$ npm run start-sandbox
Ganache CLI v6.12.1-tezos.0 (ganache-core: 2.13.2-tezos.2)

Available Accounts
==================
alice 100 MAV
  pk: edpkvGfYw3LyB1UcCahKQk4rF2tvbMUk8GFiTuMjL75uGXrpvKXhjn
  pkh: mv1Hox9jGJg3uSmsv9NTvuK7rMHh25cq44nv
bob 100 MAV
  pk: edpkurPsQ8eUApnLUJ9ZPDvu98E8VNj4KtJa1aZr16Cr5ow5VHKnz4
  pkh: mv1NpEEq8FLgc2Yi4wNpEZ3pvc1kUZrp2JWU
eve 100 MAV
  pk: edpku9qEgcyfNNDK6EpMvu5SqXDqWRLuxdMxdyH12ivTUuB1KXfGP4
  pkh: mv1AAFByPGTyFdbshgfA8ogjgf7pQKuR7ATp
mallory 100 MAV
  pk: edpkujwsG5JMrWVXQwmRMBoR9yJkokRbn6wy3monpQKBpnZTs1ogRR
  pkh: mv1DQGgRUjgKcCLvvsk9qxQNPrN5iGeaPzYc
trent 100 MAV
  pk: edpkukjpYWLPEavoTXid8KqfYAY4J1rqtbWHWrzp22FgdRQqzaRkDD
  pkh: mv1G2tqLVKtkbT34TobUUtSFA8qXdaEJRoWp
marketa 100 MAV
  pk: edpktiaGKkt8Yu6m4Gse2GhMdJCVdCtcrjtqATn3y3sf7xTBDj5g2a
  pkh: mv1U5BzAyzQjySUGSWVgMUMaJgxbEaCf1CMW
eulalie 100 MAV
  pk: edpkvCvic2obeedM7oMJaeyapEafg4dSdYuWvkZigKbcvc64q6ZKM7
  pkh: mv1TcJbhAaM8EgWijCg7LLzQE2tTesdHbEQS
stella 100 MAV
  pk: edpkvRuciP6vZeoXn1KJtBuEEtaD2SpHW59wbbCGt1SEDBL94W7AUE
  pkh: mv1WR89DhkAWa1bp4ok6r4pKjsATEcj8bDRr
carline 100 MAV
  pk: edpktxefxf3dtJEQLVb72MjV8yMiLh6DfCgNJQUV81rnsYJoZhbnK8
  pkh: mv1BmrRbPcHv12fuUtoZ82tSB3cgzF6Zud3g
tabbie 100 MAV
  pk: edpkvXobE6tQLdsm3kPu3MYT2z2XrgVchfkK2WPB9tniNXdWSRyud3
  pkh: mv1JkXG5EpgTVkHZa7DpCcRPcZ5h7Vp8geQJ

<...>

Notice all the funded accounts created by Ganache.

Migrating the Raffle contracts

The contracts can be migrated onto the private network. More details can be found about the Truffle tool in the Build a Dapp module.

First, the private network has to be defined in the truffle-config.js file. A development subsection should be found under the networks section:

    development: {
host: "http://localhost",
port: 8732,
network_id: "*",
secretKey: alice.sk,
type: "tezos"
}

It defines a localhost network, matching the ganache private network.

We're going to use the accounts from the scripts/sandbox/account.js file. Our three contracts can now be migrated with this command:

$ truffle migrate --network development

Three contracts are now deployed onto our private network:

The migration files also include some operations to open a raffle and buy a ticket automatically once the contracts are deployed.

The same smart contract is deployed twice: one with _maps_ and the other with _big maps_. Later on, these two smart contracts will be used to highlight the difference between the processing of _maps_ and _big maps_ by _Mavryk Explorer_.

Setting up a private indexer (MvIndex)

MvIndex is an open-source indexer: it can freely be used and modified. The source code is available at github.com/mavryk-network/mvindex

To start the application, proceed as follows:

$ git clone https://github.com/mavryk-network/mvindex
$ cd mvindex
$ make build
$ ls mvindex

The last command should output "mvindex" (to check if a mvindex has indeed been built) A MvIndex binary is created in the same directory. The indexer can now watch the private network with this command:

$ ./mvindex run --rpcurl 127.0.0.1:8732 --notls --enable-cors

Note the following options:

--rpcurl: url of the ganache private network rpc node
--notls: option to use http instead of https
--enable-cors: MvIndex API will be queried by Tzstat (exposed on a different port): CORS need to be enabled

MvIndex now exposes its API on http://localhost:8000.

A db/ folder is created when launching MvIndex for the first time: it contains all the data indexed about the private network.

When restarting ganache, a blank new network is created. The `db/` folder, containing the old data has to be removed in order to make _MvIndex_ index the new network.

Setting up a private explorer (Mavryk Explorer)

Mavryk Explorer is the open-source frontend made to display the data from MvIndex. The source code is available at github.com/blockwatch-cc/tzstats

Mavryk Explorer can interact with MvIndex by setting the TZSTATS_API_URL variable to the url of the MvIndex API.

The application can be launched with:

$ git clone https://github.com/blockwatch-cc/tzstats
$ cd tzstats
$ echo 'TZSTATS_API_URL=http://localhost:8000' > development.env
$ npm install
$ yarn start

An error might occur during the npm install: ```shell /bin/sh: 1: sass: not found ``` If so, modify the `sass` field under `scripts` in the `package.json` file as follows: ```json "sass": "npx sass src/styles/scss/index.scss:src/styles/css/index.css" ``` Then run `npm install` again.

Mavryk Explorer will now launch in your web browser.

Interacting with the private explorer

At this point, the working context is:

a simulated private network running through Ganache
three migrated contracts
two contract calls
MvIndex indexing the private network
Mavryk Explorer connected to MvIndex, and running

This means that there has already been some activity that we can watch on our network.

Watching activity on Mavryk Explorer

Mavryk Explorer is running at http://localhost:3000:

Mavryk Explorer indexer is syncing

Notice that the frontend is different from its public version on Mavryk Explorer.com but the inner workings are the same.

On top, the search bar allows you to search for transactions, blocks, addresses, etc. The left panel contains various information displayed, such as the number of baked blocks.

Click on it to get more info on that block: TsStats, webpage of a block

Many block details are displayed; most of them can be clicked on for even more info. The block above only contains one operation, which is a call to our smart contracts. Click the hash to open the operation page:

TsStats, webpage of a smart contract, list of contract calls

The sender does match Alice's pkh from scripts/sandbox/account.js. pkh means public key hash: it is an address on a Mavryk network. Notice the contract address matches the returned address from the migration output logs.

Click on the address to inspect the smart contract: TsStats, webpage of a smart contract

You can see the history of calls, the entrypoints, the storage, etc. Notice the two calls made by Truffle: the origination of the contract and the purchase of a ticket.

The storage page displays the storage definition, the type of each field and their associated current value. Note that one participant is registered in the raffle and the contract holds one Mav (from selling one ticket). TsStats, webpage of a smart contract storage

About big maps

Note that the sold_tickets big map should hold an address as value, but it displays 1 in the storage. Indeed, since big maps are meant to hold unbounded lists of data, they cannot be loaded directly into the storage (out of performance concerns). Each big map is indexed and the number 1 being displayed is the index number of the big map . Its data can actually be accessed by clicking on the Bigmap 1 section.

TsStats, webpage of a smart contract with a big map

Regular maps are meant to be used with limited data size as the data is directly retrieved from the storage section. You can see an example of such a map in the second migrated smart contract (baked in the fourth block):

TsStats, webpage of a smart contract storage

The Mavryk Explorer interface is user-friendly and every piece of information can easily be read by clicking on each element. Remember that all this information is coming from the data retrieved by the indexer: Mavryk Explorer just displays them.

Watching activity from MvIndex

The same pieces of information can be retrieved without the frontend by just using the MvIndex API. For each page opened on Mavryk Explorer, an API call is made to MvIndex. Each API call can be seen in the network explorer by pressing F12 in the network section.

Mavryk Explorer webpage and the developer console of the web browser showing network activity

The request URL shows the called API endpoints:

$ GET http://127.0.0.1:8000/explorer/contract/KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF?
{
  "address": "KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF",
  "manager": "mv1Hox9jGJg3uSmsv9NTvuK7rMHh25cq44nv",
  "delegate": "",
  "height": 6,
  "fee": 0.003939,
  "gas_limit": 12826,
  "gas_used": 12726,
  "gas_price": 0.30952,
  "storage_limit": 2654,
  "storage_size": 2397,
  "storage_paid": 2397,
  "is_funded": true,
  "is_vesting": false,
  "is_spendable": false,
  "is_delegatable": false,
  "is_delegated": false,
  "first_in": 7,
  "first_out": 0,
  "last_in": 7,
  "last_out": 0,
  "first_seen": 6,
  "last_seen": 7,
  "delegated_since": 0,
  "first_in_time": "2021-04-17T17:06:27Z",
  "first_out_time": "0001-01-01T00:00:00Z",
  "last_in_time": "2021-04-17T17:06:27Z",
  "last_out_time": "0001-01-01T00:00:00Z",
  "first_seen_time": "2021-04-17T17:06:26Z",
  "last_seen_time": "2021-04-17T17:06:27Z",
  "delegated_since_time": "0001-01-01T00:00:00Z",
  "n_ops": 1,
  "n_ops_failed": 0,
  "n_tx": 1,
  "n_delegation": 0,
  "n_origination": 0,
  "token_gen_min": 1,
  "token_gen_max": 1,
  "bigmap_ids": [
    1
  ],
  "op_l": 3,
  "op_p": 0,
  "op_i": 0,
  "iface_hash": "d30a2146",
  "call_stats": [
    1,
    0,
    0
  ]
}

A lot of information is displayed, such as the contract's address, but some data is also missing, like the storage. Another API call has to be made for that:

$ GET http://127.0.0.1:8000/explorer/contract/KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF/storage?
{
  "meta": {
    "contract": "KT1HJ8VJ9rHkoi4FfzHPburSe1VdYn8AU4AF",
    "time": "2021-04-17T17:06:27Z",
    "height": 7,
    "block": "BLwCojKb2fv7Ph88kxMoYXeTRYdtavvzCNh3ZzsJNpfFgqq5ey8"
  },
  "value": {
    "admin": "mv1Qe9CGXQZFm2xvmG6sJFGNv6512usP8vhT",
    "close_date": "1618679185827",
    "contract_name": "Raffle smart contract with big map",
    "description": "",
    "jackpot": "100",
    "players": [
      "mv1Hox9jGJg3uSmsv9NTvuK7rMHh25cq44nv"
    ],
    "raffle_is_open": "true",
    "sold_tickets": "1",
    "winning_ticket_number_hash": "00"
  }
}

All the available endpoints can be found here: explorer.mavryk.network/docs/api#explorer-endpoints

Setting up a private indexer for a public network

If you don't want your infrastructure to rely on third-party public indexers to monitor public networks, you can also use a local indexer. It just has to monitor a node: either a local node, or a public node (listed at taquito.mavryk.org/docs/rpc_nodes).

$ ./mvindex run --rpcurl <node_url> --notls --enable-cors

There are three operation modes, which retrieve more or less data:

Full regular: all indexes are built (default mode)
Light light-weight: consensus and governance indexes are not built (use --light in the CLI)
Validate: state validation mode for checking accounts and balances at each block/cycle (use --validate in the CLI)

Note that whichever mode you choose, indexing a public network will take a significant amount of time due to the size of data to be fetched.

According to the Tzstats team, it is possible to index 13 000 blocks per minute from a local node connected to the mainnet (Cost and benefice section from this tzstats post). When indexing the mainnet from a public node (Smartpy node on our case), we could index about 10 blocks per seconde.

The mainnet was about 1 400 000 blocks, so to index the mainnet, it should take:

about an hour and fifty minutes from a local node
about a day and a half from a public node.

Of course, these figures depend on your network connectivity and hardware.

Conclusion

Running a private network does not condemn its user to be blind to whatever is happening on their network: private indexers and explorers can be set up to index and monitor a private network.

Public networks (mainnet, testnets) are indexed by public tools ([https://explorer.mavryk.network/] for instance). However, some use cases may required not to rely on third party services: private indexers and explorers can be used to that purpose.

Finally, indexers, since they expose a complete API, can be used to replace libraries (such as Taquito) or wallets. Decentralized apps can take advantage of these API calls to fetch data from a Mavryk network, especially when it comes down to big maps.