Filter JSON-RPC requests based on method with NGINX / OpenResty

While exploring the Avalanche P-Chain API, we came across an issue: requesting the platform.getValidatorsAt method at an early height is a costly operation and often fails with timeouts. The bigger problem in this case is that while this call is being processed, the entire P-Chain API (/ext/bc/P) becomes unavailable for querying on the Avalanche node.

This is a big issue for RPC providers, and we needed to address it. Is there a way to filter JSON-RPC requests based on the method that is being used? The goal here is to have a configurable system in which we can allow/deny methods on a JSON-RPC API endpoint.

Note: In this article, we will focus on applying filters to AvalancheGo's API interface but the same logic can be applied to any software implementing the JSON-RPC specification.

State of JSON-RPC reverse proxy options

While Googling for an answer, we realized that there are not many options available. This was confirmed by Claus Strommer's article: The bleak landscape of protocol aware JSON RPC proxies:

There is still a clear need for proxies that can do payload filtering on arbitrary protocols.

We could not agree more!

Though this article came out in 2020, we still came across some interesting projects that have been released since. Before expanding on the solution we ended up choosing, here are some honorable mentions:

• Subway: a project built in Rust for the Substrate framework. While promising, it seemed that the JSON-RPC method filtering was not implemented yet.

• Apache Tuweni: an Apache-incubated Java library for Blockchain-related development. While it seemed that JSON-RPC method filtering is a possible use case for Tuweni, the laconic documentation and having to run a Java application made us look for another solution.

Note: We did not try these projects for this specific need.

Digging on GitHub, we found the ethereum-nginx-proxy by adetante. This one gained our interest because it is using NGINX, an industry standard reverse proxy.

As we were already using NGINX as part of our infrastructure, we decided to give this project a try.

Installing OpenResty

OpenResty is an NGINX distribution packed with extra modules and their external dependencies. Since the ethereum-nginx-proxy project is essentially a LUA script leveraging the lua-nginx-module, it makes sense to install OpenResty.

Note: It is also possible to manually compile the module into NGINX but we have not tried this option.

As IT automation maximalists, we set out to develop a quick Ansible role to manage the installation/configuration of OpenResty. If you'd rather install OpenResty manually, you can follow the official installation documentation.

Note: The following resources can be found on our GitHub.

---
- hosts: validator01
  become: true
  tasks:
    - name: Import OpenResty GPG key
      get_url:
        url: https://openresty.org/package/pubkey.gpg
        dest: /etc/apt/trusted.gpg.d/openresty.asc

    - name: Add OpenResty APT repository
      apt_repository:
        repo: "deb http://openresty.org/package/ubuntu jammy main"
        filename: openresty
        state: present

    - name: Install OpenResty
      apt:
        name: openresty
        state: present

    - name: Template Nginx OpenResty configuration
      template:
        src: nginx.conf.j2
        dest: /etc/openresty/nginx.conf
        owner: root
        group: root
        mode: 0644

    - name: Download eth-jsonrpc-access.lua
      get_url:
        url: https://raw.githubusercontent.com/adetante/ethereum-nginx-proxy/master/eth-jsonrpc-access.lua
        dest: /usr/local/openresty/nginx/eth-jsonrpc-access.lua
        owner: root
        group: root
        mode: 0644

    - name: Start OpenResty
      service:
        name: openresty
        state: started
        enabled: true

This simple playbook installs OpenResty from its APT repository and templates the following configuration file:

user  www-data;

worker_processes  "12";


events {
    worker_connections  1024;
    multi_accept off;
}

http {
    include       /usr/local/openresty/nginx/conf/mime.types;
    default_type  application/octet-stream;

    server_names_hash_bucket_size 64;

    client_max_body_size 64m;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                       '$status $body_bytes_sent "$http_referer" '
                       '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /usr/local/openresty/nginx/access.log main buffer=16k flush=2m;

    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;

    keepalive_timeout  75;
    keepalive_requests 600;

    server_tokens on;

    # validator01.ash.local
    server {
        listen       80;
        server_name  validator01.ash.local;

        location / {
            set $jsonrpc_blacklist '{{ jsonrpc_blacklist }}';
            access_by_lua_file 'eth-jsonrpc-access.lua';
            proxy_pass http://localhost:9650;
        }
    }
}

We can now run the following command to install OpenResty and configure it to deny access to the platform.getValidatorsAt method:

ansible-playbook playbooks/openresty.yml -i inventories/local -e jsonrpc_blacklist=platform.getValidatorsAt

Note: In this case, we only blacklist a single method but the jsonrpc_blacklist parameter can accept a comma-separated list of methods.

We can then check that it's working by trying to call the platform.getValidatorsAt method through NGINX/OpenResty:

curl -X POST --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"platform.getValidatorsAt",
    "params" :{ 
        "height": 0,
        "subnetID": "11111111111111111111111111111111LpoYY"
    }
}' -H 'content-type:application/json;' -H 'Host: validator01.ash.local' http://localhost/ext/bc/P

Perfect! We successfully managed to filter JSON-RPC requests based on the used method.

Note: In this case, the methods blocked through NGINX/OpenResty can still be reached by reaching the AvalancheGo HTTP API port directly (9650 in this case). A good practice is to keep --http-host to localhost and use NGINX/OpenResty to filter outside requests.

What's next?

We have just seen how to use the ethereum-nginx-proxy script to filter JSON-RPC requests based on the method used. At Ash, we are planning to improve this script to allow using glob expressions to match multiple methods (e.g.: jsonrpc_blacklist=platform.* ) or having in-depth filtering based on params (e.g.: platform.getValidatorsAt is allowed but with height > X)

JSON-RPC method filtering will be a native feature of the Ash Console, coming out later this year.

Stay tuned for more content by subscribing below or by following us on Twitter @ash_avax!