---
name: nodexo-operator
description: Set up and operate Nodexo miners and validators with PM2, nginx endpoint hardening, wallets, chain config, and production safety checks.
---

# Nodexo Operator

Use this skill when an agent is helping an operator install, configure, verify,
or troubleshoot a Nodexo miner or validator.

This skill is for infrastructure operation. Do not use it for renter-side GPU
rentals; use `nodexo-rental` for that.

## Safety Rules

- Never ask for or print private keys, seed phrases, wallet mnemonic material,
  raw API tokens, or `NODEXO_ADMIN_TOKEN`.
- Do not delete wallets, validator databases, `~/.nodexo`,
  `executor_identity.json`, `containers.json`, or synced subtensor data.
- Do not run destructive git or filesystem commands unless the operator
  explicitly asks for that operation.
- Prefer PM2 for long-running services.
- Prefer nginx in front of miner and validator APIs, with the Python daemon
  bound to an internal interface behind the proxy.
- Do not expose validator control routes publicly. Validators should
  expose signed miner ingress and status only.

## Repository

Operator repository root:

```bash
cd nodexo
```

If working from a private/internal checkout, export the operator-facing release
tree before testing third-party install behavior.

## Miner Setup

Required inputs:

- Bittensor wallet name and hotkey name.
- Target network: `test` or `finney`.
- Public miner endpoint domain or IP.
- Open miner API port, default `8091/tcp`.
- Open rental SSH range, default `20000-20100/tcp`.
- NVIDIA driver already installed; `nvidia-smi` must work.

Chain config files contain public contract addresses and deployment metadata
only. Select the network with `--subtensor-network test` or
`--subtensor-network finney`. If an operator deliberately uses a local
subtensor, pass that endpoint at runtime; do not write private RPC endpoints
into `chain_config_*.json`.

Install host dependencies and proof artifact:

```bash
bash scripts/setup_miner.sh
```

Recommended endpoint hardening:

```bash
bash scripts/setup_endpoint_proxy.sh --role miner --public-port 8091
```

Then run the miner backend behind the proxy and advertise the public endpoint:

```text
--bind-host <INTERNAL_BIND_HOST>
--endpoint https://YOUR_PUBLIC_MINER_DOMAIN
```

Start with PM2:

```bash
cp ecosystem.config.example.cjs ecosystem.config.cjs
# edit wallet, hotkey, network, endpoint
pm2 start ecosystem.config.cjs --only miner-nodexo
pm2 logs miner-nodexo --lines 80
pm2 save
```

Post-start checks:

```bash
curl https://YOUR_PUBLIC_MINER_DOMAIN/health
pm2 logs miner-nodexo --lines 80
nodexo fleet --subtensor-network test
```

The setup script installs the ZkGEMM artifact from the shipped manifest URL,
verifies SHA-256, imports the extension, and runs a CUDA smoke test unless
explicitly disabled.

If the subnet alpha-stake gate is active, `nodexo fleet` and the web
operator dashboard show the hotkey's current alpha stake, required alpha stake,
and any grace deadline. Operators should add stake before the grace deadline;
after it expires, under-staked executors score zero.

Optional public API rental commands use the Node helper:

```bash
npm install
nodexo quote --gpu A6000 --duration 1h
```

## Link Miner Hotkey To Account

The web app account is only the destination account. Miner ownership is proven
by signing a one-time challenge with the Bittensor hotkey on the operator host.

Flow:

1. Sign in to the web app.
2. Open `/operator`, enter the miner hotkey SS58, and create a challenge.
3. Run the generated command shown on the Operator page on the machine that has
   the miner hotkey.
4. Return to the Operator page to finish the link.

Do not move coldkeys or hotkeys into browser wallets for this claim.

## Validator Setup

Required inputs:

- Bittensor wallet name and hotkey name.
- Target network: `test` or `finney`.
- Validator endpoint domain or IP.
- Chain config in repo root for the target network.
- Durable validator database path or Postgres URL.

Use the same chain config rule as miners: the config is public contract
metadata, while RPC selection is a runtime network/endpoint setting.

Install:

```bash
bash scripts/setup_validator.sh
```

Recommended endpoint hardening:

```bash
bash scripts/setup_endpoint_proxy.sh --role validator --public-port 9443
```

Start with PM2:

```bash
cp ecosystem.config.example.cjs ecosystem.config.cjs
# edit wallet, hotkey, network, endpoint, and DB_URL
# proof-verifying validators normally use VALIDATOR_NO_EVM=1
pm2 start ecosystem.config.cjs --only vali-nodexo
pm2 logs vali-nodexo --lines 100
pm2 save
```

Proof-verifying validators use:

```env
VALIDATOR_NO_EVM=1
NODEXO_VALIDATOR_DISCOVERY_MODE=native
```

Validators that need to advertise a custom HTTPS/domain endpoint can publish
through ValidatorRegistry:

```env
VALIDATOR_NO_EVM=0
NODEXO_VALIDATOR_DISCOVERY_MODE=evm
```

Follower validators are disabled by default. Use them only when the validator
should set weights from the primary validator's published state instead of
running local proof verification:

```env
VALIDATOR_FOLLOWER_MODE=1
VALIDATOR_FOLLOWER_STATE_URL=https://nodexo.ai/api/instances
```

Follower mode does not run local proof verification, rental orchestration,
offline reports, or validator control-plane jobs.

Post-start checks:

```bash
curl https://validator.nodexo.ai/health
pm2 logs vali-nodexo --lines 100
nodexo fleet --chain-direct --subtensor-network test
```

## Cloudflare And Domains

Recommended production pattern for HTTPS API endpoints:

1. Use a real domain such as `gpu1.example.com` or
   `validator1.example.com`.
2. Put Cloudflare in front of the HTTPS API endpoint on port `443`.
3. Run nginx on the host and proxy to the loopback Python backend.
4. Use a real origin certificate. Cloudflare Origin Certificates are fine only
   when all traffic goes through Cloudflare; use Let's Encrypt for direct
   validator/miner clients.
5. Keep Cloudflare Access, browser challenges, bot fights, and interactive
   challenges disabled on protocol endpoints. Validators and miners are
   non-browser clients.
6. Disable caching and request/response mutation on protocol endpoints.
7. If using Cloudflare proxy, firewall the origin HTTPS port to Cloudflare IP
   ranges.

Important limitation:

- Standard Cloudflare HTTP proxy protects HTTP/HTTPS API ports only.
- It does not proxy the rental SSH range `20000-20100/tcp`.
- Rental SSH ports must remain directly reachable, or require a separate TCP
  proxy product such as Spectrum or another dedicated tunnel design.

For a proxied HTTPS miner endpoint:

```bash
bash scripts/setup_endpoint_proxy.sh --role miner \
  --tls existing \
  --server-name gpu1.example.com \
  --public-port 443 \
  --cert-path /etc/letsencrypt/live/gpu1.example.com/fullchain.pem \
  --key-path /etc/letsencrypt/live/gpu1.example.com/privkey.pem
```

Advertise:

```text
--endpoint https://gpu1.example.com
```

## Troubleshooting

- Inventory missing: check miner health, validator logs, recent heartbeat,
  proof recency, canary status, and endpoint reachability.
- Miner not registering: check wallet/hotkey, netuid, chain config, EVM mirror
  balance, and endpoint validation.
- Validator not setting weights: check subtensor RPC, registered hotkey,
  metagraph UID, chain config, and PM2 logs.
- Rental SSH fails: check host firewall, provider firewall, rental port range,
  Docker port allocation, and whether the SSH range is direct TCP reachable.
- x402 or credits issues belong to the web app, not validator admin routes.

## References

- `docs/MINER_QUICKSTART.md`
- `docs/VALIDATOR_QUICKSTART.md`
- `docs/ENDPOINT_SECURITY.md`
- `docs/PRODUCTION.md`
- `docs/CLI_REFERENCE.md`