Collector Usage
Introduction
The Collector gathers crypto assets (strictly limited to Ethereum, native tokens, and ERC-20 tokens) from multiple deposit addresses across many EVM networks (such as Ethereum, Polygon, Avalanche, and others). These funds are then transferred to the main MPC or cold address associated with their respective omnibus account.
%%{
init: {
'theme': 'base',
'themeVariables': {
'primaryColor': '#9B69FF',
'primaryTextColor': '#EFEFEF',
'secondaryColor': '#CCCCCC',
'secondaryTextColor': '#CCCCCC',
'lineColor': '#9B69FF',
'arrowColor': '#9B69FF',
'fontFamily': 'Arial, sans-serif, bold',
'fontSize': '16px',
}
}
}%%
flowchart LR
A[Deposit Wallet 1] --Collector--> B[MPC Main Wallet]
D[Deposit Wallet 2] -- Collector --> B
E[Deposit Wallet 3] -- Collector --> B
While you can run a collector at any time, running a collector at a low-demand time lowers the cost of the various fees associated with transferring your funds. Running a collector maximizes your main wallet's liquidity.
This article outlines how to operate the collector from the command line.
Limitations
Your Custody Infrastructure deployment can have as many collectors configured as you have Omnibus accounts. There are some notable limitations when it comes to using those collectors:
- A user must be included in the "Collector & Exchange user allowlist" to create, use, or check the performance of collectors.
- Each account can have only one collector.
- Only one collector can run on your Custody Infrastructure platform at any given time.
- ERC-20 collectors configured with lower fees can take a few hours to run.
Summary
Configuring a collector involves defining the account and main wallet address and the fees you are willing to pay during collection. Configuring a given collector only needs to be done once, before the collector's initial run. The configuration process is detailed here.
Once configured, the collector is run from the command line by calling startCollector and specifying which uploaded configuration to run.
The collector may take a few hours to run. This is normal. The collector performs its transfers in three sequential batches, and waits for each batch to be fully finalized on-chain before sending the next round of transactions. After it finishes running, you can use getCollectorHistory and getCollectorTransactions to check the results.
This article covers:
- Introduction
- Limitations
- Summary
- The Collector Procedure
- Detailed Process Diagram
- CollectorConfig.py
- Task-by-task Walkthrough
- Add and Run a Collector
- Endpoint Deep Dive*
- Further Reading
*Endpoint Deep Dive - Expanded:
- addCollector
- updateCollector
- startCollector
- getCollectorHistory
- getCollectorTransactions
- getCollectors
- deleteCollector
The Collector Procedure
Utilizing the collector involves two phases, the configuration and the collection. Before you can configure the collector, you need to make some decisions regarding the collector's settings.
Key Prerequisite Information
- creator_user_id - The email address of the user creating the collector. They need Collector Initiator permissions.
- collector_name - This is a unique name that makes it easier to track the specific collector's performance through various logs and audits.
- addCollectorAddress - Where the collector will transfer the gathered assets to
- addNetworkLimits - For every network you have assets on, what is the total amount of fees you are willing to pay each time the collector runs?
- addCurrencyLimits - For each specific currecy you are collectiong, you need to define a fee ratio and a fee cap.
- Fee ratio - This is the proportion of the fee to the asset (in fiat value) that you are willing to collect, expressed as a decimal with a leading zero, i.e. 0.25 if you only want to collect if the asset is worth at least four times the fee.
- Fee cap - This is the total amount of a selected asset you are willing to spend in collection fees.
Detailed Process Diagram
%%{
init: {
'theme': 'base',
'themeVariables': {
'primaryColor': '#9B69FF',
'primaryTextColor': '#EFEFEF',
'secondaryColor': '#CCCCCC',
'secondaryTextColor': '#CCCCCC',
'lineColor': '#9B69FF',
'arrowColor': '#9B69FF',
'fontFamily': 'Arial, sans-serif, bold',
'fontSize': '16px',
}
}
}%%
graph TD
A([Start]) --> B{Is the collector configured?}
B -- No --> C["User runs <code style='padding: 4px; border-radius: 8px; background-color: #F0F0F0; color: black'>addCollectorAddress</code>"]
C --> D["User runs <code style='padding: 4px; border-radius: 8px; background-color: #F0F0F0; color: black'>addNetworkLimits</code>"]
D --> E["User runs <code style='padding: 4px; border-radius: 8px; background-color: #F0F0F0; color: black'>addCurrencyLimit</code>"]
E --> B
B -- Yes --> F["User runs <code style='padding: 4px; border-radius: 8px; background-color: #F0F0F0; color: black'>startCollector</code>"]
F --> G[Sort deposit wallets into batches]
G --> H[Batch 1: Native currencies pay fees in same currency]
H --> I{Can Batch 1 be collected within fee parameters?}
I -- No --> J([Collector Complete: No batches collected])
I -- Yes --> K[Batch 1: Execute transfers]
K --> L[Batch 1: Wait for finalization]
L --> M{Can Batch 2 be collected within remaining fee parameters?}
M -- No ---> N([Collector Complete: Only Batch 1 collected])
M -- Yes --> O[Transfer ETH to deposit wallets for Batch 2 fees]
O --> P[Batch 2: Execute transfers using transferred ETH]
P --> Q[Batch 2: Wait for finalization]
Q --> R{Can Batch 3 be collected within remaining fee parameters?}
R -- No --> S([Collector Complete: Batch 1 and 2 collected])
R -- Yes --> T[Batch 3: Execute transfers without ETH transfer]
T --> U[Batch 3: Wait for finalization]
U --> V([Collector Complete: All 3 batches collected])
J --> W["Optional: User checks results"]
N --> W
S --> W
V --> W
W --> X["User checks results with <code style='padding: 4px; border-radius: 8px; background-color: #F0F0F0; color: black'>getCollectorHistory</code>"]
W --> Y["User checks transactions with <code style='padding: 4px; border-radius: 8px; background-color: #F0F0F0; color: black'>getCollectorTransactions</code>"]
As illustrated above, the collector runs the collection in three batches.
The collector sorts the deposit wallets into batches as follows:
- Wallets that contain native coins - Collection transaction fees for these wallets come from the assets present in the wallets.
- Wallets that contain ERC-20 tokens but no native coins - The collector first transfers native coins from the main wallet to these deposit wallets, then uses that same native currency to pay the collection fees. Collecting from these wallets incurs fees twice: once when the native coin is sent out from the main wallet and a second time when the ERC-20 is transferred to the main wallet.
- Wallets that contain both ERC-20 tokens and native coins - These wallets pay the fee for collecting the ERC-20 assets from their own native coin reserves.
For each batch in order, the collector checks to see if all deposit wallets can be collected from while remaining within all configured limits. The limits are cumulative; the collector counts all fees paid for batch 1 before assessing whether there is enough left for batch 2.
If the collector cannot afford to collect from all wallets in the batch, no wallets from the batch are collected, and the collector stops.
CollectorConfig.py
A collector only needs to be configured once per account. Once configured, that collector can be run any number of times. If changes need to be made to an account's collector, it can be updated.
You can check ~/apiclient/Collector/CollectorConfig.py to see everything that needs to be configured for a collector laid out in a single file. Configuring the collector isn't done by editing the Python file; you just run a few commands to define the new collector's parameters.
When configuring a collector, some aspects are set automatically while others need to be manually populated. The parameters outlined in Key Prerequisite Information are manually configured, and the choices you make will influence how the collector runs. Other parameters will be automatically generated.
To configure a collector from your API command line:
Configure the empty collector by running:
collector_config_1 = CollectorConfig.Config(type=CollectorType.CollectorType.ON_DEMAND,creator_user_id="<[email protected]>",collector_id=uuid.uuid4(),collector_name="<Choosecollector",creation_time_in_seconds=int(time.time()))Where:
- type - This must be set to ON_DEMAND
- creator_user_id - The email address of the user creating the collector. They need Collector Initiator permissions.
- collector_id - This must be set to uuid.uuid4()to automatically generate a UUID (v4) identifier.
- collector_name - A unique and recognizable name you assign to the collector.
- creation_time_in_seconds - This must be set to int(time.time()).
Indicate what account the collector should run in and what destination it should send collected assets to by running:
collector_config_1.addCollectorAddress(account_id=1, collector_address="0x60B5287afED37E3493B60eA091AFEbC75F26Eb1F")Where:
- account_id - This is the account you want the collector to work with.
- collector_address - This is the address of the selected account's main MPC wallet.
Configure the maximum fees you are willing to pay for a collection operation on each network. This command needs to be run once for each network you will be collecting assets on:
collector_config_1.addNetworkLimits(symbol="ETH", max_total_fee=999999999, max_priority_fee=9999)Where:
- symbol - The symbol of the network being collected on, such as SOL.
- max_total_fee - The total fees you are willing to pay to collect assets on this network.
- max_priority_fee -
Configure the fees you are willing to pay to collect each specific asset type. This command needs to be run once for each asset you will be collecting:
collector_config_1.addCurrencyLimits(symbol="SOL", fee_ratio="0.01", fee_cap=100)Where:
- symbol - The symbol of the asset being collected, such as SOL.
- fee_ratio - The ratio of fee to asset, in fiat value of both. This must be expressed as a decimal with a leading zero, and must be between 0 and 1. Wallets that don't meet this criteria are ignored by the collector.
- fee_cap - The highest fee you are willing to pay to collect this type of asset.
The commands will not show any return, but the configuration will be shown when you add it to the system.
Task-by-task Walkthrough
Once the collector is configured, you interact with it via the standard API client. This section will demonstrate how to:
- Add and Run a collector
- Check a collector's results
- Review collectors' configurations
- Change or delete configured collectors.
Add and Run a Collector
After a collector is configured (see CollectorConfig.py for details), the collector needs to be added to the Custody Infrastructure backend. Once added, it can be run any number of times to collect funds from the configured account's deposit wallets.
Add a collector configuration to your system by running:
Where <collector_config_name> is the collector you configured with CollectorConfig.py.
A given config only needs to be added once.
Start the collector by running:
Where <collector_config_name> is the collector you configured with CollectorConfig.py.
A collector may take a few hours to run. This is normal.
Your Custody Infrastructure system can only run one collector at a time.
Check a Collector's Results
There are two API calls you can run to check the collector's results.
- getCollectorHistory - This retrieves and displays information about all collectors' runs. You can specify which records to retrieve by using offset=<int> to specify the earliest record you want to see and limit=<int> to specify how many total records you wish to display.
- getCollectorTransactions - This allows you to specify a run ID and Collector ID (both included in the results of getCollectorHistory) and see every transaction that occurred as part of that collection.
Review Collectors' Configurations
getCollectors is used to display every collector configuration that has been added to the system.
Change or Delete Configured Collectors
An account can only have one collector that gathers assets from its wallets. If you want to make changes to how an account's assets are collected, you can either modify its current collector or delete its current collector and create a new one.
- updateCollector - Update an existing collector on the MPC after updating its configurations with CollectorConfig.py. This command only needs to be run once per edited config.
- deleteCollector - Delete an existing collector configuration.
Status Code Interpretation
Running any of the collector endpoints will return a status code, which are to be interpreted as follows:
Code | Meaning |
|---|---|
0 | Ok |
1 | Account exists |
2 | Collector is already running |
3 | Not exists |
4 | Permission denied |
400 | Bad request |
500 | Technical error |
Endpoint Deep Dive
{
"FunctionData": {
"Config": {
"Accounts": {
"1": {
"CollectorAddress": "0x8B...217d"
}
},
"CollectorId": "dd0d6b88-484b-404d-8fe5-b7cef2721e94",
"CreationTime": 1729512592,
"CreatorUserId": "[email protected]",
"Limits": {
"Currencies": {
"ETH": {
"FeeCap": "1000000000000000000",
"FeeRatio": "0.5"
}
},
"Networks": {
"ETH": {
"MaxPriorityFee": "9999",
"MaxTotalFee": "999999999"
}
}
},
"Name": "TestCollector",
"Type": 0
}
},
"Meta": {
"AccessToken": "38b3...a9ce",
"Destination": "a6ac...5660",
"FunctionName": "addCollector",
"RandomData": "1909...40e0e",
"Source": "04a2...8eb0",
"Time": "2024-10-21 12:09:52.403398"
}
}{
"FunctionData": {
"Config": {
"Accounts": {
"1": {
"CollectorAddress": "0x8B...217d"
}
},
"CollectorId": "dd0d6b88-484b-404d-8fe5-b7cef2721e94",
"CreationTime": 1729512592,
"CreatorUserId": "[email protected]",
"Limits": {
"Currencies": {
"ETH": {
"FeeCap": "1000000000000000000",
"FeeRatio": "0.5"
}
},
"Networks": {
"ETH": {
"MaxPriorityFee": "9999",
"MaxTotalFee": "999999999"
}
}
},
"Name": "TestCollector",
"Type": 0
}
},
"Meta": {
"AccessToken": "38b3...a9ce",
"Destination": "a6ac...5660",
"FunctionName": "updateCollector",
"RandomData": "1909...40e0e",
"Source": "04a2...8eb0",
"Time": "2024-10-21 12:09:52.403398"
}
}{
"FunctionData": {
"Config": {
"Accounts": {
"1": {
"CollectorAddress": "0x8B....217d"
}
},
"CollectorId": "dd0d6b88-484b-404d-8fe5-b7cef2721e94",
"CreationTime": 1729512592,
"CreatorUserId": "[email protected]",
"Limits": {
"Currencies": {
"ETH": {
"FeeCap": "1000000000000000000",
"FeeRatio": "0.5"
}
},
"Networks": {
"ETH": {
"MaxPriorityFee": "9999",
"MaxTotalFee": "999999999"
}
}
},
"Name": "TestCollector",
"Type": 0
},
"InitiatorUserId": "[email protected]"
},
"Meta": {
"AccessToken": "cb63....d0f9",
"Destination": "a6ac....5660",
"FunctionName": "startCollector",
"RandomData": "1612....2f6f",
"Source": "04a2....8eb0",
"Time": "2024-10-21 12:09:52.576552"
}
}{
"FunctionData": {
"Limit": 100,
"Offset": 0
},
"Meta": {
"AccessToken": "b804...a06c",
"Destination": "a6ac...5660",
"FunctionName": "getCollectorHistory",
"RandomData": "280a...a138",
"Source": "04a2...8eb0",
"Time": "2024-10-21 13:44:08.222320"
}
}{
"FunctionData": {
"CollectorId": "dd0d6b88-484b-404d-8fe5-b7cef2721e94",
"Limit": 100,
"Offset": 0,
"RunId": "7f67df06-a4a4-42d7-8827-331655332565"
},
"Meta": {
"AccessToken": "fe7f....9214",
"Destination": "a6acd....5660",
"FunctionName": "getCollectorTransactions",
"RandomData": "2168....a4ba",
"Source": "04a2a....8eb0",
"Time": "2024-10-21 13:49:40.083524"
}
}{
"FunctionData": {},
"Meta": {
"AccessToken": "d4ca...3e46",
"Destination": "a6ac...5660",
"FunctionName": "getCollectors",
"RandomData": "f6c8...121b",
"Source": "04a2...8eb0",
"Time": "2024-10-21 12:09:52.637540"
}
}{
"FunctionData": {
"CollectorId": "dd0d6b88-484b-404d-8fe5-b7cef2721e94"
},
"Meta": {
"AccessToken": "1162...d8f3",
"Destination": "a6ac...5660",
"FunctionName": "deleteCollector",
"RandomData": "1d39...9ada",
"Source": "04a2...8eb0",
"Time": "2024-10-21 13:52:20.814815"
}
}Further Reading
To use collectors via the GUI, see Collector User Guide.