aboutsummaryrefslogtreecommitdiff
path: root/src/main.rs
Commit message (Collapse)AuthorAgeFilesLines
* fix: CLUSTER_USERNAME when running from frontenddiogo4642026-03-171-0/+4
| | | | | the CLUSTER_USERNAME variable was not being used when running from the frontend.
* fix: namespace tmp directorydiogo4642026-03-171-12/+13
| | | | | | | since the /tmp directory is shared by everyone then some left over files remain. when a different user attempts to remove or write to those files but does not have permission it causes failures. this commit just uses /tmp/$USER instead of using /tmp directly
* cargo fmtdiogo4642026-02-111-14/+14
|
* Add support for differing username between cluster and local machinejbordalo2026-02-111-2/+12
|
* feat: added --version flagdiogo4642025-11-091-0/+1
|
* fix: added --force flag to docker swarm leavediogo4642025-11-091-1/+1
|
* fixed clippy warningsdiogo4642025-10-301-15/+9
|
* warn if running from job machinediogo4642025-10-301-2/+13
|
* fix: compile warningdiogo4642025-10-301-0/+1
|
* fix: improve error message when job has 0 machinesdiogo4642025-10-301-0/+4
| | | | | | | | when a job is not yet in running state the program runs but the function that lists the job's machines returns zero machines which can cause a panic from division by 0 which is not a very helpful message. this commit improves the error/warn messages when no machines are listed for a job.
* fix: dont panic when unable to obtain hostnamediogo4642025-10-301-1/+2
| | | | | | | | | if we are unable to obtain the hostname of the local machine from the /etc/hostname file or HOSTNAME env var then an empty string is used as the hostname and a warning is shown instead of a panic. since the hostname is just used to determine if we are executing on the cluster, and the cluster machines all have their hostnames set, then using the empty string should not be a problem.
* remove docker networks and leave swarm on setupdiogo4642025-10-171-0/+4
| | | | | | | | | | | | | | | | | | | some docker networks and the swarm can create routing rules that conflict with the ones we create leading to errors such as: 59:2025-10-15T15:07:33.745853Z WARN Failed to connect to 10.0.1.243:4000: No route to host (os error 113) 89:2025-10-15T15:08:30.197646Z WARN Failed to connect to 10.0.1.32:4000: No route to host (os error 113) 92:2025-10-15T15:08:30.837360Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 95:2025-10-15T15:08:33.905356Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 98:2025-10-15T15:08:36.981419Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 101:2025-10-15T15:08:40.049335Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 104:2025-10-15T15:08:43.121680Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 107:2025-10-15T15:08:46.197394Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 110:2025-10-15T15:08:49.265514Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 113:2025-10-15T15:08:52.337454Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 116:2025-10-15T15:08:55.409444Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 119:2025-10-15T15:08:58.481471Z WARN Failed to connect to 10.0.1.91:4000: No route to host (os error 113) 1
* increased ssh connection attempts to 10 and increased verbositydiogo4642025-10-141-1/+2
|
* added extra debug logging when reading/parsing latency matrixdiogo4642025-10-141-0/+8
|
* feat: write timestamp to signal filediogo4642025-09-181-4/+22
|
* remove existing containers on net up/downdiogo4642025-08-231-5/+10
|
* fixed docker repeated docker pulldiogo4642025-08-181-2/+10
|
* pull container image only once per machinediogo4642025-08-181-1/+3
| | | | | | the container image is now only pulled once, before all containers are created. this prevents hitting the container registry with thousands of requests in a very short amount of time.
* added the --matrix-wrap flagdiogo4642025-08-181-9/+24
| | | | | | I got to the point of needed more than 10k rows and since my biggest latency matrix is 10k in size this option allows the values to wrap so we can create a network bigger than the matrix.
* added clean commanddiogo4642025-08-111-0/+20
|
* improved container wait reliability with timeouts/retriesdiogo4642025-08-081-3/+9
|
* doubled tcp max orphan limitdiogo4642025-08-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the default value on the machines seems to be 262144 but on some larger experiments dmesg will sometimes show the following logs: [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets hopefully increasing this limit will fix that. https://serverfault.com/questions/624911/what-does-tcp-too-many-orphaned-sockets-mean the second answer on server faul also says it could be due to tcp memory limits: ``` The possible cause of this error is system run out of socket memory.Either you need to increase the socket memory(net.ipv4.tcp_mem) or find out the cause of memory consumption [root@test ~]# cat /proc/sys/net/ipv4/tcp_mem 362688 483584 725376 So here in my system you can see 725376(pages)*4096=2971140096bytes/1024*1024=708 megabyte So this 708 megabyte of memory is used by application for sending and receiving data as well as utilized by my loopback interface.If at any stage this value reached no further socket can be made until this memory is released from the application which are holding socket open which you can determine using netstat -antulp. ``` but for now I will just increase the max orphans and see if that is enough.
* only print last 500 lines of logs on container failurediogo4642025-08-081-1/+1
|
* fix: increase arp cache table sizediogo4642025-08-071-0/+5
| | | | | | | | | | | | dmesg was showing this messages: [Thu Aug 7 14:05:26 2025] net_ratelimit: 4328 callbacks suppressed [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! and the machines were becoming inaccessible. increase the arp cache size fixes this.
* disabled conntrack on 10.0.0.0/8 packetsdiogo4642025-08-071-0/+15
| | | | | | | | | we were hitting conntrack limits when opening lots of connections and sending UDP packets to many different hosts. this resulted in TCP packets getting dropped which would manifest itself as errors when connecting or timeouts and when sending UDP packets using `sendto` it would fail with permission denied error. disabling conntrack fixes all of these problems.
* fixed dmesg logs from tcdiogo4642025-08-071-1/+1
| | | | | | there were messages similar to: HTB: quantum of class 10020 is small. Consider r2q change. that showed up when brining up the network. this commit fixes that.
* added --interleave flag to oar-p2p net showdiogo4642025-08-021-2/+23
|
* added basic retry logic to the machine_containers_wait functiondiogo4642025-07-241-1/+17
|
* added ConnectionAttempts ssh optiondiogo4642025-07-241-1/+5
|
* replaced scp with rsyncdiogo4642025-07-241-21/+13
|
* added concurrency limit via OAR_P2P_CONCURRENCY_LIMITdiogo4642025-07-231-30/+20
| | | | | | the env var OAR_P2P_CONCURRENCY_LIMIT limits the number of parallel "operations" being done on the cluster machines. so, if it is set to 3, then we only work on 3 machines at time. setting to 0 means unlimited.
* feat: added oar job id inferencediogo4642025-07-221-2/+15
|
* fixed address listing on machines with no addressesdiogo4642025-07-171-1/+1
| | | | | | | currently the shell script used to list 10.0.0.0/8 range of addresses on a machine would fail with exit code 1 if no addresses were present in that range (i.e. grep did not match anything). this fix just makes sure that command always returns exit code 0.
* improved cli help textdiogo4642025-07-171-0/+62
|
* added custom signals to run subcommanddiogo4642025-07-131-14/+27
|
* feat: add benchmark startup analysis tools and improve demo.shdiogo4642025-07-111-0/+28
| | | | | | | | - Add generate-schedule.sh script to create container schedules from addresses.txt - Add benchmark-startup Python script for analyzing container startup times - Update demo.sh to print timestamps and wait for start signal at /oar-p2p/start - Add comprehensive statistics including startup, start signal, and waiting times - Support for synchronized container coordination via start signal file
* fix: correct shell redirection syntax from 2>1 to 2>&1diogo4642025-07-111-1/+1
|
* feat: add logging for scp command outputdiogo4642025-07-111-0/+9
|
* feat: create output directory if it doesn't exist in run commanddiogo4642025-07-111-0/+8
|
* fix: add error handling for latency matrix dimension checkdiogo4642025-07-111-4/+12
|
* fixed log copyingdiogo4642025-07-111-1/+1
|
* fixed env var quoting when setting container variablesdiogo4642025-07-111-0/+2
|
* fixed reading schedule from stdindiogo4642025-07-111-4/+9
|
* added addr allocation policydiogo4642025-07-111-16/+48
|
* fixed net container builddiogo4642025-07-111-1/+3
|
* cargo clippy --fixdiogo4642025-07-101-15/+13
|
* clean enough for nowdiogo4642025-07-101-320/+187
|
* it works, now needs cleanupdiogo4642025-07-101-39/+793
|
* rust init snapshotdiogo4642025-07-091-0/+252
|
* Convert from Rust to Python project with uv supportdiogo4642025-06-291-133/+0
| | | | | | | | Remove Rust-related files (Cargo.toml, Cargo.lock, src/, target/) and restructure as Python project using uv for dependency management. Update project structure to match nova-oar-mcp style with pyproject.toml, .python-version, and proper Python packaging conventions. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
generated by cgit (git 2.34.1) at 2026-04-05 23:23:23 +0000