aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* improved container wait reliability with timeouts/retriesdiogo4642025-08-081-3/+9
|
* doubled tcp max orphan limitdiogo4642025-08-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the default value on the machines seems to be 262144 but on some larger experiments dmesg will sometimes show the following logs: [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets [Fri Aug 8 05:01:42 2025] TCP: too many orphaned sockets hopefully increasing this limit will fix that. https://serverfault.com/questions/624911/what-does-tcp-too-many-orphaned-sockets-mean the second answer on server faul also says it could be due to tcp memory limits: ``` The possible cause of this error is system run out of socket memory.Either you need to increase the socket memory(net.ipv4.tcp_mem) or find out the cause of memory consumption [root@test ~]# cat /proc/sys/net/ipv4/tcp_mem 362688 483584 725376 So here in my system you can see 725376(pages)*4096=2971140096bytes/1024*1024=708 megabyte So this 708 megabyte of memory is used by application for sending and receiving data as well as utilized by my loopback interface.If at any stage this value reached no further socket can be made until this memory is released from the application which are holding socket open which you can determine using netstat -antulp. ``` but for now I will just increase the max orphans and see if that is enough.
* only print last 500 lines of logs on container failurediogo4642025-08-081-1/+1
|
* fix: increase arp cache table sizediogo4642025-08-071-0/+5
| | | | | | | | | | | | dmesg was showing this messages: [Thu Aug 7 14:05:26 2025] net_ratelimit: 4328 callbacks suppressed [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! [Thu Aug 7 14:05:26 2025] neighbour: arp_cache: neighbor table overflow! and the machines were becoming inaccessible. increase the arp cache size fixes this.
* disabled conntrack on 10.0.0.0/8 packetsdiogo4642025-08-071-0/+15
| | | | | | | | | we were hitting conntrack limits when opening lots of connections and sending UDP packets to many different hosts. this resulted in TCP packets getting dropped which would manifest itself as errors when connecting or timeouts and when sending UDP packets using `sendto` it would fail with permission denied error. disabling conntrack fixes all of these problems.
* fixed dmesg logs from tcdiogo4642025-08-071-1/+1
| | | | | | there were messages similar to: HTB: quantum of class 10020 is small. Consider r2q change. that showed up when brining up the network. this commit fixes that.
* added --interleave flag to oar-p2p net showdiogo4642025-08-021-2/+23
|
* added basic retry logic to the machine_containers_wait functiondiogo4642025-07-241-1/+17
|
* added ConnectionAttempts ssh optiondiogo4642025-07-241-1/+5
|
* replaced scp with rsyncdiogo4642025-07-241-21/+13
|
* added concurrency limit via OAR_P2P_CONCURRENCY_LIMITdiogo4642025-07-232-35/+69
| | | | | | the env var OAR_P2P_CONCURRENCY_LIMIT limits the number of parallel "operations" being done on the cluster machines. so, if it is set to 3, then we only work on 3 machines at time. setting to 0 means unlimited.
* feat: added oar job id inferencediogo4642025-07-223-7/+307
|
* fixed address listing on machines with no addressesdiogo4642025-07-171-1/+1
| | | | | | | currently the shell script used to list 10.0.0.0/8 range of addresses on a machine would fail with exit code 1 if no addresses were present in that range (i.e. grep did not match anything). this fix just makes sure that command always returns exit code 0.
* added the interface for machines oddish,psyduck,squirtle,bulbasaurdiogo4642025-07-171-11/+11
|
* improved cli help textdiogo4642025-07-171-0/+62
|
* set the interface for moltres machinesdiogo4642025-07-171-10/+10
|
* added custom signals to run subcommanddiogo4642025-07-132-14/+288
|
* feat: add benchmark startup analysis tools and improve demo.shdiogo4642025-07-111-0/+28
| | | | | | | | - Add generate-schedule.sh script to create container schedules from addresses.txt - Add benchmark-startup Python script for analyzing container startup times - Update demo.sh to print timestamps and wait for start signal at /oar-p2p/start - Add comprehensive statistics including startup, start signal, and waiting times - Support for synchronized container coordination via start signal file
* fix: correct shell redirection syntax from 2>1 to 2>&1diogo4642025-07-111-1/+1
|
* feat: add logging for scp command outputdiogo4642025-07-111-0/+9
|
* feat: create output directory if it doesn't exist in run commanddiogo4642025-07-111-0/+8
|
* fix: replace todo!() with bond0 interface for alakazam and kadabra machinesdiogo4642025-07-111-16/+16
|
* fix: add error handling for latency matrix dimension checkdiogo4642025-07-111-4/+12
|
* fixed log copyingdiogo4642025-07-111-1/+1
|
* fixed env var quoting when setting container variablesdiogo4642025-07-111-0/+2
|
* fixed reading schedule from stdindiogo4642025-07-111-4/+9
|
* added addr allocation policydiogo4642025-07-112-16/+204
|
* fixed net container builddiogo4642025-07-111-1/+3
|
* cargo clippy --fixdiogo4642025-07-103-20/+17
|
* clean enough for nowdiogo4642025-07-104-320/+433
|
* it works, now needs cleanupdiogo4642025-07-102-98/+890
|
* rust init snapshotdiogo4642025-07-093-0/+452
|
* Convert from Rust to Python project with uv supportdiogo4642025-06-291-133/+0
| | | | | | | | Remove Rust-related files (Cargo.toml, Cargo.lock, src/, target/) and restructure as Python project using uv for dependency management. Update project structure to match nova-oar-mcp style with pyproject.toml, .python-version, and proper Python packaging conventions. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
* Add P2P network setup script with interface and latency configurationdiogo4642025-06-271-2/+4
| | | | | | | | | | | | | - Complete Python script for OAR P2P network setup - LatencyMatrix class for loading and validating square matrices - Interface preparation and configuration with parallel execution - TC latency emulation using netem (WIP - fixing class issues) - Batch IP and TC operations for efficiency - Docker containerized execution for consistent tooling 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
* Add OAR job management and IP address allocationdiogo4642025-06-271-2/+123
| | | | | | | | | | | | | - Add clap for CLI argument parsing with job_id, addresses, and latency_matrix - Add serde/serde_json for JSON parsing of OAR job data - Implement oar_network_addresses() to get machine list from OAR job - Add address_from_index() to map indices to 10.0.0.0/8 IP addresses - Add machine list with bond0 interfaces for charmander cluster - Configure musl target build in Justfile for cluster deployment 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
* initdiogo4642025-06-271-0/+10
generated by cgit (git 2.34.1) at 2026-04-05 22:52:20 +0000