Merge #1451

1451: Work around xtensa deadlock, take 2 r=Dirbaio a=bugadani

This PR is another go at trying to do something with #1449. The commit was part of the previous attempt but mistakenly discarded as I still experienced lockups. However, after further testing, it looks like that lockup is caused by something else.

This is a manual, "cpu-local" critical section impl that should be good enough on dual-core CPUs, although the implementation still contains `SIGNAL_WORK_THREAD_MODE` which is absolutely not correct on dual-core. This approach was chosen because:
 - not taking the global lock technically allows the second core to run
 - wrapping the signal read and the sleep in a critical section prevents a race condition that would cause the CPU to sleep longer than ideal if an interrupt hits after reading, but before sleeping.

Co-authored-by: Dániel Buga <[email protected]>

1 files changed, 20 insertions, 12 deletions

diff --git a/embassy-executor/src/arch/xtensa.rs b/embassy-executor/src/arch/xtensa.rs
index 61ea92c16..017b2c52b 100644
--- a/embassy-executor/src/arch/xtensa.rs
+++ b/embassy-executor/src/arch/xtensa.rs
@@ -63,21 +63,29 @@ mod thread {
             loop {
                 unsafe {
                     self.inner.poll();
+
+                    // Manual critical section implementation that only masks interrupts handlers.
+                    // We must not acquire the cross-core on dual-core systems because that would
+                    // prevent the other core from doing useful work while this core is sleeping.
+                    let token: critical_section::RawRestoreState;
+                    core::arch::asm!("rsil {0}, 5", out(reg) token);
+
                     // we do not care about race conditions between the load and store operations, interrupts
                     // will only set this value to true.
                     // if there is work to do, loop back to polling
-                    // TODO can we relax this?
-                    critical_section::with(|_| {
-                        if SIGNAL_WORK_THREAD_MODE.load(Ordering::SeqCst) {
-                            SIGNAL_WORK_THREAD_MODE.store(false, Ordering::SeqCst);
-                        } else {
-                            // waiti sets the PS.INTLEVEL when slipping into sleep
-                            // because critical sections in Xtensa are implemented via increasing
-                            // PS.INTLEVEL the critical section ends here
-                            // take care not add code after `waiti` if it needs to be inside the CS
-                            core::arch::asm!("waiti 0"); // critical section ends here
-                        }
-                    });
+                    if SIGNAL_WORK_THREAD_MODE.load(Ordering::SeqCst) {
+                        SIGNAL_WORK_THREAD_MODE.store(false, Ordering::SeqCst);
+
+                        core::arch::asm!(
+                            "wsr.ps {0}",
+                            "rsync", in(reg) token)
+                    } else {
+                        // waiti sets the PS.INTLEVEL when slipping into sleep
+                        // because critical sections in Xtensa are implemented via increasing
+                        // PS.INTLEVEL the critical section ends here
+                        // take care not add code after `waiti` if it needs to be inside the CS
+                        core::arch::asm!("waiti 0"); // critical section ends here
+                    }
                 }
             }
         }