Basically following this guide right here: https://docs.px4.io/main/en/dev\_setup/dev\_env\_mac

• Looks like I need to use an x86 terminal. Neat.

• Also looks like we need to install home brew again for x86. Looks like it doesn't replace my existing one /opt/homebrew/bin/brew but rather adds it to /user/local/bin/brew.

• hmm... my terminal still thinks I should use the old brew even after adding control flow in the .zshrc to use the right one for the right arch.

• Ahh, I needed to update my .zprofile to not override what we just put in .zshrc.

• Looks like Python3 has tooling for x86 and Arm. woop woop.

• Hmm looks like I might need to manually link the python3 installed by brew cause right now it wants to use the the existing old one. I am going to see if px4 lets me do my thang.

Now I am installing Gazebo using this guide: https://gazebosim.org/docs/latest/install\_osx/

• ChatGPT says that I should use a docker container for "Stability or whatever". I am already this far so we will keep chugging along until something breaks.

• Also looks like there 3 flavors for Gazebo. Gazebo Classic, Gazebo Sim, and Ignite/Fortess? Classic is the clear old reliable and Sim might be too new and flaky. I am downloading sim for posterity however if it turns out to be buggy enough Ill move over to classic.

• Looks like we can skip running these steps

brew unlink tbb
sed -i.bak '/disable! date:/s/^/  /; /disable! date:/s/./#/3' $(brew --prefix)/Library/Taps/homebrew/homebrew-core/Formula/tbb@2020.rb
brew install tbb@2020
brew link tbb@2020
...

brew install px4-sim-gazebo

• Still need the following though:

brew install --cask temurin
brew install --cask xquartz

• Looks like I'll have three running processes:

1. PX4 SITL (C++)

• runs the autopilot

• outputs MAVLink on UDP

1. Gazebo (Classic or gz)

• runs physics + sensors

• talks to PX4 via a plugin/bridge

1. Your Python code (MAVSDK)

• connects to PX4 via MAVLink (UDP)

• sends offboard commands

• runs ML

• More detail on all of this can be found [[here]]

2/8/2026

Now that we have the toolchain setup we can actually see if we did it right: https://docs.px4.io/main/en/dev\_setup/building\_px4

• Okay so first step is to run make px4_sitl gz_x500.

• Looks like we can run make list_config_targets to view all of the targets available.

• Okay so things aren't working. Time for us to take a step back. I should probably use a python environment so that I avoid conflicts with my other python3 installations.

• Okay done, now I reinstalling of the python packages and making sure I don't use --user flag since that for the entire system.

• Okay that seemed to fix that issue. Now I am getting some errors related to the em package.

• Okay I got really confused with the .env stuff. I ended up neede TWO python environments. One is the Homebrew python environment (system tools) which will install PX4 toolchain stuff like px4-dev opencv gstreamer; toolchain + simulator + CMake deps. The other python environment i made was for the PX4’s Python scripts + code generation dependencies. This is where I will be running make px4_sitl gz_x500.

• Looks like cmake is looking for a libOpticalFLow.so file but I have a libOpticalFlowSYtem.dylib cause I am on a mac. I need to create a symlink using ln -sf libOpticalFlow.dylib build/px4_sitl_default/OpticalFlow/install/lib/libOpticalFlow.so and make sure i don't run make distclean or the symlink will get wiped.

• Well it looks like we got really close. PX4 started running but then the Gazebo simulator died. Looks like libOpticalFlowSystem.dylib depends on a different plugin @rpath/libOpticalFlow.dyliib. The runtime loader failed to find it. I am going to create another symlink in the place where the loader looked so that I don't have to move it around.

• Holy canoli that worked...thank god I wrote all of this down. I really don't want to have to relearn all over again.

I am seeing some warnings.

WARN  [health_and_arming_checks] Preflight Fail: ekf2 missing data
WARN  [health_and_arming_checks] Preflight Fail: No connection to the GCS
WARN  [health_and_arming_checks] Preflight Fail: No connection to the GCS
WARN  [health_and_arming_checks] Preflight Fail: No connection to the GCS
WARN  [health_and_arming_checks] Preflight Fail: No connection to the GCS

“Preflight Fail: No connection to the GCS”

This is normal if you don’t have a Ground Control Station (QGroundControl) connected.

PX4 is basically saying:

“I’m not armed / not ready because I don’t see a GCS.”

It does not mean the sim isn’t running.

we can Ignore it for now if we're controlling with MAVSDK.

Now I see this

FO  [mavlink] mode: Onboard, data rate: 4000 B/s on udp port 14280 remote port 14030
INFO  [mavlink] mode: Gimbal, data rate: 400000 B/s on udp port 13030 remote port 13280
INFO  [logger] logger started (mode=all)
INFO  [logger] Start file log (type: full)
INFO  [logger] [logger] ./log/2026-02-09/05_25_45.ulg
INFO  [logger] Opened full log file: ./log/2026-02-09/05_25_45.ulg
INFO  [mavlink] MAVLink only on localhost (set param MAV_{i}_BROADCAST = 1 to enable network)
INFO  [mavlink] MAVLink only on localhost (set param MAV_{i}_BROADCAST = 1 to enable network)
INFO  [px4] Startup script returned successfully

which means that PX4 started multiple MAVLink “instances”:

• Normal (high data rate) on one UDP port (you saw 18570 earlier)

• Onboard on udp port 14280 (low rate)

• Gimbal on udp port 13030

• plus the other onboard ports you saw earlier

So PX4 is broadcasting telemetry/accepting commands over UDP.

• Okay lets recap how we are going to be developing our Autonomous Drone:

Terminal 1 (PX4 + Gazebo)

From PX4-Autopilot/:

make px4_sitl gz_x500

Leave it running. You should see pxh>.

Terminal 2 (Your Python autonomy code)

From your autonomy project folder (NOT PX4):

cd ~/Projects/AI-Grand-Prix/your-autonomy-project
source .venv3.14/bin/activate   # or whatever venv you use for MAVSDK
python main.py

That Python script is where you call await drone.connect(...).

Important: You do not type await drone.connect(...) into the PX4 pxh> shell. That shell is only PX4 commands. To learn more about the architecture and how these systems communcate look at [[Development Environment Explained]]

Now we have to create our ML python environment.

• I created another venv for the Albatross project.

• I created a connect_test.py to test the connection to PX4 using the MAVSDK

• Looks like udpin for port 18570 is being used by px4. I think we can still use udpin but its best to go for a different upd port.

• we should use 14540 where udpin and udpout should be unused.

• We can check which port is currently being used by running lsof -i udp:14540

• I remember that whole architecture and the “3 modes” plan (full / partial / degraded) with the adapter layer + fault injector.

2/9/2026

• Alright so for the next day or so we are in system design mode. We should look into some existing autonomous vehicle repo's and try to glean as much as we can about how they structure and even solve their autonomous problems (without copying of course). This will give me a good Top-down system design that I start off with.

• I found this really nice paper on ArXiv thats called A Step-by-Step Guide to Creating a Robust Autonomous Drone Testing Pipeline

• "To manage this complexity, modern autonomous drone systems are typically organized as multi-module architectures, where each functional component (e.g., perception, localization, planning, control) operates semi-independently and interacts through well-defined interfaces (Liang et al., 2025). This design paradigm, often built on middleware platforms such as the Robot Operating System (ROS) (Quigley et al., 2009), mirrors practices in broader autonomous driving systems (ADS) (Deng et al., 2022). Indeed, the majority of ADS practitioners report working on modular systems rather than monolithic end-to-end models, citing scalability, debuggability, and safety as primary motivators (Lou et al., 2022)."

• "These nodes interact using standard ROS messages (e.g., sensor_msgs, nav_msgs, geometry_ msgs) and services, enabling asynchronous and real-time communication. Developers can also use visualization tools such as RViz (Kam et al., 2015) to inspect published topics (e.g., marker poses, planned paths) and use rosbag to record and replay test data."

I really like this Software Architecture they provide

2/11/2026

• I think to increase my understanding of modern CV, architecture, and planning algorithms I am going to use Litmaps.

• Then I need a good way to organize these documents in a way that will make it easy to synthesize the best algorithms for this drone. I think I'll use Zotero to keep track of the things I read and make a mental note of the most interesting parts for each of them. There is a way for me to use tags which will make it easy to group them up and do a real cross comparison. I might use the modules as tags since that's what I am looking to optimize. I'll even throw in an architecture tag in the case a paper proposes a new architecture.

• Finding papers and categorizing them should be a good portion of my time since the actual rules surrounding the competition are still up in the air. I still don't know if we will have the opportunity to train our models on simulations they provide or what kind of compute will be used in the Simulations and IRL drones.

• As I have been doing research I have realized that making sure my python modules works in realistic simulations is a must. After getting a simple flying module set up with Gazebo and PX4. I believe getting Isaac and AirSim working next week should be my goal.

2/12/2026

• Research, Research, Research.

• hmm, the closest competition I can find related to this one seems to be surrounding the A2RL which contained teams to only use a single rolling shutter CMOS camera and prohibiting any external aids at any stage.

• We should learn about the top competitors from that competition and what they used for their system architecture.

• Hmm... so the team was TU Delfts Mavlab. They've really been pushing the envelope reaching top speeds of 28 m/s. Well my goal should be to get to 40 m/s. High hopes but I think that's what it will take to ensure a victory. At least from a speed standpoint.

• Hmm so during the AlphaPilot/AIRR competition they used motion capture and in the A2RL they didn't. I feel like they some how use motion capture to train supervise a NN using it as Y and comparing it to Y-hat.

• They have definetly set the bar I can't argue with that so I need to meet that requirement. This might mean that I might need to get motion capturing figured out. Oh boy! I wonder if I can do that with Isaac.

2/24/2026

• Dang does time fly. Other things have kept me away from this little project but I have some time now.

• Moved my repo to github and tried to use git submodule to keep px4 as a dependency. Has turned out to be very annoying. Currently trying to get px4 to build right now.

• I first had to update px4_add_*.cmake file to use Wno-double instead of Wdouble.

• Okay so I had to nuke my old .venv-px4 cause it wasn't being able to find the right python3

• Great got it working again, only took like 3 hours...

• Going to set out to do a few things I mention back on 2/12/26 tomorrow.

• Also seems like running make px4_sitl gz_x500 doesn't automatically open up the simulator, so we need to open up a third terminal and use

2/25/2026

• Well after our whole refactoring yesterday, the simulator still isn't showing up properly. I see the console but not the the Gazebo Simulator.

• I may have found some easier instructions on how to get this project setup next time. https://docs.px4.io/main/en/dev\_setup/dev\_env\_mac

• Based on that documentation there is this script ./Tools/setup/macos.sh --sim-tools which should install anything I may have missed.

• Im going to be pretty cheesed if this works. Sometimes I need to just find the documentation ya know. Anyway I learned a little bit about Git, venv, and symbolic links which is pretty nice so growth none the less.

2/26/2026

• Okay so I sorta lied. Not really, I just thought I had done something I hadn't. So the simulator still isn't running and now I can't even get the console to work properly. Now I am having Codex take a look at it so I don't waste more of my after noon working on getting a damn simulator to work on my Mac. I have better things to do.

• Okay looks like Codex was able to do it. Took a while and it definitely had to do some messing with but it looks to be running again.

• Okay lets see where we are in terms of architecture. Oh yeah I was looking into this research paper monoracer . I should ask chat to read it and help me try and reproduce a similar architecture.

2/27/2026

• I think Chat and I have come up with a pretty good architecture for now. I think the best thing we can do is get an adapter working and have the virtual drone fly. That way we can learn what works and what doesn't for an adapter. Also making is that we don't have to change anything in the code to switch between simulators would be the end goal however we are not there yet.

2/28/2026

• Working on getting this virtual pony to fly

• Okay I need to take a step back cause Codex is starting to go crazy over here with the architecture. I need to get my barrings on what exactly the input and output should be for this project.

3/1/2026

Okay so this a quick overview of what we will be getting:

INPUT:

• image

• position

• velocity

• orientation

• IMU - accelerometer

• IMU - gyroscope

PROCESS:

• neural network

OUTPUT:

• roll

• pitch

• yaw

• throttle

So basically:

Coming in: Camera + Pose + Velocity

Processing: Neural network decides how to tilt.

Going out: Roll, Pitch, Yaw, Throttle

• ChatGPT gives a pretty good example of what we might see as an input.

One concrete “frame” example:

At `t_cam = 12.400s`:

**Incoming**
- Image shows gate center offset: `dx=+0.18, dy=-0.12`
- Velocity: `v = [6.0, 0.2, 0.0] m/s`
- Orientation: (quat) `q = [0.98, 0.01, 0.10, 0.15]`

- IMU window last 100ms:
	- gyro_mean `[0.02, -0.01, 0.35]`
	- accel_mean `[0.30, 0.05, -9.50]`

**Policy outputs**
- roll `+0.30`, pitch `+0.20`, yaw `+0.10`, throttle `0.64`

**Outgoing**
- same 4 commands (converted/scaled by adapter)

• Okay sick we got the propellers to spin however I keep getting arming issues and etc.

• Okay we got something working. That's progress. The drone can take off and fly. Simply little script does it with the help of chat. Seems pretty straight forward.

• I need to read through the docs on set_attitude and do some learning on Hz and and mavlink commands.

• Okay so stop the timer. It took 11 days or just about 40 hours to get the simulator working and the drone to fly. I could have probably done this on day 5 but I decided to refactor and start working on architecture for a little while. Anyway its nice that it didn't take too long. I was able to use ChatGPT to iterate through different versions of what works and what doesn't so now I have a really good example of what needs to happen to make commanding this drone to work. This will be a great reference when building my models and architecture.

• Since its march I think it should make sense to get a good underlying system for sending and retrieving messages. I think I will make data classes for input and outputs and make sure they implement base classes that have methods that do quarternion conversion and Frame conversions which should make my life much easier.

• Time to call it a night.

3/2/2026

• Okay so we got our drone to fly. Now I need to understand how its flying. I will need to do a bit of learning today and tomorrow to make sure it all make sense.

Mavlink Learnings:

• Mavlink is the protocol used between our python code and PX4 which is the controller and model in the simulation.

• Messages are defined by xml files and are also known as dialects

• Px4 is speaking Mavlink version 2 we see this when we run mavlink status

• The dialect is determined by the pymavlink which I believe is pymavlink/dialects/v20/ardupilotmega.py that means our pymavlink install is currently using the ardupilotmega dialect module as the generated Python definitions for messages/fields. That does not mean our vehicle is ArduPilot. It just means pymavlink is using that generated module as the message definitions

• PX4 telemetry basics & Offboard control examples (Python) These cover exactly what we’re doing with SET_POSITION_TARGET_LOCAL_NED and SET_ATTITUDE_TARGET.

Streaming concepts

Control stream

• 50–100 Hz setpoints (PX4 requires >2 Hz, but racing needs faster for control)

• Use one setpoint type consistently per run:

• SET_POSITION_TARGET_LOCAL_NED (velocity mode) for simple racing baseline

• or SET_ATTITUDE_TARGET (attitude+thrust) for more direct control

• don’t mix unless you’re doing the keepalive trick on purpose

Telemetry stream

• 30–100 Hz (depends what you subscribe to / what PX4 is sending)

• You should treat telemetry as “latest sample wins”:

• a background receiver updates latest_state

• the controller reads latest_state each tick

Camera stream

• 30–60 FPS typical

• Same approach: background thread decodes frames, stores latest

Therefore our architecture has 2–3 loops

1. Rx loop (telemetry)

   • blocks on recv_match()

   • updates thread-safe latest_telemetry

2. Camera loop

   • decodes frames

   • updates latest_frame

3. Control loop (fixed Hz) - reads latest inputs (frame + telemetry) - runs policy - sends setpoint - sleeps to maintain rate

The control loop should be the only thing that “owns” timing.

3/3/2026

• Okay now that I have a clearer idea of of streaming I think we can look at the Mono-racer research and as ourselves what methods are available for us to use and possibly understand their streaming process and understand their differences.

• Lets look over MonoRacers methods with our new found knowledge.

• Hmm still a lot of this I don't understand. EKF, PnP, QuAdGate corner extraction, GateNet segmentation.

• What I can understand now is that they probably had something like 6 asynchronoous loops running at different Hz sharing some state to perform operations. I wrote some notes on my ipad so I won't pollute the blog here but I think I have a good idea of what goes on in each loop at a high level. I think tonight I need to be clear on what the input and outputs are going to be and just write something up that works.

• Something really important that we need to look out for when they publish their module is whether they are using a step-based API or a push-based API.

3/5/2026

• Alright took a couple days off. Time to get back to work.

• Just got confirmation that we will be getting most of the simulation and API stuff on March 19th which give me only a couple days to get this foundation setup. Since programming takes time, I won't be writing much here unless I think it makes sense to for me to add for posterity.

• Okay in trying to write this adapter I've realized I've needed to basically build the underlying architecture and i've built up too many requirements in my head to keep track of. Time to write a little design spec for myself that I can update while I work on this code.

3/6/2026

• Okay time to glance through the design spec a little more and we should be able to get started on making this thing work without chatGPT. Starting small of course. I had chat gpt give a few structural pointers and I think we have come up with something I could adapt for my goals. First thing first is getting the adapter to work properly and getting this drone to fly.

3/7/2026

• Okay so in order to get the adapter to work properly I need to get a high level understanding of what the control loop and runner are going to be doing. I have been writing psuedo code for what I will need and I realized that a drone class will be very helpful for modularity and making the running code very readable. I think it will make sense if the drone takes most of the objects we will be using ie modules, sharedState, and adapter.

3/8/2026

• Alright, we are making good process. I think I have an okay architecture right now that should at least let me test a few unfinished modules out using a gazebo adapter. The question is whether the abstractions I took care to create will end up making adapting this code to other simulators and drones easy-ish. We will cross that bridge when we get there. I probably didn't have to do all this abstraction however, for readability purposes I decided that putting in the effort would way off later for my sanity. One part that I really wanted to get right was the multiple threads running at the same time. I have never worked with multiple threads like this so I have no idea how their characteristics will influence the Autonomy Stack during flight. I have some guesses where issues will appear with thread loops running at different Hz but I could just be making it up. I should look into this at some point.

• Alright sick, we got all of the code written out. Writing the adapter was quite easy given the fact that we had written a test script that essentially had all of the functions we need like sending attitude data, request_arm, etc.. This will probably be MO when making an adapter for other simulators as well 1. test script 2. transfer to adapter.

• Super excited to test this out. Tomorrow is my Birthday so I may or may not have enough to do so but if I do I will most likely be debugging random issues. I expect to have to wrangle with python syntax and and timing issues. I added a lot of timing functionality based on what chat had recommended to me but now that I have most of it written out I need to do some investigating into the best solution.

• Anyway with this written I think we completed out second goal which was Create a system architecture for Albatross in total it took around 1 month of development to get to this point. not sure how many hours but 17 days x 3 hours per day is 57 hours give or take. Not too bad. A weeks worth of work and some change.

• When the competition comes around I'll have to put in more elbow grease. I hope the work that I have put it now will help me focus on the main goal of the competition which is to create an autonomy stack for a drone. This will no doubt be way harder to get right and will most likely require me to rewrite some parts of my architecture but hopefully not too much.

• I think after getting a simple take off and fly forward working with my control loop and new architecture, I'll start working on getting adapters created for Issac and AirSim. I will try to make this my main goal for the week. The competition really starts on March 19th which is when we get the API from Andruil. So that gives me about 10 days to get familiar with Issac and AirSim which may or may not be helpful when training my Autonomy Stack.

3/9/2026

• I really like this project a lot. Its the evening of my birthday and I can't help but sit down and work on it. There's just something about playing music and losing myself in designing and coding.

• Yesterday I was able to connect all of the layers of my Autonomy stack so today I'll be mostly debugging. I will add some updates here if I think them important.

• Nice thing about this project log is if I am ever curious about the code I wrote I can always just look at the commits along with each of these logs.

• Okay so most of the fixes I did were import fixes and some cleaning up. The script runs no problem but now the drone won't fly. The drone gets armed but it won't take off. I know my environment is working cause the old test script I wrote still works.

3/10/2026

• This is what I get from my new architecure from PX4 Logs:

pxh> INFO  [mavlink] partner IP: 127.0.0.1
INFO  [commander] Armed by external command
INFO  [tone_alarm] arming warning
INFO  [commander] Disarmed by external command
INFO  [tone_alarm] notify neutral
INFO  [logger] closed logfile, bytes written: 10334763

• And this is what I get from the test script code:

INFO  [commander] Armed by external command
INFO  [tone_alarm] arming warning
INFO  [logger] Start file log (type: full)
INFO  [logger] [logger] ./log/2026-03-11/05_21_22.ulg
INFO  [logger] Opened full log file: ./log/2026-03-11/05_21_22.ulg
INFO  [commander] Takeoff detected
WARN  [commander] Disarming denied: not landed
INFO  [tone_alarm] notify negative
WARN  [failsafe] Failsafe activated
INFO  [navigator] RTL: start return at 31 m (30 m above destination)
INFO  [tone_alarm] battery warning (fast)
INFO  [navigator] RTL: land at destination
INFO  [commander] Landing detected
INFO  [commander] Disarmed by landing
INFO  [tone_alarm] notify neutral
INFO  [logger] closed logfile, bytes written: 14709756

• What is super weird is that if I try to run my new architecture directly after running my test script, the drone flies perfectly.

INFO  [commander] Armed by external command
INFO  [tone_alarm] arming warning
INFO  [logger] Start file log (type: full)
INFO  [logger] [logger] ./log/2026-03-11/05_32_18.ulg
INFO  [logger] Opened full log file: ./log/2026-03-11/05_32_18.ulg
INFO  [commander] Takeoff detected
WARN  [commander] Disarming denied: not landed
INFO  [tone_alarm] notify negative
WARN  [failsafe] Failsafe activated
INFO  [navigator] RTL: start return at 76 m (76 m above destination)
INFO  [tone_alarm] battery warning (fast)
INFO  [navigator] RTL: land at destination
INFO  [commander] Landing detected
INFO  [commander] Disarmed by landing
INFO  [tone_alarm] notify neutral
INFO  [vehicle_magnetometer] 0 (197388) EST:0 offset: [0.000, 0.000, 0.000]->[0.003, -0.004, 0.002] (full [0.INFO  [logger] closed logfile, bytes written: 28366799

• I think I found the problem. Syntax Syntax Syntax.

# runner.py
...
self.drone.offboard <-- do you see the problem here?
...

• Wasn't actually calling the offboard function. The test script was setting this up for the new architecture. Silly mistake but it didn't take me too long to find it so I'm a happy camper.

• Okay now that we got this thing flying, what should I clean up.

• Right now I have no idea of telling if my drone is actually doing what I want it to. Sure it gets up and flies away but I don't know if its doing what I intended. For instance I don't tell it to rotate but some how it ends up making 90 degrees turn and then flies upward. Why 90 degrees? I wonder what the standardized way for logging out flight data and how I can make it easier to understand.

• Chat says that since I am setting the Yaw to be 0 it could be sending the drone to NED yaw 0 which I guess the drone doesn't start off facing. I will need to read through the docs to understand this more. I will also need to make sure that this is standard for other simulators or things could get complicated.

• I guess now it makes sense to set up an extendable testing infrastructure.

• Also if you want to learn more about what the architecture looks like at this stage, I had copilot create some documentation outlining it. [[Albatross Autonomy Stack v1]]

3/11/2026

• Before going crazy and cleaning up the code too much, I think its best I start finalizing what models I will need and what modules they should go into. The goal is that each module performs a specified task.

• Time to do some more research.

• ...Couple Hours Later...

• Wow, I just got some great advice from ChatGPT using its extended thinking. I have a pretty clear idea about what modules I will need to create now, what I need to refactor now to make those modules work and what phases I should build things in. All of this will be added to a separate document for me to reference later. Truly some great advice.

3/12/2026

• Today I have just been reading through all of the advice Chat gave me and trying to piece together a good to do list for myself. I also want to do my best to understand why I am implementing everything. I want to understand what's happening.

• I started writing a Albatross V2 - Design Specification which should come in handy when I start developing. Its mainly based on Chatgpts advice with my own questions and thoughts sprinkled in so that I can revisit them during development.

3/13/2026

• Started working a little late so I am a little tired. Going to try and walk through the new architecture and see if there is anything I am missing.

• I am sort of slow walking through chatGPTs proposal and making notes of anything I don't understand or implementation examples that are missing core ideas mentioned in the document.

3/14/2026

• Just reviewing the new architecture to make sure everything looks good.

3/16/2026

• Took two days to rest up a little. I like to think that breaks are healthy. The API should be coming out this Thursday so I have to get moving on implementing my new architecture. I think I need to do a little review on a few things before getting started, but I definitely want to get started today or tomorrow.

3/19/2026

• Had a few things come up, back to work. Today the design spec came out. Good news! They want to use Mavlink with UDP SITL which is exactly what our first adapter is built to handle. Got super lucky there.

• Before going off the rails I think I'll read through specification first and get an idea of what needs to change in the architecture. Based on my brief look through I seem to be on the right track.

What the spec tells us:

• communication interfaces between contestant software and the simulator

• control input requirements

• telemetry interfaces

• vision data interfaces

• simulation timing and performance constraints

• virtual race environment definition

• qualification run requirements

There are a few things that caught my eye

1. Sections 4.1-4.3 are about MAVLINK. We basically have the adapter close to perfect and what we might need for this competition which is great.

   • Although there are things I am unfamiliar with like:

     • SET_POSITION_TARGET_LOCAL_NED

     • TIMESYNC

     • ODOMETRY

2. Section 4.4 Timing

   • Physics simulation rate: 120hz <-- What does this tell us exactly? Telemetry rate?

   • Recommended command rate: 50-120 Hz <-- This is signifacantly lower than my 800hz goal lol

   • Minimum heartbeat rate: 2hz <-- I am not sure if we are implementing this properly atm

3. Section 4.5 Telemetry

   • vehicle attitude <-- we will be getting this using ATTITUDE

   • orientation <-- I think this will come from where? IMU?

   • linear velocities <-- This will come from the ATTITUDE as well I believe

   • system status flags <-- where is this coming from?

   • simulator navigation reference data <-- What does this mean?

4. Section 4.7 Software-in-the-Loop Bridge: The simulator provides a low-latency UDP SITL bridge enabling external AI controllers to exchange telemetry and control commands. <-- What do they mean by external AI controllers? Is this just referencing the normal setup for reading and sending controls and telemetry or is this something else entirely.

5. Section 5 is all about the environment which is really important.

6. Section 5.1 Runtime Environment. We get to choose what python environment we get to use. interesting.

7. Section 5.2 Client Responsibilities

   • Establish MAVLink communication (GOOD TO GO)

   • maintain heartbeat messages <-- Not super confident I understand this in our current code base.

   • Send control commands

   • process telemetry data

   • process vision stream data

8. 5.3 Intended Control Architecture.

   • Vision + Telemetry <-- (platform and vision adapters take of this)

   • Perception

   • Planning

   • Control

   • Pilot Commands

   • Stabalized Controller <-- Is this refereing to our safe_guard policy?

9. Section 6 Example Control Session

   • This should be easy to implement using an specific competition orchestrator.

   1. Client initializes MAVSDK

   2. Client connects to simulator endpoint

   3. Simulator transmits HEARTBEAT

   4. Client streams control commands

   5. Simulator applies commands

   6. Telemetry and vision streams returned

10. Section 7. Compliance. NO HUMAN INTERACTION DURING TIMED RUNS.

11. Section 8. Round One Qualification Phase

    • Round One verifies that contestants software can successfully navigate the racecourse.

    • Start Gate intermediate gates and finish gates

    • Maximum Run Duration is 8 minutes. The difficulty of this requirement will depend on how many gates there are and how fast my drone must be able to fly during this round.

I asked chat how this might change our current architecture and plan and its answer is [[Architecture Review based on Tech Spec|here]]

Module Refactor + SharedState Update per Tech Spec:

• The new architecture should still be green to go.

• Here is the first thing we need to do, update how modules work. Reason is because right now our modules are all in one place and the idea is that these modules should be used by other services such as a training and replay service. This won't be possible if they are being used soley as a part of the racing pipeline. Moving the logic for each module out of the core/ directory into a library directory will help with this division for training and replay purposes.

• The Plan:

1. Add all necessary files I think I will need for this new architecture

2. Move logic out of the control_module.py so that it becomes a thin wrapper meant to direct to the actual logic somewhere else. This can also be paired with updating the sharedstate object and using the newly released design spec.

3. Update the current module_base.py too

Replay Logging and Replay Execution:

• Also something really important I learned is that logging is also used for replay. This will probably be the second thing I implement.

• Why would we want replay?

  • You want replay so you can do things like:

    • run one sim flight once

    • save all sensor inputs and module outputs

    • later rerun just perception on the same frames

    • later rerun estimation on the same IMU + detections

    • later compare old policy vs new policy on the same observations

• and so replay logging should let us answer:

  • what did the sensors produce?

  • when did they produce it?

  • what did each module see?

  • what did each module output?

  • what command got sent?

• You might wonder if we could use a global tick to keep track of what was going on at each tick but this is essentially difficult because we have multiple threads and a single global tick could become misleading. We currently have platform telemetry pump, camera frame pump, perception thread, estimation thread, control thread, command stream thread etc. these do not all run in lock step.

• Now a tick for each thread will still be useful since you we would be able to log

  • module name

  • local tick index

  • tick start time

  • tick end time

  • input ages

  • output sequence

  • deadline miss yes/no

• however replay itself will still be driven mostly by timestamps and event order.

• There are two ways to use the replay once we have the logs for a module after a flight run.

  1. Mode 1: Fast-as-possible replay.

     1. Used for offline testing

     • read next logged event

     • publish it immediately

     • run modules as fast as possible.

     1. This is best for:

     • debugging logic

     • unit/integration tests

     • comparing versions

  2. Mode 2: Real-time replay

     1. Used for timing validation

     • preserve original event timing gaps

     • sleep according to logged timestamps

     • simulate orginal cadence

     1. This is best for:

     • Checking latency behavior

     • testing freshness logic

     • observing thread behavior close to live execution.

• Here is a [[Modules Implementing Replay Logging#10) What exactly should modules log for replay/debugging|list]] of modules that will need replay/debugging implemented. I will of course start off with the control_module.py to test the overall logging and replay system. I also had questions about the difference between a sequence and a local tick and chat answered [[logging and replay#What is the difference between a lock tick and a sequence?|here]]

• Side note, when we are implementing a monotonic timer we have two options:

  • monotonic() and per_counter_ns()

• Both are monotonic timers but perf is newer semantically for benchmarking while monotonic is the clearer API semantically for deadlines/timeouts/scheduling. More details from chat here about using either one. End of the day we should probably choose one to keep the code consistent and clear.

• We'll use it for these things:

  • module tick timing

  • record ordering within a run

  • freshness checks

  • replay timing

  • measuring compute durations

• To keep track of human time (as in wall clock) we should use time.time()

• More information about using wall clock and the performance counter can be found [[Using the wall time and performance counter|here]]

3/21/2026

• Past two days I have been traveling so haven't had much time to sit down and work. Making time for fun things like this require time that I don't always have. Consistency is more fun when its challenging.

• There are a few things I want to jot down right now that I will need to come back to. After reading through chat's advice after reading the specification [[Architecture Review based on Tech Spec|here]] it mentions that the command loop have a cap of running at 120Hz which is very low. It also mentions something called a Watchdog which I am not sure its purpose. I also need to look into the importance of the TimeSyncing and how to use it my advantage. I will also need to look into "Stabilized Controller" which is new vocabulary for a layman such as myself. This may or may not be something I have planned, but its a module so I can move forward with coding.

• Implementing V2 now.

• Okay I actually need help getting a clear SharedState written up based on the spec. [[SharedData Revamp|here]]

• Okay nice, I think we cleanly refactored the sharedstate and command classes. shared data will have base_types which will be used either as a LatestValue or RingBuffer Topic. LatestValue and RingBuffer Topic will abstract away locks and the complexity of publishing, getting freshness, and storing sequence information for future logging and replay purposes.

• We also updated the types for commands by adding a safe_action_type.py as another layer along side action_type.py as an internal version of what will become the command_type.py.

• Its getting late so I am going to call it there. Tomorrow I would like to start getting the control_module revamped keeping the logging and replay in mind which may or may not get worked on at the same time.

3/22/2026

• Okay time to work on the control_module with the SharedState Refactor in mind. Right now I am following this guide I got from chatGPT [[How to refactor using existing architecture#Example control_module.py|here]].

• Didn't get too much work done. Had to stop and do some vacation stuff.

3/24/2026

• Alright we are back. I had my nice 4 day vacation which really made it hard to focus on this project. Not to mention the headache of using one screen for 4 windows.

• My schedule is so limited I really need to be consistent so that I don't have to go through an entire project day ramping back up. Thankfully there shouldn't be too much else getting in the way as far as I can tell.

• Last I remember I was starting to work on moving the core logic for the control_module.py out into a policy library. I remember getting a little stuck cause I had already created a pretty good shared data models but in order for the controller to work I need an observation that matches the type of data type but I haven't made it yet.

• I also started getting really confused on what the purpose of the observation is. I had chat make a dumb heuristic policy and even that had me scratching my head. So I guess I need to do a little bit of learning before diving deeper into implementation. Through out this process I need to keep in mind the logging replay.

What an Observation is:

An observation is a compact summary of the world that the controller consumes.

It is not:

• raw MAVLink Telemetry

• Raw Camera Pixels

• Raw IMU History It is:

• the "few numbe rthe policy needs right now" For our current design, that means numbers like:

• bearing: how far left/right the next gate is in the image

• elevation: how far up/down the gate is

• scale: how large the gate appears, as a proxy for distance

• yaw_error how rotate the gate looks

• derivatives of those values

• confidence So the flow would be:

• telemetry + vision → estimation logic → Observation(vec, mask, valid) → control policy

• That is why our ControlModule reads shared_state.estimation.observation.

• Eventually we will need that observation to come from a real estimation script but for now I need to do a little end-to-end testing with this new setup to make sure I can confidently start building more.

• For now I will create a temporary module called DummyObservationModule which will read one or two easy topics like attitude construct a valid observation and publish it at a fixed rate. for now we can make the observation show that the gate is straight ahead at all time to keep this relatively deterministic.

• This will let us verify that our topics work, module scheduling works, ControlModule can read observations, heuristic policy generates actions, safely converts action to commands and command loop sends them.

It should only prove that the framework dataflow works:

1. adapters publish sensor topics

2. dummy module publishes observation

3. control publishes policy action

4. safety publishes command

5. adapter sends command

6. logs show healthy rates and freshness

What our Heuristic Policy is doing

• Our heuristic policy is basically a hand-written “fly toward the gate” controller.

• So the whole policy is:

  • line up horizontally

  • line up rotationally

  • trim height

  • move forward when alignment is decent

• Its already late so I need to pause here but tomorrow the goal is to finish understanding the observation module and then test everything out. Then the next goal is to start getting a logging and replay working for the Control_Module.py and working on understand how something like that would work.

3/25/2026

• Got about 2 hours to get some real work done here. We just finished looking through the main functionalities that come from the dummy_observation_module.py.

• Okay just for self reference, the new module_type (which is a base type inherited by all modules) by default defines tick, create_thread, and a run_loop. The run_loop method is a private method used by the create_thread method to make sure we don't have to rewrite too much of the thread logic.

1. create_thread is called by pipeline to start up all modules (except for the Control and SafetyModule)

2. create_thread calls _run_loop which in turn calls tick.

3. tick is implemented by the module and should route to the specified library with implementations for that module.

• Okay I understand how the observation module works. Now I need to get a grasp of what's going on with the HealthModule stuff going on. I am assuming that it might be good to have the _publish_health as an method that can be added to other methods like tick to store state information. The problem is that I don't think i've created a ModuleHealth even though its referenced in shared_data.py. I will need to ask chat what this was meant to be and what an implementation would look like.

• Anyway, another short day but good progress. Its getting late so calling it for tonight.

3/26/2026

• Alright back to it. Time to learn about the ModuleHealth and what its doing and why we need it.

• Okay so chat says that the ModuleHealth is basically our module's status report to the rest of the runtime. Its not part of the racing/control algorithm itself. It is part of the runtime supervision/ observability layer.

• Each module is doing real-time work in its own loop. ModuleHealth Lets the system answer questions like:

  • Is this module running at all?

  • is it running recently, or has it stalled?

  • is it getting stale inputs?

  • Is it currently healthy, stale, or faulted?

  • What was the last thing it complained about?

• So the purpose is:

  • debugging

  • runtime monitoring

  • dashboard / prints

  • Watchdog logic

  • log/replay diagnostics.

• Without ModuleHealth, if something stops working you just see:

  • The drone is not behaving right

  • maybe commands stopped making sense

  • maybe no detections are happing

• But we do not know which module is the problem

• Although we said that ModuleHealth is not meant to part of the racing/control algorithm itself. we can use it to drive automatic safety behavior using WatchDogs.

• “Stale” usually means:

  • the module itself is still running

  • but its inputs are too old or invalid to produce good output

• Also ModuleHealth is not "health of the algorithm" its not saying. the controller is mathematically correct, the detector is accurate, the estimate is optimal. All its saying is "runtime module is functioning in an operational sense" which is the simplest thing we need working for the algorithm as a whole to work.

• The important part now is how do we use the ModuleHealth information. For now printing the health of the modules seems fine for now, later we can have it so that a module prints its own heartbeat with some optimizations.

• Okay I think I am getting really close to being able to test out this new Architecture. All I need to do is do my best to fix all of the imports and _init_.py files. I also need to make sure that the pump logic uses the new shared_state, topic system and make sure that each module is reading and writing using this new system. If a module isn't using this new version of the shared_state module then I should make as TODO.

• After doing all that then I can start working on logging records for replay capabilities.

• Cool, calling it a night.

3/27/2026

• Started feeling a little sick this morning but really want to work on this so here we are. Going to work as much as I can without overworking myself. I want to use this weekend to finish this architecture revision.

• Right now I am fixing imports and making sure that sharedstate is updated across the board since we updated it to use topics.

• First thing that needed fixing was the main.py but it was a simple import fix

• Next thing on my mind was fixing the GazeboPx4Mavlink Platform Adapter to use the updated version of the sharedState. Chat gave be some good advice [[architecture v2 gazebo_px4_sim_adapter.py|here]].

• Okay things got a little bit more complicated. I asked chat to help me refactor and it started adding a whole bunch of stuff that I need to learn about because it mostly comes from our sharedstate update. There are a few ideas that are new to me.

• Like what's difference between pilot_command, command and command_tx_history? Also what heatbeat_rx? What is SystemStatusData and how is this different from HeartbeatData? What is heartbeat_tx? and what is timeSync and why do we store it?

pilot_command is our internal representation of:

• roll

• pitch

• yaw rate/ yaw angle

• thrust

  • It is the desired pilot-style setpoint command is the command that the command loop.adapter should treat as the current official outbound command.

In a simple stack, pilot_command and command may be identical. For now we can think:

• pilot_command = sematic layer name

• command = what the transport thread actually reads and sends

Why keep both? Because later we may want a tiny extra step between them, such as:

• transport-specific conversion <-- what the heck is this

• command-rate resampling <-- what the heck is this

• offboard-loss fallback insertion <-- what the heck is this

• command validity checks <-- what the heck is this

command_tx_history This is just a ring buffer of commands that were actually transmitted. We will use it for:

• debugging "what did we really send?"

• replay/logging

• rate checking

• detecting repeated stale commands

• comparing inteded command vs transmitted command

heartbeat_rx vs heartbeat_tx These are just:

• received heartbeat

• transmitted heartbeat

Note these are both type HeartBeatData

heartbeat_rx The latest heartbeat you received from the simulator/autopilot side. We'll Use it to know:

• connection is alive

• armed status

• mode-ish status

• system status-ish fields

heartbeat_tx The latest heartbeat your client sent outward.

Why store it?
Because the spec says the client is responsible for maintaining heartbeat messages, and it gives a minimum heartbeat rate of 2 Hz.

So storing heartbeat_tx lets us answer:

• are we actually sending them?

• when did we last send one?

• is our heartbeat loop stale?

So:

• heartbeat_rx = “what we heard from them”

• heartbeat_tx = “what we sent to them”

HeartBeatData is the normalized representation of the MAVLink Heartbeat message itself It is about:

• link presence

• base mode / custom mode

• armed bit

• autopilot/type identifiers

• general heartbeat state So it answers:

• did we receive a heartbeat?

• what did the raw heartbeat say?

SystemStatusData This is our cleaner, normalized operational status view derived from one or more telemetry sources, often including heartbeat.

Why keep both?

Because sometimes you want:

• the raw-ish transport message (HeartbeatData)

• and also a simplified internal status object (SystemStatusData)

That separation is useful for debugging.

Example:

• HeartbeatData says base_mode=209

• SystemStatusData says armed=True, offboard_enabled=False

The second one is much easier for modules/watchdogs to use.

TimeSyncData/timesync The spec explicitly lists TIMESYNC as a supported MAVLink message.

What it is It is timing information exchanged between endpoints so you can estimate:

• time offset

• round-trip delay

• maybe latency/jitter characteristics

Why store it? Even if you do nothing fancy with it at first, it is useful for:

• logging/debugging latency

• understanding sim/client time relationship

• later delay compensation

• better replay analysis

For example, if you later notice:

• observation age is weird

• control is lagging

• frame timestamps and telemetry timestamps do not line up well timesync data gives you a place to investigate that.

Do you need to use it immediately? No. You can store it now and ignore it at first. That is still worthwhile because:

• the spec names it explicitly

• timing matters in your architecture

• it is much easier to start logging it now than retrofit later

• A question I need answered at some point is what exactly are we supposed to be sending 2hz as a heartbeat? I don't think in our initial implementation we sent anything out other than our command which I guess may have been high enough to beat 2hz. Do I need to change anything here? Chat actually answered this [[architecture v2 gazebo_px4_sim_adapter.py#2. Heartbeat TX should be explicit|here]]. Seems like we should explicitly be sending heartbeats back to the simulator using HearBeatData and keeping track using `hearbeat_tx.

• Okay I updated how we print hearbeat, systemstatus and positional information in the pipeline. I used [[Architecture v2 - Print Loop Refactor|this]] advice from chat.

• I also created a send_hearbeat thread to meet that 2hz requirement.

• I also updated the send_command thread to use our updated SharedState type. [[Architecture v2 - Command Loop Refactor|This]] is advice I got from chat.

• Okay so I think we were able to update the platform adapter fairly well. All I need to do is ask chat to make me some types for the HeartBeatData, OdemetryData, TimeSyncData, and SystemStatusData.

• From there I can start piecing everything together and mentally run through the whole process. I will add some comments in places where I need more context. Fixing imports is still something that needs to be done.

• Again, after all this I will be testing the new architecture and then jumping to logging records and adding a replay capabilities.

3/28/2026

• Not feeling sick thankfully. Okay so it running through the whole setup again.

• Looks like we have everything setup well for another test. Im pretty nervous. I hope it won't be too much of a hassle to get it working.

• Okay while I am working on this I started asking chat on the replay system we will be potentially implementing next.

• Okay so for some reason our x86 terminal which should be running using rosetta isn't working. When I run uname -m it shows arch64 instead of x86. It seems like after restarting my computer, rosetta was disabled for iterm. To reenable it all I had to do was open x86 Iterm in finder cmd+i and select rosetta.

• Great we got the drone to fly using our new architecture and using some dumbed down versions of control, observation, and safety modules.

• I added colorama as a python dependency so that I can view the logs a little easier it was really hard to see right now. I can add more tomorrow. Once I have that setup I can go through the logs with a fine comb and make sure everything is working as intended.

3/29/2026

• Okay so I finished cleaning up the logging. Colorama does make things easier.

• the age_ms for hb_tx is messed up and should something I revisit.

• So it looks like we have our drone flying solely on the take off bump cause we don't ever receive any commands from the control module. The reason is because because the control module is in a stale state. I know this because of my logs.

INFO [moh] control: status=stale, age_ms=4.50225, info=no_observation

• Why this is happening will probably be something I investigate tomorrow since I had a long day and I am pretty tired right now. Working on this should give me a good opportunity to learn more about observations and how versions of the control module will use it.

Next Steps

• A day or so ago I started a conversation with ChatGPT about the logging and replay system. I want to make sure I understand it well before going down a week long project to implement it. I have to move faster now that we are getting close to April. Everyday has more weight.

• If implementing this replay feature will only provide minimal returns then I will omit it completely. I may start working on other parts of the drone's autonomy stack and grow it out and the system may be implemented if needed.

• I can see that there are a few modules that would deeply benefit from a system like replay. Identifying what those are ahead of time will help me decide if this is worth sinking time into.

• I need to review the required modules for the first competition and hone in on working on those. That will help me answer if replay is necessary.

• I need figure out at a high level how each of the modules will work and what the the pipeline will look like between them. This will help me determine the testing boundary between all of them and determine of replay works. Replay works for individual modules and stack slices to see how things improve as a whole when things get put together. It might always make sense to test a module on its own using replay and then as part of a stack slice all the way till we reach output commands. How will we keep track of improvements across stack slices and individual modules is something I need to ask chat about.

• Here is the workflow that ChatGPT thinks is crucial for developing the autonomy stack.

Step 1: replay

Use replay for:

• fast debugging

• regression testing

• module comparison

• offline metric improvement

Step 2: live sim

Then validate promising changes in the simulator for true closed-loop impact. That combination is what makes replay valuable.

• I agree this seems very valuable. Something to think about.

3/30/2026

• Okay today is Logging and Replay investigation day.

• Time to decide how we are going to implement this.

• Below are two questions that I had chatgpt answer:

Question 1: Can you remind me how the logging and replay is going to work? how am I to actually use the replay for testing and debugging once its working and how will I be able to use it? will it rerun a flight for a single module like its inputs and outputs? will it run all modules? Will it work with the sim? [[Logging and Replay Overview|Chats Answer]]

• The core idea is: live run → log timestamped events → replay those events later through the same topic system

• What gets logged: Topic publish records and Module tick records.

• Topic publish records are the important ones. They capture data flowing through the system, like:

  • IMU

  • attitude

  • local position

  • camera frames

  • detections

  • observations

  • actions

  • commands

• These are the records that let you reconstruct what happened.

• Module tick records are for introspection. They tell you:

  • when a module ran

  • what input seq it used

  • how long it took

  • whether it was stale or okay

• These are mostly for timing/debugging, not for driving replay.

• By default, replay should mainly republish source inputs, not everything.

• So the main replay inputs should be:

  • sensors.imu

  • sensors.attitude

  • sensors.local_pos

  • sensors.heartbeat

  • sensors.camera_frame

  • sensors.camera_info

• Then your current modules regenerate:

  • detections

  • state estimates

  • observations

  • actions

  • safe actions

  • commands

• That is the most useful form of replay, because it lets you test new code on old runs.

• The three main forms of replay will be [[Logging and Replay Overview#1. Whole-stack replay|Whole-stack replay]], [[Logging and Replay Overview#2. Single-module replay|Single-module replay]], and [[Logging and Replay Overview#3. Timing replay|Timing replay]].

• Replay is best for debugging:

  • perception bugs

  • estimator bugs

  • stale-data problems

  • controller behavior on the same observations

  • refactors that might have broken timing or dataflow

• It is much faster than rerunning the whole simulator every time

A concrete example

• Say you do one live run and log:

  • camera frames

  • IMU

  • attitude

  • observations

  • actions

• Later you improve your detector.

• Now you can:

  • use replay to republish the recorded frames

  • run the new detector

  • compare new detections vs old ones

  • optionally continue through the estimator and control modules

• That tells you whether the detector change actually improves the pipeline, using the exact same scenario.

Question 2: If we are changing the code won't this affect the imu and telemetry and frames? How will this let us effectively test whether there has been an improvement? [[Replay's Limitations and Focus|Chats Answer]]

• Taking a look at an older conversation I had with chat [[logging and replay|here]] is what we came up with as an implementation for logging and replay. I used this older conversation as context for some ideas of what the code would look like. I think we've come up with a fair idea, tomorrow we will implement it.