Maybe you are adding the same point multiple times to to_visit. I don’t know ruby but couldn’t see a check for visited points before adding, and to_visit appears to be an array instead of set, which can store the same point multiple times.
Language: Python
Decided to use a graph to solve (which expanded the size). Part 1 was cycle detection, and part 2 was flooding of the outside.
Language: Python
Python, used a number map to search positions near symbols.
https://github.com/massahud/advent-of-code-2023/blob/main/day03/day03.ipynb
Cars should come with ✖️⭕🟪🔺