Yes, panels can be connected directly to the Target on their own network (or common to the simulation hosts), and indeed this would result in the optimum configuration from a latency perspective. There is no need to pass the data to the host and have that forwarded.

If there was a requirement to receive some of the panel data in the host simulation, the Target can forward some, all, or any data to the host as needed.