Linux Kernel. U-Boot. BSP. Device Tree. Drivers. Yocto. Buildroot. systemd. Networking. PREEMPT_RT. Security. Containers. GPU/NPU. Multimedia. OTA. A/B Updates. Production Provisioning. Long-Term Maintenance.
An Embedded Linux product is not simply: Linux Kernel; running on: ARM Processor.
A production platform can include: SoC Boot ROM → Trusted Firmware / First-Stage Boot → SPL / U-Boot → Secure / Verified Boot → Linux Kernel → Device Tree → Kernel Drivers → Root Filesystem → Init / systemd → System Services → Product Applications → Networking / Cloud → Diagnostics → OTA Update → Field Maintenance
Every layer depends on the layers beneath it.
A Product That Boots Is Not Automatically a Production-Ready Linux Platform.
365PCB Embedded Linux Development therefore treats Linux as: A Complete Hardware-to-Lifecycle Product Platform.
Don't Start With "Which Linux Distribution?"
First define:
What must the product do?
What interfaces exist?
How fast must it boot?
Does it need graphics?
Does it need AI acceleration?
Does it need cameras?
Does it need hard or soft real-time behavior?
How will it be updated?
How long must it be supported?
What is the cybersecurity requirement?
Linux Distribution Comes After Product Requirements.
Embedded Linux often runs on:
Arm Cortex-A
x86-64
RISC-V
or other Linux-capable processor architectures.
The SoC choice influences:
kernel support
bootloader
drivers
GPU
NPU
memory
security
vendor BSP
lifecycle
Selecting the Processor Also Selects a Software Ecosystem.
A powerful SoC can still be a poor product choice if:
Linux support is immature
drivers are proprietary
BSP maintenance is weak
security updates are uncertain
lifecycle is short.
Silicon Capability Without Software Support Is Incomplete Capability.
A semiconductor vendor often provides a: Board Support Package — BSP
containing some combination of:
Bootloader
Linux Kernel
Device Tree
Drivers
Firmware Blobs
RootFS Integration
Multimedia Libraries
GPU / NPU Software
The BSP Is the First Software Bridge Between the SoC and the Product.
Vendor BSPs are usually built to demonstrate: Silicon Capability.
The product team must transform that into: Production Software.
That can require:
removing unnecessary packages
hardening
updating
adding board support
custom drivers
OTA
diagnostics
application services.
One major architectural decision is: Upstream / Mainline-Oriented Linux vs. Vendor BSP Kernel
A vendor kernel may support:
proprietary accelerators
camera ISP
multimedia
special peripherals
earlier.
Mainline can provide better:
long-term maintainability
upstream fixes
portability.
Kernel Choice Is a Lifecycle Trade-Off.
As of August 30, 2026, kernel.org lists: 7.2.2 as current stable,
while: 6.18
and: 6.12
are current longterm lines projected through December 2028.
For an embedded product: Newest Kernel ≠ Best Production Kernel.
The right kernel is the one whose: SoC Support + Drivers + Security + Lifecycle + Product Validation
match the platform.
Before launch, decide:
Which kernel branch will this product live on?
How will CVE fixes be integrated?
When will the next major kernel migration occur?
Kernel Maintenance Should Begin Before Product Shipment.
Many embedded products carry:
board patches
vendor patches
driver patches
feature patches.
A large private patch stack creates long-term maintenance cost.
Every Out-of-Tree Patch Is Future Merge Work.
Where product/business/IP constraints allow, upstream-compatible engineering reduces future rebasing.
Maintain Less Private Kernel Code.
This can become a major lifecycle advantage.
A typical Arm Linux product may follow: Reset → Boot ROM → First-Stage Firmware → Trusted Firmware → SPL → U-Boot → Linux Kernel → Userspace
Linux Boot Is a Chain of Trust and Responsibility.
The immutable Boot ROM is generally part of the SoC.
It may determine:
boot device
authentication root
recovery behavior
depending on silicon.
Product Security Can Begin Before Any Customer Code Runs.
Early boot code may initialize:
clocks
DRAM
storage
security state
before full bootloader operation.
The First Milliseconds of Boot Establish the Platform.
U-Boot architectures may use smaller early stages such as:
TPL / SPL
before the complete bootloader.
This is useful when early memory/resources are limited.
Bootloader Architecture Can Be Multi-Stage.
U-Boot remains one of the central bootloaders in Embedded Linux.
Its current documentation includes:
SPL/TPL
Device Tree
FIT images
DFU
measured boot
boot flows
update mechanisms.
The Bootloader Is Product Infrastructure — Not a Development Menu.
A production configuration should differ from an unrestricted development environment.
Review:
console access
environment modification
arbitrary boot
network boot
debug interfaces.
Development Convenience Can Become Production Attack Surface.
Boot variables can determine:
kernel image
root partition
console
update slot.
Boot Configuration Is Product Configuration.
A robust product may count failed boot attempts.
Example: Try New Image → Boot Fails Repeatedly → Return to Previous Image
Boot Failure Should Have Recovery Logic.
Some products care strongly about: Power-On-to-Useful-Function Time.
Optimization can span:
Boot ROM
Bootloader
Kernel
Drivers
systemd
Applications
Boot Time Is an End-to-End Requirement.
Don't guess where boot is slow.
Measure:
bootloader duration
kernel init
driver probe
service startup.
Optimize the Largest Boot Delay First.
Independent services can sometimes initialize concurrently.
But: Parallelism Should Follow Dependency Analysis.
Not every service must start before the user/product function becomes available.
Boot the Critical Product Path First.
Linux Device Tree is a hardware-description structure allowing platform configuration to be data-driven instead of hard-coded directly into the kernel.
Official kernel documentation describes it as a structure for hardware information including buses, interrupts, GPIOs and peripherals.
Device Tree Describes the Hardware the Kernel Is Actually Running On.
A board-level Device Tree Source can define:
CPU
Memory
I²C
SPI
UART
GPIO
Regulators
Displays
Cameras
Ethernet
and other hardware.
Hardware Configuration Becomes Machine-Readable.
Reusable SoC or platform definitions can live in shared include files.
Don't Copy an Entire SoC Description for Every Board Variant.
Bindings define the schema/contract expected by drivers.
The current Linux Device Tree documentation includes JSON-schema-based binding development and validation.
Device Tree Should Match Driver Architecture — Not Invent Properties Ad Hoc.
One product family may contain:
Rev A
Rev B
Industrial Variant
Display Variant
Wireless Variant
Device Tree architecture can help manage these differences.
Hardware Variant Management Is Software Architecture.
Overlays can modify an existing tree for modular hardware where architecture supports it.
But production systems should carefully control which configurations are valid.
Flexibility Needs Configuration Governance.
Linux has thousands of configuration options.
Do not ship:
Everything Enabled.
Select features according to:
hardware
product requirements
security
memory
boot time.
Kernel Configuration Is Product Architecture.
Unused:
protocols
filesystems
drivers
can be disabled where practical.
Benefits can include:
smaller image
lower attack surface
faster initialization.
Feature Reduction Can Improve Both Engineering and Security.
Drivers can be: built into kernel or loadable modules.
Boot-critical drivers may need to be built in depending on architecture.
Module Strategy Affects Boot, Updates and Security.
Modules provide flexibility but introduce:
loading
dependency
signing
version
considerations.
Dynamic Kernel Code Should Be Controlled Like Any Other Executable Code.
A Linux driver translates: Kernel Subsystem API
into: Hardware Behavior.
Drivers Are the Boundary Between Operating System and PCB.
Linux already has established subsystems for:
GPIO
I²C
SPI
IIO
V4L2
DRM
ALSA
Network
and many others.
Use the Linux Driver Model Instead of Rebuilding Linux Inside One Proprietary Driver.
SoC-integrated peripherals are often exposed through platform devices tied to Device Tree or ACPI descriptions.
Hardware Description and Driver Binding Work Together.
Typical devices include:
sensors
PMICs
EEPROM
codecs
controllers.
Bus Communication Is Only One Part of a Proper Linux Driver.
SPI can support:
sensors
ADC/DAC
displays
storage
controllers.
Linux integration should select the correct kernel subsystem for the device's actual function.
SPI Is a Transport — Not an Application API.
Modern Linux GPIO frameworks should be used rather than depending on obsolete assumptions or uncontrolled register access.
A GPIO Still Belongs to the Kernel Resource Model.
PWM can control:
backlight
motors
fans
power-related functions.
Hardware Timing Should Remain in Hardware Where Possible.
Linux IIO supports many:
ADCs
DACs
sensors
and buffered acquisition models.
Sensor Integration Can Use Standard Kernel Infrastructure.
High-bandwidth peripherals need efficient transfer between: device and memory.
Don't Ask the CPU to Copy Every Byte When Hardware Can Move It.
With caches and accelerators: CPU-visible memory and DMA-visible memory must remain coherent according to architecture.
Memory Is Shared Between Several Masters — Not Only the CPU.
An IOMMU can map and restrict device memory access.
Benefits can include:
isolation
virtualization
safer DMA architecture.
DMA Devices Can Have Memory Permissions Too.
Contiguous Memory Allocator-style strategies may support:
cameras
display
multimedia
requiring large physically contiguous regions.
Memory Architecture Should Follow Peripheral Data Flow.
Linux DMA-BUF enables buffer sharing between subsystems/devices.
This is important for: Camera → ISP → GPU → Display
or: Camera → NPU
pipelines.
Zero-Copy Architecture Can Remove Entire Memory-Copy Stages.
Every full-frame copy costs:
memory bandwidth
CPU
latency
power.
The Fastest Copy Is the Copy the Architecture Does Not Need.
Embedded Linux products increasingly integrate:
NVMe
network controllers
FPGAs
accelerators.
PCIe Is Both Hardware SI and Linux Software Architecture.
Boot must correctly discover:
endpoints
BARs
interrupts
link state.
Electrical Link Training Is Only the Beginning.
Modern PCIe devices often use message-signaled interrupts.
Interrupt Architecture Scales With Device Throughput.
High-performance embedded platforms may use NVMe for:
local storage
logging
AI data
database workloads.
Storage Throughput Can Become a PCIe + Filesystem + Thermal Problem.
Linux can operate as: USB Host or USB Device / Gadget
depending on hardware and product.
USB Role Is Product Architecture.
Embedded products can present themselves as:
network device
storage
serial
custom composite device.
Linux Can Become the Peripheral.
Linux offers mature networking infrastructure.
But product architecture must still handle:
PHY
MAC
timing
link state
security.
Ethernet Is Hardware + Driver + Kernel + Userspace.
Precision Time Protocol-capable systems can synchronize clocks across networks where hardware and drivers support suitable timestamping.
Network Time Can Become Measurement Time.
Industrial, sensing and distributed-control products may depend on:
PTP
NTP
GNSS
or other time sources.
Time Is a Distributed System Resource.
Linux SocketCAN provides a standard networking-style architecture for CAN interfaces.
Vehicle and Industrial Buses Can Enter the Linux Network Model.
Wireless integration includes far more than loading a driver.
Consider:
regulatory firmware
calibration
antenna
roaming
provisioning
security.
Wireless Hardware Requires Product-Level Software Integration.
Bluetooth may involve:
controller
host stack
profiles
product services.
Connectivity Stack Selection Is Product Architecture.
Cellular modems may interface over:
USB
PCIe
UART
with networking and management services layered above.
Modem Integration Is a Device-Management Problem as Well as a Network Problem.
Embedded products commonly use:
eMMC
SD
NAND
NOR
NVMe
UFS
depending on platform.
Storage Choice Determines Filesystem and Update Architecture.
Raw NAND requires management of:
bad blocks
wear
ECC
through suitable flash subsystems.
Raw Flash Is Not a Disk.
Linux Memory Technology Device infrastructure supports flash-oriented devices.
Storage Software Should Match Flash Physics.
For suitable raw-NAND architectures, UBI/UBIFS can manage:
bad blocks
wear
volume abstraction.
Flash Reliability Requires Flash-Aware Software.
Managed flash simplifies some NAND management internally.
But products still need to consider:
endurance
power failure
lifecycle.
Managed Flash Does Not Mean Infinite Write Endurance.
Possible choices include:
ext4
F2FS
SquashFS
UBIFS
and others.
Selection depends on:
storage
write behavior
integrity
update architecture.
Filesystem Is Product Behavior.
A read-only root can improve:
robustness
predictability
security.
Writable product data can live elsewhere.
Separate Product Software From Product Data.
An advanced architecture can treat the operating-system image as: Immutable.
Updates replace complete verified images rather than allowing uncontrolled in-field package drift.
Predictable Systems Are Easier to Validate.
Some designs combine: read-only base
with: writable overlay.
Useful, but persistence/update semantics must be understood.
Convenience Layers Still Need Recovery Rules.
User/product data should often have a lifecycle separate from the OS.
Updating Linux Should Not Erase Product Identity or Calibration.
A production RootFS should contain: What the Product Needs.
Not: every package available in the build system.
Linux needs a userspace initialization system.
Many modern embedded distributions use: systemd.
Yocto 6.0 now defaults its init-manager definition to systemd as part of that release's platform changes.
systemd can manage:
services
dependencies
sockets
timers
watchdogs
startup.
Product Startup Should Be a Dependency Graph — Not a Shell Script Accident.
Example: Database
must start after: persistent storage.
Network application may depend on: etwork-online.
Explicit Dependencies Beat Startup Delays Like "sleep 5."
If a service exits:
should it:
restart?
stay failed?
trigger product fault?
Process Recovery Policy Is Product Behavior.
A crashing service configured to restart forever can create:
CPU load
flash logging
unstable product behavior.
Recovery Needs Bounds.
A Linux system can still hang below userspace.
A hardware watchdog can provide an independent recovery layer where appropriate.
Process Supervision and Hardware Supervision Solve Different Failures.
Possible architecture: Application Health → systemd / Supervisor → Kernel Watchdog → Hardware Watchdog
Health Supervision Can Be Layered.
Linux provides strong process isolation.
Unlike a typical MCU RTOS: One Application Does Not Need to Own the Entire Address Space.
Separate critical services according to:
responsibility
trust
restart behavior.
Process Isolation Creates Fault-Containment Boundaries.
Within one process, threads can share:
memory
file descriptors
while executing concurrently.
Process and Thread Boundaries Serve Different Architectural Goals.
Processes need communication.
Possible mechanisms include:
Unix Domain Sockets
Shared Memory
D-Bus
Pipes
Message Queues
Network Sockets
IPC Architecture Is Product Architecture.
Useful for structured local service communication.
A Process Boundary Does Not Require TCP/IP Overhead.
Shared memory can provide high throughput.
But synchronization and ownership become critical.
Zero Copy Trades Copy Cost for Concurrency Complexity.
D-Bus can provide structured service communication on suitable Linux products.
System Services Benefit From Defined Interfaces.
If process A and process B can be updated independently: Their Interface Needs a Version Strategy.
Instead of allowing every application to: open /dev/mem or directly manipulate devices, build defined service/API boundaries.
Userspace Architecture Should Preserve Hardware Ownership.
A display application does not necessarily need: root.
A telemetry process does not necessarily need raw device access.
Least Privilege Applies to Embedded Linux.
Linux can divide traditional root privileges into narrower capability sets.
Root vs Non-Root Is Not the Only Security Choice.
System-call filtering can reduce what a process is permitted to ask the kernel to do.
Limit Software to the Kernel Interfaces It Actually Needs.
Linux's LSM framework supports policy systems including: SELinux and AppArmor, among others.
Process Isolation Can Be Policy-Enforced.
AppArmor uses profiles to restrict application behavior.
A Compromised Service Should Not Automatically Have Access to the Whole Product.
SELinux provides mandatory access-control architecture suitable for systems requiring stronger policy segmentation.
Security Policy Can Be an Engineering Artifact.
Embedded Linux can use containers where they create value.
Containers can isolate:
applications
dependencies
versions.
Containerization Is Deployment Architecture — Not Automatically Good Architecture.
Containers can increase:
storage
RAM
update complexity.
Use Containers When Isolation and Deployment Benefits Exceed Their Resource Cost.
Linux namespaces isolate system views such as:
processes
mounts
networking.
Containers Are Built From Kernel Isolation Mechanisms.
Control groups can constrain:
CPU
memory
process resources.
One Service Should Not Be Able to Consume the Entire Product.
A high-level product should have budgets for:
RAM
CPU
Storage
Network
Linux Resources Are Large — Not Infinite.
Linux can behave dramatically when system memory is exhausted.
Memory Exhaustion Needs Product-Level Prevention and Diagnosis.
A small leak that appears insignificant during a one-hour test may crash a product after months.
Long-Life Linux Products Need Long-Duration Memory Testing.
Many embedded products avoid or carefully control swap depending on:
flash
latency
endurance.
Desktop Memory Assumptions Do Not Automatically Belong in Embedded Products.
Now we reach one of the most important modern areas.
Since Linux 6.12, PREEMPT_RT support became part of official Linux. The Linux Foundation's current RT documentation explicitly states that since 6.12 the official kernel supports PREEMPT_RT.
Linux Can Now Be Engineered Much More Directly for Deterministic Real-Time Workloads.
Official kernel documentation explains that PREEMPT_RT moves much more kernel execution under scheduler control through mechanisms including:
preemptible locking
rtmutex
priority inheritance
threaded interrupts.
Real-Time Linux Reduces Uncontrolled Kernel Latency.
Installing an RT kernel does not automatically create: A Real-Time Product.
Application architecture still matters.
Linux supports real-time scheduling policies such as: SCHED_FIFO.
A higher-priority runnable task can preempt lower-priority work under the RT scheduling model.
Priority Architecture Still Matters.
Round-robin real-time scheduling can provide time sharing among suitable equal-priority tasks.
Scheduler Policy Should Match the Workload.
Advanced Linux systems can also use deadline-oriented scheduling where architecture and platform support it.
Linux Real-Time Engineering Can Move Beyond Fixed Priorities.
PREEMPT_RT uses RT mutex mechanisms to reduce certain priority-inversion effects by transferring priority to a lock owner when needed.
Kernel Locking Can Participate in Real-Time Scheduling.
PREEMPT_RT moves much interrupt processing into schedulable thread context.
Interrupt Handling Can Become Priority-Controlled Work.
A real-time application can still be delayed by incorrectly prioritized interrupt threads.
RT Kernel Configuration Requires IRQ Architecture Too.
On multicore processors, interrupts may be assigned to selected CPUs.
Keep Noisy Interrupts Away From Critical Real-Time CPUs Where Architecture Requires It.
Real-time tasks can be pinned to selected CPU cores.
Scheduling Freedom Can Be Traded for Timing Predictability.
Critical workloads may use dedicated CPUs with reduced:
background tasks
interrupts
housekeeping.
A Core Can Become a Real-Time Resource.
An architecture might dedicate: CPU Core A to high-priority control,
while other cores handle: Networking + UI + Storage.
Multicore Allows Functional Timing Isolation.
Not every real-time requirement belongs in Linux.
A product can combine:
Linux MPU/SoC
MCU / RTOS
for demanding low-latency control.
Use the Right Compute Domain for Each Deadline.
Modern SoCs may combine:
Cortex-A
with:
Cortex-M / R-class cores
or other dedicated engines.
One Chip Can Contain Several Computing Models.
Linux can coordinate secondary processor firmware on suitable platforms.
Linux Can Be the System Manager Without Running Every Real-Time Loop Itself.
Heterogeneous cores can exchange structured messages through shared-memory/mailbox frameworks.
Processor Boundaries Need Protocols Too.
Page faults can create unpredictable latency.
Critical real-time processes may therefore need memory architecture that avoids unexpected faulting.
Virtual Memory Convenience Can Become Timing Variability.
Real-time workloads may lock critical memory into RAM where appropriate.
Don't Let a Deadline Wait for Memory to Become Resident.
Runtime allocation can introduce variable latency.
Real-Time Linux Applications Still Need Resource Discipline.
CPU frequency changes can alter:
execution time
latency
power.
DVFS Is a Timing Variable.
A processor that passes timing cold can miss timing when: Thermally Throttled.
Real-Time Verification Must Include Thermal State.
Cache misses create execution-time variation.
Gigahertz CPU Frequency Does Not Mean Constant Instruction Latency.
GPU, NPU, camera and CPU may compete for DDR bandwidth.
Shared Memory Can Become a Real-Time Contention Resource.
The Linux RT documentation itself emphasizes that real-time performance depends not only on kernel scheduling but also on hardware factors including memory, buses, virtualization and networking.
Software Cannot Schedule Away a Saturated Hardware Bus.
Do not say:"Linux real-time latency is 20 µs."
without platform definition.
Measure:
hardware
kernel
load
configuration
thermal state.
Real-Time Capability Is Platform-Specific.
As with RTOS: Average Latency Is Not the Requirement.
The worst observed / engineered bound is usually far more relevant.
Real-time testing should run alongside:
networking
storage
GPU
memory pressure
representing the real product workload.
Measure Timing While the Product Is Busy.
Linux provides sophisticated tracing infrastructure for understanding:
scheduling
interrupts
latency
kernel functions.
Linux Is One of the Most Observable Embedded Operating Systems Available.
Kernel tracing can reveal:
function timing
scheduling behavior
interrupts.
Turn Kernel Timing Into Evidence.
Modern kernel tracing tools can investigate:
operating-system noise
timer latency
for real-time workloads.
Real-Time Debugging Should Locate the Source of Latency.
Performance counters and profiling can identify:
CPU hotspots
cache behavior
execution bottlenecks.
Optimize From Measurements.
eBPF increasingly enables deep observability and controlled programmable kernel instrumentation.
Production Diagnostics Can Become More Powerful Without Recompiling Every Application.
Use only where product complexity/resources justify it.
Modern Embedded Linux platforms increasingly combine CPUs with GPUs.
Applications can include:
GUI
vision
visualization
compute.
GPU Support Is Kernel + Driver + Userspace + Application Integration.
A Linux graphics pipeline can involve: Kernel DRM/KMS → Userspace Graphics Libraries → Compositor → Application.
A Display Is a Pipeline — Not One Framebuffer Call.
Linux DRM/KMS provides standard kernel infrastructure for modern display pipelines.
Display Hardware Should Integrate With the Linux Graphics Model.
Many modern embedded GUI platforms use Wayland-based compositing architectures.
GUI Architecture Should Match GPU, latency and product UI requirements.
Embedded vision products may include: Image Sensor → MIPI CSI-2 → Receiver → ISP → Memory → Application / NPU
Camera Linux Development Is an End-to-End Data Pipeline.
Video4Linux2 provides a major Linux framework for cameras/video devices.
Camera Drivers Should Integrate Into Standard Media Infrastructure Where Practical.
Complex camera systems may have multiple processing entities:
sensor
CSI receiver
ISP
scaler.
Camera Pipelines Can Be Graphs — Not Single Devices.
The PCB must first reliably deliver the high-speed camera interface.
Linux must then correctly configure:
clocks
lanes
sensor
receiver
pipeline.
Camera Bring-Up Is Hardware + SI + Driver + ISP Engineering.
Image Signal Processors can perform:
demosaic
color correction
noise reduction
HDR
depending on platform.
Image Quality Is Software-Controlled Hardware Behavior.
A camera that produces an image is not automatically: A Finished Camera System.
Real products may require:
exposure
white balance
color
noise
lens
tuning.
Multimedia applications frequently use pipeline architectures for:
camera
codec
streaming
display.
Preserve Zero-Copy Paths Through the Multimedia Pipeline Where Performance Requires It.
Hardware encode/decode can dramatically reduce CPU load for:
H.264
H.265/HEVC
other supported formats.
Use Dedicated Silicon for Dedicated Work.
Linux ALSA provides standard audio infrastructure.
Product implementation may include:
Codec
I²S/TDM
Amplifier
Mixer
Userspace Audio.
Audio Is Hardware + Clock + Driver + Userspace.
Modern SoCs increasingly include: Neural Processing Units.
These accelerate edge inference workloads.
NPU functionality can depend on:
vendor driver
runtime
compiler
model format
memory architecture.
"SoC Has NPU" Does Not Mean AI Application Is Finished.
A real architecture might be: Camera → ISP → DMA-BUF → NPU → Post-Processing → Application.
AI Performance Depends on Data Movement as Much as TOPS.
Repeatedly copying a 4K frame between:
CPU
GPU
NPU
can waste enormous memory bandwidth.
Accelerate the Data Path — Not Only the Neural Network.
Measure:
Capture
Preprocess
Inference
Postprocess
Decision
TOPS Is Not End-to-End Product Latency.
NPU/GPU performance can throttle with heat.
Benchmark AI in the Final Thermal Environment.
AI models can change after shipment.
That creates version relationships among:
Application
Runtime
Model
NPU Compiler
Hardware.
Model Version Is Product Configuration.
For complex long-life Embedded Linux products, Yocto is one of the most important custom-distribution build frameworks.
Current Yocto 6.0 Wrynose is an LTS line supported until April 2030.
Yocto Builds a Product Distribution — Not Just a Root Filesystem.
Metadata can be organized into layers.
Conceptually:
SoC Layer
Board Layer
Product Layer
Application Layer
Separate Responsibilities in the Build System.
Recipes define how software is:
fetched
configured
compiled
packaged.
Build Knowledge Should Be Machine-Readable.
BitBake executes the dependency/build graph.
Embedded Linux Build Is Dependency Engineering.
Machine metadata can describe:
CPU architecture
kernel
bootloader
board-specific behavior.
Hardware Variant Should Be Represented in the Build System.
Distribution configuration can define product-wide choices such as:
init
security
packaging
libraries.
The Operating-System Policy Should Be Explicit.
The image defines what actually ships.
If It Is Not Required by the Product, Ask Why It Is in the Image.
Wrynose 6.0 moved its baseline to:
Linux 6.18
GCC 15.2
glibc 2.43
LLVM 22.1.2
and more than 300 recipe upgrades.
Product Linux Platforms Continue to Move Rapidly.
Yocto 6.0 expanded SPDX metadata support, including options for exporting kernel configuration and PACKAGECONFIG data into SPDX output.
Build System Can Become Software-Supply-Chain Evidence.
Buildroot provides another strong embedded Linux build approach.
As of August 30, 2026:
stable: 2026.05.2
release candidate: 2026.08-rc2
LTS: 2025.02.17.
Buildroot and Yocto Solve Similar Product Problems With Different Complexity Models.
A simplified comparison: Buildroot
Excellent when:
architecture is relatively controlled
development team values simplicity
image is rebuilt as a whole.
Yocto
Strong where:
multiple products
complex dependencies
long lifecycle
reusable layers
richer package/compliance metadata
matter.
Choose the Build Architecture for the Product Organization — Not Only the First Prototype.
Embedded Linux commonly builds software on: x86 development host
for: Arm / RISC-V target.
Build Architecture and Runtime Architecture Are Different.
A controlled SDK can provide application developers with:
compiler
sysroot
libraries
headers.
Application Builds Should Target the Exact Product Platform.
Compiler/library changes can affect:
performance
ABI
bugs.
Toolchain Is Product Configuration.
The C library can influence:
compatibility
size
behavior.
Userspace ABI Choice Has Long-Term Consequences.
Native applications remain common in performance-sensitive embedded Linux.
Application Language Should Follow Latency, Safety, Ecosystem and Maintainability Requirements.
Rust adoption continues to expand in systems software and can be considered for selected userspace/security-sensitive components where the toolchain and ecosystem fit.
Memory-Safe Languages Can Reduce Certain Defect Classes — but Architecture Still Matters.
Python can be valuable for:
orchestration
diagnostics
rapid product logic
where CPU/storage/startup requirements allow.
Not Every Embedded Function Needs to Be Written in C.
A mature Linux system may split functionality into services such as:
Hardware Manager
Connectivity Manager
Update Manager
Diagnostics Manager
Application Engine
Product Software Can Become a Service Architecture.
One process should usually have clear ownership of critical hardware.
Avoid Five Daemons Fighting Over the Same Device.
Define product-level states:
BOOTING
READY
ACTIVE
DEGRADED
UPDATE
FAULT
SHUTDOWN
Linux Processes Still Need One Coherent Product State.
A process manager starting successfully does not mean: Product Ready.
The application should understand:
sensor readiness
network state
calibration
time synchronization.
Linux products with writable storage should manage:
service stop
data flush
hardware output
filesystem state.
Pulling Power Is Not a Shutdown Strategy.
Nevertheless, field products must consider sudden power loss.
The Filesystem and Persistent Data Architecture Should Assume Power Can Disappear.
Journaling filesystems can improve recovery of filesystem metadata.
But: Journaling Does Not Make Every Application Write Transactional.
Applications using databases should define power-loss and transaction behavior explicitly.
Storage Consistency Is Application Architecture Too.
A robust configuration update can use: write new → sync / verify → atomic replace
rather than destroying the only valid copy first.
Persistent State Needs Transaction Semantics.
Production platforms should increasingly consider an authenticated boot chain.
U-Boot's official Verified Boot documentation describes verification from bootloader onward, including signed kernels and integration with dm-verity-protected root filesystems.
Boot Only Software the Product Is Authorized to Run.
Conceptually: Hardware Root of Trust → Boot Firmware → Bootloader → Kernel → RootFS
Trust Should Be Transferred Through Each Boot Stage.
U-Boot can use FIT-based images for structured kernel/device-tree and signing architectures.
Boot Artifacts Can Carry Cryptographic Identity.
Terminology varies by platform.
The engineering principle is: Authenticate the Software Before Trusting It.
Linux dm-verity provides transparent integrity checking of read-only block devices using cryptographic hashes.
Detect Root Filesystem Modification at Read Time.
A read-only system image can be cryptographically verified.
If a protected block does not match its expected hash: The Product Knows the Image Is No Longer What Was Signed.
fs-verity provides per-file read-only integrity/authenticity protection and is supported by filesystems including ext4, F2FS and btrfs.
Integrity Protection Can Operate at File Level as Well as Block Level.
Encryption answers: Can an unauthorized party read the data?
Integrity answers: Has the data changed? Confidentiality and Integrity Are Different Security Requirements.
Sensitive product/user data may use encrypted storage depending on threat model.
Protect Data at Rest Where Its Value Justifies It.
Encryption only works if keys are appropriately protected.
Potential architectures include:
secure element
TPM
SoC hardware security
TEE.
A Secret Stored Beside the Ciphertext in Plaintext Is Not a Security Architecture.
On suitable Arm platforms, OP-TEE provides a TEE companion to non-secure Linux using hardware isolation such as TrustZone.
Linux Does Not Need to Hold Every Secret in the Normal World.
Potential uses include:
secure storage
cryptographic services
trusted applications
depending on product architecture.
Place High-Value Operations in the Smallest Practical Trusted Computing Base.
Platform security configuration still matters.
Even platform vendors note that generic TEE configurations must be hardened to match the specific end product rather than assumed production-secure by default.
Security Technology Still Needs Security Engineering.
Another architecture can measure boot components into a trusted hardware record.
Verified Boot Asks "May This Execute?"
Measured Boot Records "What Executed?"
U-Boot's current documentation includes measured-boot support.
A Trusted Platform Module can support:
measurements
keys
attestation
where the product requires it.
Device Trust Can Become Cryptographically Verifiable.
Each product may receive a unique:
Certificate
Key
Serial Identity
during secure provisioning.
The Device Should Be Able to Prove Which Device It Is.
Identity provisioning should be integrated with:
serial number
hardware revision
production database
secure credentials.
Digital Identity Is a Manufacturing Output.
Private keys should not casually appear in:
source code
build logs
shared production folders.
Security Supply Chain Begins Inside the Factory.
A connected product requires a plan for: How Software Changes After Shipment.
OTA is therefore not simply a cloud feature.
It is: Product Lifecycle Infrastructure.
A robust architecture may maintain: Slot A and Slot B.
For example: Running A → Write New Image to B → Verify B → Boot B → Confirm Health → Commit B.
Never Destroy the Running System Before the Replacement Is Proven.
If the new system fails: Return to Known Good.
This requires cooperation among: Bootloader + RootFS + Update Service + Application Health.
Successful kernel boot does not mean the update is good.
The product may need to confirm:
application starts
critical hardware works
configuration migrates.
Update Success Should Be Defined at Product Level.
Ask: What happens if power disappears at 37% update? Every Update Stage Must Have Recoverable State.
A good update system avoids half-new / half-old uncontrolled combinations.
Product Software Should Move Between Known Configurations.
Bootloader updates are especially sensitive because bootloader corruption can remove normal recovery.
Update Risk Increases the Lower You Go in the Boot Chain.
A product can maintain an independent:
recovery partition
USB recovery
controlled network recovery
according to architecture.
A Field Update Strategy Needs a Failure Strategy.
Cybersecurity may require preventing installation of: Known-Vulnerable Historical Images.
But service/recovery requirements must also be considered.
New software may require new:
database schema
configuration.
OS Rollback Is Harder If User Data Cannot Roll Back.
Update design must coordinate: Application Version with Persistent Data Version.
Software Lifecycle Includes Data Lifecycle.
Platforms can use established update frameworks or a custom architecture depending on requirements.
The important engineering requirements are:
Authenticity
Atomicity
Recovery
Rollback Policy
Power-Failure Safety
Version Compatibility.
Framework Name Is Less Important Than Update Guarantees.
A deployed fleet may contain:
10
10,000
or:
1,000,000
units.
Software management must understand:
version distribution
update status
failures.
Embedded Linux Becomes Fleet Software After Shipment.
Instead of updating every product simultaneously: Internal → Small Canary Group → Larger Group → Fleet.
Limit the Blast Radius of Software Failure.
Track:
download success
verification
boot success
rollback
health.
An OTA System Should Know Whether Its Own Update Worked.
Field devices can report selected:
kernel version
hardware revision
temperature
storage
service health
fault state.
Field Service Can Become Data-Driven.
Collect only telemetry the product actually needs.
Observability Should Not Become Unnecessary Data Collection.
Linux can generate enormous logs.
Define:
what to log
retention
persistence
rotation.
Unlimited Logging Eventually Becomes a Storage Failure.
Decide whether system logs live:
in RAM
persistently
according to service requirements.
Diagnostic Value and Flash Wear Must Be Balanced.
Without limits: Logs Can Fill the Filesystem.
Resource governance matters.
Kernel crash information can be preserved across reboot on suitable platforms.
A Kernel Panic Should Leave Evidence.
Userspace crashes can preserve:
signal
stack
binary version
for investigation.
"Application Restarted" Should Not Erase the Root Cause.
Build artifacts should allow addresses to be translated into:
functions
source code
for the exact released binary.
Debug Information Must Match the Product Version.
Every released executable/image should be identifiable.
You Cannot Debug a Binary Whose Source Revision Is Unknown.
A production platform may contain thousands of software components.
Know What Software You Ship.
This is especially important for vulnerability management and licensing.
Record:
version
source
license
patches.
Third-Party Software Becomes Product Responsibility When It Ships.
A Linux product needs an ongoing process: New Vulnerability → Affected? → Risk? → Patch / Mitigate → Build → Test → Deploy
Cybersecurity Continues After Product Launch.
Not every CVE has equal product risk.
Consider:
component enabled?
reachable?
privilege?
product exposure?
Vulnerability Management Should Be Context-Aware.
A mature platform should make software builds deterministic enough to reproduce released artifacts according to the project's control strategy.
U-Boot's current build documentation explicitly includes reproducible-build support among its developer tooling.
A Product Release Should Be Reconstructable.
Release images can be cryptographically signed.
Build Output Should Carry Provenance.
Every source change can trigger: Build → Static Checks → Unit Tests → Image Creation → Automated Hardware Tests
Linux Platform Quality Should Be Continuously Verified.
Real hardware can be connected to automated test infrastructure.
Run the Real Image on the Real Board Automatically.
Linux contains extensive self-testing infrastructure that can complement product-specific tests.
Platform Validation Should Test Both the Operating System and the Product.
Custom drivers should be tested for:
probe/remove
suspend/resume
errors
stress.
A Driver That Works Once During Boot Is Not Fully Validated.
Controlled test environments can emulate:
device timeout
network loss
storage errors
process crash.
Test Recovery Logic Before the Field Does.
Repeated abrupt power interruption can expose:
filesystem
database
update
boot
weaknesses.
Power Failure Is a Storage Test.
Heavy logging or databases can wear flash.
Write Amplification Can Become Product Lifetime.
Test:
reconnect
packet loss
DNS failure
server outage.
The Internet Is Not a Deterministic Peripheral.
Connected products should define: What Happens When the Cloud Is Gone?
The product should not necessarily become useless.
If NTP fails:
does TLS work?
do logs make sense?
can certificates be validated?
Wall Clock Can Be a Security Dependency.
A hardware RTC may preserve approximate time across power loss.
Timekeeping Can Require Hardware Support.
Embedded Linux may use low-power states.
But suspend/resume requires cooperation from:
kernel
drivers
hardware
applications.
Sleep Is a Distributed System State.
Possible wake sources include:
GPIO
RTC
network
peripheral.
The Product Should Know Why It Woke Up.
A driver that works after cold boot may fail after 500 suspend/resume cycles.
Resume Is Another Initialization Path.
Unused devices can sometimes be powered down independently.
Power Management Should Follow Actual Device Use.
Dynamic voltage/frequency changes can reduce power.
But they influence:
performance
thermal
timing.
Energy Management Is Scheduling + Hardware Management.
Some SoCs dynamically scale:
GPU
memory/interconnect
accelerators.
CPU Is Not the Only Dynamic Power Domain.
Linux can coordinate:
sensors
cooling
throttling
on suitable platforms.
Thermal Behavior Can Become Operating-System Policy.
Product control can connect: Temperature → Cooling Policy → Fan / Performance
Thermal Control Is a Closed-Loop Product Function.
Field data can reveal whether deployed products regularly throttle.
Thermal Margin Can Be Measured in the Fleet.
A Linux product can require programming of:
Bootloader
Kernel
Device Tree
RootFS
Recovery Image
Device Credentials
Software Loading Is a Manufacturing Process.
Production images may include special:
factory test
calibration
provisioning
functions.
Factory Software and Customer Software Need Controlled Boundaries.
Linux can automate hardware testing:
RAM
Storage
Ethernet
Wi-Fi
USB
Camera
Audio
GPIO
depending on product.
Linux Can Be the Manufacturing Test Platform for Its Own Hardware.
Production calibration data can be generated and stored per unit.
Calibration Must Follow the Physical Product It Describes.
The system can bind:
Serial Number
to:
MAC
Device Certificate
Hardware Revision
Test Record.
Physical Identity and Digital Identity Should Agree.
Production should control access to:
certificates
private keys
credentials.
Security Must Survive the Factory.
Products with multiple PCB revisions can identify the hardware using:
EEPROM
GPIO strapping
identity data
depending on design.
Software Should Know Which PCB It Is Running On.
Before loading an image: Is this software compatible with this board?
Prevent Configuration Mismatch Before It Becomes Product Failure.
One Linux platform can support several models.
But variant differences should be represented through controlled:
build configuration
Device Tree
hardware ID
feature configuration.
Product Families Need Configuration Architecture.
EVT focuses on: Hardware Bring-Up.
Typical work:
boot
DDR
storage
Ethernet
USB
PCIe
display
camera
sensors.
EVT Linux Should Maximize Hardware Visibility.
Initial boot often uses:
Serial Console
as one of the most valuable debug interfaces.
Before the Display Works, UART Tells the Story.
Linux cannot compensate for an unstable DDR subsystem.
Hardware Memory Integrity Comes Before Operating-System Stability.
Before blaming Linux:
investigate:
DDR
power
clocks
thermal
storage
silicon errata.
Kernel Panic Can Be a Hardware Symptom.
If identical software behaves differently between units: Manufacturing Variation Has Become a Software Symptom.
This is especially important for365PCB.
DVT moves toward:
production kernel
security
OTA
final peripherals
final enclosure
stress.
DVT Validates the Complete Platform.
Test: old version → new and update failure and rollback.
OTA Must Be Qualified Before Shipment.
Validate:
unauthorized image rejection
debug restriction
filesystem integrity
credentials
according to threat model.
Security Features Must Be Tested as Behaviors.
PVT verifies:
programming
provisioning
production test
image version
hardware variants.
Linux Becomes a Manufacturing Configuration at PVT.
A release can identify:
Bootloader
Kernel
DTB
RootFS
Application
Certificates / Policy Versions
"Firmware Version 1.3" Is Often Too Little Information for a Linux Product.
Shipping the product starts: The Longest Phase of Linux Engineering.
Long-term supported upstream components can reduce lifecycle risk.
Yocto 6.0's current LTS support window runs to April 2030; kernel.org currently maintains multiple longterm kernels rather than one universal LTS.
Lifecycle Version Selection Should Be Intentional.
A vendor may stop maintaining the BSP before the physical product reaches end-of-life.
Software Support Life Should Be Evaluated When Selecting the SoC.
Long-lived products eventually need to move to another kernel.
Upgrade Before the Old Platform Becomes Unmaintainable.
Linux kernel-internal driver APIs can change across versions.
Out-of-Tree Drivers Create Migration Work.
Another reason to align with upstream architecture where practical.
Some hardware depends on proprietary:
GPU
NPU
wireless
multimedia
components.
Closed Dependencies Need Vendor-Lifecycle Risk Analysis.
For long-lived industrial products: Control as Much of the Software Stack as the Business Requires to Maintain the Product.
For critical dependencies, organizations may need contractual lifecycle planning.
Technology Risk Can Be Supplier Risk.
A field failure can originate from:
Hardware
Kernel
Driver
Userspace
Cloud
Configuration
Power
Thermal
"Linux Crashed" Is a Symptom — Not a Root Cause.
Collect:
Kernel Log
Application State
Hardware Data
Reset Cause
Temperature
Cross-Layer Evidence Solves Cross-Layer Problems.
Examples include: Marginal DDR → random segmentation faults.
Power droop → reboot.
PCIe SI margin → endpoint disappears.
Overheating → performance collapse.
Embedded Linux Debugging Must Understand Hardware Physics.
Conversely: bad driver
can look like:
broken peripheral.
wrong Device Tree
can look like:
PCB pin problem.
Hardware and Linux Must Be Debugged Together.
A pure software house may see: Kernel Error.
A PCB manufacturer may see: Electrical Test Pass.
But an integrated ODM engineering team can ask:
Is this a driver bug?
DDR timing?
power integrity?
clocking?
Device Tree?
SI?
manufacturing variation?
The Real Product Exists Across Disciplines.
At the highest level: Product Requirements → SoC Architecture → Hardware Design → Boot ROM / Security Root → Trusted Firmware → U-Boot → Verified / Measured Boot → Linux Kernel → Device Tree → BSP → Drivers → Storage Architecture → Root Filesystem → systemd / Services → IPC → Networking → Multimedia → GPU / NPU → Real-Time Linux → Process Isolation → TEE / Keys → Security Hardening → SBOM → Build System → CI → A/B OTA → Recovery → Manufacturing Provisioning → EVT → DVT → PVT → Fleet Monitoring → Security Maintenance → Kernel / BSP Lifecycle → Long-Term Production Linux Platform
That is the difference between: Booting Linux on a Board
and: Engineering an Embedded Linux Product.
Depending on project scope, a 365PCB ODM Embedded Linux program may include:
Embedded Linux Requirements
SoC / Software Platform Trade Study
Vendor BSP Assessment
Mainline vs Vendor Kernel Strategy
Linux Kernel Selection
LTS Strategy
Boot Architecture
Trusted Firmware Inputs
SPL / TPL Configuration
U-Boot Porting
U-Boot Configuration
Boot-Time Optimization
Boot-Failure Recovery
Kernel Configuration
Kernel Porting
Custom Kernel Integration
Kernel Patch Management
Device Tree Development
Device Tree Bindings
Hardware Variant Support
BSP Development
Board Bring-Up
Kernel Driver Development
GPIO / I²C / SPI Drivers
PWM Drivers
IIO / Sensor Integration
Ethernet Drivers / Integration
CAN / CAN FD Integration
USB Host / Gadget Integration
PCIe Integration
NVMe Integration
Wi-Fi Integration
Bluetooth Integration
Cellular Modem Integration
Storage Architecture
eMMC / NAND / NOR Integration
MTD / UBI / UBIFS Inputs
Filesystem Selection
Read-Only RootFS Design
Data Partition Architecture
RootFS Customization
systemd Integration
Service Architecture
Hardware Management Services
IPC Architecture
D-Bus / Socket Integration
Linux Networking
PTP / Time-Synchronization Inputs
Process Isolation
Resource Control
Containers where appropriate
GPU Integration
DRM / KMS
Display Integration
Camera / V4L2 Integration
MIPI CSI-2 Software Integration
ISP Integration
Multimedia Pipeline
GStreamer Integration
Audio / ALSA Integration
NPU / AI Runtime Integration
Edge AI Pipeline
DMA / Zero-Copy Design
DMA-BUF Integration
IOMMU Inputs
Real-Time Linux Architecture
PREEMPT_RT
Real-Time Scheduling
IRQ Threading / Affinity
CPU Isolation
Real-Time Latency Analysis
Real-Time Trace / Profiling
Heterogeneous Linux + MCU Architecture
Inter-Core Communication Inputs
Power Management
Suspend / Resume
Runtime PM
DVFS Inputs
Thermal Management
Watchdog Architecture
Service Supervision
Logging Architecture
Crash Diagnostics
Kernel Trace
Performance Profiling
Secure Boot
Verified Boot
FIT Signing Inputs
dm-verity Integration
fs-verity Inputs
Storage Encryption Inputs
TEE / OP-TEE Integration
TPM Inputs
Secure Storage Inputs
Device Identity
Secure Provisioning
Linux Security Hardening
SELinux / AppArmor Inputs
seccomp / Capability Restrictions
Read-Only System Architecture
Software Component Minimization
SBOM / SPDX Inputs
Vulnerability Management Inputs
Yocto Project Development
Yocto Layers
Yocto Recipes
Machine / Distro Configuration
Image Development
SDK Generation
Buildroot Development
Custom RootFS
Cross-Compilation Toolchain
Reproducible Build Inputs
CI / Automated Build
Automated Hardware Testing
A/B Update Architecture
OTA Update
Update Signing
Rollback / Recovery
Power-Fail-Safe Update Design
Persistent Data Migration
Fleet Update Inputs
Remote Diagnostics
Factory Linux Image
Production Programming
Production Hardware Test
Calibration Integration
Secure Credential Provisioning
Product Configuration Traceability
EVT Linux Bring-Up
DVT Platform Validation
PVT Production Validation
Long-Duration Testing
Storage Endurance Testing
Power-Loss Testing
Network Stress Testing
Real-Time Stress Testing
Kernel / BSP Maintenance
Security Patch Maintenance
Hardware Revision Support
Software Lifecycle Planning
Field Failure Analysis
Embedded Linux Release Documentation
The actual engineering depth should follow: SoC + Peripheral Complexity + Real-Time Requirements + Multimedia + AI + Security + Connectivity + Boot Time + Update Model + Product Lifecycle.
Embedded Linux capability is platform-specific. Kernel support, boot time, real-time latency, multimedia performance, hardware acceleration, security architecture, power consumption and lifecycle depend on the selected SoC, vendor BSP, kernel version, hardware design, memory architecture, peripheral drivers, software workload, build system and product requirements.
We Don't Judge an Embedded Linux Platform by Whether It Reaches a Login Prompt.
We Judge It by Whether the Complete Product Can Boot, Operate, Recover, Update, and Be Maintained Reliably.
Bring Us the Product Platform — Not Just the Linux Version
You can begin with:
SoC
Custom PCB
Schematic
Existing BSP
Linux Kernel
U-Boot
Device Tree
Existing Drivers
Yocto / Buildroot Project
Boot Log
Kernel Panic
Peripheral Bring-Up Issue
Real-Time Requirement
Security Requirement
OTA Requirement
or simply: Tell Us What Hardware Must Become a Product — and How Long That Product Must Be Maintained.
365PCB can help translate: Custom Hardware → Boot Chain → Linux Platform → Drivers → Services → Security → OTA → Manufacturing → Lifecycle.
Don't Just Boot Linux.
Define the Product Platform.
Choose the Kernel for the Lifecycle.
Control the Boot Chain.
Describe the Hardware Correctly.
Engineer the BSP.
Integrate the Drivers.
Control the Memory and Data Flow.
Build the Root Filesystem Intentionally.
Structure the Services.
Protect the Processes.
Engineer Real-Time Behavior Where Required.
Integrate the GPU and NPU.
Secure the Boot Chain.
Verify the Filesystem.
Protect the Device Identity.
Make Updates Recoverable.
Make Builds Reproducible.
Program It Correctly in Manufacturing.
Diagnose It in the Field.
Maintain It for the Product Lifetime.
365PCB Embedded Linux Development connects: SoC + Hardware + Bootloader + Linux Kernel + Drivers + Real-Time + Multimedia + AI + Security + OTA + Manufacturing + Lifecycle
into one coordinated product-platform engineering process.
Embedded Linux Is Not a Desktop Operating System Put Into a Product.
It Is a Custom Software Platform Built Around the Hardware, Boot Chain, Drivers, Services, Security, and Lifecycle of the Product.
And:
The Kernel Boots the Hardware.
The Platform Makes It a Product.