TY_HOME14
China 365PCB Technology Co., Ltd.
  • contract box build assembly
  • contract box build assembly

Embedded Linux Development

Linux Kernel. U-Boot. BSP. Device Tree. Drivers. Yocto. Buildroot. systemd. Networking. PREEMPT_RT. Security. Containers. GPU/NPU. Multimedia. OTA. A/B Updates. Production Provisioning. Long-Term Maintenance.

An Embedded Linux product is not simply: Linux Kernel; running on: ARM Processor.

A production platform can include: SoC Boot ROM → Trusted Firmware / First-Stage Boot → SPL / U-Boot → Secure / Verified Boot → Linux Kernel → Device Tree → Kernel Drivers → Root Filesystem → Init / systemd → System Services → Product Applications → Networking / Cloud → Diagnostics → OTA Update → Field Maintenance

Every layer depends on the layers beneath it.

A Product That Boots Is Not Automatically a Production-Ready Linux Platform.

365PCB Embedded Linux Development therefore treats Linux as: A Complete Hardware-to-Lifecycle Product Platform.


01 — Start With the Product Architecture

Don't Start With "Which Linux Distribution?"

First define:

What must the product do?

What interfaces exist?

How fast must it boot?

Does it need graphics?

Does it need AI acceleration?

Does it need cameras?

Does it need hard or soft real-time behavior?

How will it be updated?

How long must it be supported?

What is the cybersecurity requirement?

Linux Distribution Comes After Product Requirements.

02 — SoC Selection Is Linux Architecture

Embedded Linux often runs on:

  • Arm Cortex-A

  • x86-64

  • RISC-V

  • or other Linux-capable processor architectures.

The SoC choice influences:

  • kernel support

  • bootloader

  • drivers

  • GPU

  • NPU

  • memory

  • security

  • vendor BSP

  • lifecycle

Selecting the Processor Also Selects a Software Ecosystem.

03 — Processor Performance Is Not Enough

A powerful SoC can still be a poor product choice if:

  • Linux support is immature

  • drivers are proprietary

  • BSP maintenance is weak

  • security updates are uncertain

lifecycle is short.

Silicon Capability Without Software Support Is Incomplete Capability.

04 — Vendor BSP

A semiconductor vendor often provides a: Board Support Package — BSP

containing some combination of:

  • Bootloader

  • Linux Kernel

  • Device Tree

  • Drivers

  • Firmware Blobs

  • RootFS Integration

  • Multimedia Libraries

  • GPU / NPU Software

The BSP Is the First Software Bridge Between the SoC and the Product.

05 — BSP Is Not the Final Product

Vendor BSPs are usually built to demonstrate: Silicon Capability.

The product team must transform that into: Production Software.

That can require:

  • removing unnecessary packages

  • hardening

  • updating

  • adding board support

  • custom drivers

  • OTA

  • diagnostics

application services.

06 — Mainline Linux vs Vendor Kernel

One major architectural decision is: Upstream / Mainline-Oriented Linux vs. Vendor BSP Kernel

A vendor kernel may support:

  • proprietary accelerators

  • camera ISP

  • multimedia

  • special peripherals

earlier.

Mainline can provide better:

  • long-term maintainability

  • upstream fixes

portability.

Kernel Choice Is a Lifecycle Trade-Off.

07 — Don't Chase the Highest Kernel Number

As of August 30, 2026, kernel.org lists: 7.2.2 as current stable,

while: 6.18

and: 6.12

are current longterm lines projected through December 2028.

For an embedded product: Newest Kernel ≠ Best Production Kernel.

The right kernel is the one whose: SoC Support + Drivers + Security + Lifecycle + Product Validation

match the platform.

08 — Kernel Lifecycle Strategy

Before launch, decide:

Which kernel branch will this product live on?

How will CVE fixes be integrated?

When will the next major kernel migration occur?

Kernel Maintenance Should Begin Before Product Shipment.

09 — Patch Stack

Many embedded products carry:

  • board patches

  • vendor patches

  • driver patches

feature patches.

A large private patch stack creates long-term maintenance cost.

Every Out-of-Tree Patch Is Future Merge Work.

10 — Upstream Where Practical

Where product/business/IP constraints allow, upstream-compatible engineering reduces future rebasing.

Maintain Less Private Kernel Code.

This can become a major lifecycle advantage.

11 — Boot Chain

A typical Arm Linux product may follow: Reset → Boot ROM → First-Stage Firmware → Trusted Firmware → SPL → U-Boot → Linux Kernel → Userspace

Linux Boot Is a Chain of Trust and Responsibility.

12 — Boot ROM

The immutable Boot ROM is generally part of the SoC.

It may determine:

  • boot device

  • authentication root

  • recovery behavior

depending on silicon.

Product Security Can Begin Before Any Customer Code Runs.

13 — First-Stage Bootloader

Early boot code may initialize:

  • clocks

  • DRAM

  • storage

  • security state

before full bootloader operation.

The First Milliseconds of Boot Establish the Platform.

14 — SPL / TPL

U-Boot architectures may use smaller early stages such as:

TPL / SPL

before the complete bootloader.

This is useful when early memory/resources are limited.

Bootloader Architecture Can Be Multi-Stage.

15 — U-Boot

U-Boot remains one of the central bootloaders in Embedded Linux.

Its current documentation includes:

  • SPL/TPL

  • Device Tree

  • FIT images

  • DFU

  • measured boot

  • boot flows

update mechanisms.

The Bootloader Is Product Infrastructure — Not a Development Menu.

16 — Production U-Boot

A production configuration should differ from an unrestricted development environment.

Review:

  • console access

  • environment modification

  • arbitrary boot

  • network boot

debug interfaces.

Development Convenience Can Become Production Attack Surface.

17 — Boot Environment

Boot variables can determine:

  • kernel image

  • root partition

  • console

update slot.

Boot Configuration Is Product Configuration.

18 — Boot Count

A robust product may count failed boot attempts.

Example: Try New Image → Boot Fails Repeatedly → Return to Previous Image

Boot Failure Should Have Recovery Logic.

19 — Fast Boot

Some products care strongly about: Power-On-to-Useful-Function Time.

Optimization can span:

  • Boot ROM

  • Bootloader

  • Kernel

  • Drivers

  • systemd

  • Applications

Boot Time Is an End-to-End Requirement.

20 — Boot Profiling

Don't guess where boot is slow.

Measure:

  • bootloader duration

  • kernel init

  • driver probe

service startup.

Optimize the Largest Boot Delay First.

21 — Parallel Startup

Independent services can sometimes initialize concurrently.

But: Parallelism Should Follow Dependency Analysis.

22 — Deferred Initialization

Not every service must start before the user/product function becomes available.

Boot the Critical Product Path First.

23 — Device Tree

Linux Device Tree is a hardware-description structure allowing platform configuration to be data-driven instead of hard-coded directly into the kernel.

Official kernel documentation describes it as a structure for hardware information including buses, interrupts, GPIOs and peripherals.

Device Tree Describes the Hardware the Kernel Is Actually Running On.

24 — DTS

A board-level Device Tree Source can define:

  • CPU

  • Memory

  • I²C

  • SPI

  • UART

  • GPIO

  • Regulators

  • Displays

  • Cameras

  • Ethernet

and other hardware.

Hardware Configuration Becomes Machine-Readable.

25 — DTSI

Reusable SoC or platform definitions can live in shared include files.

Don't Copy an Entire SoC Description for Every Board Variant.

26 — Device Tree Bindings

Bindings define the schema/contract expected by drivers.

The current Linux Device Tree documentation includes JSON-schema-based binding development and validation.

Device Tree Should Match Driver Architecture — Not Invent Properties Ad Hoc.

27 — Board Variants

One product family may contain:

  • Rev A

  • Rev B

  • Industrial Variant

  • Display Variant

  • Wireless Variant

Device Tree architecture can help manage these differences.

Hardware Variant Management Is Software Architecture.

28 — Device Tree Overlays

Overlays can modify an existing tree for modular hardware where architecture supports it.

But production systems should carefully control which configurations are valid.

Flexibility Needs Configuration Governance.

29 — Kernel Configuration

Linux has thousands of configuration options.

Do not ship:

  • Everything Enabled.

  • Select features according to:

  • hardware

  • product requirements

  • security

  • memory

boot time.

Kernel Configuration Is Product Architecture.

30 — Smaller Kernel Surface

Unused:

  • protocols

  • filesystems

  • drivers

can be disabled where practical.

Benefits can include:

  • smaller image

  • lower attack surface

faster initialization.

Feature Reduction Can Improve Both Engineering and Security.

31 — Built-In vs Module

Drivers can be: built into kernel or loadable modules.

Boot-critical drivers may need to be built in depending on architecture.

Module Strategy Affects Boot, Updates and Security.

32 — Kernel Modules

Modules provide flexibility but introduce:

  • loading

  • dependency

  • signing

  • version

considerations.

Dynamic Kernel Code Should Be Controlled Like Any Other Executable Code.

33 — Device Drivers

A Linux driver translates: Kernel Subsystem API

into: Hardware Behavior.

Drivers Are the Boundary Between Operating System and PCB.

34 — Don't Create Custom APIs Unnecessarily

Linux already has established subsystems for:

  • GPIO

  • I²C

  • SPI

  • IIO

  • V4L2

  • DRM

  • ALSA

  • Network

and many others.

Use the Linux Driver Model Instead of Rebuilding Linux Inside One Proprietary Driver.

35 — Platform Drivers

SoC-integrated peripherals are often exposed through platform devices tied to Device Tree or ACPI descriptions.

Hardware Description and Driver Binding Work Together.

36 — I²C Drivers

Typical devices include:

  • sensors

  • PMICs

  • EEPROM

  • codecs

controllers.

Bus Communication Is Only One Part of a Proper Linux Driver.

37 — SPI Drivers

SPI can support:

  • sensors

  • ADC/DAC

  • displays

  • storage

controllers.

Linux integration should select the correct kernel subsystem for the device's actual function.

SPI Is a Transport — Not an Application API.

38 — GPIO

Modern Linux GPIO frameworks should be used rather than depending on obsolete assumptions or uncontrolled register access.

A GPIO Still Belongs to the Kernel Resource Model.

39 — PWM

PWM can control:

  • backlight

  • motors

  • fans

power-related functions.

Hardware Timing Should Remain in Hardware Where Possible.

40 — Industrial I/O — IIO

Linux IIO supports many:

  • ADCs

  • DACs

  • sensors

and buffered acquisition models.

Sensor Integration Can Use Standard Kernel Infrastructure.

41 — DMA

High-bandwidth peripherals need efficient transfer between: device and memory.

Don't Ask the CPU to Copy Every Byte When Hardware Can Move It.

42 — DMA Coherency

With caches and accelerators: CPU-visible memory and DMA-visible memory must remain coherent according to architecture.

Memory Is Shared Between Several Masters — Not Only the CPU.

43 — IOMMU

An IOMMU can map and restrict device memory access.

Benefits can include:

  • isolation

  • virtualization

safer DMA architecture.

DMA Devices Can Have Memory Permissions Too.

44 — CMA

Contiguous Memory Allocator-style strategies may support:

  • cameras

  • display

  • multimedia

requiring large physically contiguous regions.

Memory Architecture Should Follow Peripheral Data Flow.

45 — DMA-BUF

Linux DMA-BUF enables buffer sharing between subsystems/devices.

This is important for: Camera → ISP → GPU → Display

or: Camera → NPU

pipelines.

Zero-Copy Architecture Can Remove Entire Memory-Copy Stages.

46 — Zero Copy

Every full-frame copy costs:

  • memory bandwidth

  • CPU

  • latency

power.

The Fastest Copy Is the Copy the Architecture Does Not Need.

47 — PCIe

Embedded Linux products increasingly integrate:

  • NVMe

  • network controllers

  • FPGAs

accelerators.

PCIe Is Both Hardware SI and Linux Software Architecture.

48 — PCIe Enumeration

Boot must correctly discover:

  • endpoints

  • BARs

  • interrupts

link state.

Electrical Link Training Is Only the Beginning.

49 — MSI / MSI-X

Modern PCIe devices often use message-signaled interrupts.

Interrupt Architecture Scales With Device Throughput.

50 — NVMe

High-performance embedded platforms may use NVMe for:

  • local storage

  • logging

  • AI data

database workloads.

Storage Throughput Can Become a PCIe + Filesystem + Thermal Problem.

51 — USB

Linux can operate as: USB Host or USB Device / Gadget

depending on hardware and product.

USB Role Is Product Architecture.

52 — USB Gadget

Embedded products can present themselves as:

  • network device

  • storage

  • serial

custom composite device.

Linux Can Become the Peripheral.

53 — Ethernet

Linux offers mature networking infrastructure.

But product architecture must still handle:

  • PHY

  • MAC

  • timing

  • link state

security.

Ethernet Is Hardware + Driver + Kernel + Userspace.

54 — PTP

Precision Time Protocol-capable systems can synchronize clocks across networks where hardware and drivers support suitable timestamping.

Network Time Can Become Measurement Time.

55 — Time Synchronization

Industrial, sensing and distributed-control products may depend on:

  • PTP

  • NTP

  • GNSS

or other time sources.

Time Is a Distributed System Resource.

56 — CAN / CAN FD

Linux SocketCAN provides a standard networking-style architecture for CAN interfaces.

Vehicle and Industrial Buses Can Enter the Linux Network Model.

57 — Wi-Fi

Wireless integration includes far more than loading a driver.

Consider:

  • regulatory firmware

  • calibration

  • antenna

  • roaming

  • provisioning

security.

Wireless Hardware Requires Product-Level Software Integration.

58 — Bluetooth

Bluetooth may involve:

  • controller

  • host stack

  • profiles

product services.

Connectivity Stack Selection Is Product Architecture.

59 — Cellular

Cellular modems may interface over:

  • USB

  • PCIe

  • UART

with networking and management services layered above.

Modem Integration Is a Device-Management Problem as Well as a Network Problem.

60 — Storage Architecture

Embedded products commonly use:

  • eMMC

  • SD

  • NAND

  • NOR

  • NVMe

  • UFS

depending on platform.

Storage Choice Determines Filesystem and Update Architecture.

61 — Raw NAND

Raw NAND requires management of:

  • bad blocks

  • wear

  • ECC

through suitable flash subsystems.

Raw Flash Is Not a Disk.

62 — MTD

Linux Memory Technology Device infrastructure supports flash-oriented devices.

Storage Software Should Match Flash Physics.

63 — UBI / UBIFS

For suitable raw-NAND architectures, UBI/UBIFS can manage:

  • bad blocks

  • wear

volume abstraction.

Flash Reliability Requires Flash-Aware Software.

64 — eMMC

Managed flash simplifies some NAND management internally.

But products still need to consider:

  • endurance

  • power failure

lifecycle.

Managed Flash Does Not Mean Infinite Write Endurance.

65 — Filesystem Selection

Possible choices include:

  • ext4

  • F2FS

  • SquashFS

  • UBIFS

  • and others.

  • Selection depends on:

  • storage

  • write behavior

  • integrity

update architecture.

Filesystem Is Product Behavior.

66 — Read-Only Root Filesystem

A read-only root can improve:

  • robustness

  • predictability

security.

Writable product data can live elsewhere.

Separate Product Software From Product Data.

67 — Immutable System Direction

An advanced architecture can treat the operating-system image as: Immutable.

Updates replace complete verified images rather than allowing uncontrolled in-field package drift.

Predictable Systems Are Easier to Validate.

68 — OverlayFS

Some designs combine: read-only base

with: writable overlay.

Useful, but persistence/update semantics must be understood.

Convenience Layers Still Need Recovery Rules.

69 — Data Partition

User/product data should often have a lifecycle separate from the OS.

Updating Linux Should Not Erase Product Identity or Calibration.

70 — RootFS Architecture

A production RootFS should contain: What the Product Needs.

Not: every package available in the build system.

71 — Init System

Linux needs a userspace initialization system.

Many modern embedded distributions use: systemd.

Yocto 6.0 now defaults its init-manager definition to systemd as part of that release's platform changes.

72 — systemd

systemd can manage:

  • services

  • dependencies

  • sockets

  • timers

  • watchdogs

startup.

Product Startup Should Be a Dependency Graph — Not a Shell Script Accident.

73 — Service Dependencies

Example: Database

must start after: persistent storage.

Network application may depend on: etwork-online.

Explicit Dependencies Beat Startup Delays Like "sleep 5."

74 — Service Supervision

If a service exits:

should it:

restart?

stay failed?

trigger product fault?

Process Recovery Policy Is Product Behavior.

75 — Restart Loops

A crashing service configured to restart forever can create:

  • CPU load

  • flash logging

unstable product behavior.

Recovery Needs Bounds.

76 — Hardware Watchdog

A Linux system can still hang below userspace.

A hardware watchdog can provide an independent recovery layer where appropriate.

Process Supervision and Hardware Supervision Solve Different Failures.

77 — Watchdog Chain

Possible architecture: Application Health → systemd / Supervisor → Kernel Watchdog → Hardware Watchdog

Health Supervision Can Be Layered.

78 — Processes

Linux provides strong process isolation.

Unlike a typical MCU RTOS: One Application Does Not Need to Own the Entire Address Space.

79 — Process Boundaries

Separate critical services according to:

  • responsibility

  • trust

restart behavior.

Process Isolation Creates Fault-Containment Boundaries.

80 — Threads

Within one process, threads can share:

  • memory

  • file descriptors

while executing concurrently.

Process and Thread Boundaries Serve Different Architectural Goals.

81 — IPC

Processes need communication.

Possible mechanisms include:

  • Unix Domain Sockets

  • Shared Memory

  • D-Bus

  • Pipes

  • Message Queues

  • Network Sockets

IPC Architecture Is Product Architecture.

82 — Unix Domain Sockets

Useful for structured local service communication.

A Process Boundary Does Not Require TCP/IP Overhead.

83 — Shared Memory

Shared memory can provide high throughput.

But synchronization and ownership become critical.

Zero Copy Trades Copy Cost for Concurrency Complexity.

84 — D-Bus

D-Bus can provide structured service communication on suitable Linux products.

System Services Benefit From Defined Interfaces.

85 — API Versioning

If process A and process B can be updated independently: Their Interface Needs a Version Strategy.

86 — Hardware Abstraction Service

Instead of allowing every application to: open /dev/mem or directly manipulate devices, build defined service/API boundaries.

Userspace Architecture Should Preserve Hardware Ownership.

87 — Privilege Separation

A display application does not necessarily need: root.

A telemetry process does not necessarily need raw device access.

Least Privilege Applies to Embedded Linux.

88 — Linux Capabilities

Linux can divide traditional root privileges into narrower capability sets.

Root vs Non-Root Is Not the Only Security Choice.

89 — seccomp

System-call filtering can reduce what a process is permitted to ask the kernel to do.

Limit Software to the Kernel Interfaces It Actually Needs.

90 — Linux Security Modules

Linux's LSM framework supports policy systems including: SELinux and AppArmor, among others.

Process Isolation Can Be Policy-Enforced.

91 — AppArmor

AppArmor uses profiles to restrict application behavior.

A Compromised Service Should Not Automatically Have Access to the Whole Product.

92 — SELinux

SELinux provides mandatory access-control architecture suitable for systems requiring stronger policy segmentation.

Security Policy Can Be an Engineering Artifact.

93 — Containers

Embedded Linux can use containers where they create value.

Containers can isolate:

  • applications

  • dependencies

versions.

Containerization Is Deployment Architecture — Not Automatically Good Architecture.

94 — Container Cost

Containers can increase:

  • storage

  • RAM

update complexity.

Use Containers When Isolation and Deployment Benefits Exceed Their Resource Cost.

95 — Namespaces

Linux namespaces isolate system views such as:

  • processes

  • mounts

networking.

Containers Are Built From Kernel Isolation Mechanisms.

96 — cgroups

Control groups can constrain:

  • CPU

  • memory

process resources.

One Service Should Not Be Able to Consume the Entire Product.

97 — Resource Budget

A high-level product should have budgets for:

  • RAM

  • CPU

  • Storage

  • Network

Linux Resources Are Large — Not Infinite.

98 — Out-of-Memory

Linux can behave dramatically when system memory is exhausted.

Memory Exhaustion Needs Product-Level Prevention and Diagnosis.

99 — Memory Leak

A small leak that appears insignificant during a one-hour test may crash a product after months.

Long-Life Linux Products Need Long-Duration Memory Testing.

100 — Swap

Many embedded products avoid or carefully control swap depending on:

  • flash

  • latency

endurance.

Desktop Memory Assumptions Do Not Automatically Belong in Embedded Products.

101 — Real-Time Linux

Now we reach one of the most important modern areas.

Since Linux 6.12, PREEMPT_RT support became part of official Linux. The Linux Foundation's current RT documentation explicitly states that since 6.12 the official kernel supports PREEMPT_RT.

Linux Can Now Be Engineered Much More Directly for Deterministic Real-Time Workloads.

102 — What PREEMPT_RT Changes

Official kernel documentation explains that PREEMPT_RT moves much more kernel execution under scheduler control through mechanisms including:

  • preemptible locking

  • rtmutex

  • priority inheritance

threaded interrupts.

Real-Time Linux Reduces Uncontrolled Kernel Latency.

103 — PREEMPT_RT Does Not Guarantee Your Deadline

Installing an RT kernel does not automatically create: A Real-Time Product.

Application architecture still matters.

104 — SCHED_FIFO

Linux supports real-time scheduling policies such as: SCHED_FIFO.

A higher-priority runnable task can preempt lower-priority work under the RT scheduling model.

Priority Architecture Still Matters.

105 — SCHED_RR

Round-robin real-time scheduling can provide time sharing among suitable equal-priority tasks.

Scheduler Policy Should Match the Workload.

106 — SCHED_DEADLINE

Advanced Linux systems can also use deadline-oriented scheduling where architecture and platform support it.

Linux Real-Time Engineering Can Move Beyond Fixed Priorities.

107 — Priority Inheritance

PREEMPT_RT uses RT mutex mechanisms to reduce certain priority-inversion effects by transferring priority to a lock owner when needed.

Kernel Locking Can Participate in Real-Time Scheduling.

108 — Threaded Interrupts

PREEMPT_RT moves much interrupt processing into schedulable thread context.

Interrupt Handling Can Become Priority-Controlled Work.

109 — IRQ Priority

A real-time application can still be delayed by incorrectly prioritized interrupt threads.

RT Kernel Configuration Requires IRQ Architecture Too.

110 — IRQ Affinity

On multicore processors, interrupts may be assigned to selected CPUs.

Keep Noisy Interrupts Away From Critical Real-Time CPUs Where Architecture Requires It.

111 — CPU Affinity

Real-time tasks can be pinned to selected CPU cores.

Scheduling Freedom Can Be Traded for Timing Predictability.

112 — CPU Isolation

Critical workloads may use dedicated CPUs with reduced:

  • background tasks

  • interrupts

housekeeping.

A Core Can Become a Real-Time Resource.

113 — Real-Time Core vs Linux Core

An architecture might dedicate: CPU Core A to high-priority control,

while other cores handle: Networking + UI + Storage.

Multicore Allows Functional Timing Isolation.

114 — Linux + MCU Hybrid Architecture

Not every real-time requirement belongs in Linux.

A product can combine:

  • Linux MPU/SoC

  • MCU / RTOS

for demanding low-latency control.

Use the Right Compute Domain for Each Deadline.

115 — Heterogeneous SoCs

Modern SoCs may combine:

Cortex-A

with:

Cortex-M / R-class cores

or other dedicated engines.

One Chip Can Contain Several Computing Models.

116 — Remote Processor Architecture

Linux can coordinate secondary processor firmware on suitable platforms.

Linux Can Be the System Manager Without Running Every Real-Time Loop Itself.

117 — RPMsg / IPC Direction

Heterogeneous cores can exchange structured messages through shared-memory/mailbox frameworks.

Processor Boundaries Need Protocols Too.

118 — Linux Real-Time Memory

Page faults can create unpredictable latency.

Critical real-time processes may therefore need memory architecture that avoids unexpected faulting.

Virtual Memory Convenience Can Become Timing Variability.

119 — Memory Locking

Real-time workloads may lock critical memory into RAM where appropriate.

Don't Let a Deadline Wait for Memory to Become Resident.

120 — Dynamic Allocation

Runtime allocation can introduce variable latency.

Real-Time Linux Applications Still Need Resource Discipline.

121 — Frequency Scaling

CPU frequency changes can alter:

  • execution time

  • latency

power.

DVFS Is a Timing Variable.

122 — Thermal Throttling

A processor that passes timing cold can miss timing when: Thermally Throttled.

Real-Time Verification Must Include Thermal State.

123 — Cache

Cache misses create execution-time variation.

Gigahertz CPU Frequency Does Not Mean Constant Instruction Latency.

124 — Memory Bandwidth

GPU, NPU, camera and CPU may compete for DDR bandwidth.

Shared Memory Can Become a Real-Time Contention Resource.

125 — Hardware Bus Contention

The Linux RT documentation itself emphasizes that real-time performance depends not only on kernel scheduling but also on hardware factors including memory, buses, virtualization and networking.

Software Cannot Schedule Away a Saturated Hardware Bus.

126 — Latency Measurement

Do not say:"Linux real-time latency is 20 µs."

without platform definition.

Measure:

  • hardware

  • kernel

  • load

  • configuration

thermal state.

Real-Time Capability Is Platform-Specific.

127 — Worst-Case Latency

As with RTOS: Average Latency Is Not the Requirement.

The worst observed / engineered bound is usually far more relevant.

128 — Stress + Latency

Real-time testing should run alongside:

  • networking

  • storage

  • GPU

  • memory pressure

representing the real product workload.

Measure Timing While the Product Is Busy.

129 — Kernel Tracing

Linux provides sophisticated tracing infrastructure for understanding:

  • scheduling

  • interrupts

  • latency

kernel functions.

Linux Is One of the Most Observable Embedded Operating Systems Available.

130 — ftrace

Kernel tracing can reveal:

  • function timing

  • scheduling behavior

interrupts.

Turn Kernel Timing Into Evidence.

131 — osnoise / timerlat Direction

Modern kernel tracing tools can investigate:

  • operating-system noise

  • timer latency

for real-time workloads.

Real-Time Debugging Should Locate the Source of Latency.

132 — perf

Performance counters and profiling can identify:

  • CPU hotspots

  • cache behavior

execution bottlenecks.

Optimize From Measurements.

133 — eBPF

eBPF increasingly enables deep observability and controlled programmable kernel instrumentation.

Production Diagnostics Can Become More Powerful Without Recompiling Every Application.

Use only where product complexity/resources justify it.

134 — GPU Integration

Modern Embedded Linux platforms increasingly combine CPUs with GPUs.

Applications can include:

  • GUI

  • vision

  • visualization

compute.

GPU Support Is Kernel + Driver + Userspace + Application Integration.

135 — Graphics Stack

A Linux graphics pipeline can involve: Kernel DRM/KMS → Userspace Graphics Libraries → Compositor → Application.

A Display Is a Pipeline — Not One Framebuffer Call.

136 — DRM / KMS

Linux DRM/KMS provides standard kernel infrastructure for modern display pipelines.

Display Hardware Should Integrate With the Linux Graphics Model.

137 — Wayland Direction

Many modern embedded GUI platforms use Wayland-based compositing architectures.

GUI Architecture Should Match GPU, latency and product UI requirements.

138 — Camera Systems

Embedded vision products may include: Image Sensor → MIPI CSI-2 → Receiver → ISP → Memory → Application / NPU

Camera Linux Development Is an End-to-End Data Pipeline.

139 — V4L2

Video4Linux2 provides a major Linux framework for cameras/video devices.

Camera Drivers Should Integrate Into Standard Media Infrastructure Where Practical.

140 — Media Controller

Complex camera systems may have multiple processing entities:

  • sensor

  • CSI receiver

  • ISP

scaler.

Camera Pipelines Can Be Graphs — Not Single Devices.

141 — MIPI CSI-2

The PCB must first reliably deliver the high-speed camera interface.

Linux must then correctly configure:

  • clocks

  • lanes

  • sensor

  • receiver

pipeline.

Camera Bring-Up Is Hardware + SI + Driver + ISP Engineering.

142 — ISP

Image Signal Processors can perform:

  • demosaic

  • color correction

  • noise reduction

  • HDR

depending on platform.

Image Quality Is Software-Controlled Hardware Behavior.

143 — Camera Tuning

A camera that produces an image is not automatically: A Finished Camera System.

Real products may require:

  • exposure

  • white balance

  • color

  • noise

  • lens

tuning.

144 — GStreamer

Multimedia applications frequently use pipeline architectures for:

  • camera

  • codec

  • streaming

display.

Preserve Zero-Copy Paths Through the Multimedia Pipeline Where Performance Requires It.

145 — Video Codec

Hardware encode/decode can dramatically reduce CPU load for:

  • H.264

  • H.265/HEVC

other supported formats.

Use Dedicated Silicon for Dedicated Work.

146 — Audio

Linux ALSA provides standard audio infrastructure.

Product implementation may include:

  • Codec

  • I²S/TDM

  • Amplifier

  • Mixer

Userspace Audio.

Audio Is Hardware + Clock + Driver + Userspace.

147 — NPU

Modern SoCs increasingly include: Neural Processing Units.

These accelerate edge inference workloads.

148 — NPU Integration

NPU functionality can depend on:

  • vendor driver

  • runtime

  • compiler

  • model format

  • memory architecture.

"SoC Has NPU" Does Not Mean AI Application Is Finished.

149 — Edge AI Pipeline

A real architecture might be: Camera → ISP → DMA-BUF → NPU → Post-Processing → Application.

AI Performance Depends on Data Movement as Much as TOPS.

150 — Zero-Copy AI

Repeatedly copying a 4K frame between:

  • CPU

  • GPU

  • NPU

can waste enormous memory bandwidth.

Accelerate the Data Path — Not Only the Neural Network.

151 — AI Latency Budget

Measure:

  • Capture

  • Preprocess

  • Inference

  • Postprocess

  • Decision

TOPS Is Not End-to-End Product Latency.

152 — Thermal AI Performance

NPU/GPU performance can throttle with heat.

Benchmark AI in the Final Thermal Environment.

153 — Model Lifecycle

AI models can change after shipment.

That creates version relationships among:

  • Application

  • Runtime

  • Model

  • NPU Compiler

Hardware.

Model Version Is Product Configuration.

154 — Yocto Project

For complex long-life Embedded Linux products, Yocto is one of the most important custom-distribution build frameworks.

Current Yocto 6.0 Wrynose is an LTS line supported until April 2030.

Yocto Builds a Product Distribution — Not Just a Root Filesystem.

155 — Yocto Layers

Metadata can be organized into layers.

Conceptually:

  • SoC Layer

  • Board Layer

  • Product Layer

  • Application Layer

Separate Responsibilities in the Build System.

156 — Recipes

Recipes define how software is:

  • fetched

  • configured

  • compiled

packaged.

Build Knowledge Should Be Machine-Readable.

157 — BitBake

BitBake executes the dependency/build graph.

Embedded Linux Build Is Dependency Engineering.

158 — Machine Configuration

Machine metadata can describe:

  • CPU architecture

  • kernel

  • bootloader

board-specific behavior.

Hardware Variant Should Be Represented in the Build System.

159 — Distro Configuration

Distribution configuration can define product-wide choices such as:

  • init

  • security

  • packaging

libraries.

The Operating-System Policy Should Be Explicit.

160 — Image Recipe

The image defines what actually ships.

If It Is Not Required by the Product, Ask Why It Is in the Image.

161 — Yocto 6.0

Wrynose 6.0 moved its baseline to:

  • Linux 6.18

  • GCC 15.2

  • glibc 2.43

  • LLVM 22.1.2

and more than 300 recipe upgrades.

Product Linux Platforms Continue to Move Rapidly.

162 — SBOM / SPDX

Yocto 6.0 expanded SPDX metadata support, including options for exporting kernel configuration and PACKAGECONFIG data into SPDX output.

Build System Can Become Software-Supply-Chain Evidence.

163 — Buildroot

Buildroot provides another strong embedded Linux build approach.

As of August 30, 2026:

  • stable: 2026.05.2

  • release candidate: 2026.08-rc2

LTS: 2025.02.17.

Buildroot and Yocto Solve Similar Product Problems With Different Complexity Models.

164 — Yocto vs Buildroot

A simplified comparison: Buildroot

Excellent when:

  • architecture is relatively controlled

  • development team values simplicity

image is rebuilt as a whole.

Yocto

Strong where:

  • multiple products

  • complex dependencies

  • long lifecycle

  • reusable layers

  • richer package/compliance metadata

matter.

Choose the Build Architecture for the Product Organization — Not Only the First Prototype.

165 — Cross Compilation

Embedded Linux commonly builds software on: x86 development host

for: Arm / RISC-V target.

Build Architecture and Runtime Architecture Are Different.

166 — SDK

A controlled SDK can provide application developers with:

  • compiler

  • sysroot

  • libraries

headers.

Application Builds Should Target the Exact Product Platform.

167 — Toolchain Version

Compiler/library changes can affect:

  • performance

  • ABI

bugs.

Toolchain Is Product Configuration.

168 — glibc vs musl

The C library can influence:

  • compatibility

  • size

behavior.

Userspace ABI Choice Has Long-Term Consequences.

169 — C / C++

Native applications remain common in performance-sensitive embedded Linux.

Application Language Should Follow Latency, Safety, Ecosystem and Maintainability Requirements.

170 — Rust

Rust adoption continues to expand in systems software and can be considered for selected userspace/security-sensitive components where the toolchain and ecosystem fit.

Memory-Safe Languages Can Reduce Certain Defect Classes — but Architecture Still Matters.

171 — Python

Python can be valuable for:

  • orchestration

  • diagnostics

  • rapid product logic

where CPU/storage/startup requirements allow.

Not Every Embedded Function Needs to Be Written in C.

172 — Product Services

A mature Linux system may split functionality into services such as:

  • Hardware Manager

  • Connectivity Manager

  • Update Manager

  • Diagnostics Manager

  • Application Engine

Product Software Can Become a Service Architecture.

173 — Service Ownership

One process should usually have clear ownership of critical hardware.

Avoid Five Daemons Fighting Over the Same Device.

174 — Product State Manager

Define product-level states:

  • BOOTING

  • READY

  • ACTIVE

  • DEGRADED

  • UPDATE

  • FAULT

  • SHUTDOWN

Linux Processes Still Need One Coherent Product State.

175 — Startup

A process manager starting successfully does not mean: Product Ready.

The application should understand:

  • sensor readiness

  • network state

  • calibration

time synchronization.

176 — Graceful Shutdown

Linux products with writable storage should manage:

  • service stop

  • data flush

  • hardware output

filesystem state.

Pulling Power Is Not a Shutdown Strategy.

177 — Unexpected Power Loss

Nevertheless, field products must consider sudden power loss.

The Filesystem and Persistent Data Architecture Should Assume Power Can Disappear.

178 — Journaling

Journaling filesystems can improve recovery of filesystem metadata.

But: Journaling Does Not Make Every Application Write Transactional.

179 — Database Integrity

Applications using databases should define power-loss and transaction behavior explicitly.

Storage Consistency Is Application Architecture Too.

180 — Atomic Configuration

A robust configuration update can use: write new → sync / verify → atomic replace

rather than destroying the only valid copy first.

Persistent State Needs Transaction Semantics.

181 — Secure Boot

Production platforms should increasingly consider an authenticated boot chain.

U-Boot's official Verified Boot documentation describes verification from bootloader onward, including signed kernels and integration with dm-verity-protected root filesystems.

Boot Only Software the Product Is Authorized to Run.

182 — Chain of Trust

Conceptually: Hardware Root of Trust → Boot Firmware → Bootloader → Kernel → RootFS

Trust Should Be Transferred Through Each Boot Stage.

183 — FIT Images

U-Boot can use FIT-based images for structured kernel/device-tree and signing architectures.

Boot Artifacts Can Carry Cryptographic Identity.

184 — Verified Boot vs Secure Boot

Terminology varies by platform.

The engineering principle is: Authenticate the Software Before Trusting It.

185 — dm-verity

Linux dm-verity provides transparent integrity checking of read-only block devices using cryptographic hashes.

Detect Root Filesystem Modification at Read Time.

186 — Why dm-verity Matters

A read-only system image can be cryptographically verified.

If a protected block does not match its expected hash: The Product Knows the Image Is No Longer What Was Signed.

187 — fs-verity

fs-verity provides per-file read-only integrity/authenticity protection and is supported by filesystems including ext4, F2FS and btrfs.

Integrity Protection Can Operate at File Level as Well as Block Level.

188 — Encryption vs Integrity

Encryption answers: Can an unauthorized party read the data?

Integrity answers: Has the data changed? Confidentiality and Integrity Are Different Security Requirements.

189 — Storage Encryption

Sensitive product/user data may use encrypted storage depending on threat model.

Protect Data at Rest Where Its Value Justifies It.

190 — Key Storage

Encryption only works if keys are appropriately protected.

Potential architectures include:

  • secure element

  • TPM

  • SoC hardware security

TEE.

A Secret Stored Beside the Ciphertext in Plaintext Is Not a Security Architecture.

191 — Trusted Execution Environment

On suitable Arm platforms, OP-TEE provides a TEE companion to non-secure Linux using hardware isolation such as TrustZone.

Linux Does Not Need to Hold Every Secret in the Normal World.

192 — OP-TEE

Potential uses include:

  • secure storage

  • cryptographic services

  • trusted applications

depending on product architecture.

Place High-Value Operations in the Smallest Practical Trusted Computing Base.

193 — TEE Is Not Automatically Secure

Platform security configuration still matters.

Even platform vendors note that generic TEE configurations must be hardened to match the specific end product rather than assumed production-secure by default.

Security Technology Still Needs Security Engineering.

194 — Measured Boot

Another architecture can measure boot components into a trusted hardware record.

Verified Boot Asks "May This Execute?"

Measured Boot Records "What Executed?"

U-Boot's current documentation includes measured-boot support.

195 — TPM

A Trusted Platform Module can support:

  • measurements

  • keys

  • attestation

where the product requires it.

Device Trust Can Become Cryptographically Verifiable.

196 — Device Identity

Each product may receive a unique:

  • Certificate

  • Key

  • Serial Identity

during secure provisioning.

The Device Should Be Able to Prove Which Device It Is.

197 — Manufacturing Provisioning

Identity provisioning should be integrated with:

  • serial number

  • hardware revision

  • production database

secure credentials.

Digital Identity Is a Manufacturing Output.

198 — Secret Handling

Private keys should not casually appear in:

  • source code

  • build logs

shared production folders.

Security Supply Chain Begins Inside the Factory.

199 — OTA

A connected product requires a plan for: How Software Changes After Shipment.

OTA is therefore not simply a cloud feature.

It is: Product Lifecycle Infrastructure.

200 — A/B Update

A robust architecture may maintain: Slot A and Slot B.

For example: Running A → Write New Image to B → Verify B → Boot B → Confirm Health → Commit B.

Never Destroy the Running System Before the Replacement Is Proven.

201 — Boot Rollback

If the new system fails: Return to Known Good.

This requires cooperation among: Bootloader + RootFS + Update Service + Application Health.

202 — Health Confirmation

Successful kernel boot does not mean the update is good.

The product may need to confirm:

  • application starts

  • critical hardware works

configuration migrates.

Update Success Should Be Defined at Product Level.

203 — Power Failure During OTA

Ask: What happens if power disappears at 37% update? Every Update Stage Must Have Recoverable State.

204 — Atomic Update

A good update system avoids half-new / half-old uncontrolled combinations.

Product Software Should Move Between Known Configurations.

205 — Bootloader Update

Bootloader updates are especially sensitive because bootloader corruption can remove normal recovery.

Update Risk Increases the Lower You Go in the Boot Chain.

206 — Recovery Mode

A product can maintain an independent:

  • recovery partition

  • USB recovery

  • controlled network recovery

according to architecture.

A Field Update Strategy Needs a Failure Strategy.

207 — Anti-Rollback

Cybersecurity may require preventing installation of: Known-Vulnerable Historical Images.

But service/recovery requirements must also be considered.

208 — Data Migration

New software may require new:

  • database schema

configuration.

OS Rollback Is Harder If User Data Cannot Roll Back.

209 — Forward / Backward Data Compatibility

Update design must coordinate: Application Version with Persistent Data Version.

Software Lifecycle Includes Data Lifecycle.

210 — OTA Frameworks

Platforms can use established update frameworks or a custom architecture depending on requirements.

The important engineering requirements are:

  • Authenticity

  • Atomicity

  • Recovery

  • Rollback Policy

  • Power-Failure Safety

Version Compatibility.

Framework Name Is Less Important Than Update Guarantees.

211 — Fleet Management

A deployed fleet may contain:

10

10,000

or:

1,000,000

units.

Software management must understand:

  • version distribution

  • update status

failures.

Embedded Linux Becomes Fleet Software After Shipment.

212 — Staged Deployment

Instead of updating every product simultaneously: Internal → Small Canary Group → Larger Group → Fleet.

Limit the Blast Radius of Software Failure.

213 — Update Telemetry

Track:

  • download success

  • verification

  • boot success

  • rollback

health.

An OTA System Should Know Whether Its Own Update Worked.

214 — Remote Diagnostics

Field devices can report selected:

  • kernel version

  • hardware revision

  • temperature

  • storage

  • service health

fault state.

Field Service Can Become Data-Driven.

215 — Privacy

Collect only telemetry the product actually needs.

Observability Should Not Become Unnecessary Data Collection.

216 — Logs

Linux can generate enormous logs.

Define:

  • what to log

  • retention

  • persistence

rotation.

Unlimited Logging Eventually Becomes a Storage Failure.

217 — journal Persistence

Decide whether system logs live:

  • in RAM

  • persistently

according to service requirements.

Diagnostic Value and Flash Wear Must Be Balanced.

218 — Log Rotation

Without limits: Logs Can Fill the Filesystem.

Resource governance matters.

219 — pstore / ramoops Direction

Kernel crash information can be preserved across reboot on suitable platforms.

A Kernel Panic Should Leave Evidence.

220 — Core Dumps

Userspace crashes can preserve:

  • signal

  • stack

  • binary version

for investigation.

"Application Restarted" Should Not Erase the Root Cause.

221 — Crash Symbolication

Build artifacts should allow addresses to be translated into:

  • functions

  • source code

for the exact released binary.

Debug Information Must Match the Product Version.

222 — Build IDs

Every released executable/image should be identifiable.

You Cannot Debug a Binary Whose Source Revision Is Unknown.

223 — SBOM

A production platform may contain thousands of software components.

Know What Software You Ship.

This is especially important for vulnerability management and licensing.

224 — Software Dependency Inventory

Record:

  • version

  • source

  • license

patches.

Third-Party Software Becomes Product Responsibility When It Ships.

225 — CVE Management

A Linux product needs an ongoing process: New Vulnerability → Affected? → Risk? → Patch / Mitigate → Build → Test → Deploy

Cybersecurity Continues After Product Launch.

226 — Patch Urgency

Not every CVE has equal product risk.

Consider:

component enabled?

reachable?

privilege?

product exposure?

Vulnerability Management Should Be Context-Aware.

227 — Reproducible Builds

A mature platform should make software builds deterministic enough to reproduce released artifacts according to the project's control strategy.

U-Boot's current build documentation explicitly includes reproducible-build support among its developer tooling.

A Product Release Should Be Reconstructable.

228 — Signed Build Artifacts

Release images can be cryptographically signed.

Build Output Should Carry Provenance.

229 — CI

Every source change can trigger: Build → Static Checks → Unit Tests → Image Creation → Automated Hardware Tests

Linux Platform Quality Should Be Continuously Verified.

230 — Board Farm Direction

Real hardware can be connected to automated test infrastructure.

Run the Real Image on the Real Board Automatically.

231 — Kernel Selftests

Linux contains extensive self-testing infrastructure that can complement product-specific tests.

Platform Validation Should Test Both the Operating System and the Product.

232 — Driver Testing

Custom drivers should be tested for:

  • probe/remove

  • suspend/resume

  • errors

stress.

A Driver That Works Once During Boot Is Not Fully Validated.

233 — Fault Injection

Controlled test environments can emulate:

  • device timeout

  • network loss

  • storage errors

process crash.

Test Recovery Logic Before the Field Does.

234 — Power-Loss Testing

Repeated abrupt power interruption can expose:

  • filesystem

  • database

  • update

  • boot

weaknesses.

Power Failure Is a Storage Test.

235 — Storage Endurance Testing

Heavy logging or databases can wear flash.

Write Amplification Can Become Product Lifetime.

236 — Network Stress

Test:

  • reconnect

  • packet loss

  • DNS failure

server outage.

The Internet Is Not a Deterministic Peripheral.

237 — Offline Operation

Connected products should define: What Happens When the Cloud Is Gone?

The product should not necessarily become useless.

238 — Clock / Time Failure

If NTP fails:

does TLS work?

do logs make sense?

can certificates be validated?

Wall Clock Can Be a Security Dependency.

239 — RTC

A hardware RTC may preserve approximate time across power loss.

Timekeeping Can Require Hardware Support.

240 — Suspend

Embedded Linux may use low-power states.

But suspend/resume requires cooperation from:

  • kernel

  • drivers

  • hardware

applications.

Sleep Is a Distributed System State.

241 — Wake Sources

Possible wake sources include:

  • GPIO

  • RTC

  • network

peripheral.

The Product Should Know Why It Woke Up.

242 — Resume Reliability

A driver that works after cold boot may fail after 500 suspend/resume cycles.

Resume Is Another Initialization Path.

243 — Runtime Power Management

Unused devices can sometimes be powered down independently.

Power Management Should Follow Actual Device Use.

244 — cpufreq / DVFS

Dynamic voltage/frequency changes can reduce power.

But they influence:

  • performance

  • thermal

timing.

Energy Management Is Scheduling + Hardware Management.

245 — devfreq

Some SoCs dynamically scale:

  • GPU

  • memory/interconnect

accelerators.

CPU Is Not the Only Dynamic Power Domain.

246 — Thermal Framework

Linux can coordinate:

  • sensors

  • cooling

  • throttling

on suitable platforms.

Thermal Behavior Can Become Operating-System Policy.

247 — Fan Control

Product control can connect: Temperature → Cooling Policy → Fan / Performance

Thermal Control Is a Closed-Loop Product Function.

248 — Thermal Throttle Telemetry

Field data can reveal whether deployed products regularly throttle.

Thermal Margin Can Be Measured in the Fleet.

249 — Manufacturing Programming

A Linux product can require programming of:

  • Bootloader

  • Kernel

  • Device Tree

  • RootFS

  • Recovery Image

  • Device Credentials

Software Loading Is a Manufacturing Process.

250 — Factory Image

Production images may include special:

  • factory test

  • calibration

  • provisioning

functions.

Factory Software and Customer Software Need Controlled Boundaries.

251 — Production Test

Linux can automate hardware testing:

  • RAM

  • Storage

  • Ethernet

  • Wi-Fi

  • USB

  • Camera

  • Audio

  • GPIO

depending on product.

Linux Can Be the Manufacturing Test Platform for Its Own Hardware.

252 — Calibration

Production calibration data can be generated and stored per unit.

Calibration Must Follow the Physical Product It Describes.

253 — Serial Number

The system can bind:

Serial Number

to:

MAC

  • Device Certificate

  • Hardware Revision

Test Record.

Physical Identity and Digital Identity Should Agree.

254 — Secure Provisioning

Production should control access to:

  • certificates

  • private keys

credentials.

Security Must Survive the Factory.

255 — Board Revision Detection

Products with multiple PCB revisions can identify the hardware using:

  • EEPROM

  • GPIO strapping

  • identity data

depending on design.

Software Should Know Which PCB It Is Running On.

256 — Hardware / Software Compatibility

Before loading an image: Is this software compatible with this board?

Prevent Configuration Mismatch Before It Becomes Product Failure.

257 — Product Variants

One Linux platform can support several models.

But variant differences should be represented through controlled:

  • build configuration

  • Device Tree

  • hardware ID

feature configuration.

Product Families Need Configuration Architecture.

258 — EVT Linux

EVT focuses on: Hardware Bring-Up.

Typical work:

  • boot

  • DDR

  • storage

  • Ethernet

  • USB

  • PCIe

  • display

  • camera

sensors.

EVT Linux Should Maximize Hardware Visibility.

259 — Early Bring-Up

Initial boot often uses:

  • Serial Console

as one of the most valuable debug interfaces.

Before the Display Works, UART Tells the Story.

260 — DDR Bring-Up

Linux cannot compensate for an unstable DDR subsystem.

Hardware Memory Integrity Comes Before Operating-System Stability.

261 — Random Kernel Crashes

Before blaming Linux:

investigate:

  • DDR

  • power

  • clocks

  • thermal

  • storage

silicon errata.

Kernel Panic Can Be a Hardware Symptom.

262 — Good Board vs Bad Board

If identical software behaves differently between units: Manufacturing Variation Has Become a Software Symptom.

This is especially important for365PCB.

263 — DVT Linux

DVT moves toward:

  • production kernel

  • security

  • OTA

  • final peripherals

  • final enclosure

stress.

DVT Validates the Complete Platform.

264 — DVT Upgrade Testing

Test: old version → new and update failure and rollback.

OTA Must Be Qualified Before Shipment.

265 — DVT Security

Validate:

  • unauthorized image rejection

  • debug restriction

  • filesystem integrity

  • credentials

according to threat model.

Security Features Must Be Tested as Behaviors.

266 — PVT Linux

PVT verifies:

  • programming

  • provisioning

  • production test

  • image version

hardware variants.

Linux Becomes a Manufacturing Configuration at PVT.

267 — Production Release Manifest

A release can identify:

  • Bootloader

  • Kernel

  • DTB

  • RootFS

  • Application

Certificates / Policy Versions

"Firmware Version 1.3" Is Often Too Little Information for a Linux Product.

268 — Field Maintenance

Shipping the product starts: The Longest Phase of Linux Engineering.

269 — LTS

Long-term supported upstream components can reduce lifecycle risk.

Yocto 6.0's current LTS support window runs to April 2030; kernel.org currently maintains multiple longterm kernels rather than one universal LTS.

Lifecycle Version Selection Should Be Intentional.

270 — Vendor EOL

A vendor may stop maintaining the BSP before the physical product reaches end-of-life.

Software Support Life Should Be Evaluated When Selecting the SoC.

271 — Kernel Migration

Long-lived products eventually need to move to another kernel.

Upgrade Before the Old Platform Becomes Unmaintainable.

272 — Kernel Internal APIs

Linux kernel-internal driver APIs can change across versions.

Out-of-Tree Drivers Create Migration Work.

Another reason to align with upstream architecture where practical.

273 — Binary Blobs

Some hardware depends on proprietary:

  • GPU

  • NPU

  • wireless

  • multimedia

components.

Closed Dependencies Need Vendor-Lifecycle Risk Analysis.

274 — Source Availability

For long-lived industrial products: Control as Much of the Software Stack as the Business Requires to Maintain the Product.

275 — Escrow / Maintenance Strategy

For critical dependencies, organizations may need contractual lifecycle planning.

Technology Risk Can Be Supplier Risk.

276 — Field Root-Cause Analysis

A field failure can originate from:

  • Hardware

  • Kernel

  • Driver

  • Userspace

  • Cloud

  • Configuration

  • Power

  • Thermal

"Linux Crashed" Is a Symptom — Not a Root Cause.

277 — Correlated Diagnostics

Collect:

  • Kernel Log

  • Application State

  • Hardware Data

  • Reset Cause

  • Temperature

Cross-Layer Evidence Solves Cross-Layer Problems.

278 — Hardware Failure Appearing as Software Failure

Examples include: Marginal DDR → random segmentation faults.

Power droop → reboot.

PCIe SI margin → endpoint disappears.

Overheating → performance collapse.

Embedded Linux Debugging Must Understand Hardware Physics.

279 — Software Failure Appearing as Hardware Failure

Conversely: bad driver

can look like:

broken peripheral.

wrong Device Tree

can look like:

PCB pin problem.

Hardware and Linux Must Be Debugged Together.

280 — This Is Where 365PCB Has a Strategic Advantage

A pure software house may see: Kernel Error.

A PCB manufacturer may see: Electrical Test Pass.

But an integrated ODM engineering team can ask:

Is this a driver bug?

DDR timing?

power integrity?

clocking?

Device Tree?

SI?

manufacturing variation?

The Real Product Exists Across Disciplines.

281 — World-Class Embedded Linux Engineering

At the highest level: Product Requirements → SoC Architecture → Hardware Design → Boot ROM / Security Root → Trusted Firmware → U-Boot → Verified / Measured Boot → Linux Kernel → Device Tree → BSP → Drivers → Storage Architecture → Root Filesystem → systemd / Services → IPC → Networking → Multimedia → GPU / NPU → Real-Time Linux → Process Isolation → TEE / Keys → Security Hardening → SBOM → Build System → CI → A/B OTA → Recovery → Manufacturing Provisioning → EVT → DVT → PVT → Fleet Monitoring → Security Maintenance → Kernel / BSP Lifecycle → Long-Term Production Linux Platform

That is the difference between: Booting Linux on a Board

and: Engineering an Embedded Linux Product.

Typical Embedded Linux Development Deliverables

Depending on project scope, a 365PCB ODM Embedded Linux program may include:

  • Embedded Linux Requirements

  • SoC / Software Platform Trade Study

  • Vendor BSP Assessment

  • Mainline vs Vendor Kernel Strategy

  • Linux Kernel Selection

  • LTS Strategy

  • Boot Architecture

  • Trusted Firmware Inputs

  • SPL / TPL Configuration

  • U-Boot Porting

  • U-Boot Configuration

  • Boot-Time Optimization

  • Boot-Failure Recovery

  • Kernel Configuration

  • Kernel Porting

  • Custom Kernel Integration

  • Kernel Patch Management

  • Device Tree Development

  • Device Tree Bindings

  • Hardware Variant Support

  • BSP Development

  • Board Bring-Up

  • Kernel Driver Development

  • GPIO / I²C / SPI Drivers

  • PWM Drivers

  • IIO / Sensor Integration

  • Ethernet Drivers / Integration

  • CAN / CAN FD Integration

  • USB Host / Gadget Integration

  • PCIe Integration

  • NVMe Integration

  • Wi-Fi Integration

  • Bluetooth Integration

  • Cellular Modem Integration

  • Storage Architecture

  • eMMC / NAND / NOR Integration

  • MTD / UBI / UBIFS Inputs

  • Filesystem Selection

  • Read-Only RootFS Design

  • Data Partition Architecture

  • RootFS Customization

  • systemd Integration

  • Service Architecture

  • Hardware Management Services

  • IPC Architecture

  • D-Bus / Socket Integration

  • Linux Networking

  • PTP / Time-Synchronization Inputs

  • Process Isolation

  • Resource Control

  • Containers where appropriate

  • GPU Integration

  • DRM / KMS

  • Display Integration

  • Camera / V4L2 Integration

  • MIPI CSI-2 Software Integration

  • ISP Integration

  • Multimedia Pipeline

  • GStreamer Integration

  • Audio / ALSA Integration

  • NPU / AI Runtime Integration

  • Edge AI Pipeline

  • DMA / Zero-Copy Design

  • DMA-BUF Integration

  • IOMMU Inputs

  • Real-Time Linux Architecture

  • PREEMPT_RT

  • Real-Time Scheduling

  • IRQ Threading / Affinity

  • CPU Isolation

  • Real-Time Latency Analysis

  • Real-Time Trace / Profiling

  • Heterogeneous Linux + MCU Architecture

  • Inter-Core Communication Inputs

  • Power Management

  • Suspend / Resume

  • Runtime PM

  • DVFS Inputs

  • Thermal Management

  • Watchdog Architecture

  • Service Supervision

  • Logging Architecture

  • Crash Diagnostics

  • Kernel Trace

  • Performance Profiling

  • Secure Boot

  • Verified Boot

  • FIT Signing Inputs

  • dm-verity Integration

  • fs-verity Inputs

  • Storage Encryption Inputs

  • TEE / OP-TEE Integration

  • TPM Inputs

  • Secure Storage Inputs

  • Device Identity

  • Secure Provisioning

  • Linux Security Hardening

  • SELinux / AppArmor Inputs

  • seccomp / Capability Restrictions

  • Read-Only System Architecture

  • Software Component Minimization

  • SBOM / SPDX Inputs

  • Vulnerability Management Inputs

  • Yocto Project Development

  • Yocto Layers

  • Yocto Recipes

  • Machine / Distro Configuration

  • Image Development

  • SDK Generation

  • Buildroot Development

  • Custom RootFS

  • Cross-Compilation Toolchain

  • Reproducible Build Inputs

  • CI / Automated Build

  • Automated Hardware Testing

  • A/B Update Architecture

  • OTA Update

  • Update Signing

  • Rollback / Recovery

  • Power-Fail-Safe Update Design

  • Persistent Data Migration

  • Fleet Update Inputs

  • Remote Diagnostics

  • Factory Linux Image

  • Production Programming

  • Production Hardware Test

  • Calibration Integration

  • Secure Credential Provisioning

  • Product Configuration Traceability

  • EVT Linux Bring-Up

  • DVT Platform Validation

  • PVT Production Validation

  • Long-Duration Testing

  • Storage Endurance Testing

  • Power-Loss Testing

  • Network Stress Testing

  • Real-Time Stress Testing

  • Kernel / BSP Maintenance

  • Security Patch Maintenance

  • Hardware Revision Support

  • Software Lifecycle Planning

  • Field Failure Analysis

Embedded Linux Release Documentation

The actual engineering depth should follow: SoC + Peripheral Complexity + Real-Time Requirements + Multimedia + AI + Security + Connectivity + Boot Time + Update Model + Product Lifecycle.

Embedded Linux capability is platform-specific. Kernel support, boot time, real-time latency, multimedia performance, hardware acceleration, security architecture, power consumption and lifecycle depend on the selected SoC, vendor BSP, kernel version, hardware design, memory architecture, peripheral drivers, software workload, build system and product requirements.

We Don't Judge an Embedded Linux Platform by Whether It Reaches a Login Prompt.

We Judge It by Whether the Complete Product Can Boot, Operate, Recover, Update, and Be Maintained Reliably.

Bring Us the Product Platform — Not Just the Linux Version

You can begin with:

  • SoC

  • Custom PCB

  • Schematic

  • Existing BSP

  • Linux Kernel

  • U-Boot

  • Device Tree

  • Existing Drivers

  • Yocto / Buildroot Project

  • Boot Log

  • Kernel Panic

  • Peripheral Bring-Up Issue

  • Real-Time Requirement

  • Security Requirement

  • OTA Requirement

or simply: Tell Us What Hardware Must Become a Product — and How Long That Product Must Be Maintained.

365PCB can help translate: Custom Hardware → Boot Chain → Linux Platform → Drivers → Services → Security → OTA → Manufacturing → Lifecycle.

Don't Just Boot Linux.

Define the Product Platform.

Choose the Kernel for the Lifecycle.

Control the Boot Chain.

Describe the Hardware Correctly.

Engineer the BSP.

Integrate the Drivers.

Control the Memory and Data Flow.

Build the Root Filesystem Intentionally.

Structure the Services.

Protect the Processes.

Engineer Real-Time Behavior Where Required.

Integrate the GPU and NPU.

Secure the Boot Chain.

Verify the Filesystem.

Protect the Device Identity.

Make Updates Recoverable.

Make Builds Reproducible.

Program It Correctly in Manufacturing.

Diagnose It in the Field.

Maintain It for the Product Lifetime.

365PCB Embedded Linux Development connects: SoC + Hardware + Bootloader + Linux Kernel + Drivers + Real-Time + Multimedia + AI + Security + OTA + Manufacturing + Lifecycle

into one coordinated product-platform engineering process.

Embedded Linux Is Not a Desktop Operating System Put Into a Product.

It Is a Custom Software Platform Built Around the Hardware, Boot Chain, Drivers, Services, Security, and Lifecycle of the Product.

And:

The Kernel Boots the Hardware.

The Platform Makes It a Product.

Dedicated Engineering & Support Team

* Your Name
* E-mail Address
* Contact Phone
* Company Name
* Message Content
We use cookies to offer you a better browsing experience, analyze site traffic and personalize content. Part of the tracking is necessary to ensure SEO effectiveness,
By using this site, you agree to our use of cookies. Visit our cookie policy to learn more.
Reject Accept