Fleet Bundled Packages Fail To Install In Air-Gapped Mode

Aug 5, 2025 by Kenji Nakamura 58 views

[Fleet] Bundled Packages Installation Failure in Air-Gapped Mode

Hey guys! It seems like we've hit a snag with installing bundled packages in Fleet when running in air-gapped mode. This article dives into the nitty-gritty of the issue, how to reproduce it, and what the expected behavior should be. Let's break it down so we can get this fixed!

What's the Deal?

So, the main keyword here is air-gapped Fleet package installation. In version 9.0.0, a nifty configuration flag was introduced to mark Fleet as air-gapped. The goal? To allow the installation of bundled packages without needing to reach out to the registry. A pull request was even made to ensure this functionality: https://github.com/elastic/kibana/pull/202435. However, after some further testing, it turns out that installing bundled packages is no longer working as expected. It looks like some changes to the installation flow have inadvertently broken this feature.

Why is Air-Gapped Mode Important?

For those not fully in the loop, air-gapped environments are systems that are physically isolated from public networks like the internet. This isolation is crucial for security reasons, especially in industries dealing with sensitive data, such as government, finance, and healthcare. In such environments, accessing external registries to download packages is a no-go. Therefore, the ability to install bundled packages locally is essential. The air-gapped configuration ensures that all necessary components can be installed without risking exposure to external threats. This setup involves pre-loading all required packages and assets into the environment, making the installation process self-contained. The importance of a smooth installation process in air-gapped environments cannot be overstated, as it directly impacts the ability to deploy and maintain critical security and monitoring solutions. Without a reliable way to install bundled packages, organizations operating in these secure environments would face significant challenges in leveraging the full capabilities of tools like Fleet.

The Bug in Detail

The core of the issue lies in how Fleet handles package installations when the isAirGapped flag is enabled. The expectation is that when this flag is set to true, Fleet should bypass any attempts to connect to external registries and instead rely on the locally bundled packages. However, the current behavior indicates that the system is still trying to reach out to the registry, leading to installation failures. This problem was observed in Kibana version 9.1.0, indicating that the issue persists beyond the initial implementation in 9.0.0. The error messages clearly point to the system's inability to reach the registry, even though it should be operating in a mode where registry access is unnecessary. This not only disrupts the installation process but also raises concerns about the consistency and reliability of the air-gapped configuration.

Impact on Users

For users operating in air-gapped environments, this bug is a major headache. It prevents them from easily deploying and managing integrations like APM, which are crucial for monitoring and securing their systems. The inability to install bundled packages locally defeats the purpose of the air-gapped configuration, potentially forcing users to find cumbersome workarounds or delay critical deployments. This disruption can lead to increased operational costs, delayed security updates, and an overall reduction in the efficiency of their security posture. Therefore, resolving this issue is of paramount importance to ensure that users in secure environments can fully leverage the capabilities of Fleet without compromising their security protocols.

Steps to Reproduce the Issue

Alright, let's get our hands dirty and see how to reproduce this bug. Follow these steps:

Configure Kibana for Air-Gapped Mode:
- Open your kibana.dev.yml file.
- Set xpack.fleet.isAirGapped: true.
- Make sure you do not set registryUrl or registryProxyUrl. This is crucial to ensure that calls to the registry are skipped.
Load a Bundled Package:
- Take a bundled package (like APM) and place it in the x-pack/platform/plugins/shared/fleet/target/bundled_packages directory.
Attempt Installation:
- Go to Integrations > APM in Kibana.
- Try to install the APM integration.
Observe the Failure:
- You'll likely see an error message similar to this in your logs:

[2025-08-05T16:51:00.866+02:00][WARN ][plugins.fleet.PackagePolicyClient.create] getAgentTemplateAssetsMap error: RegistryResponseError: isAirGapped config enabled, registry not reacheable
[2025-08-05T16:51:00.866+02:00][ERROR][plugins.fleet] Error while creating package policy due to error: isAirGapped config enabled, registry not reacheable RegistryResponseError: isAirGapped config enabled, registry not reacheable
    at getResponseStreamWithSize (requests.ts:107:9)
    at fetchArchiveBuffer (index.ts:513:27)
    at Object.getPackage (index.ts:392:9)
    at getAgentTemplateAssetsMap (get.ts:947:19)
    at PackagePolicyClientWithAuthz.create (package_policy.ts:423:25)
    at createPackagePolicyHandler (handlers.ts:270:27)
    at defaultErrorHandlerWrapper (fleet_router.ts:68:14)
    at CoreVersionedRoute.handle (core_versioned_route.ts:216:20)
    at Router.handle (router.ts:215:30)
    at router.ts:198:23

Breaking Down the Reproduction Steps

The reproduction steps are designed to simulate a typical air-gapped environment setup. By explicitly setting xpack.fleet.isAirGapped: true and omitting the registryUrl and registryProxyUrl configurations, we ensure that Fleet should not attempt to access any external registries. The key here is the isolation of the environment. Loading the bundled package into the specified directory mimics the process of pre-loading all necessary resources into an air-gapped system. When the installation is triggered via the Kibana UI, the expected behavior is that Fleet should recognize the local package and proceed with the installation. However, the observed error message clearly indicates that Fleet is still trying to reach the registry, which contradicts the air-gapped configuration. This discrepancy highlights a critical bug in the system's logic for handling bundled packages in air-gapped mode. The steps are straightforward, making it easy for anyone to replicate the issue and confirm the bug's presence in their environment.

Visual Confirmation

To further illustrate the issue, here's a screenshot showing the error in action:

What We Should Expect (Expected Behavior)

Now, let's talk about what should happen. The expected behavior is that when isAirGapped is set to true, the package policy installation for a bundled package should work flawlessly – no need to reach out to any external registry. This is the whole point of air-gapped mode, right? To operate independently of external networks.

The Ideal Installation Flow

In an ideal scenario, when you kick off the installation of a bundled package in air-gapped mode, Fleet should follow a streamlined, self-contained process. First, it should detect that the isAirGapped flag is enabled. This flag acts as a signal to bypass any registry-related operations. Next, Fleet should look within the designated directory (i.e., x-pack/platform/plugins/shared/fleet/target/bundled_packages) for the required package. Upon finding the package, it should proceed with the installation using the local resources. There should be no attempt to resolve external dependencies or fetch assets from the internet. The entire process should be seamless and transparent, ensuring that the package is installed correctly without any external network connectivity. This behavior is crucial for maintaining the integrity and security of air-gapped environments. It also simplifies the installation process, making it more efficient and less prone to errors. The focus should be on utilizing the locally available resources to ensure a smooth and reliable installation experience.

Importance of Clear Error Handling

Even in the best-case scenario, things can sometimes go wrong. That's why clear and informative error handling is essential. If, for any reason, the installation cannot proceed using the bundled package (e.g., the package is missing or corrupted), Fleet should provide a specific error message that guides the user towards a solution. For instance, the error message could state,