How to Fix "Test reports were found but none of them are new. Did tests run?" in Jenkins

Dealing with build artifacts and delayed notifications in a Continuous Integration/Continuous Delivery (CI/CD) pipeline is a common headache. As developers, we often need complex workflows that involve chaining jobs—where one job generates data that another job consumes—and then conditionally notifying stakeholders. The error message you are encountering, "Test reports were found but none of them are new. Did tests run?", points directly to a synchronization and timing issue within Jenkins' artifact management system.

This post will dissect why this happens and provide robust, developer-focused strategies to bypass this limitation, allowing you to achieve your goal of delayed email notifications without compromising the integrity of your build history.

Understanding the Jenkins Artifact Staleness Problem

The core of the issue lies in how Jenkins tracks artifacts. When a job completes, it generates reports (artifacts). Subsequent jobs that reference these artifacts often check for freshness or creation time against the current state. In your scenario, you have two separate jobs: one runs tests and produces reports, and another job attempts to send an email based on those reports.

When you chain jobs where there is a significant delay between the test execution and the notification attempt (especially if the artifact import happens hours later), Jenkins' internal checks can flag the artifacts as "old" or non-existent in the immediate context of the notification step, even if they physically exist on the disk. This is especially true when dealing with specific plugin configurations that rely on immediate build data synchronization.

Strategy 1: Decoupling Artifact Generation and Notification

The most reliable solution is to stop relying on a single job to perform both the artifact import and the final notification. We need to decouple these two actions into distinct, managed steps.

Instead of having Job B directly try to access artifacts from Job A immediately upon triggering, use persistent storage or explicit signalling mechanisms.

Implementing Delayed Notification via Build Status

Since you cannot rely on immediate artifact freshness for emailing, shift your focus from "emailing the test results" to "notifying that a specific sequence has completed successfully."

1. Test Job (Job A): Focus solely on running tests and ensuring artifacts are correctly stored by Jenkins.
2. Artifact Relayer Job (Job B): This job should not try to read the reports directly for notification purposes. Instead, it should use a mechanism to signal completion.

You can achieve this by using Post-Build Actions or Parameterized Triggers combined with Timers. For example, instead of emailing immediately after the test job finishes, configure the email step in Job B to trigger only after a specific time delay has passed since the build was marked successful.

Strategy 2: Using Scripting for Asynchronous Notification

If you must use an email notification based on a delayed result, leverage Jenkins’ scripting capabilities (Groovy) to manage the state asynchronously. This pattern is highly effective in complex CI/CD setups, similar to how robust application architectures, such as those seen in Laravel projects, rely on decoupled services.

Here is a conceptual example using a simple Groovy script within your notification job:

pipeline {
    agent any
    stages {
        stage('Wait and Notify') {
            steps {
                // Wait for a specified duration (e.g., 1 hour) before proceeding
                script {
                    echo "Waiting for artifacts to stabilize..."
                    sleep(3600000L) // Sleep for 1 hour (in milliseconds)

                    // Now, attempt the notification based on known success status
                    if (env.BUILD_STATUS == 'SUCCESS') {
                        // Use a custom script or plugin method here to send the email, 
                        // referencing the build number rather than relying solely on artifact timestamps.
                        sendDelayedNotification(env.BUILD_NUMBER) 
                    } else {
                        echo "Build failed, skipping notification."
                    }
                }
            }
        }
    }
}

// Note: sendDelayedNotification would be a custom function or script 
// that handles the actual email sending logic, bypassing the immediate artifact check.

Conclusion

The error you faced is less about broken tests and more about managing the asynchronous nature of CI/CD workflows. By moving away from synchronous artifact dependency for notifications and instead implementing a time-based waiting mechanism coupled with explicit build status checks, you effectively solve the "stale report" problem. This approach gives you the necessary control to post delayed, relevant information without fighting Jenkins' internal artifact synchronization logic. For robust CI/CD pipelines, always prioritize decoupling stages and using explicit state management over implicit timing when dealing with external notifications.