An energy-efficient scheduling approach for wind-solar-hydrogen systems based on distributed reinforcement learning

Efficient load pattern identification: A PCA-enhanced and K-means clustering model

Clustering power load data is essential for identifying representative load patterns, which are fundamental to load forecasting, demand response programs, and efficient grid management. Traditional clustering techniques, such as Gaussian Mixture Models (GMM), have been extensively utilized due to their capacity to characterize complex data distributions via probabilistic assignments. However, GMM exhibits significant limitations: it is computationally burdensome for high-dimensional datasets and relies heavily on the assumption of underlying Gaussian distributions, which frequently fails to align with empirical load profiles. Furthermore, GMM requires meticulous initialization and shows high sensitivity to hyper-parameter configurations, including the number of components and covariance structures. To circumvent these challenges, we propose a framework integrating PCA with K-means clustering. PCA effectively reduces dimensionality by extracting dominant features, thereby enhancing computational throughput and mitigating the impact of noise. When applied to the reduced feature space, K-means clustering offers a rapid and interpretable methodology for partitioning load profiles into distinct, meaningful cohorts. Compared to GMM, the proposed PCA-K-means approach demonstrates enhanced scalability, reduced computational overhead, and greater ease of implementation, making it highly suitable for large-scale utility data analytics. This synergistic combination not only preserves the intrinsic structural properties of the data but also facilitates the robust identification of representative power consumption patterns.

Let $$ X \in \mathbb{R}^{n \times 24} $$ represent the matrix of normalized daily load profiles, where each row denotes a specific day and each column corresponds to an hourly interval. PCA is subsequently employed to project this data into a lower-dimensional feature space while retaining the maximum variance. The optimal number of principal components, m, is determined such that the cumulative explained variance exceeds a predefined threshold, τ. In this study, the threshold τ is established at 0.95 to ensure high information retention. Following dimensionality reduction, K-means clustering is applied to the transformed dataset $$ Z \in \mathbb{R}^{n \times m} $$. The objective is to partition the dataset into k distinct clusters, C_i, by minimizing the total intra-cluster variance.

where z_j denotes the j-th load profile in the reduced feature space, and μ_i represents the centroid of cluster C_i. For each cluster, a representative day is identified as the original profile whose lower-dimensional representation is closest to the respective cluster centroid.

where x_j signifies the original 24-hour load curve corresponding to the reduced data point z_j. The resulting set $$ \{x_1^*,\ldots,x_k^*\} $$ effectively captures the characteristic variability of annual demand with reduced computational complexity. This extracted set provides a robust foundation for subsequent multi-objective dispatch optimization.

Figure 2A illustrates the cumulative explained variance as a function of the dimensionality of the retained principal subspace. The analysis reveals that a specific quantity of principal components is essential to sustain a cumulative variance threshold of at least 95%. This provides a rigorous theoretical justification for dimensionality reduction, ensuring that the intrinsic characteristics of the original data are preserved while simplifying the data structure. As demonstrated by the results, the cumulative explained variance exceeds 95% when the principal component count is established at four. This configuration ensures the retention of the majority of informational content without incurring significant computational overhead from excessive dimensionality. Figure 2B presents the load profiles of seven representative days extracted via the K-means clustering algorithm. Each individual curve characterizes the typical load pattern of a specific cluster, reflecting temporal variations over a 24-hour horizon. A comparative analysis across these clusters reveals distinct differences, particularly in the timing of peak loads and the magnitude of fluctuations. These representative profiles offer crucial insights for load forecasting and power system scheduling, facilitating the identification of diverse consumption patterns and their operational implications for the grid.

Figure 2. Cluster results. (A) Cumulative explained variance; (B) Representative days load.

Figure 3A illustrates the distribution of the dataset within the reduced PCA space, projected onto the first two principal components. In this visualization, distinct colors signify the clustering results obtained via the K-means algorithm. The clear segregation of color-coded points demonstrates the effective identification of a robust clustering structure within the reduced-dimensional feature space. Furthermore, Figure 3B exemplifies the percentage of explained variance for the first four principal components in a bar chart format. Quantitative values for explained variance are explicitly annotated above each bar, facilitating a direct comparison of the relative contribution of each individual component. By evaluating the explained variance, the significance of each principal component in preserving the original data variability can be rigorously assessed. This analysis consequently determines the optimal dimensionality to retain during the feature extraction process.

Figure 3. Cluster analysis. (A) PCA + K-means clustering (first two principal components); (B) Explanatory variance for the first four dimensions; (C) Load profiles for all days in each cluster; (D) Number of days in each cluster. PCA: Principal Component Analysis.

Figure 3C depicts the load profiles of all days within each cluster, with the representative day’s load profile highlighted in red. Transparent curves in different colors represent the load variations of individual days within the same cluster, while the red curve signifies the typical load pattern of the cluster. By comparing the load profiles within a cluster, the consistency of load patterns can be observed, and the red curve emphasizes the core characteristics of the cluster, and Figure 3D shows the size of the data in each cluster, and makes a special cluster for the anomalous data.

Renewable generation pattern identification: a high-fidelity DTW-DBSCAN model

Traditional methods for extracting representative days from wind and solar power generation time series often utilize Euclidean distance-based clustering techniques, such as K-means or arithmetic averaging. Although computationally efficient, these methods possess notable limitations, specifically their reliance on linear temporal alignment and their inability to capture phase shifts, peak duration variations, or other nonlinear temporal distortions inherent in renewable energy profiles. In contrast, the DTW-DBSCAN framework excels at capturing local temporal distortions and can autonomously filter noise through density thresholds, thereby eliminating the need for subjective pre-specification of cluster counts. This study introduces a hybrid methodology that integrates DTW and DBSCAN. The proposed approach segments annual generation data into 24-hour daily subsequences on a seasonal basis, constructs a DTW-based shape similarity distance matrix to quantify morphological alignment, and performs unsupervised clustering via DBSCAN to categorize days with similar fluctuation patterns. Unlike conventional centroid-based methods that rely on Euclidean assumptions, representative days in this study are selected as real-world samples located closest to the DTW-geometric centroids within each cluster. Given a set of daily generation curves {x₁,x₂,…,x_n}, where each $$ x_i \in \mathbb{R}^T $$ denotes the generation series for a day with T time intervals, the DTW distance between two sequences x_i and x_j is defined as follows:

In this context, W denotes the set of all valid warping paths, while (p, q) represents the aligned index pair between sequences. A pairwise DTW distance matrix $$ D \in \mathbb{R}^{n \times n} $$ is then constructed and utilized as input for the DBSCAN clustering algorithm. DBSCAN effectively categorizes days with similar temporal patterns without the necessity of specifying a predefined number of clusters. The algorithm identifies clusters based on a neighborhood radius ε and a minimum number of points minPts required to establish a dense region. For each resulting cluster C_k, the most representative day is identified as

The optimal selection process yields representative curves $$ \{x_1^*,\ldots,x_k^*\} $$ by minimizing intra-cluster DTW distances, thereby effectively preserving the characteristic temporal patterns and inherent variability of wind and solar generation profiles. This methodology provides high-fidelity inputs for the planning and operational simulation of renewable-dominated energy systems, ensuring robustness against the temporal uncertainty and nonlinear dynamics intrinsic to wind and solar resources. The seasonal profiles for solar and wind power, along with their corresponding representative days, are illustrated in Figures 4 and 5, respectively. These profiles accurately capture the characteristic seasonal energy levels.

Figure 4. Solar power generation profiles during the year. (A) Spring; (B) Summer; (C) Autumn; (D) Winter.

Figure 5. Wind power generation profiles during the year. (A) Spring; (B) Summer; (C) Autumn; (D) Winter.

The representative power generation profiles for solar and wind resources under seasonal variations are depicted in Figure 6A and B, respectively. Observations indicate that solar generation typically peaks during daylight hours in spring and summer, whereas wind power yields higher output during the day in spring and autumn and transitions to nighttime peaks during winter. These results demonstrate that renewable energy generation is not stochastic but adheres to predictable diurnal and seasonal cycles. The presence of these inherent temporal patterns provides a robust foundation for employing clustering methods to effectively identify and characterize typical operational scenarios.

Figure 6. Diurnal power generation profiles across all seasons. (A) Solar power generation profiles; (B) Wind power generation profiles.

Hydrogen energy storage system model

The HESS configuration investigated in this research comprises an electrolyzer, a hydrogen storage vessel, and a proton exchange membrane fuel cell (PEMFC). This system serves exclusively as an energy conversion and storage medium, where the electrolyzer converts surplus renewable generation into stored hydrogen. Conversely, the fuel cell regenerates electrical power by consuming the stored hydrogen during periods of supply deficit. In industrial practice, the energy conversion efficiency of hydrogen storage is influenced by various nonlinear parameters, including operating power, thermal conditions, and current density. While high-fidelity empirical models can precisely characterize these behaviors, they introduce substantial computational complexity and non-convexity to the optimization problem. To maintain mathematical tractability and computational efficiency, this study utilizes a linearized constant efficiency approximation. The operational efficiencies for the charging and discharging cycles are assumed to be fixed coefficients, represented by η_H,c and η_H,d, respectively. For illustrative purposes, both efficiencies are assigned a value of 0.5^[43], resulting in an aggregate round-trip efficiency of 25%. To model the hydrogen storage model to fit the system, we specify the evolution of the State of Charge (SoC) model of the HESS as

where E_H,t is the hydrogen storage state of charge at time t, P_H,c,t and P_H,d,t are the charging and discharging power, representing the power input to the electrolyzer and the power output from the fuel cell, respectively, η_H,c and η_H,d are the constant charging and discharging efficiencies, Δt is the duration of the time interval. The discharging term is expressed as P_H,d,t/η_H,d because the electrical energy output from the fuel cell is only a fraction of the actual energy extracted from the hydrogen storage. Specifically, to deliver P_H,d,t units of electricity to the system, the fuel cell must consume P_H,d,t/η_H,d units of hydrogen energy.

This simplified model provides a practical balance between modeling fidelity and computational efficiency, and is particularly suitable for long-term scheduling, where large-scale dispatch problems must be solved repeatedly with limited computational resources.

Efficient optimal scheduling method for wind-solar-hydrogen systems

Optimizing the dispatch strategy for hybrid systems to achieve socio-economic benefits calls for a comprehensive consideration of grid requirements, maximization of renewable energy penetration, and minimization of carbon emission costs. With the advancement of distributed reinforcement learning (DRL), distributed frameworks have demonstrated significant efficacy in large-scale optimization tasks. As illustrated in Figure 7, the fundamental architecture of these algorithms typically comprises three core components: the Actor, the Learner, and the Coordinator. The Actor interacts with the environment to generate experience data, the Learner processes this data to update policy network parameters, and the Coordinator facilitates seamless communication between these components to ensure efficient data transfer. Parallel execution across multiple Actors significantly enhances data throughput, thereby providing a robust data foundation for training complex decision-making policies. In light of the temporal uncertainties inherent in wind and solar generation, this study proposes a multi-faceted power management policy tailored for wind-solar-hydrogen systems. The approach leverages deep reinforcement learning to govern an autonomous agent that maximizes cumulative rewards through continuous interaction with its environment. This decision-making process is guided by optimized policies that determine the most effective actions based on the current system states.

Figure 7. Basic framework of distributed reinforcement learning.

In DRL, two common paradigms exist: synchronous and asynchronous training. In synchronous algorithms, all actor processes must wait for global policy updates before continuing their interactions with the environment. This strict synchronization often leads to idle time, as actors or learners must pause until others complete their tasks, resulting in reduced system efficiency. In contrast, asynchronous algorithms allow actors to interact with their environments and update local models independently, without waiting for global synchronization. This approach significantly improves data throughput and computational efficiency, making it especially suitable for large-scale and complex tasks such as energy dispatch optimization in wind-solar-hydrogen systems. The distributed extension of the DDPG algorithm adopts this parallelism by enabling multiple actor processes to interact with independent environments simultaneously, each maintaining a local policy $$ \mu_i(s|\theta_i^\mu) $$ and value function $$ Q_i(s,a|\theta_i^Q) $$. To ensure consistency, these local parameters are periodically aggregated to update global networks. The aggregation rule averages the parameters across all actors:

where N denotes the number of distributed actors. Each actor samples experiences (s_i,a_i,r_i,s'_i) and contributes to a shared replay buffer $$ D = \bigcup_{i=1}^N D_i $$, which the learner uses to train global networks. The distributed framework significantly improves data throughput and training stability. In asynchronous implementations, actors continue to collect data without waiting for global parameter updates, further enhancing scalability for large-scale energy dispatch optimization. In the distributed implementation, a centralized-learner and multi-actor architecture is facilitated by the MATLAB Parallel Server. Specifically, three parallel actor nodes are deployed, each assigned to a distinct segment of the annual load-pattern dataset and maintaining a local copy of the target policy network to interact with its respective environmental instance. To balance computational efficiency with training stability, the synchronization and aggregation interval is established at N_sync=10. Consequently, the system enforces a global synchronization every 10 parameter iterations by broadcasting the latest Actor network weights to all parallel nodes. This mechanism ensures that each actor receives timely feedback from the global policy, effectively mitigating the gradient lag phenomenon inherent in DRL. Distributed storage and asynchronous update mechanisms further optimize training efficiency. The update rule for the online Q-network is formulated as

Here, θ^Q denotes the parameters of the online Q-network, while α_Q represents the learning rate. The temporal difference error is symbolized by δ. The term $$ \nabla_{\theta^{Q}} Q(s,a|\theta^Q) $$ refers to the gradient of the Q-value function with respect to the network parameters, characterizing how variations in θ^Q impact the Q-value. This gradient directs the parameter updates to minimize the temporal difference error δ. The online policy network is updated as

In this context, θ^μ represents the parameters of the online policy network, whereas α_μ denotes the associated learning rate. The expression $$ \nabla_{\theta^\mu} \mu(s|\theta^\mu) \nabla_a Q(s,a|\theta^Q) \bigg|_{a=\mu(s|\theta^\mu)} $$ signifies the gradient of the expected Q-value with respect to the policy network parameters, computed via the deterministic policy gradient theorem. This gradient provides the direction for updating θ^μ to maximize the expected return. The target networks are updated as

where τ denotes the soft update rate, typically a small constant significantly less than 1, which governs the update frequency of the target networks. The symbols θ^Q' and θ^μ' designate the parameters of the target Q-network and the target policy network, respectively. By adjusting the target networks incrementally, this soft update mechanism enhances training stability and facilitates convergence.

The agent-environment interaction in the DDPG algorithm’s reinforcement learning framework is formally modeled as a Markov Decision Process (MDP), represented by the five-tuple (s, a, r, p, γ). The state s encapsulates comprehensive environmental information that serves as input to the neural network, while the action a constitutes the network’s output that directly influences and modifies the environment. The reward signal r provides the optimization direction for the neural network, driving parameter updates toward higher expected returns. The state transition probability p, representing the environment’s intrinsic physical dynamics, determines the likelihood of subsequent state transitions. The discount factor γ governs the temporal value perspective, ensuring that the agent balances immediate operational costs with the long-term flexibility of energy storage over the scheduling cycle. Each of these fundamental MDP components will be rigorously defined and mathematically characterized in the subsequent subsections.

State and action space definitions

The DDPG algorithm is implemented to derive an optimal control policy within real-valued state and action domains. The reinforcement learning architecture primarily comprises two interacting entities: the environment and the agent. At each discrete time step t, the agent perceives the current system state s(t) and subsequently determines an action a(t)=[P_t(t)] within the bounded action space [-ΔP_t,ΔP_t] (MWh). To fulfill the prescribed control objectives, the state vector s(t) must encapsulate comprehensive environmental information necessary for informing effective decision-making. The constituent variables of the state space are defined as follows:

Within this framework, P_t(t) signifies the active power output of the thermal unit at time t, which fluctuates in response to operational requirements and demand variations. The variables P_w(t) and P_s(t) characterize the power generation from wind and solar energy sources at time t, respectively. Furthermore, P_e(t) denotes the quantity of external power acquired at time t to compensate for the gaps in the thermal unit’s generation. The term P_bsc(t) identifies the power contribution from the HESS, while P_dem(t) represents the instantaneous load demand.

To explicitly clarify the system dynamics, the variables within this framework are categorized into exogenous and controllable variables. The exogenous (uncontrollable) variables include the wind power output P_w(t), solar power output P_s(t), and regional electrical load demand P_dem(t), which are fundamentally driven by meteorological and stochastic operational processes. Conversely, the controllable variables encompass the active power output of the thermal unit P_t(t), the electrical power exchanged with the external grid P_e(t), and the power contribution from the hydrogen energy storage system P_bsc(t). Specifically, the agent directly dictates the dispatch of the thermal unit, while the grid exchange and HESS operations dynamically ensure system-wide load balance. To ensure a high-fidelity simulation of thermal power dynamics, the rate of change in P_t(t) is strictly governed by the thermal unit’s maximum allowable ramp rate. In instances where the thermal power output exceeds these operational boundaries, the current training episode is terminated.

Reward function design

The optimization of the energy dispatch strategy for thermal generation is primarily aimed at securing economic advantages for both societal and industrial stakeholders. Consequently, it is imperative to satisfy system load requirements, maximize the penetration of renewable energy sources, and minimize total operational expenditures. This research utilizes thermal power units and energy storage systems as synergistic components to facilitate the seamless integration of wind and solar power. Furthermore, a carbon trading mechanism is integrated into the dispatch framework, simultaneously addressing coal consumption costs and CO₂ emission levels. Such an integrative approach seeks to establish a harmonized equilibrium between economic performance and environmental sustainability through carbon reduction.

(1) Coal fuel cost: To minimize the expenditures associated with coal fuel consumption, the cost function r₁ is formulated as follows:

where each time step t corresponds to a specific interval for decision-making. Y_c is the total coal combustion cost, set at 115 $/MWh, and serves as the basis for evaluating the economic impact of fuel consumption. The coal consumption function G(P_t(t)) estimates the coal usage of the thermal unit based on its power output P_t(t), the parameters a₁, b₁, and c₁ are used in computing the thermal unit’s output and corresponding coal consumption. Since non-fossil energy sources do not directly emit CO₂, the model includes carbon costs only for thermal power generation.

(2) Carbon emission expenditure: Since non-fossil energy sources produce zero direct emissions, the model allocates carbon-related expenditures solely to the thermal power component. The total volume of CO₂ emissions is quantified as follows:

Here, M(P_t(t)) signifies the CO₂ emissions associated with the thermal unit output P_t(t) at time t. The emission volume is determined via parameters a₂, b₂, and c₂, which mathematically characterize the emission characteristics of the specific unit. Accounting for carbon market mechanisms, this study assumes that a predefined quota of CO₂ emission allowances is allocated gratis. These free-of-charge allowances are determined by the generation capacity and the specific category of the thermal unit, formulated as follows:

where M_fr(t) represents the free CO₂ emission allowance allocated to the thermal unit at time t. The coefficient β defines the amount of allowance granted per unit of electricity generated. Under this allocation scheme, any emissions exceeding the free allowance must be offset through the purchase of additional allowances, resulting in a CO₂ emission cost. To penalize excessive emissions and encourage low-carbon generation, the CO₂ emission cost function r₂(t) is defined as

where M_sp(t) denotes the payable CO₂ emissions at time t, representing the amount by which actual emissions exceed the allocated free allowance. The carbon price is denoted by φ, which is set to 11.5 $/ton^[44]. This price is used to calculate the monetary cost of excess emissions within the carbon trading framework.

(3) Electricity purchase cost: To ensure that the power supply meets the load demand, an electricity purchase cost is introduced based on the power deficit. The total power is given by

If an energy storage system is considered, P(t) also includes the power supplied by storage, P_bsc(t). The electricity purchase cost r₃(t) is defined as

Here, Y_e denotes the electricity purchase price, established at 327 $/MWh, which serves as a benchmark for comparing external grid procurement costs with internal generation expenses. The term Δt represents the dispatch interval, fixed at 1 h, which determines the temporal resolution of the simulation model. Throughout the interaction process, the agent acquires rewards by progressively modifying the environmental state. The instantaneous reward r(t) is formulated as follows:

The control policy is continuously updated by the agent based on the value of the reward function. Given that the energy management of the wind-solar-hydrogen system involves time-coupled variables, such as the SoC of the HESS, the scheduling task is formulated as a finite-horizon MDP. Accordingly, the discount factor γ is set to 1. This configuration ensures that the agent yields an unbiased cumulative reward over the entire daily scheduling cycle, effectively balancing immediate operational costs with the long-term flexibility of energy storage.

Binding conditions

To guarantee the stable and reliable operation of the integrated wind-solar-hydrogen system, several physical hard constraints must be strictly adhered to. The mathematical formulations of these constraints are presented as follows:

(1) Load balance constraint: To ensure load balance at each time t, the total power supply must be no less than the load demand. The supply includes wind power P_w(t), solar power P_s(t), purchased electricity P_e(t), storage discharge power P_bdch(t), and subtracts storage charging power P_bch(t). The load balance constraint is given by

(2) Climbing capacity constraint: To accurately replicate the operational behavior of an actual thermal power unit, a ramp rate limitation is imposed on the power output fluctuations. The governing climbing capacity constraint is formulated as follows:

Here, ΔP_t signifies the ramp rate limitation of the thermal power generating unit. This parameter establishes the maximum permissible increment or decrement in power output within a discrete time interval.

(3) Generation output constraints: The active power outputs produced by the thermal, solar, and wind units are strictly governed by their individual rated capacities. These operational boundaries are mathematically defined as follows:

In this formulation, P_t,max signifies the maximum generation capacity assigned to the thermal power unit. Analogously, P_w,max and P_s,max designate the peak output limitations for the wind and solar generating units, respectively.

(4) Storage capacity constraints: The operational capacity of the energy storage system is governed by specific physical thresholds to ensure safety and longevity. These capacity limitations are mathematically formulated as follows:

In this formulation, P_bsc(t) denotes the power supplied by the HESS at time t. As defined above, P_bch(t) and P_bdch(t) represent the charging and discharging power of the HESS at time t, respectively.

(5) Charging and discharging constraints: The operational logic presiding over the charging and discharging cycles of the energy storage system is articulated as follows:

where P_bdch,max and P_bch,max denote the maximum discharging and charging power of the energy storage system, respectively, P_bsc,max represents the maximum energy storage capacity of the system.

The neural network takes power system data as input and outputs adjustment values for thermal power generation. Through this continuous input-output interaction, the operation of the thermal unit is progressively optimized, improving overall generation efficiency within each sampling interval.

Performance validation

A detailed analysis of the economic performance across the ten scenarios is presented in Table 2. The results systematically assess the impact of different optimization algorithms, the inclusion of a HESS, and varying cost sensitivities on the system’s operational strategy and total cost.

Firstly, a comparison between scenarios with and without the HESS reveals its significant economic benefits. For instance, comparing Scenario 1 (SLSQP without HESS) to Scenario 2 (SLSQP with HESS), the total cost is marginally reduced. Similarly, the introduction of the HESS in the DDPG algorithm (Scenario 4 vs. Scenario 3) and the DRL algorithm (Scenario 6 vs. Scenario 5) leads to total cost reductions of 0.023 × 10⁷ and 0.107 × 10⁷ $ respectively. This consistently demonstrates that integrating the HESS improves energy utilization and reduces overall operational costs, regardless of the optimization algorithm used.

Secondly, among the different optimization strategies, the DRL algorithm combined with HESS (Scenario 6) achieves the lowest total cost of 5.360 × 10⁷ $. Comparing Scenario 6 with Scenario 5 explicitly demonstrates the mechanism behind this improvement: although the integration of HESS leads to a slightly higher coal fuel cost (2.823 × 10⁷ vs. 2.629 × 10⁷ $), this controlled increase in steady thermal generation enables the system to drastically scale back on costly external power purchases (2.910 × 10⁶ vs. 5.334 × 10⁶ $). Furthermore, the energy time-shifting capabilities reduce reliance on high-emission peaking periods, lowering overall carbon emission costs (2.246 × 10⁷ vs. 2.305 × 10⁷ $). This favorable trade-off confirms it is the most effective strategy for balancing coal fuel consumption, CO₂ emissions, and electricity purchasing to achieve economic optimality. Scenario 7, which introduces noise to the renewable generation data, results in a slightly higher total cost (5.379 × 10⁷ $) than Scenario 6, reflecting the economic trade-offs required to manage system operations under uncertainty.

Finally, the sensitivity analysis (Scenarios 8-10), based on the optimal framework of Scenario 7, provides critical insights into the system’s response to different cost priorities. In Scenario 8, doubling the weight of the coal fuel cost forces the system to reduce fuel consumption (from 2.626 × 10⁷ to 2.607 × 10⁷ $) and instead rely more on purchasing electricity (cost increased from 4.654 × 10⁶ to 6.376 × 10⁶ $). In Scenario 9, a higher penalty on CO₂ emissions prompts an even more significant shift away from thermal generation, achieving the lowest fuel (2.418 × 10⁷ $) and emission (2.151 × 10⁷ $) costs at the expense of having the highest electricity purchase cost (8.909 × 10⁶ $). Conversely, in Scenario 10, when the electricity purchase cost is penalized, the system minimizes its reliance on the grid, leading to the lowest purchase cost (4.231 × 10⁵ $) but increasing its own thermal generation, which in turn raises both fuel and emission costs. This demonstrates the model’s ability to intelligently adapt its energy dispatch strategy in response to varying economic and environmental signals.

Robustness verification

Scenario 7 was specifically designed to validate the robustness of the optimal dispatch strategy (developed in Scenario 6) against the inherent uncertainties of renewable energy generation. To achieve this, we introduced stochastic fluctuations, modeled via an AR(1) process, to the wind and solar power data to simulate real-world forecasting errors.

The results demonstrate strong performance stability. The total operational cost for Scenario 7 was 5.379 × 10⁷ $ [Table 2], representing a marginal increase of only 0.019 × 10⁷ $ - or approximately 0.35% - compared to the 5.360 × 10⁷ $ cost of its deterministic counterpart, Scenario 6. This minimal cost deviation, despite the introduction of significant input noise, signifies that the DRL agent has acquired a resilient and adaptive dispatch policy. This resilience is critical for practical implementation, as it confirms the model’s capability to maintain near-optimal economic performance even when faced with imperfect environmental forecasts, thereby validating its robustness for real-world deployment.

Comparison of training speeds

To assess the computational efficiency of the proposed algorithm, we evaluate the performance of multi-threaded parallel training. All experiments are conducted on a single workstation equipped with an 11th Gen Intel Core i7-11800H CPU (8 Cores, 16 Threads) and 32 GB of RAM. The performance metrics, summarized in Table 3, demonstrate a significant reduction in the total wall-clock training time as the number of parallel worker threads increases.

The baseline single-threaded training required 72.5 h to complete. By scaling the number of workers to 2, 4, and 6, the training time was dramatically reduced to 37.2, 19.1, and 11.5 h, respectively. This corresponds to empirical speedup ratios of 1.95x, 3.80x, and 5.50x. The speedup is observed to be nearly linear up to 4 threads, indicating efficient parallelization with minimal communication overhead. Beyond this point, a slight drop-off in linear scaling efficiency is noted. The results validate that the multi-threaded training architecture effectively leverages multi-core CPUs to accelerate the learning process. The observed scalability confirms that employing a moderate number of parallel workers presents an optimal trade-off between computational resource allocation and training time reduction for this task.