Graph-based methods

At the beginning stage of research in DSP, graph-based methods were used by most of the researchers. The nodes denote the parts in the graph representation, and the edges represent the relationship between those parts. The most used graph-based approach is the AND/OR graph (De Floriani and Nagy, Reference De Floriani and Nagy1989), and an extended version of the AND/OR graph was introduced by Ma et al. (Reference Ma, Jun, Kim and Lee2011). Other graph methods include the extended process graph (Kim and Lee, Reference Kim and Lee2017; Tian et al., 2019a), the disassembly precedence graph (Han et al., Reference Han, Yu and Lee2013), and the graph cut-set method (Gunji et al., Reference Gunji, Pabba, Rajaram, Sorakayala, Dubey, Deepak, Biswal and Bahubalendruni2021). The graph-based approach is preferred because it generates feasible sequences from the product data. To process the uncertainties in dynamic DSP, Vongbunyong et al. (Reference Vongbunyong, Kara and Pagnucco2012) proposed a cognitive robot-based DSP approach. Later work (Vongbunyong et al., Reference Vongbunyong, Kara and Pagnucco2013) introduced a cognition-based robot’s basic and advanced behavior control strategies for disassembly planning. The graph-based method gives a proper representation of various disassembly possibilities, but the definition for graphs needs to be given manually; also, it has the problem of combinatorial explosion when generated in an automatic manner.

Matrix-based methods

Various matrix-based representations like contact (liaison), translational (Smith et al., Reference Smith, Smith and Chen2012), precedence (Azab et al., Reference Azab, Ziout and ElMaraghy2011; Ren et al., Reference Ren, Tian, Zhao, Yu and Zhang2017), directional, and interference (Kheder et al., Reference Kheder, Trigui and Aifaoui2017) matrices are proposed by researchers. These matrices depict the pairwise relationship between the parts of a product regarding their stability, priority, and space interference. Matrix-based data are the input in the optimization and sequence generation processes. Apart from graph and matrix-based approaches, Petri net-based (Petri and Reisig, Reference Petri and Reisig2008; Kuo, Reference Kuo2013) representation is also used. The matrix-based representation is the most suitable type for different computational processes. Another advantage is that the matrix data are obtained from the respective product’s CAD models automatically.

Mathematical and meta-heuristic methods

Various computational techniques and optimization algorithms are used to generate feasible or optimal disassembly sequences. Mathematics-based computational methods like branch-and-bound (Kim and Lee, Reference Kim and Lee2017), linear (Zhu et al., Reference Zhu, Sarigecili and Roy2013), and nonlinear (Ullerich, Reference Ullerich2014) methods are used to solve DSP. The mathematical models have the capability to find the optimal solutions, but the quality of the solution is entirely based on the product’s objective function and representation format. Meta-heuristic methods like genetic algorithm (GA) (Giudice and Fargione, Reference Giudice and Fargione2007; Hui et al., Reference Hui, Dong and Guanghong2008; Tseng and Lee, Reference Tseng and Lee2018), ant colony optimization (ACO) (Wang et al., Reference Wang, Liu, Li and Zhong2003; Kheder et al., Reference Kheder, Trigui and Aifaoui2017), particle swarm optimization (PSO) (Kheder et al., Reference Kheder, Trigui and Aifaoui2017), and artificial bee colony optimization (Ren et al., Reference Ren, Zhang, Zhao, Xiao and Tian2018; Tian et al., 2019a,b; Liu et al., Reference Liu, Zhou, Pham, Xu, Yan, Liu, Ji and Liu2018, Reference Liu, Zhou, Pham, Xu, Ji and Liu2020) are used by various researchers to solve the DSP problem. The meta-heuristic approaches can be applied to complex products to get near-optimal or optimal solutions, but the quality of solutions varies based on the constraints of that particular approach.

Hence, these approaches are enhanced and combined with other meta-heuristic methods to produce hybrid approaches (Tian et al., Reference Tian, Zhou and Li2018). For instance, Tseng et al. (Reference Tseng, Chang, Lee and Huang2019) introduced a hybrid technique based on ACO. A combined version of GA and artificial fish swarm algorithm was submitted by Guo et al. (Reference Guo, Zhong, Li, Du and Guo2019). The recent work of Xing et al. (Reference Xing, Wu and Qu2021) is based on an improved version of the max–min and ant colony system (IMMAS). Hybrid methods can generate better solutions when compared to single-heuristic or meta-heuristic methods. One main problem with these methods is that they are not in a straightforward manner. They use different techniques in different stages of the algorithm, which is not a generalized approach.

Other approaches in DSP

In addition to computational and optimization approaches, various advanced techniques and technologies have been explored for DSP. They are simulation-dependent techniques like CAD (Issaoui et al., Reference Issaoui, Aifaoui and Benamara2017), decision-making-based de-manufacturing (Anil Kumar et al., Reference Anil Kumar, Bahubalendruni, Prasad and Sankaranarayanasamy2021), virtual reality (Mitrouchev et al., Reference Mitrouchev, Wang and Chen2016, Reference Mitrouchev, Wang, Chen, Eynard, Nigrelli, Oliveri, Fajarnes and Rizzuti2017), and augmented reality (AR) (Osti et al., Reference Osti, Ceruti, Liverani and Caligiana2017; Chang et al., Reference Chang, Nee and Ong2020). The robot–human collaborative approach for DSP was implemented following the artificial bee colony algorithm (Liu et al., Reference Liu, Zhou, Pham, Xu, Yan, Liu, Ji and Liu2018, Reference Liu, Zhou, Pham, Xu, Ji and Liu2020; Xu et al., Reference Xu, Tang, Liu, Liu, Zhou and Pham2020). This work tries to minimize the labor process and utilizes human knowledge for both single-objective and multi-objective problems. All these advanced techniques can be used to generate disassembly sequences for specific products with special orientations so that they cannot be used as a general technique for other products, and benchmarking with standard algorithms is a difficult process.

Synthesis of literature study

Though many works are done in DSP, most are based on traditional optimization algorithms and nature-inspired heuristic algorithms. The human–robot collaborative approaches have been employed for various disassembly problems. But specifically, for DSP only a few works have been published. More research work can be done in those areas to solve the DSP problem. Machine learning approaches are used only to a limited extent in DSP due to its problems of slow convergence and local minima. To address these challenges, the utilization of adaptive parameter techniques has emerged as a viable solution. Numerous adaptive parameter techniques have been introduced across various learning problems, including learning automata-based approaches (Beigy and Meybodi, Reference Beigy and Meybodi2000, Reference Beigy and Meybodi2001; Meybodi and Beigy, Reference Meybodi and Beigy2000, Reference Meybodi and Beigy2002). Simulations of these techniques demonstrate their feasibility and effectiveness in various learning problems. In particular, the adaptation of parameters in the back-propagation (BP) algorithm has been applied to train multilayer neural networks. Inspired by this concept of parameter adaptation to overcome the problems of local minima and slow convergence rate, an enhanced simulated annealing (ESA) algorithm has been implemented in this work. By enhancing the convergence rates, a wide range of machine learning models can be explored to address the challenges in DSP (Syed Shahul Hameed and Rajagopalan, Reference Syed Shahul Hameed and Rajagopalan2022, Reference Syed Shahul Hameed and Rajagopalan2023).

Based on the study of related works, it is evident that there is a lack of exploration of machine learning methods, particularly reinforcement learning (RL), in solving DSP problems. Additionally, there is a significant need for an efficient framework capable of effectively handling various products.

Research contributions

To address these gaps, an optimized Q-learning (OQL)-based RL approach, specifically tailored for DSP (RL-DSP), is proposed in this research. The primary objective of this work is to overcome the challenges in DSP by utilizing the potential of RL technique. The contributions in this work are as follows:

1. 1. A novel RL framework for DSP: An original RL framework has been contributed that generates optimal sequences for various products with diverse levels of complexity. While limited DSP attributes have been incorporated for processing in most existing works, all necessary attributes have been considered and appropriately weighted according to their influence in generating optimal disassembly sequences in this study.

2. 2. Implementation of proposed ESA: An ESA is proposed in this work. Its innovative approach narrows the search space over time and optimally balances between exploration and exploitation, thus resulting in avoiding local optima and enhancing the likelihood of finding global optima.

3. 3. The development of the OQL approach: The proposed OQL approach innovatively integrates an epsilon-greedy (EG) approach and the ESA. This approach improves the action selection process by introducing a flexible, adaptive learning model that increases the ability to solve problems more robustly. This leads to the OQL method outperforming the classic QL (CQL) technique in a variety of complex scenarios.

The proposed work is benchmarked against standard and state-of-the-art techniques, selected based on their demonstrated performance in current literature. The results show that the proposed RL-DSP approach provides better solutions in terms of optimality, as well as reduced time and memory consumption.

The organization of the remainder of this paper is as follows: Section “Background study” presents a background study of RL and QL in relation to the research. Section “Proposed methodology” outlines the proposed methodology, including the novel ESA approach and the OQL-based RL framework for DSP. Section “Experiments and analysis” analyzes the experimental study and discusses the results, highlighting the effectiveness of the proposed approach. Finally, the paper concludes with a summary of key findings, contributions, and suggestions for future research.

Reinforcement learning (RL)

The RL technique is explored because of its efficiency in handling nondeterministic polynomial time (NP)-hard problems (Sutton and Barto, Reference Sutton and Barto2018). This technique aims to make the agent select an appropriate action in the current state that produces the best results. Based on the reward (feedback) received for every action, the agent (decision-maker) analyzes the current state and environment (scenario) to take a particular action (step) at that time. The state of a given problem changes based on the actions picked by the agent. If the agent takes appropriate action to solve the problem, it is rewarded. The agent aims to get maximum reward points by taking the desired actions to maximize the reward based on its feedback experience. In this method, the state (S) denotes the group of all possible states in a problem, action (A(S)) represents the group of possible actions that can be taken at a particular state (S), and reward (R) denotes the reward points given to the agent for taking a desirable action; the penalty is the low reward or negative points given to the agent for taking an undesirable or wrong action, the cost is the measure that denotes the quality of the solution or state, and time indicates the period taken for the learning process. The general framework of RL is illustrated in Figure 6. The main task of an agent is policy $ (\pi $ ) learning, $ \pi $ : S → A. The policy (knowledge) learned must be able to generate a maximum of total rewards (M). Both learning rate (α) and discount factor (γ) must be 0 < α and γ < 1. The total rewards M are defined as r0 + γ*r1 + γ2*r2 + …

Figure 6. General framework of reinforcement learning process.

Classic Q-learning (CQL) approach

CQL is the standard QL technique, which is based on temporal difference (TD) learning. The updating of Q-values is based on the Q-value [Eq. (8)]. The calculated values are updated in the Q-table. From Eq. (8), s and a are the state and action at the current time (t). Q_t (s, a) is the current Q-value considering the current state and the action taken. Q_t + 1 is the Q-value of the following state (s’) and action (a’). R (s, a) denotes the reward obtained for that pair of (s, a). max Q’ (s’, a’) defines the maximum number of rewards gained given the new state (s’) and all possible actions at the new state. Algorithm 1 gives the step-by-step approach of the CQL algorithm. Figure 7 depicts the flowchart of the CQL process.

(8) $$ {Q}_{t+1}\left(s,\alpha \right)={Q}_t\left(s,\alpha \right)+\alpha \left(R\left(s,\alpha \right)+\gamma {\max}_{\alpha^{\prime }}Q\Big({s}^{\prime },{\alpha}^{\prime}\left)-{Q}_t\right(s,\alpha \Big)\right) $$

Algorithm 1. Classic Q-Learning (CQL) Algorithm.

1. 1 Set the parameters: (learning rate α, discount factor γ)

2. 2 For each pair of state (s) & action (a), Q-matrix (s,a) = 0

3. 3 Observe the state s

4. 4 repeat

5. 5 Select the action a using Epsilon-greedy method

6. 6 Take the action a

7. 7 Receive immediate reward r(s, a)

8. 8 Observe the new state s’

9. 9 Update Q (s, a) with Eq. (8)

10. 10 s = s’

11. 11 until all the stopping criterion is satisfied

Figure 7. Classic Q-learning (CQL) flowchart.

Epsilon-greedy (EG) approach

The exploitation and exploration trade-offs are the pivotal aspect of RL. The agent must choose the best from already exploited actions to maximize the rewards. Still, it is also required to explore more actions to find the other potentially best solutions for a given problem. For action selection, there are various strategies followed. One of them is the epsilon-greedy method. The epsilon $ (\varepsilon $ ) values should be in the range of $ 0<\varepsilon <1 $ . Initially, when the epsilon rates are higher (~1), the agents explore the environment more; eventually, the epsilon rate decreases, and consequently, the agent starts to exploit more.

Based on the increasing exploration process, the agent gets more knowledge about the environment and the required policy is built. The policy $ \pi $ (s) is applied according to the given Eq. (9) (Ottoni et al., Reference Ottoni, Nepomuceno, de Oliveira and de Oliveira2021). However, this EG approach lacks efficiency in its epsilon-decreasing structure.

(9) $$ \pi \left(\mathrm{s}\right)=\left\{\begin{array}{l}a\ast, with\ probability\hskip0.4em 1-\varepsilon \\ {} ar, with\ probability\hskip0.4em \varepsilon\;\end{array}\right. $$

$ \pi (s) $ denotes the decision policy for the current state s, a* denotes the best-estimated action for the state s at the current time, and ar denotes the random action selected with probability $ \varepsilon $ .

Proposed enhanced simulated annealing (ESA) algorithm

The proposed ESA algorithm is the enhanced version of the standard SA method. In the standard SA method, the new solution is found using the solution function and compared with the old for calculating the difference value (Kirkpatrick et al., Reference Kirkpatrick, Gelatt and Vecchi1983). The difference value is compared with the temperature (temp) factor, and the next decision is taken. For annealing, the temperature reduction factor (β) is used. The standard SA algorithm is given in Algorithm 2.

In SA, the temperature has to be reduced in a phased manner from its initial rate. This helps the algorithm achieve convergence. In ESA, the additional parameters ε, λ, and t_r are used. The SA needs a more structured decreasing rate for the temperature. In order to solve this issue, λ and t_r are added. λ and t_r denote the temperature decay factor and the temperature regularizing criterion, respectively. Their values are given between 0 and 1. The temperature regularizing criterion is used to reduce the temperature rate in a slow-phased and structured manner, whereas the temperature decay factor is used to maintain the temperature value within the positive range of values.

Based on these parameters, the equation for ESA is formulated. The epsilon gets decreased for each run based on Eq. (10). Then, a decision is taken based on the comparison with random values as given in Algorithm 3.

(10) $$ \varepsilon =\varepsilon -\Big(1/\left(\mathit{\log}\left( temp+\lambda \right)+{t}_r\right) $$

Proposed optimized Q-learning (OQL) approach

The existing EG approach in CQL uses immense randomness to decrease the epsilon rate, resulting in unstructured decay of epsilon values. A structured decrease in the epsilon rate is required to obtain more accurate results from the algorithm. The proposed ESA approach acts as a good optimization technique to gradually reduce the epsilon value. Its cooling schedule, which reduces the exploration rate over time, aligns well with the RL learning process. This approach is used in conjunction with the standard EG approach to make the eventual decay of epsilon value more controlled and organized. To define the standard values for the various parameters in this proposed OQL technique, the temperature (temp) is defined as the total number of iterations. This is because the more iterations the program runs, the less exploration is required to find the best solution.

Thus, 1/log(temp) decreases the epsilon value based on the number of iterations consumed by the problem. Additional parameters are used to add more regularization to the decaying epsilon values. The value of decay rate $ (\lambda $ ) is set as (1-α) due to the interdependence of exploration and learning factors. The decreasing rate of epsilon is also based on the learning rate value of the algorithm. As the agent learns more about the environment, less effort is required for exploration. There is a need for a regularizing value that further adds structure to the decreasing rate. For that, the temperature regularizing criterion (t_r) is set to the range (0, 1) to maintain the same proportion as the other parameters used in RL. The algorithm for the proposed OQL technique is given in Algorithm 4.

For the proposed ESA [Eq. (10)], the following values are taken: epsilon (ε) = 1.0; temperature (temp) = no. of iterations; $ \lambda $ = 1 - learning rate (α); and temperature regularizing criterion (t_r) = 0.1995.

To determine the optimal value for t_r, a range of values from 0.01 to 0.2 was tested through a series of experiments. It was observed that initially the values of (D_f) showed significant variation, but they eventually converged when the value of t_r exceeded 0.1995. These experiments were conducted across all eight different products, and the results consistently indicated that the OQL method achieved better convergence and higher quality solutions when t_r was equal to or greater than 0.1995. Based on these findings, t_r is chosen as 0.1995. Since the OQL technique offers better convergence and higher quality solutions by controlling the decay of epsilon values in a structured manner, this proposed OQL technique helps in obtaining the required results in less time compared to CQL.

Proposed RL framework for DSP

RL is preferred to solve DSP because of its efficiency in solving optimization problems. The supervised and unsupervised learning techniques are not used because of the unavailability of huge datasets for the DSP problem. DSP requires the processing of disassembly attributes and generating the solution based on the objectives but without any training process, which is different from the usual problems solved using supervised/unsupervised learning techniques. For applying RL to DSP, the precedence, stability, liaison, and geometric feasibility data are considered and taken as conditions to generate the reward and penalty values. The initial state is the complete product with all parts connected, and the terminal state is the disassembled product with individual parts. The disassembly sequence is the required output. The primary objective of the RL agent is to generate a sequence that consumes less disassembly time and cost. Every part of a product’s disconnection is taken as an action to perform. So, the state and action of DSP are formulated based on the parts disassembled and the remaining parts to be disassembled.

The main objective of this RL framework is to prepare the agent to predict an efficient DSP sequence that is feasible and optimal with minimal disassembly directions. The RL structure for DSP is given as follows:

States: The states are based on the total number (n) of parts of the product to be disassembled. Initially, the complete product with all the products is the starting state. As each part is removed from the product, the state gets changed eventually, and at the final state, only the last part will be left.

Action: Action is the disassembling of one part from the remaining parts of the product. Action is taken based on the information of the set of parts that can be possibly disassembled given the current state. In each state, an action is taken (a part is removed), and it is carried out till there is no part left to disassemble.

Rewards: These functions are defined to associate the disassembly time/cost with the dismantling process. For each action (disassembly of a part), a reward is given. This reward generation is based on the disassembly fitness (D_f) function given in Eq. (11). The better the optimality, the higher the reward is obtained.

(11) $$ Reward\;(R)=1000\ast \left({D}_f\right) $$

RL-DSP methodology

The RL-DSP methodology proposed in this work is built on four steps.

1. 1. The stability, liaison, and geometric data are processed to generate the cost (D_c) and time values (D_t) of the disassembly. Then, the disassembly fitness (D_f) value is calculated based on the D_c and D_t values.

2. 2. The D_f values are given as the reward matrix to the RL program; then, the actions are taken based on the OQL method. This is followed by the feasibility checking process using precedence data.

3. 3. Once the feasibility conditions are satisfied, the Q-table is generated based on the states, actions, and the QL formula given in Eq. (8).

4. 4. Based on the Q-table values, the RL agent generates the disassembly sequences. The parameters t_r, α, γ, and λ are tuned to get better optimality.

The flowchart of the proposed RL-DSP framework is shown in Figure 8. The calculation process of fitness and reward values is given in Algorithm 5. The DSP attributes are considered, and the D _f value is calculated based on the D_c and D_t values. The D_c and D_t values are allotted to each attribute based on their importance in generating feasible and optimal sequences. Reward values are declared as directly proportional to D_f. So, the main objective of the agent will be to maximize the rewards, thereby increasing the fitness value.

Algorithm 5. Proposed RL-DSP Algorithm.

1. 1 Initialize RL-DSP product data; (stability, liaison, geometric feasibility); D_p = 1;

2. 2 Calculate D_c, T_p, D_t, D _f

3. 3 repeat

4. 4 If L = 0,

   D_c = 15;

   Else,

   D_c = 1;

5. 5 If S = 0,

   D_c = D_c + 10

   Else if S = 1,

   D_c = D_c + 5

   Else D_c = D_c

6. 6 If Oc_ij < 2, then T_p = T_p + 3

   Else, T_p = 0

7. 7 D_t = T_p + D_p

8. 8 D _f = 1/(D_c + D_t)

9. 9 Reward = 1000*D _f

10. 10 until all the stopping criterion is satisfied

Figure 8. Proposed reinforcement learning framework for DSP – flowchart.

The rewards’ data are taken as the OQL algorithm’s input. Randomly, a part is selected, and Q-value calculation is initiated. It is followed by the generation of subsequent part connection’s Q-values. The OQL is followed for taking proper action after analyzing the exploration and exploitation factors. Based on that analysis, the algorithm either searches for new solutions in the entire search space or searches for the best solution within the explored space. Then, the action is checked for feasibility conditions based on the precedence matrix (P_r). If it satisfies the criteria, the action is selected, else another action is taken. This process is repeated until all the part connections are processed. Then, the Q-values are calculated for all the part disassembles, and the final sequence is generated. Finally, a disassembly sequence with the highest reward is chosen as the solution by the agent.

Experimental setup

The RL-based DSP program is implemented as Python program and tested on different products using the PyCharm IDE running on a Windows 10 system. The results are analyzed and visualized using the Matplotlib library. The system used for testing has the following specifications: 8GB random access memory (RAM), 1 TB hard disk drive (HDD), and an Intel i5 6th Generation processor. For testing the efficiency of the proposed RL-DSP technique, eight different products with disassembly attributes are taken into consideration. The product information is given in Table 2.

Table 2. Product details

Out of eight products, five are taken from the literature work from which the data are extracted manually, and for the remaining three products, the attributes are obtained from their CAD models. All these products are of different complexities with multidirectional disassembly feasibilities.

Influence of parameters

The proposed RL-DSP framework has been tested on eight different products with multiple parts across various episodes (100, 200, 500, and 1000). Along with the number of episodes, parameters such as temperature regularizing criterion (t_r), alpha (α), gamma (γ), and epsilon (ε) also play a crucial role in generating solutions. Different parameter settings have been explored to generate optimal sequences for the diverse set of products. Considering the satisfactory results obtained within 100 episodes, the default number of episodes is set to 100. To determine the appropriate t_r value, fitness values of various disassembly sequences are compared with their respective t_r values across 10 runs. The experimental study reveals that t_r values below 0.1895 exhibit significant variation in solution quality. However, for t_r values of 0.1995 and 0.2095, the quality of solutions remains consistent. This can be attributed to t_r, which facilitates the reduction of temperature and structured decay of epsilon rate. Consequently, the algorithm achieves a more organized and controlled epsilon decay, leading to improved convergence. Similarly, the values of $ \alpha =0.825 $ , $ \gamma =0.35 $ , and $ \lambda =0.0125 $ are chosen based on their ability to produce high-quality results.

Based on the experimentation, it is concluded that higher learning rates (α) and lower discount factors (γ) contribute to the generation of optimal solutions. On the other hand, higher values of λ result in increased randomness of epsilon (ε), which negatively affects the convergence rate, leading to higher iterations and time consumption.

Evaluation metrics

For the evaluation of the proposed RL framework, three metrics are considered. They are time consumption, memory consumption, and the optimality (fitness value) of the solution. The time consumption given in seconds (s) denotes the period required by the various optimization methods to give the results. The memory consumption given in terms of megabytes (MB) refers to the amount of RAM consumed by the application (optimization methods) to run the program and provide the results. The optimality of the solution is determined by fitness values, where higher values indicate better quality results. An optimal sequence with a high fitness value exhibits feasibility, stability, and minimal directional changes. In this work, the optimality criterion is categorized as no-results, infeasible, near-optimal, and best near-optimal results for clarity and explanation purposes. Based on these three metrics, the performance of the RL-DSP framework (CQL and OQL) and other algorithms is compared.

Results and discussion

A detailed analysis and a comprehensive discussion interpreting the results based on its performance metrics are given in this section.

Time and memory observation

The exact methods explore all the possible solutions. Hence, it produces correct results all the time but consumes more time for the huge number of parts. The optimization algorithms such as ACO, GA, and IMMAS process only a selected set of possible solutions based on their strategies. Being strategic in their processing of solutions, these algorithms give near-optimal results for products of high complexity and matched optimal solutions for smaller or less complex products. Thus, they demonstrate a lower consumption of time and memory compared to the exact methods. The gap in metrics such as time and memory between the tested algorithms is mainly due to the nature of their processing and searching mechanisms. Table 3 and Figure 9 show the time consumption comparisons of the various algorithms for the considered products. Similarly, the memory consumption comparison is given in Table 4 and Figure 10.

Table 3. Time consumption comparison of various algorithms

No Solution Infeasible Solution Near-Optimal Solution Best Near-Optimal Solution

Table 4. Memory consumption comparison of various algorithms

No Solution Infeasible Solution Near-Optimal Solution Best Near-Optimal Solution

Figure 9. Time consumption analysis chart.

Figure 10. Memory consumption analysis chart.

From Tables 3 and 4, it is evident that traditional algorithms exhibit increased time and memory consumption for products with larger part counts. The BF approach requires more than 500 seconds and 900 MB of memory for products with more than nine parts. The DP approach experiences longer execution time and memory usage for 21-part products and does not consistently yield optimal results. Similarly, the GM consumes more time and memory for products with 16 and 21 parts, failing to provide optimal solutions for products with fewer parts (8, 12, and 13). In terms of optimization algorithms, IMMAS demonstrates superior time performance compared to ACO and GA.

Additionally, IMMAS delivers best near-optimal solutions for products of up to 13 parts. Following IMMAS, GA exhibits satisfactory performance in terms of time and memory consumption for sequence generation compared to ACO. ACO performs better than all the traditional algorithms but falls short when compared to other optimization algorithms such as GA and IMMAS. In the case of RL methods, both the CQL and the proposed OQL outperform other algorithms in terms of solution quality. The advantage of RL methods is their efficient learning and decision-making processes. They offer best near-optimal solutions for all products by learning from the consequences of past actions, while minimizing time and memory consumption.

When comparing CQL and the proposed OQL, it is observed that OQL consumes less time and memory than CQL. From the observation, the OQL approach seems to give better performance for products with more parts. The graphical representation of these data in Figures 9 and 10 indicates that the overall time and memory consumption of RL methods are lesser than those of other algorithms. Moreover, the time and memory consumption for RL methods steadily increases as the number of parts in the product rises. When comparing CQL and OQL methods, the OQL technique demonstrates a slight advantage and performs better for products with more parts.

Fitness and reward value observation

The fitness values for the disassembly sequences generated by all the considered algorithms are presented in Table 5 and visualized in Figure 11. High fitness values indicate stable sequences with minimal directional changes. From Figure 11, it is observed that both RL methods (CQL and OQL) consistently generate sequences with high fitness values for all the products considered. On the other hand, traditional algorithms, due to their high time consumption, are limited to generating solutions within 900 seconds, resulting in poorer fitness values. Both RL methods, CQL and OQL, provide best near-optimal solutions with slight differences in fitness values for large products. Although this difference may seem small, it can have a significant impact on the performance of larger and more complex products.

Table 5. Fitness value comparison of various algorithms

No Solution Infeasible Solution Near-Optimal Solution Best Near-Optimal Solution

Figure 11. Fitness value analysis chart.

To compare the performance of CQL and OQL techniques in the RL aspect, the rewards obtained over 100 episodes by both RL methods for four products with 4, 8, 12, and 21 parts are analyzed and visualized in Figure 12. These four products are selected specifically to showcase the variation in rewards obtained in both RL methods.

Figure 12. Reward vs episode comparison – CQL and OQL.

Observing Figure 12, it is evident that for the 4-part product, the highest reward is achieved at the 40^th episode and starts converging in the OQL method, while the CQL approach reaches its highest reward only at the end of 80 episodes. In the case of the 8-part product, convergence occurs within 50 episodes with OQL, while CQL starts converging only from the 80^th episode. For the 12-part product, convergence begins at 50 episodes with OQL, whereas CQL achieves convergence only after 70 episodes. In the case of a large product with 21 parts, OQL produces the best result after 50 episodes, while CQL, with variations, fails to achieve high reward result within 100 episodes.

Summarizing the results, it is clear that OQL achieves good solutions within 100 episodes, with the rewards stabilizing beyond 60 episodes on average for all the considered products. In contrast, CQL exhibits more variations in rewards and only reaches good solutions toward the end of the 100 episodes in most of the cases. Based on this experimental study, it is evident that the OQL method shows significant improvements over CQL with the EG approach. The introduction of the ESA technique in OQL allows for a more structured and organized decay of epsilon values, resulting in faster convergence and higher-quality solutions compared to the CQL method. The results clearly indicate that the OQL method can provide better results faster than CQL, making it a more efficient approach for addressing the DSP problem.