A Margin-based Model with a Fast Local Search\newline for Rule Weighting and Reduction in Fuzzy\newline Rule-based Classification Systems

Document Type : Research Paper


Computer Science & Engineering & IT Department of Shiraz University, Shiraz, Fars, Iran


Fuzzy Rule-Based Classification Systems (FRBCS) are highly investigated by researchers due to their noise-stability and  interpretability. Unfortunately, generating a rule-base which is sufficiently both accurate and interpretable, is a hard process. Rule weighting is one of the approaches to improve the accuracy of a pre-generated rule-base without modifying the original rules. Most of the proposed methods by now, may over-fit on training data due to generating complex decision boundaries. In this paper, a margin-based optimization model is proposed to improve the performance on unseen data. By this model, fixed-size margins are defined along the decision boundaries and the rule weights are adjusted such that the marginal space would be empty of training instances as much as possible. This model is proposed to support the single-winner reasoning method with a special cost-function to remove undesired effects of noisy instances. The model is proposed to be solved by a fast well-known local search method. With this solving method, a huge amount of irrelevant and redundant rules are removed as a side effect.Two artificial and 16 real world datasets from UCI repository are used to show that the proposed method significantly outperforms other methods with proper choice of the margin size, which is the single parameter of this method.


A. Cano, A. Zafra and S. Ventura, {\it An EP algorithm for learning highly interpretable classifiers},
11th Inter. Conf. on Intelligent Systems Design and Applications (ISDA), (2011), 325--330.

S. M. Chen, {\it Generating weighted fuzzy rules from relational database systems for estimating values using genetic algorithms}, IEEE Trans. on Fuzzy Systems, {\bf 11}\textbf{(4)} (2003), 495--506.

S. M. Chen, {\it A new weighted fuzzy rule interpolation method based on GA-based weights-learning techniques}, proced. of ICMLC, {\bf 5} (2010), 2705-2711.

S. M. Chen, {\it Weighted fuzzy rule interpolation based on GA-based weight-learning techniques},
IEEE Trans. on Fuzzy Systems, {\bf 19}\textbf{(4)} (2011), 729--744.

Z. Chi, H. Yan and T. Pham, {\it Fuzzy algorithms: with applications to image processing and pattern recognition}, World Scientific, Singapore, 1996.

C. Cortes and V. Vapnik, {\it Support vector networks}, Machine Learning, {\bf 20} (2004).

J. Demsar, {\it Statistical comparisons of classifiers over multiple data sets}, Journal of Machine Learning Research, {\bf 7} (2006), 1--30.

S. M. Fakhrahmad and  M. Z. Jahromi, {\it A new rule-weight learning method based on gradient descent},
Proc. of World Congress on Engineering, {\bf I} (2009).

G. Forman and I. Cohen, {\it Learning from little: comparison of classifiers given little training},
PKDD 2004, LNAI 3202, Springer-Verlag Berlin Heidelberg 2004, {2004}, 161--172.

L. Fu, {\it Rule generation from neural networks}, IEEE Transaction on systems, Man, and Cybernetics,
{\bf 24}\textbf{(8)} (1994).

S. Garcia and F. Herrera, {\it An extension on statistical comparison of classifiers over multiple data sets for all pair wise comparisons}, Journal of Machine Learning Research, {\bf 9} (2008), 2677--2694.

C. Hsu and C. Lin, {\it A comparison of methods for multiclass support vector machines}, IEEE Transaction on neural networks, {\bf 13}\textbf{(2)} (2002).

Q. Hu, P. Zhu, Y. Yang and D. Yu, {\it Large-margin nearest neighbor classifiers via sample weight learning},
Neurocomputing, {\bf 74}\textbf{(4)} (2011), 656--660.

H. Ishibuchi, T. Murata and I. B.Turksen, {\it Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems}, Fuzzy Sets and Systems, {\bf 89}\textbf{(2)} (1997), 135--150.

H. Ishibuchi and T. Nakashima, {\it Effect of rule weights in fuzzy rule-based classification systems},
IEEE Transactions on Fuzzy Systems, {\bf 9}\textbf{(4)} (2001), 506--515.

H. Ishibuchi, T. Nakashima and M. Nii, {\it Classification and modelling with linguistic information granules: advanced approaches to linguistic data mining}, Springer Verlag, 2004.

H. Ishibuchi and M. Nii, {\it Techniques and applications of neural networks for fuzzy rule approximation},
Fuzzy Theory Systems, (1999), 1491--1519.

H. Ishibuchi and Y. Nojima, {\it Analysis of interpretability-accuracy tradeoff of fuzzy systems by multiobjective fuzzy genetics-based machine learning}, International Journal of Approximate Reasoning,
{\bf 44}\textbf {(1)} (2007), 4--31.

H. Ishibuchi, K. Nozaki and H. Tanaka, {\it Distributed representation of fuzzy rules and its application to pattern classification}, Fuzzy Sets and Systems, {\bf 52}\textbf{(1)} (1992),21--32.

H. Ishibuchi, K. Nozaki, N. Yamamoto and H. Tanaka, {\it Selecting fuzzy if-then rules for classification problems using genetic algorithms}, IEEE Transactions on Fuzzy Systems, {\bf 3}\textbf{(3)} (1995),260--270.

H. Ishibuchi and T. Yamamoto, {\it Rule weight specification in fuzzy rule-based classification systems},
IEEE Trans. on Fuzzy Systems, {\bf 13}\textbf{(4)} (2005), 428--435.

H. Ishibuchi and T. Yamamoto, {\it Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining}, Fuzzy Sets and Systems, {\bf 141}\textbf{(1)} (2004), 59--88.

J. Langford, {\it Tutorial on practical prediction theory for classification}, Journal of Machine Learning Research, {\bf 6} (2005), 273--306.

R. Mikut, J. Jakel and L. Groll, {\it Interpretability issues in data-based learning of fuzzy systems},
Elsevier, Fuzzy Sets and Systems, {\bf 150} (2005), 179--197.

T. Nakashima, G. Schaefer, Y. Yokota and H. Ishibuchi, {\it A weighted fuzzy classifier and its application to image processing tasks}, Fuzzy Sets and Systems, {\bf 158}\textbf{(3)} (2007), 284--294.

K. Nozaki, H. Ishibuchi and H. Tanaka, {\it Adaptive fuzzy rule-based classification systems}, IEEE Transactions on Fuzzy Systems, {\bf 4}\textbf{(3)} (1996), 238--250.

M. Taheri, H. Azad, K. Ziarati and R. Sanaye, {\it A quadratic margin-based model for weighting fuzzy classification rules inspired by support vector machines}, Iranian Journal of Fuzzy Systems,
{\bf 10}\textbf{(4)} (2013), 41--55.

{\it UCI machine learning repository, http://www.ics.uci.edu/~mlearn/databases.}

V. Vapnik, {\it The nature of statistical learning theory}, Springer Verlag, New York, 1995.

K. Q. Weinberger, J. C. Blitzer and L. K. Saul, {\it Distance metric learning for large margin nearest neighbor classification}, Advances in Neural Information Processing Systems (NIPS), {\bf 18} (2006), 1473--1480.

J. Weston and C. Watkins, {\it Support vector machines for multi-class pattern recognition}, ESANN'1999 proceedings - European Symposium on Artificial Neural Networks, Bruges (Belgium),
isbn:2-600049-9-X, (1999), 219--224.

L. Xu, K. Crammer and D. Schuurmans, {\it Robust support vector machine training via convex outlier ablation}, In Proc. of the National Conf. on Artificial Intelligence, {\bf 21}\textbf{(1)} (2006), 536--542.

L. Yu and J. Xiao, {\it Trade-off between accuracy and interpretability: experience-oriented fuzzy modeling via reduced-set vectors}, Elsevier, Computers and Mathematics with Applications, {\bf 57} (2009), 885--895.

M. J. Zolghadri and M. Taheri, {\it A proposed method for learning rule weights in fuzzy rule-based classification systems}, Fuzzy Sets and Systems, {\bf 159} (2008), 449--459.