Support vector regression with random output variable and probabilistic constraints

Document Type: Research Paper

Authors

1 Department of Statistics, Ferdowsi University of Mashhad, Mashhad, Iran

2 Department of Applied Mathematics, Ferdowsi University of Mashhad, Mashhad, Iran

Abstract

Support Vector Regression (SVR) solves regression problems based on the concept of Support Vector Machine (SVM). In this paper, a new model of SVR with probabilistic constraints is proposed that any of output data and bias are considered the random variables with uniform probability functions. Using the new proposed method, the optimal hyperplane regression can be obtained by solving a quadratic optimization problem. The proposed
method is illustrated by several simulated data and real data sets for both models (linear and nonlinear
) with probabilistic constraints.

Keywords


[1] A. R. Arabpour and M. Tata, Estimating the parameters of a fuzzy linear regression model,
Iranian Journal of Fuzzy Systems, 5(2) (2008), 1–19.
[2] K. Bache and M. Lichman, UCI machine learning repository, Available on-line at:
http://archive.ics.uci.edu/ml/machine-learning-databases, 2013.
[3] A. Ben-Tal, S. Bhadra, C. Bhattacharyya and J. S. Nath, Chance constrained uncertain
classification via robust optimization, Math. Program., 127(1) (2011), 145–173.
[4] P. Bosch, J. Lopez, H. Ramirez and H. Robotham, Support vector machine under uncertainty:
an application for hydroacoustic classification of fish-schools in Chile, Expert Systems with
Applications, 40 (2013), 4029–4034.
[5] K. D. Brabanter, J. D. Brabanter, J. A. K. Suykens and B. D. Moor, Approximate confidence
and prediction intervals for least squares support vector regression, IEEE Transactions on
Neural Networks, 22 (2011), 110–120.
[6] E. Carrizosa, J. E. Gordillo and F. Plastria, Kernel support vector regression with imprecise
output, Dept. MOSI. Vrije Univ. Brussel. Belgium. Tech. Rep., Available on-line at:
http://www.optimization-online.org/DB_FILE/2008/01/1896.pdf, 2008.
[7] E. Carrizosa, J. E. Gordillo and F. Plastria, Support vector regression for imprecise
data, Dept. MOSI. Vrije Univ. Brussel. Belgium. Tech. Rep., Available on-line at:
http://www.optimization-online.org/DB_HTML/2007/11/1826.html, 2007.
[8] J. H. Chiang and P. Y. Hao, Support vector learning mechanism for fuzzy rule-based modeling:
a new approach, IEEE Trans. Fuzzy Syst., 12(1) (2004), 1–12.
[9] H. Drucker, Ch. J. C. Burges, L. Kaufman, A. Smola and V. Vapnik, Support vector regression
machines, Adv. Neural Inform. Process. Syst., 9 (1997), 155–161.
[10] B. Efron, Bootstrap methods: Another look at the jackknife, Annals of Statistics, 7 (1979)
1–26.
[11] A. Farag and R. M. Mohamed, Classification of multispectral data using support vector machines
approach for density estimation, IEEE Seventh International Conference on Intell.
Eng. Syst., (2003), 4–6.
[12] J. B. Gao, S. R. Gunn, C. J. Harris and M. Brown, A probabilistic framework for SVM
regression and error bar estimation, Machine Learning, 46 (2002), 71–89.
[13] P. Y. Hao and J. H. Chiang, A fuzzy model of support vector regression machine, International
Journal of Fuzzy Systems, 9(1) (2007), 45–49.
[14] H. P. Huang and Y. H. Liu, Fuzzy support vector machines for pattern recognition and data
mining, International Journal of Fuzzy Systems, 4 (2002), 826–835.
[15] G. Huang, S. Song, C. Wu and K. You, Robust support vector regression for uncertain input
and output data, IEEE Transactions on Neural Networks and Learning Systems, 23(11)
(2012), 1690–1700.
[16] R. K. Jayadeva, R. Khemchandani and S. Chandra, Twin support vector machines for pattern
classification, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(5)
(2007), 905–910.
[17] Y. Jinglin, H. X. Li and H. Yong, A probabilistic SVM based decision system for pain diagnosis,
Expert Systems with Applications, 38 (2011), 9346–9351.
[18] A. F. Karr, probability, Springer, New york, (1993), 52–74.
[19] M. A. Kumar and M. Gopal, Least squares twin support vector machines for pattern classification,
Expert Systems with Applications, 36(4) (2009), 7535–7543.
[20] J. T. Y. Kwok, The evidence framework applied to support vector machines, IEEE Transactions
on Neural Networks, 11 (2000), 1162–1173.
[21] G. R. G. Lanckriet, L. E. Ghaoui, Ch. Bhattacharyya and M. I. Jordan, A robust minimax
approach to classification, J. Mach. Learn. Res., 3 (2002), 555–582.

[22] Y. J. Lee and S. Y. Huang, Reduced support vector machines: a statistical theory, IEEE
Transactions on Neural Networks, 18 (2007), 1–13.
[23] H. Li, J. Yang, G. Zhang and B. Fan, Probabilistic support vector machines for classification
of noise affected data, Information Sciences, 221 (2013), 60–71.
[24] C. F. Lin and S. D. Wang, Fuzzy support vector machine, IEEE Transactions on Neural
Networks, 13 (2002), 464–471.
[25] W. Y. Liu, K. Yue and M. H. Gao, Constructing probabilistic graphical model from predicate
formulas for fusing logical and probabilistic knowledge, Information Sciences, 181(18) (2011),
3828–3845.
[26] M. Lobo, L. Vandenberghe, S. Boyd and H. Lebret, Applications of second-order cone programming,
Linear Algebra Its Appl., 284 (1998), 193–228.
[27] O. L. Mangasarian, Nonlinear Programming, McGraw-Hill, New York, (1969), 69–75.
[28] S. Mehrotra, On the implementation of a primal-dual interior point method, SIAM J. Optim.,
2 (1992), 575–601.
[29] X. Peng, TSVR: an efficient twin support vector machine for regression, Neural Networks.,
23(3) (2010), 365–372.
[30] J. C. Platt, Probabilistic outputs for support vector machines and comparisons to regularized
likelihood methods, Advances in Large Margin Classifiers, 10(3) (1999), 61–74.
[31] Z. Qi, Y. Tian and Y. Shi, Robust twin support vector machine for pattern classification,
Pattern Recognition, 46 (2013), 305–316.
[32] H. Sadoghi Yazdi, S. Effati and Z. Saberi, The probabilistic constraints in the support vector
machine, App. Math. Comput., 194 (2007), 467–479.
[33] P. K. Shivaswamy, Ch.Bhattacharyya and A.J.Smola, Second order cone programming approaches
for handling missing and uncertain data,J. Mach. Learn. Res.,7 (2006), 1283-1314.
[34] P. Sollich, Bayesian methods for support vector machines: evidence and predictive class
probabilities, Machine Learning, 46 (2002), 21–52.
[35] J. A. K. Suykens and J. Vandewalle, Least squares support vector machine classifiers, Neural
Processing Letters, 9(3) (1999), 293–300.
[36] T. B. Trafalis and S. A. Alwazzi, Support vector regression with noisy data: a second order
cone programming approach, Int. J. General Syst., 36 (2007), 237–250.
[37] T. B. Trafalis and R. C. Gilbert, Robust classification and regression using support vector
machines, Eur. J. Oper. Res., 173(3) (2006), 893–909.
[38] URL http://www.csie.ntu.edu.tw/ cjlin/libsvmtools/datasets/regression.html.
[39] URL http://www.dcc.fc.up.pt/ ltorgo/Regression/DataSets.html.
[40] URL http://www.itl.nist.gov/div898/strd/nls/nls_main.shtml.
[41] V. Vapnik, The nature of statistical learning theory, Springer-Verlag, New York, (1995),
123-146, 181-186.
[42] V. Vapnik, S. Golowich and A. Smola, Support vector method for multivariate density estimation,
Adv. Neural Inform. Process. Syst., 12 (1999), 659–665.
[43] Y. Xu and L. Wang, A weighted twin support vector regression, Knowledge-Based Syst., 33
(2012), 92–101.
[44] Y. Xu, W. Xi, X. Lv and R. Guo, An improved least squares twin support vector machine,
Journal of information and computational science, 9(4) (2012), 1063–1071.
[45] X. Yang, L. Tan and L. He, A robust least squares support vector machine for regression and
classification with noise, Neurocomputing, 140 (2014), 41–52.