Anil Kumar Jain (born 1948) is an Indian-American computer scientist and University Distinguished Professor in the Department of Computer Science and Engineering at Michigan State University. He is one of the most highly cited researchers in computer science, and is internationally recognized for his foundational contributions to pattern recognition, computer vision, and biometric recognition, particularly in fingerprint recognition and face recognition. Jain is a member of the United States National Academy of Engineering, a Foreign Member of the Chinese Academy of Sciences, and a Foreign Fellow of the Indian National Academy of Engineering. He is a Fellow of the ACM, IEEE, AAAS, IAPR, and SPIE. His research has shaped the field of biometrics and has been applied in systems used worldwide for identity verification, law enforcement, and border security. In 2024, he was awarded the BBVA Foundation Frontiers of Knowledge Award in the category of Information and Communication Technologies. == Early life and education == Born in Basti, India, Jain received his Bachelor of Technology in electrical engineering from the Indian Institute of Technology, Kanpur in 1969. He then moved to the United States, where he earned his M.S. in 1970 and Ph.D. in 1973 from Ohio State University. His doctoral dissertation, titled Some Aspects of Dimensionality and Sample Size Problems in Statistical Pattern Recognition, was supervised by Robert B. McGhee and laid the groundwork for his subsequent research in pattern recognition. == Career == Jain began his academic career at Wayne State University, where he taught from 1972 to 1974. In 1974, he joined the faculty of Michigan State University, where he has remained for over five decades and currently holds the position of University Distinguished Professor. Throughout his career, Jain has conducted pioneering research in data clustering, fingerprint recognition, and face recognition. His work has been published in leading scientific journals including Scientific American, Nature, IEEE Spectrum, and MIT Technology Review. He served as Editor-in-Chief of the IEEE Transactions on Pattern Analysis and Machine Intelligence from 1991 to 1994. Jain has also contributed to national security and policy through his service on several advisory bodies. He served as a member of the U.S. National Academies panels on Information Technology, Whither Biometrics, and Improvised Explosive Devices (IED). He has also served on the Defense Science Board, the Forensic Science Standards Board, and the AAAS Latent Fingerprint Working Group. In 2014, Jain was named Innovator of the Year at Michigan State University for transferring several technologies on face and fingerprint recognition to major players in the biometrics industry. He holds eight U.S. and Korean patents related to biometric technologies. == Research contributions == Jain's research spans pattern recognition, computer vision, machine learning, and biometric recognition. His contributions have been particularly influential in several areas: === Biometric recognition === Jain is considered one of the foremost authorities on biometric recognition systems. His research group at Michigan State University has developed algorithms and systems for fingerprint, face, and iris recognition that have been widely adopted in both academic research and commercial applications. His work on fingerprint matching algorithms has been instrumental in establishing standards for automated fingerprint identification systems (AFIS) used by law enforcement agencies worldwide. In recent years, Jain and his research team have made significant advances in child fingerprint recognition, demonstrating that digital scans of a young child's fingerprint can be correctly recognized one year later with over 99 percent accuracy for children as young as six months old. This research has important implications for child identification in developing countries, where it can be used to track immunization records and provide access to medical care. === Data clustering === Jain's survey article "Data clustering: a review" (1999), co-authored with M. N. Murty and P. J. Flynn, is one of the most highly cited papers in computer science. His 2010 paper "Data Clustering: 50 Years Beyond K-Means" provided a comprehensive overview of the evolution of clustering methods and remains an essential reference in the field. === Statistical pattern recognition === Jain's work on statistical pattern recognition, including his influential survey "Statistical pattern recognition: A review" (2000) with R. P. W. Duin and Jianchang Mao, has shaped the theoretical foundations of the field. == Citation metrics and academic impact == Jain is among the most highly cited researchers in computer science. Based on his Google Scholar profile, he had an h-index of 200 in 2020, which was the highest among computer scientists identified in a survey published by UCLA at the time. As of August 2023, his h-index on Google Scholar is 211. He has since been surpassed by Yoshua Bengio, a researcher of similar subjects (neural networks and deep learning for artificial intelligence), who had an h-index of 224 as of August 2023. Another source reported that as of December 2022, he had the highest discipline h-index (D-index) in computer science. == Honors and awards == Jain has received numerous awards and honors recognizing his contributions to computer science and engineering: === Academy memberships === Member, United States National Academy of Engineering (2016) — elected "for contributions to the engineering and practice of biometrics" Foreign Fellow, Indian National Academy of Engineering (2016) Foreign Member, Chinese Academy of Sciences (2019) Member, The World Academy of Sciences (2019) Fellow, National Academy of Inventors === Professional society fellowships === Fellow, ACM Fellow, IEEE (1988) — for contributions to image processing Fellow, AAAS Fellow, International Association for Pattern Recognition Fellow, SPIE === Major awards === BBVA Foundation Frontiers of Knowledge Award in Information and Communication Technologies (2024) IAPR King-Sun Fu Prize (2008) IEEE W. Wallace McDowell Award (2007) — the highest technical honor awarded by the IEEE Computer Society, for pioneering contributions to theory, technique, and practice of pattern recognition, computer vision, and biometric recognition systems IEEE Computer Society Technical Achievement Award (2003) IAPR Pierre Devijver Award (2002) Humboldt Research Award (2002) Guggenheim Fellowship (2001) Fulbright Fellowship (1998) IEEE ICDM Research Contribution Award (2008) === Best paper awards === IEEE Transactions on Neural Networks (1996) Pattern Recognition journal (1987, 1991, 2005) === Honorary doctorates === Universidad Autónoma de Madrid (2018) Hong Kong University of Science and Technology (2021) == Legacy and endowments == Two endowed funds have been established in Jain's honor at Michigan State University, recognizing his lasting impact on the field and the university. In 2015, a former visiting scholar from Jain's laboratory made an anonymous $400,000 gift to create the Anil K. Jain Endowed Graduate Fellowship, which supports doctoral-level research in pattern recognition, computer vision, and biometric recognition. In 2022, the Anil K. and Nandita K. Jain Endowed Professorship was established through $1 million in contributions from multiple donors, including a substantial gift from the Jain family, to support faculty recruitment and retention in the Department of Computer Science and Engineering. == Selected publications == === Books === 1988. Algorithms For Clustering Data. With Richard C. Dubes. Prentice Hall. 1993. Markov Random Fields: Theory and Applications. With Rama Chellappa eds. Academic Press. 1999. Biometrics: Personal Identification in Networked Society. With Ruud M. Bolle and Sharath Pankanti eds. Springer. 2003. Handbook of Fingerprint Recognition. (2nd edition 2009). With D. Maio, D. Maltoni, S. Prabhakar. Springer. 2005. Handbook of Face Recognition. (2nd edition 2011). With S. Z. Li ed. Springer. 2006. Handbook of Multibiometrics. With A. Ross and K. Nandakumar. Springer. 2007. Handbook of Biometrics. With P. Flynn and A. Ross eds. Springer. 2011. Introduction to Biometrics. With A. Ross and K. Nandakumar. Springer. 2015. Encyclopedia of Biometrics (Second Edition). With Stan Li. Springer. === Research articles === Cross, George R. and Anil K. Jain. "Markov random field texture models". IEEE Transactions on Pattern Analysis and Machine Intelligence (1983): 25–39. Jain, Anil K., and Farshid Farrokhnia. "Unsupervised texture segmentation using Gabor filters". Pattern Recognition 24.12 (1991): 1167–1186. Jain, Anil K., and Douglas Zongker. "Feature selection: Evaluation, application, and small sample performance". IEEE Transactions on Pattern Analysis and Machine Intelligence, 19.2 (1997): 153–158. Jain, Anil K., L. Hong, S. Pankanti, R. Bolle. "An Identity-A
Differentiable imaging
Differentiable imaging is a method within computational imaging that incorporates differentiable programming to design imaging systems. It treats the entire imaging process - from light passing through optical components to the numerical reconstruction—as a differentiable programming problem. This approach links optical hardware with numerical reconstruction, enabling joint optimization of both parts through differentiable programming. Differentiable imaging additionally extends the scope of computational imaging beyond image reconstruction, such as by aiding in characterization of optical components. == Background == Computational imaging combines optical hardware and computational algorithms to capture and reconstruct information that conventional imaging system cannot. This is achieved from a combination of the imaging system and the software used in the image reconstruction. Since the captured information may not directly show the image of the target, these systems often rely on numerical models that describe how light encodes the target. In practice, such models may deviate from the physical systems due to uncertainties such as noise, misalignments, manufacturing imperfections, environmental variations, etc. These uncertainties can cause a mismatch between the physical system and its numerical model, which may degrade reconstruction quality and limit the effectiveness of the hardware–software co-design. Uncertainty quantification is also studied in other hybrid physical–numerical systems, such as digital twin. While numerical modeling imaging systems date back to the several decades, such as the multislice method in electron microscopy or X-Ray nanotomography, differentiable imaging emphasizes jointly modeling uncertainties and solving inverse problems with image reconstruction simultaneously. Differentiable imaging transforms the traditional encoding model y = f ( x ) {\textstyle y=f(x)} into a more comprehensive formulation y = f ( x , θ ) {\textstyle y=f(x,\theta )} , where θ {\displaystyle \theta } represents a parameter set of mismatches between physical systems and numerical models. The forward model captures the entire imaging pipeline through a series of interconnected component functions: y = f ( x , θ ) , f = f n o i s e ∘ f c ∘ f o c ∘ f x ∘ f o i ∘ f i , {\displaystyle y=f(x,\theta ),\qquad f=f_{noise}\circ f_{c}\circ f_{oc}\circ f_{x}\circ f_{oi}\circ f_{i},} where the function composition operator ∘ {\displaystyle \circ } connects each system component, and θ = { θ c , θ o c , … } {\displaystyle \theta =\{\theta _{c},\theta _{oc},\ldots \}} encompasses uncertainty system parameters. Each component corresponds to specific physical processes within the imaging system, from illumination through object interactions to sensor behavior and noises. This forward model enables the formulation of an inverse problem that simultaneously optimizes system parameters while reconstructing images: x ∗ , θ ∗ = argmin x , θ L ( f ( x , θ ) , y ) + ∑ n = 1 N β n R n ( x ) {\displaystyle x^{},\theta ^{}={\text{argmin}}_{x,\theta }{\mathcal {L}}(f(x,\theta ),y)+\sum _{n=1}^{N}\beta _{n}{\mathcal {R}}_{n}(x)} s . t . x ∈ Ω x , θ ∈ Ω θ {\displaystyle s.t.\quad x\in \Omega _{x},\theta \in \Omega _{\theta }} Here, L ( f ( x , θ ) , y ) {\displaystyle {\mathcal {L}}(f(x,\theta ),y)} represents the fidelity term that quantifies the discrepancy between the model predictions and measured data. The whole process of the y = f ( x , θ ) {\displaystyle y=f(x,\theta )} is constructed as a computer graph based on differentiable programming, and the inverse problem is solved with gradient based algorithm, while the gradient is calculated with automatic differentiation. == Applications == One application of differentiable imaging is uncertainty management, which seeks to quantify and mitigate the impact of factors induce reality-numerical mismatch. Explicitly accounting for uncertainties can improve reconstruction accuracy and system robustness. Examples include: Model-related uncertainties: unknown or unmeasurable variables—for instance, optical system quantities that differ from the design specifications Data and system uncertainties: artifacts introduced during image acquisition, such as low-quality data, noise, or hardware imperfections Manufacturing uncertainties: variability in the production of imaging hardware—such as slight deviations in lens curvature or sensor alignment—that alters the physical system's behavior
Generalized blockmodeling of valued networks
Generalized blockmodeling of valued networks is an approach of the generalized blockmodeling, dealing with valued networks (e.g., non-binary). While the generalized blockmodeling signifies a "formal and integrated approach for the study of the underlying functional anatomies of virtually any set of relational data", it is in principle used for binary networks. This is evident from the set of ideal blocks, which are used to interpret blockmodels, that are binary, based on the characteristic link patterns. Because of this, such templates are "not readily comparable with valued empirical blocks". To allow generalized blockmodeling of valued directional (one-mode) networks (e.g. allowing the direct comparisons of empirical valued blocks with ideal binary blocks), a non–parametric approach is used. With this, "an optional parameter determines the prominence of valued ties as a minimum percentile deviation between observed and expected flows". Such two–sided application of parameter then introduces "the possibility of non–determined ties, i.e. valued relations that are deemed neither prominent (1) nor non–prominent (0)." Resulted occurrences of links then motivate the modification of the calculation of inconsistencies between empirical and ideal blocks. At the same time, such links also give a possibility to measure the interpretational certainty, which is specific to each ideal block. Such maximum two–sided deviation threshold, holding the aggregate uncertainty score at zero or near–zero levels, is then proposed as "a measure of interpretational certainty for valued blockmodels, in effect transforming the optional parameter into an outgoing state". Problem with blockmodeling is the standard set of ideal block, as they are all specified using binary link (tie) patters; this results in "a non–trivial exercise to match and count inconsistencies between such ideal binary ties and empirical valued ties". One approach to solve this is by using dichotomization to transform the network into a binary version. The other two approaches were first proposed by Aleš Žiberna in 2007 by introducing valued (generalized) blockmodeling and also homogeneity blockmodeling. The basic idea of the latter is "that the inconsistency of an empirical block with its ideal block can be measured by within block variability of appropriate values". The newly–formed ideal blocks, which are appropriate for blockmodeling of valued networks, are then presented together with the definitions of their block inconsistencies. Two other approaches were later suggested by Carl Nordlund in 2019: deviational approach and correlation-based generalized approach. Both Nordlund's approaches are based on the idea, that valued networks can be compared with the ideal block without values. With this approach, more information is retained for analysis, which also means, that there are fewer partitions having identical values of the criterion function. This means, that the generalized blockmodeling of valued networks measures the inconsistencies more precisely. Usually, only one optimal partition is found in this approach, especially when it is used by homogeneity blockmodeling. Contrary, while using binary blockmodeling on the same sample, usually more than one optimal partition had occurred on several occasions.
Non-negative matrix factorization
Non-negative matrix factorization (NMF or NNMF), also non-negative matrix approximation is a group of algorithms in multivariate analysis and linear algebra where a matrix V is factorized into (usually) two matrices W and H, with the property that all three matrices have no negative elements. This non-negativity makes the resulting matrices easier to inspect. Also, in applications such as processing of audio spectrograms or muscular activity, non-negativity is inherent to the data being considered. Since the problem is not exactly solvable in general, it is commonly approximated numerically. NMF finds applications in such fields as astronomy, computer vision, document clustering, missing data imputation, chemometrics, audio signal processing, recommender systems, and bioinformatics. == History == In chemometrics non-negative matrix factorization has a long history under the name "self modeling curve resolution". In this framework the vectors in the right matrix are continuous curves rather than discrete vectors. Also early work on non-negative matrix factorizations was performed by a Finnish group of researchers in the 1990s under the name positive matrix factorization. It became more widely known as non-negative matrix factorization after Lee and Seung investigated the properties of the algorithm and published some simple and useful algorithms for two types of factorizations. == Background == Let matrix V be the product of the matrices W and H, V = W H . {\displaystyle \mathbf {V} =\mathbf {W} \mathbf {H} \,.} Matrix multiplication can be implemented as computing the column vectors of V as linear combinations of the column vectors in W using coefficients supplied by columns of H. That is, each column of V can be computed as follows: v i = W h i , {\displaystyle \mathbf {v} _{i}=\mathbf {W} \mathbf {h} _{i}\,,} where vi is the i-th column vector of the product matrix V and hi is the i-th column vector of the matrix H. When multiplying matrices, the dimensions of the factor matrices may be significantly lower than those of the product matrix and it is this property that forms the basis of NMF. NMF generates factors with significantly reduced dimensions compared to the original matrix. For example, if V is an m × n matrix, W is an m × p matrix, and H is a p × n matrix then p can be significantly less than both m and n. Here is an example based on a text-mining application: Let the input matrix (the matrix to be factored) be V with 10000 rows and 500 columns where words are in rows and documents are in columns. That is, we have 500 documents indexed by 10000 words. It follows that a column vector v in V represents a document. Assume we ask the algorithm to find 10 features in order to generate a features matrix W with 10000 rows and 10 columns and a coefficients matrix H with 10 rows and 500 columns. The product of W and H is a matrix with 10000 rows and 500 columns, the same shape as the input matrix V and, if the factorization worked, it is a reasonable approximation to the input matrix V. From the treatment of matrix multiplication above it follows that each column in the product matrix WH is a linear combination of the 10 column vectors in the features matrix W with coefficients supplied by the coefficients matrix H. This last point is the basis of NMF because we can consider each original document in our example as being built from a small set of hidden features. NMF generates these features. It is useful to think of each feature (column vector) in the features matrix W as a document archetype comprising a set of words where each word's cell value defines the word's rank in the feature: The higher a word's cell value the higher the word's rank in the feature. A column in the coefficients matrix H represents an original document with a cell value defining the document's rank for a feature. We can now reconstruct a document (column vector) from our input matrix by a linear combination of our features (column vectors in W) where each feature is weighted by the feature's cell value from the document's column in H. == Clustering property == NMF has an inherent clustering property, i.e., it automatically clusters the columns of input data V = ( v 1 , … , v n ) {\displaystyle \mathbf {V} =(v_{1},\dots ,v_{n})} . More specifically, the approximation of V {\displaystyle \mathbf {V} } by V ≃ W H {\displaystyle \mathbf {V} \simeq \mathbf {W} \mathbf {H} } is achieved by finding W {\displaystyle W} and H {\displaystyle H} that minimize the error function (using the Frobenius norm) ‖ V − W H ‖ F , {\displaystyle \left\|V-WH\right\|_{F},} subject to W ≥ 0 , H ≥ 0. {\displaystyle W\geq 0,H\geq 0.} , If we furthermore impose an orthogonality constraint on H {\displaystyle \mathbf {H} } , i.e. H H T = I {\displaystyle \mathbf {H} \mathbf {H} ^{T}=I} , then the above minimization is mathematically equivalent to the minimization of K-means clustering. Furthermore, the computed H {\displaystyle H} gives the cluster membership, i.e., if H k j > H i j {\displaystyle \mathbf {H} _{kj}>\mathbf {H} _{ij}} for all i ≠ k, this suggests that the input data v j {\displaystyle v_{j}} belongs to k {\displaystyle k} -th cluster. The computed W {\displaystyle W} gives the cluster centroids, i.e., the k {\displaystyle k} -th column gives the cluster centroid of k {\displaystyle k} -th cluster. This centroid's representation can be significantly enhanced by convex NMF. When the orthogonality constraint H H T = I {\displaystyle \mathbf {H} \mathbf {H} ^{T}=I} is not explicitly imposed, the orthogonality holds to a large extent, and the clustering property holds too. When the error function to be used is Kullback–Leibler divergence, NMF is identical to the probabilistic latent semantic analysis (PLSA), a popular document clustering method. == Types == === Approximate non-negative matrix factorization === Usually the number of columns of W and the number of rows of H in NMF are selected so the product WH will become an approximation to V. The full decomposition of V then amounts to the two non-negative matrices W and H as well as a residual U, such that: V = WH + U. The elements of the residual matrix can either be negative or positive. When W and H are smaller than V they become easier to store and manipulate. Another reason for factorizing V into smaller matrices W and H, is that if one's goal is to approximately represent the elements of V by significantly less data, then one has to infer some latent structure in the data. === Convex non-negative matrix factorization === In standard NMF, matrix factor W ∈ R+m × k， i.e., W can be anything in that space. Convex NMF restricts the columns of W to convex combinations of the input data vectors ( v 1 , … , v n ) {\displaystyle (v_{1},\dots ,v_{n})} . This greatly improves the quality of data representation of W. Furthermore, the resulting matrix factor H becomes more sparse and orthogonal. === Nonnegative rank factorization === In case the nonnegative rank of V is equal to its actual rank, V = WH is called a nonnegative rank factorization (NRF). The problem of finding the NRF of V, if it exists, is known to be NP-hard. === Different cost functions and regularizations === There are different types of non-negative matrix factorizations. The different types arise from using different cost functions for measuring the divergence between V and WH and possibly by regularization of the W and/or H matrices. Two simple divergence functions studied by Lee and Seung are the squared error (or Frobenius norm) and an extension of the Kullback–Leibler divergence to positive matrices (the original Kullback–Leibler divergence is defined on probability distributions). Each divergence leads to a different NMF algorithm, usually minimizing the divergence using iterative update rules. The factorization problem in the squared error version of NMF may be stated as: Given a matrix V {\displaystyle \mathbf {V} } find nonnegative matrices W and H that minimize the function F ( W , H ) = ‖ V − W H ‖ F 2 {\displaystyle F(\mathbf {W} ,\mathbf {H} )=\left\|\mathbf {V} -\mathbf {WH} \right\|_{F}^{2}} Another type of NMF for images is based on the total variation norm. When L1 regularization (akin to Lasso) is added to NMF with the mean squared error cost function, the resulting problem may be called non-negative sparse coding due to the similarity to the sparse coding problem, although it may also still be referred to as NMF. === Online NMF === Many standard NMF algorithms analyze all the data together; i.e., the whole matrix is available from the start. This may be unsatisfactory in applications where there are too many data to fit into memory or where the data are provided in streaming fashion. One such use is for collaborative filtering in recommendation systems, where there may be many users and many items to recommend, and it would be inefficient to recalculate everything when one user or one item is added to the system. The cost function for o
Multiple kernel learning
Multiple kernel learning refers to a set of machine learning methods that use a predefined set of kernels and learn an optimal linear or non-linear combination of kernels as part of the algorithm. Reasons to use multiple kernel learning include a) the ability to select for an optimal kernel and parameters from a larger set of kernels, reducing bias due to kernel selection while allowing for more automated machine learning methods, and b) combining data from different sources (e.g. sound and images from a video) that have different notions of similarity and thus require different kernels. Instead of creating a new kernel, multiple kernel algorithms can be used to combine kernels already established for each individual data source. Multiple kernel learning approaches have been used in many applications, such as event recognition in video, object recognition in images, and biomedical data fusion. == Algorithms == Multiple kernel learning algorithms have been developed for supervised, semi-supervised, as well as unsupervised learning. Most work has been done on the supervised learning case with linear combinations of kernels, however, many algorithms have been developed. The basic idea behind multiple kernel learning algorithms is to add an extra parameter to the minimization problem of the learning algorithm. As an example, consider the case of supervised learning of a linear combination of a set of n {\displaystyle n} kernels K {\displaystyle K} . We introduce a new kernel K ′ = ∑ i = 1 n β i K i {\displaystyle K'=\sum _{i=1}^{n}\beta _{i}K_{i}} , where β {\displaystyle \beta } is a vector of coefficients for each kernel. Because the kernels are additive (due to properties of reproducing kernel Hilbert spaces), this new function is still a kernel. For a set of data X {\displaystyle X} with labels Y {\displaystyle Y} , the minimization problem can then be written as min β , c E ( Y , K ′ c ) + R ( K , c ) {\displaystyle \min _{\beta ,c}\mathrm {E} (Y,K'c)+R(K,c)} where E {\displaystyle \mathrm {E} } is an error function and R {\displaystyle R} is a regularization term. E {\displaystyle \mathrm {E} } is typically the square loss function (Tikhonov regularization) or the hinge loss function (for SVM algorithms), and R {\displaystyle R} is usually an ℓ n {\displaystyle \ell _{n}} norm or some combination of the norms (i.e. elastic net regularization). This optimization problem can then be solved by standard optimization methods. Adaptations of existing techniques such as the Sequential Minimal Optimization have also been developed for multiple kernel SVM-based methods. === Supervised learning === For supervised learning, there are many other algorithms that use different methods to learn the form of the kernel. The following categorization has been proposed by Gonen and Alpaydın (2011) ==== Fixed rules approaches ==== Fixed rules approaches such as the linear combination algorithm described above use rules to set the combination of the kernels. These do not require parameterization and use rules like summation and multiplication to combine the kernels. The weighting is learned in the algorithm. Other examples of fixed rules include pairwise kernels, which are of the form k ( ( x 1 i , x 1 j ) , ( x 2 i , x 2 j ) ) = k ( x 1 i , x 2 i ) k ( x 1 j , x 2 j ) + k ( x 1 i , x 2 j ) k ( x 1 j , x 2 i ) {\displaystyle k((x_{1i},x_{1j}),(x_{2i},x_{2j}))=k(x_{1i},x_{2i})k(x_{1j},x_{2j})+k(x_{1i},x_{2j})k(x_{1j},x_{2i})} . These pairwise approaches have been used in predicting protein-protein interactions. ==== Heuristic approaches ==== These algorithms use a combination function that is parameterized. The parameters are generally defined for each individual kernel based on single-kernel performance or some computation from the kernel matrix. Examples of these include the kernel from Tenabe et al. (2008). Letting π m {\displaystyle \pi _{m}} be the accuracy obtained using only K m {\displaystyle K_{m}} , and letting δ {\displaystyle \delta } be a threshold less than the minimum of the single-kernel accuracies, we can define β m = π m − δ ∑ h = 1 n ( π h − δ ) {\displaystyle \beta _{m}={\frac {\pi _{m}-\delta }{\sum _{h=1}^{n}(\pi _{h}-\delta )}}} Other approaches use a definition of kernel similarity, such as A ( K 1 , K 2 ) = ⟨ K 1 , K 2 ⟩ ⟨ K 1 , K 1 ⟩ ⟨ K 2 , K 2 ⟩ {\displaystyle A(K_{1},K_{2})={\frac {\langle K_{1},K_{2}\rangle }{\sqrt {\langle K_{1},K_{1}\rangle \langle K_{2},K_{2}\rangle }}}} Using this measure, Qui and Lane (2009) used the following heuristic to define β m = A ( K m , Y Y T ) ∑ h = 1 n A ( K h , Y Y T ) {\displaystyle \beta _{m}={\frac {A(K_{m},YY^{T})}{\sum _{h=1}^{n}A(K_{h},YY^{T})}}} ==== Optimization approaches ==== These approaches solve an optimization problem to determine parameters for the kernel combination function. This has been done with similarity measures and structural risk minimization approaches. For similarity measures such as the one defined above, the problem can be formulated as follows: max β , tr ( K t r a ′ ) = 1 , K ′ ≥ 0 A ( K t r a ′ , Y Y T ) . {\displaystyle \max _{\beta ,\operatorname {tr} (K'_{tra})=1,K'\geq 0}A(K'_{tra},YY^{T}).} where K t r a ′ {\displaystyle K'_{tra}} is the kernel of the training set. Structural risk minimization approaches that have been used include linear approaches, such as that used by Lanckriet et al. (2002). We can define the implausibility of a kernel ω ( K ) {\displaystyle \omega (K)} to be the value of the objective function after solving a canonical SVM problem. We can then solve the following minimization problem: min tr ( K t r a ′ ) = c ω ( K t r a ′ ) {\displaystyle \min _{\operatorname {tr} (K'_{tra})=c}\omega (K'_{tra})} where c {\displaystyle c} is a positive constant. Many other variations exist on the same idea, with different methods of refining and solving the problem, e.g. with nonnegative weights for individual kernels and using non-linear combinations of kernels. ==== Bayesian approaches ==== Bayesian approaches put priors on the kernel parameters and learn the parameter values from the priors and the base algorithm. For example, the decision function can be written as f ( x ) = ∑ i = 0 n α i ∑ m = 1 p η m K m ( x i m , x m ) {\displaystyle f(x)=\sum _{i=0}^{n}\alpha _{i}\sum _{m=1}^{p}\eta _{m}K_{m}(x_{i}^{m},x^{m})} η {\displaystyle \eta } can be modeled with a Dirichlet prior and α {\displaystyle \alpha } can be modeled with a zero-mean Gaussian and an inverse gamma variance prior. This model is then optimized using a customized multinomial probit approach with a Gibbs sampler. These methods have been used successfully in applications such as protein fold recognition and protein homology problems ==== Boosting approaches ==== Boosting approaches add new kernels iteratively until some stopping criteria that is a function of performance is reached. An example of this is the MARK model developed by Bennett et al. (2002) f ( x ) = ∑ i = 1 N ∑ m = 1 P α i m K m ( x i m , x m ) + b {\displaystyle f(x)=\sum _{i=1}^{N}\sum _{m=1}^{P}\alpha _{i}^{m}K_{m}(x_{i}^{m},x^{m})+b} The parameters α i m {\displaystyle \alpha _{i}^{m}} and b {\displaystyle b} are learned by gradient descent on a coordinate basis. In this way, each iteration of the descent algorithm identifies the best kernel column to choose at each particular iteration and adds that to the combined kernel. The model is then rerun to generate the optimal weights α i {\displaystyle \alpha _{i}} and b {\displaystyle b} . === Semisupervised learning === Semisupervised learning approaches to multiple kernel learning are similar to other extensions of supervised learning approaches. An inductive procedure has been developed that uses a log-likelihood empirical loss and group LASSO regularization with conditional expectation consensus on unlabeled data for image categorization. We can define the problem as follows. Let L = ( x i , y i ) {\displaystyle L={(x_{i},y_{i})}} be the labeled data, and let U = x i {\displaystyle U={x_{i}}} be the set of unlabeled data. Then, we can write the decision function as follows. f ( x ) = α 0 + ∑ i = 1 | L | α i K i ( x ) {\displaystyle f(x)=\alpha _{0}+\sum _{i=1}^{|L|}\alpha _{i}K_{i}(x)} The problem can be written as min f L ( f ) + λ R ( f ) + γ Θ ( f ) {\displaystyle \min _{f}L(f)+\lambda R(f)+\gamma \Theta (f)} where L {\displaystyle L} is the loss function (weighted negative log-likelihood in this case), R {\displaystyle R} is the regularization parameter (Group LASSO in this case), and Θ {\displaystyle \Theta } is the conditional expectation consensus (CEC) penalty on unlabeled data. The CEC penalty is defined as follows. Let the marginal kernel density for all the data be g m π ( x ) = ⟨ ϕ m π , ψ m ( x ) ⟩ {\displaystyle g_{m}^{\pi }(x)=\langle \phi _{m}^{\pi },\psi _{m}(x)\rangle } where ψ m ( x ) = [ K m ( x 1 , x ) , … , K m ( x L , x ) ] T {\displaystyle \psi _{m}(x)=[K_{m}(x_{1},x),\ldots ,K_{m}(x_{L},x)]^{T}} (the kernel distance between the labe
Free boundary condition
In image processing, the free boundary condition is the convention used when applying a convolution kernel to a digital image in which pixel locations that lie outside the image boundaries are interpreted as having a value of zero.[1] The question of what value to assign out-of-bounds pixels may arise, for instance, when applying a 3×3 kernel to the corner pixel in an image.
Crossover (evolutionary algorithm)
Crossover in evolutionary algorithms and evolutionary computation, also called recombination, is a genetic operator used to combine the genetic information of two parents to generate new offspring. It is one way to stochastically generate new solutions from an existing population, and is analogous to the crossover that happens during sexual reproduction in biology. New solutions can also be generated by cloning an existing solution, which is analogous to asexual reproduction. Newly generated solutions may be mutated before being added to the population. The aim of recombination is to transfer good characteristics from two different parents to one child. Different algorithms in evolutionary computation may use different data structures to store genetic information, and each genetic representation can be recombined with different crossover operators. Typical data structures that can be recombined with crossover are bit arrays, vectors of real numbers, or trees. The list of operators presented below is by no means complete and serves mainly as an exemplary illustration of this dyadic genetic operator type. More operators and more details can be found in the literature. == Crossover for binary arrays == Traditional genetic algorithms store genetic information in a chromosome represented by a bit array. Crossover methods for bit arrays are popular and an illustrative example of genetic recombination. === One-point crossover === A point on both parents' chromosomes is picked randomly, and designated a 'crossover point'. Bits to the right of that point are swapped between the two parent chromosomes. This results in two offspring, each carrying some genetic information from both parents. === Two-point and k-point crossover === In two-point crossover, two crossover points are picked randomly from the parent chromosomes. The bits in between the two points are swapped between the parent organisms. Two-point crossover is equivalent to performing two single-point crossovers with different crossover points. This strategy can be generalized to k-point crossover for any positive integer k, picking k crossover points. === Uniform crossover === In uniform crossover, typically, each bit is chosen from either parent with equal probability. Other mixing ratios are sometimes used, resulting in offspring which inherit more genetic information from one parent than the other. In a uniform crossover, we don’t divide the chromosome into segments, rather we treat each gene separately. In this, we essentially flip a coin for each chromosome to decide whether or not it will be included in the off-spring. == Crossover for integer or real-valued genomes == For the crossover operators presented above and for most other crossover operators for bit strings, it holds that they can also be applied accordingly to integer or real-valued genomes whose genes each consist of an integer or real-valued number. Instead of individual bits, integer or real-valued numbers are then simply copied into the child genome. The offspring lie on the remaining corners of the hyperbody spanned by the two parents P 1 = ( 1.5 , 6 , 8 ) {\displaystyle P_{1}=(1.5,6,8)} and P 2 = ( 7 , 2 , 1 ) {\displaystyle P_{2}=(7,2,1)} , as exemplified in the accompanying image for the three-dimensional case. === Discrete recombination === If the rules of the uniform crossover for bit strings are applied during the generation of the offspring, this is also called discrete recombination. === Intermediate recombination === In this recombination operator, the allele values of the child genome a i {\displaystyle a_{i}} are generated by mixing the alleles of the two parent genomes a i , P 1 {\displaystyle a_{i,P_{1}}} and a i , P 2 {\displaystyle a_{i,P_{2}}} : α i = α i , P 1 ⋅ β i + α i , P 2 ⋅ ( 1 − β i ) w i t h β i ∈ [ − d , 1 + d ] {\displaystyle \alpha _{i}=\alpha _{i,P_{1}}\cdot \beta _{i}+\alpha _{i,P_{2}}\cdot \left(1-\beta _{i}\right)\quad {\mathsf {with}}\quad \beta _{i}\in \left[-d,1+d\right]} randomly equally distributed per gene i {\displaystyle i} The choice of the interval [ − d , 1 + d ] {\displaystyle [-d,1+d]} causes that besides the interior of the hyperbody spanned by the allele values of the parent genes additionally a certain environment for the range of values of the offspring is in question. A value of 0.25 {\displaystyle 0.25} is recommended for d {\displaystyle d} to counteract the tendency to reduce the allele values that otherwise exists at d = 0 {\displaystyle d=0} . The adjacent figure shows for the two-dimensional case the range of possible new alleles of the two exemplary parents P 1 = ( 3 , 6 ) {\displaystyle P_{1}=(3,6)} and P 2 = ( 9 , 2 ) {\displaystyle P_{2}=(9,2)} in intermediate recombination. The offspring of discrete recombination C 1 {\displaystyle C_{1}} and C 2 {\displaystyle C_{2}} are also plotted. Intermediate recombination satisfies the arithmetic calculation of the allele values of the child genome required by virtual alphabet theory. Discrete and intermediate recombination are used as a standard in the evolution strategy. == Crossover for permutations == For combinatorial tasks, permutations are usually used that are specifically designed for genomes that are themselves permutations of a set. The underlying set is usually a subset of N {\displaystyle \mathbb {N} } or N 0 {\displaystyle \mathbb {N} _{0}} . If 1- or n-point or uniform crossover for integer genomes is used for such genomes, a child genome may contain some values twice and others may be missing. This can be remedied by genetic repair, e.g. by replacing the redundant genes in positional fidelity for missing ones from the other child genome. In order to avoid the generation of invalid offspring, special crossover operators for permutations have been developed which fulfill the basic requirements of such operators for permutations, namely that all elements of the initial permutation are also present in the new one and only the order is changed. It can be distinguished between combinatorial tasks, where all sequences are admissible, and those where there are constraints in the form of inadmissible partial sequences. A well-known representative of the first task type is the traveling salesman problem (TSP), where the goal is to visit a set of cities exactly once on the shortest tour. An example of the constrained task type is the scheduling of multiple workflows. Workflows involve sequence constraints on some of the individual work steps. For example, a thread cannot be cut until the corresponding hole has been drilled in a workpiece. Such problems are also called order-based permutations. In the following, two crossover operators are presented as examples, the partially mapped crossover (PMX) motivated by the TSP and the order crossover (OX1) designed for order-based permutations. A second offspring can be produced in each case by exchanging the parent chromosomes. === Partially mapped crossover (PMX) === The PMX operator was designed as a recombination operator for TSP like Problems. The explanation of the procedure is illustrated by an example: === Order crossover (OX1) === The order crossover goes back to Davis in its original form and is presented here in a slightly generalized version with more than two crossover points. It transfers information about the relative order from the second parent to the offspring. First, the number and position of the crossover points are determined randomly. The resulting gene sequences are then processed as described below: Among other things, order crossover is well suited for scheduling multiple workflows, when used in conjunction with 1- and n-point crossover. === Further crossover operators for permutations === Over time, a large number of crossover operators for permutations have been proposed, so the following list is only a small selection. For more information, the reader is referred to the literature. cycle crossover (CX) order-based crossover (OX2) position-based crossover (POS) edge recombination voting recombination (VR) alternating-positions crossover (AP) maximal preservative crossover (MPX) merge crossover (MX) sequential constructive crossover operator (SCX) The usual approach to solving TSP-like problems by genetic or, more generally, evolutionary algorithms, presented earlier, is either to repair illegal descendants or to adjust the operators appropriately so that illegal offspring do not arise in the first place. Alternatively, Riazi suggests the use of a double chromosome representation, which avoids illegal offspring.