Evaluation of Search Engines: A Conceptual Model and … - uncensored search engine reddit

---

International Journal of Business and Management; Vol. 8, No. 6; 2013
ISSN 1833-3850 E-ISSN 1833-8119
Published by Canadian Center of Science and Education
Evaluation of Search Engines: A Conceptual Model and
Research Issues
Ramaraj Palanisamy1
1 Department of Information Systems, St Francis Xavier University, Canada
Correspondence: Ramaraj Palanisamy, Department of Information Systems, The Gerald Schwartz School of
Business, St Francis Xavier University, Antigonish, Canada. Tel: 1-902-867-2184. E-mail: rpalanis@stfx.ca
Received: September 19, 2012 Accepted: December 12, 2012 Online Published: February 20, 2013
doi:10.5539/ijbm.v8n6p1 URL: http://dx.doi.org/10.5539/ijbm.v8n6p1
Abstract
This paper examines the evaluation of search engines by developing a conceptual model based on the literature.
The model identifies the key factors that influence user evaluation of search engines, effective and efficient
criteria for evaluation by considering user satisfaction and usage as the search engine success variables. The
model attempts to identify the attributes that determine a good search engine, why users repeatedly visit their
favorite search engines, and why users switch between different search engines. The research issues are evolved
out of the conceptual model; the implications for searchers and search engine providers are given.
Keywords: search engine performance, evaluation of search engine, criteria for evaluation, search engine
success, conceptual model
1. Introduction
The most performed activities by Internet users are searching for specific information together with email
services. Search engines are one of the major information retrieval tool that users use to find various kind of
information from the Internet (Rangaswamy et al., 2009). For instance, in e-commerce, search engines often act
as the gate keepers for gathering information about products, browse products, or make purchases only after the
websites of interest have been identified using search engines (Kraut et al., 1996). Users interact with online
informational retrieval systems using powerful tool such as search engines (Liaw & Huang, 2006) to retrieve
better information in an efficient way (Jansen et al., 2008). An international survey (OCLC, 2005) report that 89
percent of information searches undertaken by college students begin with a search engine and the same trend is
observed for faculty and researchers. Internet search engines are the first option for people who want to find
information about anything (Kim, 2009). Thereby search engines have become an integral part of information
environment and they are replacing the role of libraries in facilitating information retrieval and access (Rieger,
2009). As a result, there has been a rapid growth in the Web Search engine market and search engines continue to
attract a large number of online users / Web searchers and regularly rank as heavily visited sites in terms of the
number of visitors (Alexa Internet Inc., 2008).
There are numerous search engines including the generalist (Bing, Google, Sapo and Yahoo!), health-specific
(Medline Plus, Sapo Sau?de and Web MD) and other search engines available on the Internet (Lopes & Ribeiro,
2011) that can help users find any information, anywhere on the web, or even beyond it. Web-based information
retrieval systems would collapse if search engines were not available on the Internet (Liaw & Huang, 2003).
Among the various search engines, Google is the most visited website in the world (Alexa, 2010) and Googling
has become synonymous with researching (Mostafa, 2005). Other search engine companies such as Yahoo!,
Excite, Lycos, Altavista, HotBot, Bing, Ask, AOL, and other portal sites believe that they gain competitive
advantage by acquiring long-term repeat users. Though the business model followed by these companies relies
on the fact that users will be repeatedly choosing their favorite search engines, it is not clear as to what might
lead to such repeat visits or how users interact with search engines for information retrieval.
With the tremendous amount of diversity content on the Internet, retrieving relevant information is so important
for the users. So, in the recent years, attention has shifted away from information retrieval (IR) to evaluation of
search engine performance (Leroy et al., 2007). The existing IR research evaluates the performance of search
engine tools in terms of precision and recall (Baujard et al., 1998), compares the retrieval effectiveness of online
search engines with single keyword and questions- answering tasks (Bin & Lun, 2001). As the experience with a
1
www.ccsenet.org/ijbm International Journal of Business and Management Vol. 8, No. 6; 2013
search engine will determine to use and reuse the search engine, it becomes important to study how Internet
users interact with and evaluate their experience with search engines.Though the user's interactions with search
engines are often brief, users will however form an opinion of the search tools that they have used. Besides, for
the progression of web retrieval system development mainly depends on how the end-users perceive the search
engine tools (Johnson & Crudge, 2007) and evaluating information retrieval systems is central to information
science (Su, 2003a). There is a lack of knowledge on how users evaluate search engines and tools when they use
them for information searching and retrieval. To fill this gap, a conceptual model for evaluating search engines
has been developed based on a review of literature. The model identifies the key factors that influence user
evaluation of search engines and eventually identify the attributes that determine a good search engine, why
users repeatedly visits their favorite search engines, and why users switch between different search engines.The
rest of the paper is organized as follows. The section 2 describes the research methodology; the section 3 reviews
the literature on the evaluation of search engines; section 4 evolves a conceptual model for evaluating search
engines; based on the conceptual model section 5 discusses the various issues for research followed by the
implications for practice and research with conclusions in section 6.
2. Research Methodology
Developing an appropriate methodology for evaluating information retrieval system is central to information
science and information retrieval systems (Su, 2003a). To facilitate this process, this research falls into
answering the following questions: (1) What are the influences for users to use a search engine? (2) What are the
intentions of a user in using a search engine? (3) What are the criteria used in evaluating a search engine? (4)
How a search engine could be effectively used? (5) How a search engine could be efficiently used? (6) How the
satisfaction of the users could be increased in using a search engine? Accordingly, this review concentrates on
the following aspects: the influences (environmental and user-related) for using a search engine, criteria
(effectiveness and efficiency) used in evaluating a search engine, intentions to use a search engine, user
satisfaction and usage in using a search engine.
A literature survey was employed in the study to explore these aspects from perspective of evaluating
information retrieval system such as search engine. The literature was collected primarily from publications in
the area of information retrieval systems, web searching, user characteristics in web search, information seeking
behavior, human interaction with web, search engine performance, user satisfaction, and system usage. The
literature search includes journals published by numerous publishers in particular Emerald, ACM, IEEE and
Elsevier, as they are well reputed. The aim of literature search was to explore the evaluation of search engines by
developing a conceptual model thereby identify the research issues based on the model.
3. Evaluation of Search Engines
As the Internet provides unlimited amounts of information that can be accessed with low effort and cost, search
engine represents powerful tool that assists the Internet users in their interaction with the online environment
(Liaw & Huang, 2006). A search engine is an information retrieval system with a set of programs with search
tools used to perform searches (Liaw & Huang, 2003), designed, developed, and used for finding information
from the web (Katz, 2010) using different strategies. Search engines perform the basic retrieval task including
the acceptance of a query, a comparison with a database and the production of retrieved digital information such
as text, audio, video, data and simulations (Rieger, 2009; Rowley, 1998). A search engine tool is a utility
available in the Internet in which the user inputs specific name, subject, and/or key words for the purpose of
retrieving a list of web links that match the user's query terms (Vaughan, 1999). The search engine facilitates the
users to apply their criteria to a database to build a set of matches. The examples include Google, Yahoo!,
AltaVista, Webcrawler, Lycos, Excite, Infoseek, Excite, HotBot, Bing, Ask, AOL, and other portal sites. Usually
the search engine indexes with automatic software and the catalog is built manually with human input.Search
engines gather web pages that form the universe from which the users retrieve information by issuing queries
and the required information is retrieved by using information retrieval algorithms (Gordon & Pathak, 1999).
Usually the search engines provide search services on the web based on directory services and query based
(Liaw & Huang, 2003). The directory services (e.g. yahoo) provide a hierarchical organization of resources
developed by human cataloguers (Callery & Tracy-Proulx, 1997); the query based services (e.g., Excite) provide
broad coverage of the Internet through intensive automation of query retrieval process (Jenkins et al., 1998).
Users usually search for information trying to maximize the accuracy of search outcome with minimum effort
exerted to acquire it (Bettman, 1979).
It is important to focus on how end users interact with and evaluate their experience in using the search engines
(Leroy et al., 2007). A framework for evaluating search engines is provided by Sutcliffe & Ennis (1998)
2
www.ccsenet.org/ijbm International Journal of Business and Management Vol. 8, No. 6; 2013
consisting of four steps: (i) problem identification (ii) need articulation (iii) query formulation and (iv) results
evaluation. During problem identification, a user realizes the need to retrieve information; in the `need
articulation' stage the user consult external sources for the required information or formulate the terms (from the
user's own domain knowledge) to search; during query formulation the user combines the terms in query
suitable for a search engine; and finally during results evaluation stage the required information is compared
with the retrieved information.
Though the user's interactions with search engines are often brief, users will form an opinion of the search
engine and tools used (Spink et al., 2001) and these impressions form "mental models" resulting from the user's
interactions with search engines (Johnson & Crudge, 2007). These mental models facilitate the users in
evaluating the search engines and represent the user's evaluative view which can be used to understand the
user-based criteria for evaluation. Some of the user-based/ user-centered criteria are effectiveness, efficiency,
interaction/ usability, and overall satisfaction and/or success (Su, 2003a; Spink, 2002; Johnson et al., 2001).
Effectiveness refers to the impact of users' interactions with search engines and usability refers to the capabilities
of the search engine. In general, the criteria are diminishing the effort and cost of search and improving the
decision-making process in using the search results (Haubl & Trifts, 2000). In particular, the following criteria
are used for evaluating search engines:
value of the search results - major features (options that make the site unique, informative, and any value
addition to the search results) - quality of retrieved items (proving current and authoritative information)
(Jones & Timm, 2008)
convenience of various search tools
ability of search engines to locate information on the Internet
the usability of search tools (interface design -availability of basic and advanced search features with
instructions for effective searches - overall ease of use) (Su, 2003b; Vaughan, 1999) - most frequently
applied measure
effectiveness (number and precision or relevance of returned results) (Ziff-Davis, 1995)
search engines' capacity to retrieve the information that matches user's informational needs (Pan et al.,
2007)
comprehensiveness of the Web engines by number of documents indexed (Venditto, 1996)
search capability options, how the results were displayed (readability), update frequency of information
(Courtois et al., 1995)
browsability (ease of understanding results) -navigation (easy-to-use format for finding and viewing the
information) (Jones & Timm, 2008)
customizability (ability to construct a search to weed out irrelevant results)
name (the name of the site), on-screen help, speed (response time/ timeliness), database coverage, number
of links, accessibility and others.
4. A Conceptual Model for Evaluating Search Engines
A conceptual model for evaluating search engines is shown in Figure 1. This model is based on user evaluation
of web search engines model developed by Su (2003a, 2003b) and technology acceptance model (TAM)
developed by Davis (1989) and Davis et al., (1989). Su (2003a) suggested that there are many factors that
influence user evaluation of search engines including: system features, search interface, documentation, output
display, relevance of results, interface-ease, expectancy, connectivity, and user-prior experience. TAM is widely
used for understanding user behavior in technology usage and the model shows how users gain the willingness to
accept and use a new system or technology. In the TAM, the user tries the computer system and forms
perceptions about its usefulness (perceived usefulness) and ease of use. These perceptions on ease of use and
usefulness then influence the user's intention to use the computer system. Since a search engine is a web-based
information system, the proposed model includes satisfaction and usage as search engine (information systems)
success variables.
Since the proposed model is part on the TAM, here is a logical scenario of the proposed model. A user tries a
targeted search engine at least once, and then evaluates the search engine's performance in terms of effectiveness
and efficiency. In this process the user forms an (emotional) opinion about the search engine, develop an attitude
and estimate the learning cost. As a result, if the user is satisfied with the search engine, she or he intends to
3


• College Textbook Search Engine - edocs.
• Search Engine Optimization Web Design .pdf - edocs.utsa
• ANALYSIS
• Development of a web-based card game engine
• Homepage - Reddit