WikiISI/Evaluation

From AI Wiki

Evaluation

Contents 1 Abstract 2 Introduction 3 Literature Review 3.1 Potential Uses of Evaluation 3.1.1 Context Evaluation 3.1.2 Implementation Evaluation 3.1.3 Cost Evaluation 3.2 Evaluation Methodology 4 Unique Challenges in ISI Evaluation 4.1 Intelligence and Warning 4.2 Border and Transportation Security 4.3 Domestic Counter-terrorism 4.4 Protecting Critical Infrastructure and Key Assets 4.5 Defending Against Catastrophic Terrorism 4.6 Emergency Preparedness and Response 5 Research Design 5.1 Algorithm Level Evaluation 5.2 System Level Evaluation 5.3 User Level Evaluation 5.4 Group Level Evaluation 5.5 Organization Level Evaluation 5.6 Country Level Evaluation 6 Case Study 6.1 Law-Enforcement: COPLINK 6.2 Dark Web: Authorship Analysis 7 Conclusions and Future Directions 8 References

Abstract

In the past few years, intelligence and security informatics (ISI) research, which, as has been discussed in other chapters, is experiencing tremendous growth.^[1]. As the community matures, attention is now moving towards the evaluation of the research, and the evaluation of the applications that emerge from the research. In this chapter, we look at current evaluation purposes, practices, and methodologies, as well as the unique challenges and issues that need to be adressed when dealing with evaluation of ISI technologies in particular.

Introduction

Federal authorities are actively implementing comprehensive strategies and measures in order to achieve the following three objectives:

• Preventing future terrorist attacks
• Reducing the nation’s vulnerability
• Minimizing the damage and recovering from attacks that occur

Science and technology have been identified in the “National Strategy for Homeland Security” report as the keys to win the new counter-terrorism war. Based on the crime and intelligence knowledge discovered, the federal, state, and local authorities can make timely decisions to select effective strategies and tactics as well as allocate the appropriate amount of resources to detect, prevent, and respond to future attacks. Because of the abundant research and funding opportunities in ISI, there has been a spurt of research interest, and subsequent application development, in this field. Agencies across the United States have begun to adopt these innovative technologies. However, the acquisition and integration of these technologies can be expensive, both in terms of time spent on requirement analysis and technology selection, and in terms of cost. Evaluation of ISI technologies is therefore a very important aspect for technology selection. Also, after the technology has been integrated in the department or agency, validation of the ISI is required to justify the cost to the taxpayers. Therefore, in this chapter, we will discuss the evaluation of ISI technologies.

Literature Review

Evaluation is defined as a process which attempts to determine as systematically and objectively as possible the relevance, effectiveness, efficiency, and impact of activities in the light of specified objectives. It is a learning and action-oriented management tool and organizational process for improving both current activities and future planning, programming, and decision-making (UNICEF Guide). The most common questions asked during an evaluation of any technology are ^[2]:

1. Can the technology lead to increased productivity?
2. How is the ISI useful?
3. What should be the focus of future development?

Potential Uses of Evaluation

Formative evaluation activities can increase community participation, provide motivation for networking among community agencies and promote new coalitions/partnerships (Evaluation Handbook). They can take the following forms:

Context Evaluation

Context evaluation is conducted to determine whether a selection of technologies or interventions is supported by its targeted community. In the early stage, context evaluation focuses on:

• Assessing the needs, assets, and resources of an identified community to plan relevant and effective interventions within the community context
• Identifying the political environment, financial, strategic and structural context of the identified community to increase the likelihood that interventions will be supported by community leaders and organizations

After the selection of technologies or intervention, it may focuse on:

• Gathering contextual information to modify project plans and/or explain past identified risks/problems
• Identifying the financial, organizational, structural, political, social, and environmental strengths and weaknesses the community and the project
• Examining the impact of changing federal and state climates on project implementation and success

Implementation Evaluation

Implementation evaluations focus on examining the core activities undertaken to achieve project goals and intended outcomes. It focuse on:

• Identifying and maximizing strengths in development
• Identifying and minimizing barriers to implementing activities
• Determining if project goals match target population needs
• Assessing whether available resources can sustain project activities
• Measuring the performance and perceptions of the staff
• Measuring the community's perceptions of the project
• Ascertaining the quality of services provided by the project
• Documenting systemic change
• Monitoring clients' and other stakeholders' experiences with the project, and their satisfaction with and utilization of project services

Without such information, it can be difficult to make informed decisions about how to improve the project. Furthermore, if environmental barriers to project implementation are understood, seemingly troubled projects might be deemed successful based on the barriers they overcame.

Cost Evaluation

It assesses the short- and long-term results of a project and seeks to measure the changes brought about by the project. In the early stage, it focuses on:

• determining what outcomes are expected or hoped for from the project
• thinking through how individual participant/client outcomes connect to specific program or system-level outcomes

These types of early evaluation activities increase the likelihood that implementation activities are linked to the outcomes that the project is trying to achieve, and help staff and stakeholders stay focused on the changes they are attempting to make.

In later phases of project maturity, an outcome evaluation focuses on:

• demonstrate the effectiveness of the project and making a case for its continued funding or for expansion/replication
• answer questions about what works, for whom, and in what circumstances, and how to improve program delivery and services
• determine which activities and contextual factors are supporting or hindering outcomes and overall effectiveness

Evaluation Methodology

Evaluation of ISI technologies may use one or more of these methodologies during various phases of design, development, and deployment:

• Pilot testing: Pilot testing is a trial run of procedures and instruments. The main purpose of pilot testing is to catch potential problems before they become costly mistakes. It is typically used if an instrument or method of data collection is being used for the first time or for the first time with a particular group.

• Surveys: The first step in producing a survey is to define the purpose or objective of the survey. “A clear statement of purpose is necessary not only as a justification/explanation of the project, but also as a guideline to determine if future actions in the project are in support of the original purpose” ^[3].

• Experiments: Experiments may be used either to validate the performance factors of the ISI, or to compare performance with competing technologies.

• Case Studies: Case study is an empirical inquiry that investigates a phenomenon within its real-life context. Case study research means single- and multiple case studies, can include quantitative evidence, relies on multiple sources of evidence and benefits from the prior development of theoretical propositions. They can be based on any mix of quantitative and qualitative evidence ^[4].

• Interviews: An interview is a structured social interaction between a researcher and a subject who is identified as a potential source of information, in which the interviewer initiates and controls the exchange to obtain quantifiable and comparable information relevant to an emerging or previously stated hypothesis. It can be used get detailed feedback on user expectations, reactions, etc regarding the ISI.

Unique Challenges in ISI Evaluation

The “National Strategy for Homeland Security” report has identified six critical mission areas where ISI technologies can make a big contribution^[5]. :

• Intelligence and Warning
• Border and Transportation Security
• Domestic Counter-terrorism
• Protecting Critical Infrastructure and Key Assets
• Defending Against Catastrophic Terrorism
• Emergency Preparedness and Response

Below, we discuss some unique challanges faced when evaluating ISI technologies in each of these areas.

Intelligence and Warning

Within this area, cross-jurisdictional information-sharing is of critical importance. For example, local police departments might have data of value to regional or national agencies. A relevant ISI technology needs to support real analysis tasks using data sets that can be realistically collected and shared^[6]. Hence the technology needs to be evaluated, not only in its intrinsic functionality, but also to see whether security guidelines are adhered to, and whether legal policies, protocols, and procedures have been followed.

Border and Transportation Security

The national strategy for homeland security^[5] calls for the creation of "smart borders" that provide "greater security through better intelligence and coordinated national efforts". Several agencies are involved in these efforts, including, but not limited to, The Department of Homeland Security (DHS), Customs and Border Protection (CBP), Border Patrol, Department of Transportation (DOT), Transportation Security Administration (TSA), state Highway Patrol agencies, and local law enforcement agencies from border-area jurisdictions^[7]. This makes it critical to evaluate the existance and effectiveness of the information sharing and analysis framework as part of the overall evaluation of the ISI technology.

Domestic Counter-terrorism

In the past few years, the world wide web has emerged as a widely-used forum for extremist groups to spread their ideologies and recruit new members^[8]. Researchers have employed content analysis and web structure analysis in order to study the dark web. However, because of the amorphous structure of the web, there is no standardazed methodology for the evaluation of research in this area. Most evaluation thus is at the algorithm or system level, and not at the user level.

Protecting Critical Infrastructure and Key Assets

There are several streams of research within this area of intelligence and security informatics. Several new technologies are emerging that defend personal computers or networks from unlawful intrusions or malicious attacks. Evaluations of such technologies are usually at the algorithm or system level. Other research focuses on infrastructure by guarding borders. For example, Kaza, Wang and Chen (2006)^[7] studied mutual information in order to aid suspect vehicle identification at land border crossings. Anand, Madigan, et al. (2006) ^[9] developed an algorithm to aid in Port of Entry inspection procedures. Evaluations in such cases can run the gamut from algorithm evaluation, to country-wide implementation, policy and effectiveness evaluation.

Defending Against Catastrophic Terrorism

“Terrorist attacks can cause devastating damage to a society through the use of chemical, biological, or radiological weapons.” ^[10] “Information systems that can efficiently and effectively collect, access, analyze, and report relevant data can help prevent, detect, respond to and manage these attacks”. Given the wide range of technologies, as well as implementation scope and strategies, evaluation of these technologies can also take many forms, ranging from algorithm evaluation to country-level evaluation. These various strategies are further discussed in the research design.

Emergency Preparedness and Response

Emergency preparedness and response is one of the hardest areas for evaluation. Applications and technologies are evaluated using either experiments, or simulation drills. However, because of the lack of established guidelines, there are no standardized mechanisms for reporting the results of the evaluations.

Research Design

Evaluation research methodology depends on the level of the ISI technology being studied. There are several ways that the different levels can be described:

• Algorithm level evaluation
• System level evaluation
• User level evaluation
• Group level evaluation
• Organizational level evaluation
• Country level evaluation

Below, we discuss common research designs based on the evaluation level of the ISI.

Algorithm Level Evaluation

Evaluation of an algorithm usually involves measuring specific performance characteristics of the algorithm using a well-described test-bed, in a manner that allows statistical comparison. Different tests are run for parameters such as speed, sensitivity, accuracy, precision, recall, etc. Such kinds of evaluations are necessary to test the basic functionalities of an ISI technology. For instance, in identity deception detection and resolution, the algorithm is evaluated in terms of speed (time required to resolve N1, N2,…N distinct identities), compression ratio (resolved distinct identities / original number of identities in the dataset), and precision, accuracy and recall (normally measured against a gold standard prepared by an expert)^[11]. In analysis of criminal networks, cox regression may be used to identify and validate factors that are significant in link formation.

System Level Evaluation

This level of evaluation targets the entire system, usually from a software engineering point of view. Factors evaluated at this stage of testing may include benchmarking for system response time, response accuracy, reliability, evaluation of user interface, programming code quality, documentation etc. Research at this level normally will involve laboratory settings, as well as some user-testing, when carried out by developers of the technology. When used to make investment decisions, law enforcement agencies will also include requirements analysis, functionality comparisons, and investment utilization. An example of this is the evaluation of COPLINK done by Martinez et al. in 2001^[12], which is described in the case studies.

User Level Evaluation

Evaluation at this level focuses on whether a law enforcement official intends to use a technology, and the factors that affect that decision. User acceptance refers to an individual’s psychological state with regard to his or her voluntary acceptance of a technology ^[13]. There are several theories that analyze behavioral intention. The most widely accepted theories are the Theory of Reasoned Action (TRA^[14]), the Theory of Planned Behavior (TPB^[15][16]), the Technology Acceptance Model (TAM ^[17][18][19]), and the Unified Theory of Acceptance and Use of Technology (UTAUT^[20]).

The research design involves definition and identification of the independent and dependent variables, recruitment of subjects (law enforcement officials), instrument development and data collection (usually through surveys), and analysis of data (using structured equation modeling^[21]). One example is the acceptance study done on COPLINK (described in greater detail in the case study section)^[22].

Group Level Evaluation

Any ISI technology is also usually evaluated by the group using it, in order to justify past expenditures, and to identify areas for adding more necessary functionality in the future in order to plan further investment. In the case of ISI, such evaluation is usually anecdotal in nature.

Organization Level Evaluation

This type of evaluation is similar to group level evaluation, but is carried out across the entire organization.

Country Level Evaluation

Since 9/11, the US government has spent billions of dollars in order to improve security and intelligence capabilities. However, the evaluation of such an investment is extremely complicated. Identifying and defining measures for evaluation is very complex because of several reasons, including lack of past efforts, lack of agreement among various stakeholders, reliance on anecdotal evidence, lack of standardized reporting mechanisms etc. (Associated Press: Quantifying danger preparedness a tough task). However, partly because of a push from Congress, several agencies are in a push toward developing these measures. As the country gains more experience in various aspects of this process, it will become easier to apply that experience to different evaluations at the country level.

Case Study

Law-Enforcement: COPLINK

Law-enforcement officers face the following challenges when they perform their jobs ^[23]:

• Knowledge-intensive
  • Need to access all related information about current crime investigation across jurisdiction (information sharing)
  • Need to extract/retrieve useful cues from all available information resources (information retrieval)
• Time critical
  • Crime investigation needs to be completed in a relative short period of time (timely-information access)

Therefore, an ISI system in law-enforcement is aimed to provide solutions for three dimensions:

• Information sharing
  • A standardized platform for information sharing
• Information Retrieval
  • Analysis tools to retrieve potential correlation between cases
• Timely information access
  • An ease-to-use, integrated interface for law-enforcement officers to access

From Table 1, we can see that most of ISI systems only focus on one dimension, such as information retrieval or timely information access. However, COPLINK provides an integrated solution for all three dimensions. Therefore, we use COPLINK to illustrate the evaluation of ISI systems.

Table 1: Brief Summary of Current ISI Systems in Law-Enforcement ^[23]

ISI System	Information Sharing	Information Retrieval	Timely Information Access
Time Analysis System (TAS)		Use visualization and time analysis to examine information
AICAMS		Aid in information retrieval by drawing upon human heuristics to investigate tasks
INFORTECH	A window based system to improve information sharing between agencies
Future Alert Contact Network (FALCON)			Receive request, monitor all incoming record relevant to the request, and notify officers by email or pager
Consolidated Criminal History Reporting System (CCHRS)		Information is organized by several dimensions for officers to query
COPLINK	Allow access to multiple databases through information sharing between various agencies	Provide several analysis tools, such as crime analysis and geo-mapping et al.	Allow access to multiple databases through one interface

Since ISI systems in law-enforcement are designed to support officers in job performance by information sharing, retrieval, and timely access, the evaluation of ISI systems focuses on:

• User perceptions of usability (System Level Evalaution)
  • Effectiveness (impact of system on job performance)
  • Ease of use (measures of effort required to complete a task)
  • Efficiency (speed of completing tasks)
• User acceptance (User Level Evaluation)
  • Examine the Technology Acceptance Model (TAM)
    • Users' behavioral intention, which is affected by a technology's usefulness and ease of use, determines whether users accept a technology or not.

COPLINK is an ISI system which provides law-enforcement officers timely information access and effective knowledge sharing within and beyond agency boundaries ^[22]. It was developed by the Artificial Intelligence Lab at the University of Arizona and jointly found by the National Institute of Justice and the National Science Foundation’s Digital Government Initiative. Current implementation of COPLINK has two distinct but complementary modules: COPLINK Connect and Detect.

• COPLINK Connect
  • Allow the officers to access data/information from multiple sources, above and beyond system heterogeneity or geographical constraints
• COPLINK Detect
  • Build upon COPLINK Connect and supports the officers to monitor or analyze a particular crime and its related activities via information integration and sharing

Table 1: Summary of COPLINK Evaluation

Study	Component	Evaluation Level	Data Collection Methods	Examine Factors
Hauck and Chen, 1999 [24]	COPLINK Concept Space	System	1.Documentation 2.Interviews 3.Direct Observation	1.Usefulness 2.Usability
Hauck, 1999 [25]	COPLINK DB	System	1.Questionnaires 2.Interviews 3.Direct Observation	1.Usability 1.1.Effectiveness 1.2.Ease of Use 1.3. Efficiency
Chen, 2000 [26]	COPLINK DB and Concept Space	System	COPLINK DB 1.Questionnaires COPLINK Concept Space 1.Documentation 2.Interviews 3.Direct Observation	COPLINK DB 1.Usability 1.1.Effectiveness 1.2.Ease of Use 1.3. Efficiency COPLINK Concept Space 1.Function 2.Design
Martinez and Moosman, 2001 [12]	COPLINK Detect	System	1.Verbal Reports 2.Search Notes 3.Transaction Log	1.Pattern and Frequency of Use 2.Manner of Use 3.System Performance 4.Users' Assessment
Chau et al., 2002 [27]	COPLINK Entity Extractor	System	1.Police Narrative Reports	1.Performance 1.1.Precision 1.2.Recall
Chen et al., 2002 [28]	COPLINK Connect	System	1.Interviews 2.Questionnaires	1.Effectiveness 2.Ease of Use 3.Efficiency
Xu et al., 2004 [29]	COPLINK Criminal Network Analysis	System	1.Field Study	1.Usefulness
Xianga et al., 2005 [30]	COPLINK Criminal Network Analysis	System	1.Experiment	1.Effectiveness 2.Efficiency
Hu et al., 2005 [22]	COPLINK	User	1.Questionnaire	1.Technology Context 1.1.Efficiency Gain 1.2.Perceived Usefulness 1.3. Perceived Ease of Use 2.Organizational Context 2.1.Subjective Norm 2.2.Availability 3.Attitude 4.Intention to Accept

In order to understand how well COPLINK interacts with officers, several user-oriented evaluation has been conducted as summarized in Table 2. Here we use three studies to illustrate their procedures and results.

• System Level Evaluation

In 2001, Martinez et al. ^[12] conducted a user study of COPLINK Detect during the stress test of COPLINK Detect 1.0 beta. In their study, they wanted to know (1) how the subjects used COPLINK, (2) the types of queries they performed, (3) successes they had in solving cases, and (4) their perceived ease of use of COPLINK. The data was collected through three methods: (1) verbal reports, (2) search notes, and (3) electronic transaction logs. In their analysis, they found that users performed 73% of sessions on Tuesdays, Wednesdays, and Thursdays. This finding was consistent with the fact that all detectives were scheduled to work on those three days. Throughout a day, COPLINK Detect sessions were fairly evenly spread with 40% before 12 PM and 60% after 12 PM. In the relative amount of sessions conducted by each job classification, crime analysts initiated the most sessions, followed by robbery, patrol, homicide and gangs, and adult sexual assault.

In 2002, Chen et al. ^[28] evaluated the usability of COPLINK Connect with RMS system. Three measures of usability are used: effectiveness (impact of system on job performance, productivity, and information accuracy), ease of use (measures of effort required to complete a task, ease of learning how to use the application, ability to navigate easily through the different screens, and satisfaction with the interaction), and efficiency (speed of completing tasks, organization of the information on the screens, ability to find information and the interface design itself). A group of 52 law-enforcement personnel was recruited to participate in this study. The data collection had three major stops. First, all subjects were asked to complete a pre-interaction questionnaire concerning their background and level of computer experience. Then participants were given a questionnaire targeting their perceived usability of the RMS system. In the final step, after a brief introduction of COPLINK Connect, subjects were asked to complete at least two search tasks by COPLINK Connect and then the usability of COPLINK Connect were collected through questionnaires and interviews.

In their results, the participants rated COPLINK significantly higher than they rated RMS in all three measures. In other words, the use of COPLINK improved users’ performance over the use of RMS. Besides, many participants rated the information found in COPLINK was more useful than that in RMS. It was interesting finding because all the information in COPLINK were from RMS.

• User Level Evaluation

In 2005, Hu et al. ^[22] investigated user acceptance of ISI technologies via COPLINK. They stated that user acceptance was a critical factor that influenced the success of an ISI technology. In their investigation, they adopted the Technology Acceptance Model (TAM) in which an individual’s decision on the acceptance of a technology is explained by his/her behavioral intention. They suggested that whether an officer accepts an ISI technology, such as COPLINK, depends on the technological, individual, and organizational contexts, the research model shown in Figure 1. The technological context is related to an officer’s perception of the technology, including perceived usefulness, ease of use, and efficiency gain. The individual context examines the attitude of an officer toward the use of the technology. In the organizational context, they studied subject norm and availability. Subject norm refers to an officer’s perception of significant referents’ opinions on his/her acceptance. Availability measures an officer’s perception of availability of computers for accessing the technology. The intention of an officer to accept COPLINK is influenced by subjective norm, availability, perceived usefulness, and attitude.

Hu et al. ^[22] used a self-administered survey to collect data and verify their research model. In their questionnaire, there were totally 23 items measured by a 7-point Likert scale. To reduce potential ceiling effects that might induce monotonous responses from subjects, they randomized the items in the questionnaire and negated half of the questions. They received 283 effective responses with a 68.9% response rate.

In their results, perceived usefulness appeared to be the most important factor for an officer to accept COPLINK. They suggested that an officer makes his/her decision on the acceptance of COPLINK from a utility perspective. They also found that an officer is not likely to consider an ISI technology to be useful because it is easy to use. Efficiency gain significantly influences an officer’s perception of usefulness. In other words, an officer would think COPLINK to be useful only if COPLINK helps him/her to increase his/her task performance. Besides, in their analysis, subjective norm had a negative direct effect on an officer’s intention to accept COPLINK. They suggested that this result may stem from the autonomy of law-enforcement officers and that administrators and technology professionals should cautiously anchor subjective norm from the utility aspect rather than complain acceptance decisions.

Dark Web: Authorship Analysis

More and more terrorists utilize World Wide Web to communicate with each other. How to effectively monitor those activities on WWW became a critical issue for ISI. Authorship analysis can assist such a monitor by automatically extracting linguistic features from online messages and evaluating stylistic details for patterns of terrorist communication ^[31]. However, authorship analysis faces two major challenges ^[31]:

1. Authorship analysis techniques are based on literary text, which is totally different from online communication.
2. The global nature of terrorist activities facilitates the analysis of multilingual content.

To solve these two issues, Abbasi and Chen (2005) ^[31] proposed a special multilingual model to identify Arabic messages and incorporated a complex message extraction component to extract a set of comprehensive set of features on online messages. They conducted an algorithm-level evaluation with experiments to test the efficacy of authorship identification techniques in an online setting. Their experimental objectives are:

1. Determine whether techniques proposed in authorship analysis could identify author writing in Arabic
2. How identification performance differed between English and Arabic
3. Identify the important feature difference between the English and Arabic groups and language models

Their test bed consisted of English and Arabic data sets extracted from Web forum messages. The features of the test bed are:

1. For both languages, they extracted 20 messages for each of 20 authors, resulting in a total of messages per language.
2. They extracted the English messages from a US forum belonging to the White Knights.
3. They extracted the Arabic data set from forum messages associated with the Palestinian Al-Aqsa Martyrs group.

In their experiments, they adopted two well-established classifiers: C4.5 and SVM. Four types of features, lexical, syntactic, structural, and content-specific features, are extracted for each message. After extracting the feature values, they categorized them into four features sets:

• F1: lexical features
• F2: lexical and syntactic features
• F3: lexical, syntactic, and structural features
• F4: lexical, syntactic, structural, and content-specific features

Therefore, each classifier was trained with these four features sets to see how those features affect the classifier. Figure 2 shows their main experimental results. From Figure 3, we can see that:

1. With the increase of message features, both classifiers had better performance in message classification.
2. SVM performed better than C4.5 in average.
3. English data set achieved better performance than Arabic data set

In 2006, Abbasi and Chen ^[32] further extended authorship analysis to visualization level called Writeprint analysis. They suggested that Writeprint visualization created unique writing style patterns that can be automatically identified in a manner similar to fingerprint biometric systems. In order to demonstrate that Writeprint is better suited for larger quantities of information, they conducted an algorithm level evaluation with experiments. Since the main focus of Writeprint analysis is on visualization, they created a Writeprint Comparison Algorithm to compare author/message writing style patterns by Writeprint. Their test bed consisted of data from three online forums:

• USENET forum consisting of software sales and distribution messages
• A Yahoo group forum for Al-Aqsa Martyrs
• A website forum for the White Knights

For each forum, 30 messages were collected for each of 10 authors.

They used SVM as benchmark to compare the performance of Writeprint. The result is shown in Figure 3. From Figure 3, we can see that Writeprint performed much better than SVM. It should also noted that Writeprint was not able to perform on messages shorter than 250 characters due to the need to maintain a minimum sliding window size and gather sufficient data points for the evaluation algorithm.

Conclusions and Future Directions

In this chapter, we have reviewed the potential uses of evaluation, as well as various evaluation methodologies. We have also studied the unique challenges faced in evaluation of various ISI technologies that address six critical areas of current focus identified by the DHS, namely: intelligence and warning, border and transportation security, domestic counter-terrorism, protecting critical infrastructure and key assets, defending against catastrophic terrorism, and emergency preparedness and response. We have seen that there are various standardized methodologies in place for evaluation up to the organization level. In the future, we believe there will be a strong focus on measures and mechanisms for broader evaluations at the country level.

References

1. ↑ IEEE International Conference on Intelligence and Security Informatics, ISI 2006, San Deigo, CA, USA, May 2006, Proceedings
2. ↑ Hauck, R. V. and Chen, H. (1999). Coplink: a Case of Intelligent Analysis and Knowledge Management Proceeding of the 20th International Conference on Information Systems Charlotte, North Carolina, United States
3. ↑ Department of the Air Force. (1974). A Guide for the Development of the Attitude Opinion Survey. Washington, D.C., pp.2.
4. ↑ Yin, R. K. (2002). Case study research. Design and methods. California, Sage Publications.
5. ↑ ^5.0 5.1 National Strategy for Homeland Security. Office of Homeland Security, 2002.
6. ↑ B. Marshall and H. Chen, "Using Importance Flooding to Identify Interesting Networks of Criminal Activity", IEEE International Conference on Intelligence and Security Informatics, ISI 2006, San Deigo, CA, USA, May 2006, pp. 14-25, Proceedings.
7. ↑ ^7.0 7.1 S. Kaza, Y. Wang, and H. Chen, "Suspect Vehicle Identification for Border Safety with Modified Mutual Information", IEEE International Conference on Intelligence and Security Informatics, ISI 2006, San Deigo, CA, USA, May 2006, pp. 308-318, Proceedings.
8. ↑ J. Xu, H. Chen, Y. Zhou, and J. Qin, "On the topology of the Dark Web of Terrorist Groups", IEEE International Conference on Intelligence and Security Informatics, ISI 2006, San Deigo, CA, USA, May 2006, pp. 367-376, Proceedings.
9. ↑ S. Anand, D. Madigan, R. Mammone, S. Pathak, and F. Roberts, "Experimental Analysis of Sequential Decision Making Algorithms for Port of Entry Inspection Procedures", IEEE International Conference on Intelligence and Security Informatics, ISI 2006, San Diego, CA, USA, May 2006, pp. 319-330, Proceedings.
10. ↑ H. Chen, "Intelligence and Security Informatics for International Security", Springer 2005.
11. ↑ G. Wang, H. Chen, and H. Atabakhsh, "Automatically detecting Deceptive Criminal Identities", Communications of the ACM, vol 47, pp. 71-76, 2004.
12. ↑ ^{12.0 12.1 12.2} Martinez, J. and Moosman, A. COPLINK Detect User Study Report. University of Arizona, AI Lab Technical Report, 2001.
13. ↑ Gattiker, U. E. (1984). Managing computer-based office information technology: A process for management. . Human factors in organizational design H. Hendrick and O. Brown. Amsterdam, Elsevier Science: 395-403.
14. ↑ Fishbein, M. and I. Ajzen (1975). Belief, attitude, intention, and behavior : an introduction to theory and research. Reading, MA, Addison-Wesley Publishing Co.
15. ↑ Ajzen, I. (1985). From intentions to actions: A theory of planned behavior. Action control: From cognition to behavior. J. Kuhl and J. Beckmann. New York, Springer-Verlag: 11-39.
16. ↑ Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes 50(2): 179-211.
17. ↑ Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3): 319-340.
18. ↑ Bagozzi, R. P., F. D. Davis and P. R. Warshaw (1992). Development and test of a theory of technological learning and usage. Human Relations 45(7): 659.
19. ↑ Davis, F. D., R. P. Bagozzi and P. R. Warshaw (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science 35(8): 982-1003.
20. ↑ Venkatesh, V., M. G. Morris, G. B. Davis and F. D. Davis (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly 27(3): 425-478.
21. ↑ Hair, J. F., R. E. Anderson, R. L. Tatham and W. C. Black (1995). Multivariate data analysis (4th ed.): with readings, Prentice-Hall, Inc.
22. ↑ ^{22.0 22.1 22.2 22.3 22.4} Hu, P., Lin, C., and Chen, H. (2005). "User Acceptance of Intelligence and Security Informatics Technology: A Study of COPLINK" Journal of The American Society for Information Science and Technology (JASIST) 56(3): 235-244.
23. ↑ ^23.0 23.1 Chen, H., Schroeder, J. et al. (2002). "COPLINK Connect: Information and Knowledge Management for Law Enforcement," Decision Support Systems, 34:271-285.
24. ↑ Hauck, R.V. and Chen, H. (1999). "COPLINK: A Case of Intelligent Analysis and Knowlege Management," Proceeding of the International Conference on Information Systems, 15-28.
25. ↑ Hauck, R.V. (1999). "COPLINK: Exploring Usability of a Multimedia Database Application for Law Enforcement," Report Prepared for NIJ Site Visit.
26. ↑ Chen, H. et al. (1999). "COPLINK: Information and Knowledge Management for Law Enforcement," Photonics East Conference, SPIE, Technologies for Law Enforcement, 2000.
27. ↑ Chau, M. et al., (2002). "Extracting Meaningful Entities from Police Narrative Reports," In Proceedings of the Second National Conference on Digital Government Reserach, May 2002.
28. ↑ ^28.0 28.1 Chen, H. et al., (2002). "COPLINK: Visualization and Collaboration for Law Enforcement," DSS, 34(3): 271-285.
29. ↑ Xu, J. et al., (2004). "Analyzing and Visualizing Criminal Network Dynamics: A Case Study," IEEE Conference on Intelligence and Security Informatics, Tucson, AZ, 2004.
30. ↑ Xianga, Y. et al., (2005). "Visualizing Criminal Relationships: Comparison of a Hyperbolic Tree and a Hierarchical List," DSS, 41: 69-83.
31. ↑ ^{31.0 31.1 31.2 31.3} Abbasi, A. and Chen, H. (2005). "Applying Authorship Analysis to Extremist-Group Web Forum Messages," The IEEE Computer Society, SEPTEMBER/OCTOBER, pp.67-75.
32. ↑ ^32.0 32.1 Abbasi, A. and Chen, H. (2006). "Visualizing Authorship for Identification," In the Proceedings of the Intelligence and Security Informatics: IEEE International Conference on Intelligence and Security Informatics, San Diego, CA, USA, May 23-24, 2006.