Exploring Splunk

Table of Contents Preface About This Book i What’s In This Book? ii Conventions ii Acknowledgments iii PART I: EXPLORING SPLUNK 1 The Story of Splunk Splunk to the Rescue in the Datacenter 3 Splunk to the Rescue ...

Table of Contents Preface About This Book i What’s In This Book? ii Conventions ii Acknowledgments iii PART I: EXPLORING SPLUNK 1 The Story of Splunk Splunk to the Rescue in the Datacenter 3 Splunk to the Rescue in the Marketing Department 4 Approaching Splunk 5 Splunk: The Company and the Concept 7 How Splunk Mastered Machine Data in the Datacenter 8 Operational Intelligence 9 Operational Intelligence at Work 11 2 Getting Data In Machine Data Basics 13 Types of Data Splunk Can Read 15 Splunk Data Sources 15 Downloading, Installing, and Starting Splunk 15 Bringing Data in for Indexing 17 Understanding How Splunk Indexes Data 18 3 Searching with Splunk The Search Dashboard 23 SPL™: Search Processing Language 27 Pipes 27 Implied AND 28 top user 28 fields – percent 28 The search Command 29 Tips for Using the search Command 30 Subsearches 30 4 SPL: Search Processing Language Sorting Results 33 sort 33 Filtering Results 35 where 35 dedup 36 head 38 Grouping Results 39 transaction 39 Reporting Results 41 top 41 stats 43 chart 45 timechart 47 Filtering, Modifying, and Adding Fields 48 fields 49 replace 50 eval 51 rex 52 lookup 53 5 Enriching Your Data Using Splunk to Understand Data 55 Identifying Fields: Looking at the Pieces of the Puzzle 56 Exploring the Data to Understand its Scope 58 Preparing for Reporting and Aggregation 60 Visualizing Data 65 Creating Visualizations 65 Creating Dashboards 67 Creating Alerts 68 Creating Alerts through a Wizard 68 Tuning Alerts Using Manager 71 Customizing Actions for Alerting 74 The Alerts Manager 74 PART II: RECIPES 6 Recipes for Monitoring and Alerting Monitoring Recipes 79 Monitoring Concurrent Users 79 Monitoring Inactive Hosts 80 Reporting on Categorized Data 81 Comparing Today’s Top Values to Last Month’s 82 Finding Metrics That Fell by 10% in an Hour 84 Charting Week Over Week Results 85 Identify Spikes in Your Data 86 Compacting Time-Based Charting 88 Reporting on Fields Inside XML or JSON 88 Extracting Fields from an Event 89 Alerting Recipes 90 Alerting by Email when a Server Hits a Predefined Load 90 Alerting When Web Server Performance Slows 91 Shutting Down Unneeded EC2 Instances 91 Converting Monitoring to Alerting 92 7 Grouping Events Introduction 95 Recipes 97 Unifying Field Names 97 Finding Incomplete Transactions 97 Calculating Times within Transactions 99 Finding the Latest Events 100 Finding Repeated Events 101 Time Between Transactions 102 Finding Specific Transactions 104 Finding Events Near Other Events 107 Finding Events After Events 108 Grouping Groups 109 8 Lookup Tables Introduction 113 lookup 113 inputlookup 113 outputlookup 113 Further Reading 114 Recipes 114 Setting Default Lookup Values 114 Using Reverse Lookups 114 Using a Two-Tiered Lookup 116 Using Multistep Lookups 116 Creating a Lookup Table from Search Results 117 Appending Results to Lookup Tables 117 Using Massive Lookup Tables 118 Comparing Results to Lookup Values 120 Controlling Lookup Matches 122 Matching IPs 122 Matching with Wildcards 123 Appendix A: Machine Data Basics Application Logs 126 Web Access Logs 126 Web Proxy Logs 127 Call Detail Records 127 Clickstream Data 127 Message Queuing 128 Packet Data 128 Configuration Files 128 Database Audit Logs and Tables 128 File System Audit Logs 128 Management and Logging APIs 129 OS Metrics, Status, and Diagnostic Commands 129 Other Machine Data Sources 129 Appendix B: Case Sensitivity Appendix C: Top Commands Appendix D: Top Resources Appendix E: Splunk Quick Reference Guide CONCEPTS 137 Overview 137 Events 137 Sources and Sourcetypes 138 Hosts 138 Indexes 138 Fields 138 Tags 138 Event Types 139 Reports and Dashboards 139 Apps 139 Permissions/Users/Roles 139 Transactions 139 Forwarder/Indexer 140 SPL 140 Subsearches 141 Relative Time Modifiers 141 COMMON SEARCH COMMANDS 142 Optimizing Searches 142 SEARCH EXAMPLES 143 EVAL FUNCTIONS 146 COMMON STATS FUNCTIONS 151 REGULAR EXPRESSIONS 152 COMMON SPLUNK STRPTIME FUNCTIONS 153 i Preface Splunk Enterprise Software (“Splunk”) is probably the single most powerful tool for searching and exploring data that you will ever encounter. We wrote this book to provide an introduction to Splunk and all it can do. This book also serves as a jumping off point for how to get creative with Splunk. Splunk is often used by system administrators, network administrators, and security gurus, but its use is not restricted to these audiences. There is a great deal of business value hidden away in corporate data that Splunk can liberate. This book is designed to reach beyond the typical techie reader of O’Reilly books to marketing quants as well as everyone interested in the topics of Big Data and Operational Intelligence. About This Book The central goal of this book is to help you rapidly understand what Splunk is and how it can help you. It accomplishes this by teaching you the most important parts of Splunk’s Search Processing Language (SPL™). Splunk can help technologists and businesspeople in many ways. Don’t expect to learn Splunk all at once. Splunk is more like a Swiss army knife, a simple tool that can do many powerful things. Now the question becomes: How can this book help? The short answer is by quickly giving you a sense of what you can do with Splunk and pointers on where to learn more. But isn’t there already a lot of Splunk documentation? Yes: • If you check out http://docs.splunk.com, you will find many manuals with detailed explanations of the machinery of Splunk. • If you check out http://splunkbase.com, you will find a searchable database of questions and answers. This sort of content is invaluable when you know a bit about Splunk and are trying to solve common problems. This book falls in between these two levels of documentation. It offers a basic understanding of Splunk’s most important