Foreword xxi

Preface xxiii

About the Authors xxxi

Acknowledgments xxxiii

PART ONE: Observability Methods 1

Chapter 1: Introduction to Observability Tools 3

1.1 Observability Tools 4

1.2 Drill-Down Analysis 7

1.3 About Part One 8

Chapter 2: CPUs 11

2.1 Tools for CPU Analysis 11

2.2 vmstat Tool 13

2.3 CPU Utilization 14

2.4 CPU Saturation 15

2.5 psrinfo Command 15

2.6 uptime Command 15

2.7 sar Command 16

2.8 Clock Tick Woes 19

2.9 mpstat Command 20

2.10 Who Is Using the CPU? 23

2.11 CPU Run Queue Latency 24

2.12 CPU Statistics Internals 26

2.13 Using DTrace to Explain Events from Performance Tools 29

2.14 DTrace Versions of runq-sz, %runocc 31

2.15 DTrace Probes for CPU States 33

Chapter 3: Processes 35

3.1 Tools for Process Analysis 35

3.2 Process Statistics Summary: prstat 37

3.3 Process Status: ps 41

3.4 Tools for Listing and Controlling Processes 45

3.5 Process Introspection Commands 47

3.6 Examining User-Level Locks in a Process 52

3.7 Tracing Processes 53

3.8 Java Processes 60

Chapter 4: Disk Behavior and Analysis 67

4.1 Terms for Disk Analysis 67

4.2 Random vs. Sequential I/O 69

4.3 Storage Arrays 70

4.4 Sector Zoning 71

4.5 Max I/O Size 72

4.6 iostat Utility 73

4.7 Disk Utilization 74

4.8 Disk Saturation 75

4.9 Disk Throughput 76

4.10 iostat Reference 76

4.11 Reading iostat 82

4.12 iostat Internals 85

4.13 sar -d 87

4.14 Trace Normal Form (TNF) Tracing for I/O 88

4.15 DTrace for I/O 88

4.16 Disk I/O Time 97

4.17 DTraceToolkit Commands 101

4.18 DTraceTazTool 108

Chapter 5: File Systems 109

5.1 Layers of File System and I/O 109

5.2 Observing Physical I/O 111

5.3 File System Latency 112

5.4 Causes of Read/Write File System Latency 114

5.5 Observing File System “Top End” Activity 118

5.6 File System Caches 119

5.7 NFS Statistics 133

Chapter 6: Memory 135

6.1 Tools for Memory Analysis 135

6.2 vmstat(1M) Command 137

6.3 Types of Paging 138

6.4 Physical Memory Allocation 142

6.5 Relieving Memory Pressure 144

6.6 Scan Rate as a Memory Health Indicator 146

6.7 Process Virtual and Resident Set Size 148

6.8 Using pmap to Inspect Process Memory Usage 149

6.9 Calculating Process Memory Usage with ps and pmap 150

6.10 Displaying Page-Size Information with pmap 153

6.11 Using DTrace for Memory Analysis 154

6.12 Obtaining Memory Kstats 157

6.13 Using the Perl Kstat API to Look at Memory Statistics 158

6.14 System Memory Allocation Kstats 158

6.15 Kernel Memory with kstat 160

6.16 System Paging Kstats 161

6.17 Observing MMU Performance Impact with trapstat 163

6.18 Swap Space 164

Chapter 7: Networks 173

7.1 Terms for Network Analysis 173

7.2 Packets Are Not Bytes 175

7.3 Network Utilization 176

7.4 Network Saturation 177

7.5 Network Errors 177

7.6 Misconfigurations 177

7.7 Systemwide Statistics 178

7.8 Per-Process Network Statistics 189

7.9 TCP Statistics 191

7.10 IP Statistics 196

7.11 ICMP Statistics 199

Chapter 8: Performance Counters 201

8.1 Introducing CPU Caches 203

8.2 cpustat Command 206

8.3 cputrack Command 215

8.4 busstat Command 216

Chapter 9: Kernel Monitoring 221

9.1 Tools for Kernel Monitoring 221

9.2 Profiling the Kernel and Drivers 222

9.3 Analyzing Kernel Locks 223

9.4 DTrace lockstat Provider 227

9.5 DTrace Kernel Profiling 229

9.6 Interrupt Statistics: vmstat -i 230

9.7 Interrupt Analysis: intrstat 230

PART TWO: Observability Infrastructure 233

Chapter 10: Dynamic Tracing 235

10.1 Introduction to DTrace 235

10.2 The Basics 236

10.3 Inspecting Java Applications with DTrace 257

10.4 DTrace Architecture 265

10.5 Summary 271

10.6 Probe Reference 271

10.7 MDB Reference 294

Chapter 11: Kernel Statistics 295

11.1 C-Level Kstat Interface 295

11.2 Command-Line Interface 307

11.3 Using Perl to Access kstats

11.4 Snooping a Program’s kstat Use with DTrace 317

11.5 Adding Statistics to the Solaris Kernel 317

11.6 Additional Information 323

PART THREE: Debugging 325

Chapter 12: The Modular Debugger 327

12.1 Introduction to the Modular Debugger 327

12.2 MDB Concepts 330

Chapter 13: An MDB Tutorial 335

13.1 Invoking MDB 335

13.2 MDB Command Syntax 336

13.3 Working with Debugging Targets 353

13.4 GDB-to-MDB Reference 357

13.5 dcmd and Walker Reference 359

Chapter 14: Debugging Kernels 367

14.1 Working with Kernel Cores 367

14.2 Examining User Process Stacks within a Kernel Image 382

14.3 Switching MDB to Debug a Specific Process 385

14.4 kmdb, the Kernel Modular Debugger 388

14.5 Kernel Built-In MDB dcmds 395

APPENDICES

Appendix A Tunables and Settings 401

Appendix B DTrace One-Liners 407

Appendix C Java DTrace Scripts 409

Appendix D Sample Perl Kstat Utilities 413

Bibliography 429

Index 433

Solaris  [PTG: PRENTICE HALL PROFESSIONAL]   (Computer Science)

"In total, the two books Solaris Performance and Tools & Solaris Internals reviewed here present a new level of knowledge about the internals of Solaris, what they do, how they behave, and how to analyze that behavior. The books are a must for developers, system programmers, and systems administrators who work with Solaris 8, 9, or 10. They are especially useful for users of Solaris 10 and OpenSolaris because of their exploration of the new tools in those releases. These books receive my highest recommendation. "—Peter Baer Galvin, Contributing Editor, Sys Admin Magazine

Richard McDougall is a Distinguished Engineer at Sun Microsystems, specializing in operating systems technology and systems performance.

Jim Mauro is a Senior Staff Engineer in the Performance, Architecture, and Applications Engineering group at Sun Microsystems, where his most recent efforts have focused on Solaris performance on Opteron platforms.

Brendan Gregg is a Solaris consultant and instructor teaching classes for Sun Microsystems across Australia and Asia. He is also an OpenSolaris contributor and community leader, and has written numerous software packages, including the DTraceToolkit.

"The Solaris™Internals volumes are simply the best and most comprehensive treatment of the Solaris (and OpenSolaris) Operating Environment. Any person using Solaris--in any capacity--would be remiss not to include these two new volumes in their personal library. With advanced observability tools in Solaris (like DTrace), you will more often find yourself in what was previously unchartable territory. Solaris™ Internals, Second Edition, provides us a fantastic means to be able to quickly understand these systems and further explore the Solaris architecture--especially when coupled with OpenSolaris source availability."

--Jarod Jenson, chief systems architect, Aeysis

"The Solaris™ Internals volumes by Jim Mauro and Richard McDougall must be on your bookshelf if you are interested in in-depth knowledge of Solaris operating system internals and architecture. As a senior Unix engineer for many years, I found the first edition of Solaris™ Internals the only fully comprehensive source for kernel developers, systems programmers, and systems administrators. The new second edition, with the companion performance and debugging book, is an indispensable reference set, containing many useful and practical explanations of Solaris and its underlying subsystems, including tools and methods for observing and analyzing any system running Solaris 10 or OpenSolaris."

--Marc Strahl, senior UNIX engineer

Solaris™ Performance and Tools provides comprehensive coverage of the powerful utilities bundled with Solaris 10 and OpenSolaris, including the Solaris Dynamic Tracing facility, DTrace, and the Modular Debugger, MDB. It provides a systematic approach to understanding performance and behavior, including:

• Analyzing CPU utilization by the kernel and applications, including reading and understanding hardware counters
• Process-level resource usage and profiling
• Disk IO behavior and analysis
• Memory usage at the system and application level
• Network performance
• Monitoring and profiling the kernel, and gathering kernel statistics
• Using DTrace providers and aggregations
• MDB commands and a complete MDB tutorial

The Solaris™ Internals volumes make a superb reference for anyone using Solaris 10 and OpenSolaris.

Pearson Higher Education offers special pricing when you choose to package your text with other student resources. If you're interested in creating a cost-saving package for your students contact your Pearson Higher Education representative.