The tests are designed to be run with the printer switched off, so that Write IRPs do not complete straightaway. WdmIoCancel opens the WdmIo device for overlapped I/O, using the GetDeviceViaInterface01 function. It then issues two overlapped Write requests and does not wait for either of them to complete. It then exits straightaway. This behavior is designed to test the IRP cancelling.
Figure 16.2 shows the DebugPrint output that demonstrates that IRP cancelling works correctly. At points (1) in the diagram, one of the IOCTL IRPs has been issued. They are passed to WdmIoStartIo, which processes them immediately. At point (2), the first Write IRP is issued. At point (3), WdmIoStartIo starts processing this first Write. At point (4), the second Write IRP is issued. As the first Write is still in progress (the printer being off), this second Write IRP is queued.
Point (5) is where it gets interesting. WdmIoCancel has just exited. The I/O Manager tries to cancel all pending IRPs. It calls the cancel routine of the second Write. WdmIoCancelIrp finds that it is not the current IRP. It therefore removes it from the StartIo queue and completes the IRP with a cancelled status.
The I/O Manager then calls the cancel routine of the first Write IRP at point (6). WdmIoCancelIrp finds that this IRP is the current IRP. In this design, WdmIoCancelIrp simply exits. In due course, the Timeout1s time-out routine runs. Timeout1s finds that the IRP Cancel flag is set. The WdmIoDpcForIsr routine is called to complete the IRP with a cancelled status.
Finally, the Cleanup IRP is issued at point (7). However, this finds that it has no work to do as the device queue is empty and there is no interrupt driven I/O in progress.
If you look carefully at the DebugPrint output, you will notice that the IRP pointer for the first four IRPs issued is the same, 0xC8548180. The I/O Manager is obviously reusing the IRP structures from its pool of available IRPs. The second write needs another IRP structure, which is at 0xC17C3E00.
Figure 16.2 Cleanup handling DebugPrint output
If you remove the comments around the CloseHandle call at the end of WdmIoCancel, you will be able to test the Cleanup IRP handling correctly. In this case, the IRP cancel routines will not be called. Instead, the Cleanup IRP handler removes the second Write IRP from the device queue. It then goes on to cancel the current IRP (i.e., the first Write).
Finally, the WdmIoCancel code contains a commented out CancelIO call after the second Write is issued. Uncomment this and recompile to check that this cancels the Write IRPs correctly.
Supplemental Device Queues
You can set up your own device queues, if need be. They are usually termed supplemental device queues to differentiate them from the standard device queue for StartIo requests. Supplemental device queues might be used in the following situations.
1. A full duplex driver will need a separate queue of IRPs for each direction of transfer. This lets a Read IRP and a Write IRP be processed at the same time. Interrupt routines need to be designed carefully to ensure that interrupts are handled correctly. The standard device queue could be used for Write IRPs. A supplemental device queue could then be used for the Read IRPs.
2. The I/O Manager Cancel spin lock must be acquired whenever there is an operation on any standard device queue. This is a considerable bottleneck for the whole system. Supplemental device queues can be designed to operate with a much-reduced use of the Cancel spin lock. The example in the SRCGENERALCANCEL directory of the Windows 2000 DDK shows this technique in action.
3. When a Plug and Play device is stopped for resource reassignment, all I/O IRPs should be queued[41]. The IRPs should be run when the device is restarted with new resource assignments. All the Plug and Play drivers in this book avoid this complication by not letting a device be stopped if there are any open handles. These IRPs can be queued in the standard device queue. However, you may deem that a supplemental device queue is needed instead of, or in addition to, the standard device queue.
These queued IRPs may not need to be processed serially. When the device is started, all the IRPs are eligible for running straightaway. To achieve this, system worker thread work items may be scheduled to process all the IRPs. System worker threads are covered briefly in Chapter 14.
Do not forget that any queued IRPs need to be cancellable. Adding a queue is not necessarily a quick operation.
Implementing a Supplemental Device Queue Let's see how to set up and use a supplemental device queue. This queue will perform exactly the same function as the kernel-managed device queue for StartIo requests and so shows what must be happening behind the scenes in the kernel IoStartPacket call, etc. The aim is to serialize processing of IRPs in the AbcStartIo2 function. An IRP is put in the queue for processing using the AbcStartPacket2 routine. AbcStartNextPacket2 starts the processing of the next packet. Listing 16.8 shows the code for these routines.
These routines also handle the cancelling of IRPs correctly using the Cancel spin lock and cancel routines.
Declare a KDEVICE_QUEUE field somewhere in nonpaged memory. In this case, it is in the device extension in a field called AbcIrpQueue. Initialize it when the device object has been created by calling KeInitializeDeviceQueue at PASSIVE_LEVEL
KeInitializeDeviceQueue(&dx->AbcIrpQueue);
The device extension also needs a current IRP pointer, here in a field called AbcCurrentIrp.
Inserting IRPs into the Queue
AbcStartPacket2 uses KeInsertDeviceQueue to insert the IRP into the device queue. Note that KeInsertDeviceQueue must be called at DISPATCH_LEVEL IRQL. The call to IoAcquireCancelSpinLock in AbcStartPacket2 does this. Subsequent calls to IoReleaseCancelSpinLock return to the old IRQL.
If the device queue is empty, KeInsertDeviceQueue returns FALSE, but it sets the queue into a busy state. Subsequent insertion attempts return TRUE. In AbcStartPacket2, if KeInsertDeviceQueue returns FALSE, then the IRP is passed for processing in AbcStartIo2 straightaway. First, the IRP pointer is stored as the current IRP in the AbcCurrentIrp field in the device extension. Then the Cancel spin lock is released, returning the IRQL to its old level. StartIo routines must run at DISPATCH_LEVEL, so calls to KeRaiseIrql and KeLowerIrql are put around the call to AbcStartIo2.
If KeInsertDeviceQueue returns TRUE, the IRP has been put in the device queue. In this case, no further processing is required in AbcStartPacket2, apart from releasing the Cancel spin lock.
IRP Processing
The AbcStartIo2 function initially does some IRP cancel checks. Then, it sets about processing the IRP. Eventually, it completes the IRP and calls AbcStartNextPacket2 to begin processing the next IRP. The call to AbcStartNextPacket2 can occur in a different routine (e.g., after a hardware interaction).
Starting the Next IRP
AbcStartNextPacket2's job is to see if there are any more IRPs queued for processing. If there are, it dequeues one and sends it for processing in AbcStartIo2. Note that the calls to AbcStartNextPacket2 and AbcStartIo2 are recursive, which I trust will not be a problem. AbcStartNextPacket2 must be called at DISPATCH_LEVEL IRQL.
AbcStartNextPacket2 calls KeRemoveDeviceQueue to try to remove an IRP from the device queue. KeRemoveDeviceQueue must be run at DISPATCH_LEVEL IRQL. When running an IRP device queue, it is usual to call KeRemoveDeviceQueue while holding the Cancel spin lock. The call to IoAcquireCancelSpinLock raises the IRQL to the correct level, as before.
If KeRemoveDeviceQueue returns NULL, no IRPs are queued. The AbcCurrentIrp field is reset to NULL and the Cancel spin lock is released.
If KeRemoveDeviceQueue returns non-NULL then the return value represents the device queue entry in the IRP structure. Get the actual IRP pointer using the CONTAINING_RECORD macro. Store this in AbcCurrentIrp, release the Cancel spin lock, and call AbcStartIo2.
Listing 16.8 Supplemental device queue routines
VOID AbcStartPacket2(IN PDEVICE_OBJECT DeviceObject, IN PIRP Irp) {
PABC_DEVICE_EXTENSION dx = (PABC_DEVICE_EXTENSION)fdo->DeviceExtension;
KIRQL OldIrql;
IoAcquireCancelSpinLock(&OldIrql);
IoSetCancelRoutine(Irp, AbcCancelIrp);
if (KeInsertDeviceQueue(&dx->AbcIrpQueue, &Irp->Tail.Overlay.DeviceQueueEntry)) IoReleaseCancelSpinLock(OldIrql);
else {
DeviceExtension->AbcCurrentIrp = Irp;
IoReleaseCancelSpinLock(OldIrql);
KeRaiseIrql(DISPATCH_LEVEL, &OldIrql);
AbcStartIo2(DeviceObject, Irp);
KeLowerIrql(OldIrql);
}
}
VOID AbcStartNextPacket2(IN PDEVICE_OBJECT DeviceObject) {
PABC_DEVICE_EXTENSION dx = (PABC_DEVICE_EXTENSION)fdo->DeviceExtension;
KIRQL OldIrql;
IoAcquireCancelSpinLock(&OldIrql);
PKDEVICE_QUEUE_ENTRY QueueEntry = KeRemoveDeviceQueue(&dx->AbcIrpQueue);
