You may wish to read the VERY LAST paragraph first!
On Thu, 06 Jan 2005 04:02:05 -0700, Ritchie Annand
<
XXXX@XXXXX.COM>writes:
Quote
How bare-bones do you need to run the
machine you develop on?
Well, the development machine is of course, not the target machine. As
far as the target machine is concerned...
As little or as much as you want. Realistically though, you don't
normally want to have loads of stuff running when you're testing,
unless you're stress or load testing. I simply have a machine that
dual boots 2K and XP SP2, with the Platform SDK, DirectX DSK, and a
copy of WinDbg installed.
Debugging stuff between user & kernel mode gets tricky though: you end
up running the user mode process in WinDbg on the target machine, just
like an ordinary de{*word*81}, and then you kernel debug the box remotely.
However, you have to remember that if you've stopped the machine in
the kernel de{*word*81}, trying to play with the user mode de{*word*81} is not
going to be entirely successful!
Quote
How much of the XP kernel comes from the NT/2K line, I often puzzle.
All of it. The 2K and XP kernel are surprisingly similar - the latter
is basically the former with an extra few years of development thrown
in.
As far as driver development goes, the NT kernel is a slightly
different kettle of fish, because it doesn't support the newest driver
models.
Quote
I
puzzle mostly because running Delphi 8 can still run me out of Windows
resources (!), which it doesn't seem to under 2K :)
Oh really? That *does* surprise me. I would suspect a bug somewhere.
Quote
Apart from that, I am left to negotiate things on a case-by-case basis, e.g.
in a remoting session, it is the responsibility of the client to ask for
termination only after it knows it has done all its tasks, and it is the
server's to terminate only upon request but to delay the reply until its
own tasks are complete.
Experience in designing comms protocols is astually a great benefit in
this regard, because all the same techniques apply. In fact, you can
view messages in threads the same way as messages between machines -
in terms of synchronisation design, there's no real difference, except
that communication is more likely to be guaranteed.
Quote
Actually, we've got a bit of a naming conundrum, where the pieces are named
for their historical/technical purpose, not necessarily their function.
It always happens. Every successful project undergoes at least 3
changes of direction, and 4 different sets of engineering or marketing
names.
Quote
The inventor of the What's-In-My-Head Projector will be rich! :)
It's technically possible now - it is been done with a cat. The thing
is, humans don't take kindly to have the back of their skull levered
off, and thousand of electroned implanted in their V1 visual cortex.
Quote
That's sweet :) I wish there was something more robust than BoundsChecker
like that for user mode apps :)
Well I think BC isn't too bad - it is just not as comprehensive as it
could be. There's nothing to stop it hooking the process / thread
Win32API calls.
Quote
Mind you, I will bet it is a piece of code they took from that, put it in SP2
that made my laptop unbootable with an unrecoverable "Unsafe Driver" BSOD
that required an OS reinstallation. Grrr :)
Shouldn't require a reinstall - but having said that, you'd have to
know how to how to poke around with the registry after mounting the
drive on another machine - second thoughts ... perhaps reinstalling is
better!
Quote
It sure sounds like you rose to the challenge :) On multiple fronts too, no
doubt. Attacking a multithreaded kernel-mode driver isn't something the
garden variety CS grad is going to be a good fit for ;)
I suspect it is probably only really well suited to people with
Asperger's syndrome - the rest of us are just too fallible!
Quote
That makes sense. There's a whole lot of prefix and postfix code just to
start playing, and a whole lot of skeleton code to fill in, no doubt :)
For a device that handles PnP and Power management properly, about
2,000 lines of boilerplate code.
Quote
Are the samples still straight-C? Or is there an inkling of OOP set up
already - if not in the performance-sensitive pieces, then at least in the
performance-insensitive pieces?
There are plenty of OOP inklings, and at lot of the code is object
based. C++ is used in the kernel, and some of the more experienced
driver writers write the drivers in C++.
However, whilst C++ is not forbidden, it is not officially supported
either. There are many areas in the language where you have to know
how the compiler implements a feature before you can use it safely in
the kernel. Huge swathes of the standard library and STL are not
suitable for use in the kernel. C++ Exception handling mechanisms
likewise are not suitable. There are issues with the compiler using a
lot more stack in C++ - and you only have 16 of kernel stack to play
with.
Quote
I'm scared :)
I'm scared you're asking questions ;-)
Quote
So it looks somewhat like overlapped I/O in files or sockets?
Very much so. Take a look at this bit of code (first random example I
could find on the web):
www-ivs.cs.uni-magdeburg.de/~trikalio/opencbm/PortAccess_8c-source.html
And particularly the function: "ParPortIoctlInOut". This function
builds an IOCTL, sends it down the stack, and if it doesn't complete
immediately, it waits on an event. Looks like overlapped I/O yeah?
This is of course, exactly what the synchronous versions of ReadFile /
WriteFile and DeviceIoControl do internally at the top levels of the
kernel to turn asynchronous I/O into synchronous I/O.
Quote
>1. Complete it now.
>2. Pass it down the stack now.
Pass it down the stack? Ah, just had to Google on that - that is a fairly
overloaded word, that. (here I was thinking "uh, down the stack? isn't that
just a function call?" :) So this is the 'device stack' - would it be fair
to make a rough analogy to mouse clicks? If it is irrelevant for a child
object, then it keeps getting passed on until someone is interested?
Yes and no. In principle, that is what happens. In practice, (alas)
it's more complicated. The driver stack does actually bear a close
resemblance to the call stack on the way down, but not typically on
the way back up.
Let's imagine that we have 3 drivers in a stack. The top one (A)
passes down to the middle (B), and the middle passes down to the
bottom(C). Control works basically like this:
The IO Manager calls IOCallDriver for driver A
Driver A entry point starts.
Driver A calls IoSkipCurrentIrpStackLocation and returns
Driver A calls IoCallDriver for B
IoCallDriver in the IO manager calls entry point for B
Driver B calls IoSkipCurrentIrpStackLocation and returns
Driver B calls IoCallDriver for C
Driver C calls IoCompleteRequest
Driver C returns value from IoCompleteRequest
Driver B returns value from IoCallDriver
Driver A returns value from IoCallDriver
IoManager handles the return code.
Seems simple, huh? Well it is, until you notice that drivers may want
to process the IRP on its way back up the stack when it is returning
results. In order to this you need to register a completion routine.
The completion routines gets stored in the IRP stack allocated. Now,
depending on exactly what the drivers want to do, and whether they
have to wait, the completion routines could get called synchronously,
or asynchronously.
When I/O CompleteRequest is called, it calls the various completion
routines for the drivers in the stack, from bottom to top. In our
example above, we'd have something like:
...
Driver B calls IoCallDriver for C
Driver C calls IoCompleteRequest
IoManager calls Driver C's completion routine.
C's routine returns
IoManager calls Driver B's completion routine.
B's routine returns
IoManager calls Driver A's completion routine.
A's routine returns
IoCompleteRequest returns
Driver C returns value from IoCompleteRequest
Driver B returns value from IoCallDriver
...
Now of course, the IRP may not be completed synchronously. It's
possible for the call stack to unwind because the IRP has been set to
STATUS_PENDING before IoCompleteRequest has been called. In that case,
the IRP will be stored somewhere, and IoCompleteRequest for that IRP
may be called by the bottom driver (C) at some later point (and in a
different thread context!).
Of course, The event setting done by the IoManager in response to
IoBuildDeviceIoControlRequest is going to look remarkably like what's
done by the IoCompleteRequest if all the completion routines have
indicated that they've finished ;-)
Quote
Are there any major differences in how you pend it in those two
circumstances?
There may be, and I would have to look into the books to find out. It
depends what IRQL you're called at. In most cases, you don't want to
block, or pass the IRP down, so you simply stash the IRP away in a
queue, call IoMarkIRPPending and return STATUS_PENDING (or something
like that!).
Quote
IRP = I/O Request Packet, gotcha.
Yep - you got it.
Uh oh - I have given you another post with lots of new function names
that you can google for ;-)
Quote
So this would be a callback set by DriverObject.MajorFunction[IRP_MJ_READ]
:= HandleNewIrp; (approximately?)
Exactly that. You set that pointer in your driver object, and then
pass your driver object into IoCreateDevice, and that hooks it up. Of
course, if the driver verifier is running, it also inserts an extra
bit of hooking in to do some checking.
Quote
Is this a queue of your own devising, or is there a standard IRP queue you
can use?
It could be either. XP now has Cancel safe queues, so you'd use those,
but many existing drivers implement their own queues. When I say
"their own", they'll use interlocked list functions like
"ExInterlockedInsertHeadList" which allow you to maintain a
thread-safe doubly linked list, and then they'll later their own IRP
handling on top of that.
Quote
Process as many as possible... within a given timeframe? Or is the check a
little rougher than that?
Well, given the model I showed you above, you normally deal with one
IRP at a time. There are two cases:
Pending an IRP in an indeterminate thread context (which is most of
the time): one does so in order to send another IRP off eg: "To handle
this request to enable my device, I as the function driver, need to
send a whole load of IRP's to the PCI bus driver", and once the last
completion routine for the IRP you sent off has completed, you'll
complete the IRP you were given. Typically, you complete as fast as
possible, because you're not sure which thread context, and hence,
whose time you're stealing - especially in a completion routine.
The other situation is when you're in a known thread context. This is
common for me, when, for example, I can complete read requests on my
device when a new video frame comes in. In such situations, I queue
the read requests up. I then have a worker thread that waits on an
event. An interrupt comes from the device saying I have a new video
frame, and in that, I can set the event, and kick the worker thread
off. My worker thread can then fill out the data in the read requests,
and call IoCompleteRequest for those IRP's at its leisure.
It's actually more complicated than that (surprise surprise) because
you can not play with kernel dispatcher objects (events) at interrupt
time, so you have to schedule a DPC (deferred procedure call), and
theye are yet more syncornisation mechanisms (like interrupt spin
locks) that you might also have to deal with.
Easy, aint it? ;-)
Quote
So ownership = allowed to process?
Yes - ownership means that you're allowed to manipulate the "head" end
of the queue. This is pretty easy: just implement a queue with a lock
round it, and then implement ownership by checking the ThreadID.
Quote
Isn't the process of adding an item to a queue a separate concern from
taking ownership and processing?
Adding to the tail is a separate concern.
Quote
Just wondering how that would make for a subtle race condition :) I will trust
you on this point, though!
Well, you can protect all the calls with a single lock, just like any
shared datastructure - you just need inside that to Check ThreadID's,
and implement a suitable "ClaimOwnership" and "ReleaseOwnership"
function.
Of course, if you wanted a "WaitForOwnership" function, then you're
back to allocating another event for "CurrentlyOwned".
Quote
Now *that's* a situation I have encountered. Was a good 1 1/2 hours of
whiteboarding and "could we break *this*?" with a coworker. I should dig up
my research notes. I think we ended up with two flags and a lot of jumping
in and out of a critical section.
Yep - makes sense. Depending on how you wanted to do things, you might
even have ended up with a "try again" flag.
Quote
I've had subtle errors manage to skip items (at about a 0.25% rate) I put on
a queue with a single line-of-Pascal bad exposure.
In my current Delphi app, I actually have the opposite race condition:
A queue has Get and Put functions, and because of other constraints, I
couldn't do strict counting with semaphores, and wanted instead to
have a "BlockWhileEmpty" function. As it turns out, you can do:
BlockWhileEmpty
NewItem := GetItem;
However, these aren't atomic (you've left & re-entered the lock), so
you have to be prepared for someone else to have got there first:
loop
BlockWhileEmpty;
NewItem := GetItem();
if NewItem ...
endloop
And you just need to be aware that if a lot of threads are trying to
take stuff out at once, they may zip round the loop a small number of
times before actually getting an item. Actually - the block is in the
conditional, cos it is more efficient, but that is another story.
Quote
That's a sensible model for backpedalling. Seems a lot like parsing or
run-of-the-mill packet detection in that way... except for the concurrency
part. Pop, pop, pop... end of queue... dang, I need something more - better
wait for the next IRP... stuff, stuff, stuff.
Yes - that is it, and of course you need ownership to make sure no-one
nicks anything else off the queue before you decide to backpedal -
which would result in requests in the wrong order.
Quote
It would be an easier problem if the cancellation were simply a queued-up
IRP itself. I have thought about a similar solution to some of my troubles,
but in many cases, you just can not trust it to get that far; sometimes the
cause of the cancellation is something that makes the task wait, or fails
to spur the task on.
Would be nice - but the problem is that you can forward IRP's onto
other stacks, duplicate them, hide them away, and do all sorts of
stuff, and the problem is that once you're actually doing stuff in a
tree of devices, you then start having to duplicate and forward
cancellation requests to multiple places. Whilst it *could* work fine,
in that you'd then have a set of rules that dictated how to handle
farming the cancellation IRP's out (and get them back...) it is then
another huge set of cancellation completion routines etc etc ... and
the hair on your head starts falling out - and in that respect, a
central registration system seems to work better.
In my current Delphi app that does a lot of queueing, cancellation is
fairly immediate, and via pointer. Instead of flushing through the
whole way, cancelled packets typically get disposed of next time they
get to a queue of some description.
Quote
How doe the cancellation 'notification' arrive? Is it a held-onto state
somewhere?
There's both some cancellation state, and a routine that gets invoked
at cancellation time - as you've probably discovered.
Quote
So, just before you release ownership of the queue, you decide that you need
more information, so you do your backpedal, and then drop out. Would the
driver be effectively "suspended" at this point if a cancellation
interceded?
At this point, you need to synchronise the cancel handling and the
lock on the queue, hence cancel locks. If a cancellation intervenes,
then you need to set a cancelled flag (whilst not actually putting it
into or taking it out of the queue) such that if you were going to
backpedal it so that you put it in the queue, instead of doing that,
you free it.
Quote
>Due to the way the I/O manager deals with calls down the driver stack
>(which is WAAAY too complicated for me to discuss in detail here),
Oh well, got there anyway.
Quote
Got a link to any diagrams? :)
I *wish*. This is a good page:
msdn.microsoft.com/library/default.asp
But unfortunately, is not that hot on useful diagrams.
By the way, this is, IMHO the best book I have seen on the subject, when
it comes to real understanding:
www.amazon.com/exec/obidos/tg/detail/-/0735618038/qid=1105056486/sr=8-1/ref=sr_8_xs_ap_i1_xgl14/103-2293265-6327068?v=glance&s=books&n=507846
It does some to have received a couple of bad reviews, but it looks to
me like the people that wrote those reviews were looking for a "bullet
point explanation", instead of long prose ... unfortunately, it seems
that you need to ponder and wade through the prose in order to
understand enough to really "get it". For more concise info, MSDN
seems to do fine.
Even better - the sample code contains a GREAT library module which
actually implements cancel-safe queues for you.
Unfortunately, the drivers I am working on got written before the book
came out ;-)
Quote
>c) Ahhh. This is where it gets very interesting. In order to deal with
>this, we need to actually make each queue be *two* queues: a "waiting
>processing" queue, and a "currently processing" queue. If you do this,
>you can then deal with cancellation requests which occur whilst
>something is being processed - e.g. set a flag for that item. If you
>then subsequently backpedal the item, you can take note of its
>cancelled status, and bin it instead.
Do the items actually get cancelled *while* you're processing them, or is
this check entirely up to you? As long as an IRP is marked "cancelled"
properly, you're okay? Is there any lock you can hold onto to make sure the
cancel status can not be updated while you're looking at it?
The cancellation is actually done by setting a flag, and your own
cancel handler for the IRP being called. I need to look more carefully
at cancel locks, because as you can see, the issue of synchronisation
boils down to synchronisation between three things.
- The cancelled flag / status of the IRP.
- The execution of the cancellation handler.
- The synchronisation of cancellation with queuing operations.
It's kinda complicated, but this section deals with it - as you can
see, it spans several pages.
msdn.microsoft.com/library/default.asp
This is, of course, the kind of stuff where once you have your
cancel-safe queueing library, you leave it at that!
Quote
Ah, so there IS a global cancel lock :)
Yes, but it is use is deprecated. The recommended method nowadays is
for drivers to maintain their own cancel locks: the system cancel lock
turned out to be a fairly serious bottleneck.
Quote
>Those who feel the need for a bit of MSDN {*word*36} can consult:
>tinyurl.com/6dbxj
Not enough {*word*36} ;)
Well that is okay, I have included plenty more for you :-P
Quote
I've got an odd-job other case called a "shared queue". This is the case
where multiple processing objects all need to process the same queue items,
locking constraints prevent them from simply using multiple queues, but you
want it to otherwise *behave* like multiple queues. I wish I could remember
more of the details - it was an interesting piece of work, and cleaned up
the queue's "tail" as the consumers consumed :)
Sounds rather similar to what we've been discussing here.
Quote
Doing non-blocking async, I find I can cut a number of things short simply
by getting rid of the objects, because they aren't "actively waiting" (the
callback that would start them back up simply goes in the trash). The very
last item that is actually being processed occasionally holds up the game,
but I tend to ensure that those items *can't* wait forever (though what am
I going to do if network file access goes off into la-la land?)
Yep - my queueing systems seem similar to this: ditching dtuff in
queues is relatively easy - with the exception that you might have to
inform other components that are waiting for them to "pop out"
somewhere else having completed, but the tricky bit is cancelling
things that aren't in queues and are actually in progress.
Quote
Mind you, I think you can avoid some of the tribulations if you can
completely encapsulate a "processing unit" inside a single item in a queue
- this would likely cut down the number of cases to something more easily
manageable :)
There are areas in the video capture drivers I am working on which do
this: there's only one queue, and no passing from one queue to the
next, and it then becomes relatively simple (i.e. "only" a CS final's
exam question of difficulty).
Quote
Hmmm... I know we're nowhere close to a Guiness World Record of longest
post, but we're trying :) Feel free to ignore some stuff on the next round
- these are taking a long time to reply to *grin*
No problem. I would meant to dash off a quick reply, and then found myself
getting rather engrossed - and wanting to check/recheck the details on
how it worked. I suspect this post may be even longer, but I won't
know the line count until I have sent it!
No need to trawl thru all my points if you don't want to - because
it's at that point where the discussion of minutiae could get scary.
What I think gives you a nice flavour of the issues however is this
document (which is readable narrative, not techie) which explains why
there's _yet_another_ driver model coming out soon, WDF, which
apparently will enable us to get rid of many of these woes - and he
nicely describes some of the challenges faced:
download.microsoft.com/download/e/b/a/eba1050f-a31d-436b-9281-92cdfeae4b45/WDFpreview_BMc.doc
MH.