Sampling theory—"best" explanation

[stratum]

"once in the digital domain, samples represent perfect impulses"

Apparently that's in the very idea that has caused so much disagreement.
I consider it to be linguistic problem:)
What is a perfect or an ideal impulse? One cannot define that without resorting to calculus, but calculus is about continuous domains, not discrete. One can only fix that problem by imposing a mental picture(*) from the analog world into the digital, and indeed this is what earlevel does in that blog. Sort of, this is as best as the job could be done, there is no other way or you have to drop that word. Once you drop that idea though, what sense does it make to talk about the spectrum of the sampled digital signal and say that this specific interpretation is preferable? It's a totally arbitrary representation then, so there is no way around it. The math you find in text books would only say "DFT of it", with whatever further meaning you can assign is totally up to you. One can say DFT is periodic and therefore the spectra repeats just as its time domain does, but that's a requirement from the definition of DFT anyway. Where does DFT itself come from? Discrete analogous version of continuous Fourier transform, of course. Somehow it looks like there is no escaping from that analog mental picture, after all the whole story of DSP begins with sampling an analog signal.

(*) http://www.iep.utm.edu/intentio/

[earlevel]

"once in the digital domain, samples represent perfect impulses"

Apparently that's in the very idea that has caused so much disagreement.
I consider it to be linguistic problem:)
What is a perfect or an ideal impulse? One cannot define that without resorting to calculus, but calculus is about continuous domains, not discrete. One can only fix that problem by imposing a mental picture(*) from the analog world into the digital, and indeed this is what earlevel does in that blog. Sort of, this is as best as the job could be done, there is no other way or you have to drop that word. Once you drop that idea though, what sense does it make to talk about the spectrum of the sampled digital signal and say that this specific interpretation is preferable? It's a totally arbitrary representation then, so there is no way around it. The math you find in text books would only say "DFT of it", with whatever further meaning you can assign is totally up to you. One can say DFT is periodic and therefore the spectra repeats just as its time domain does, but that's a requirement from the definition of DFT anyway. Where does DFT itself come from? Discrete analogous version of continuous Fourier transform, of course. Somehow it looks like there is no escaping from that analog mental picture, after all the whole story of DSP begins with sampling an analog signal.

(*) http://www.iep.utm.edu/intentio/

Well...not to quibble with your comment, it's fine, but maybe worth discussing: I didn't use the analog world in order to impose a mental picture of, or describe, the digital domain. Analog sampling came first, and that's what I represented. Digitizing is just a small detail (it ends up being an incredibly useful detail, because analog computing and storage is more difficult than digital) on top of that.

I wasn't there, but I believe this wasn't a case of mathematicians figuring things out and then we did it. We did it, then had to account for the interesting limitations observed. There is nothing new added by taking it digital. I don't think the mathematical notion of an impulse is any more difficult that of infinity, but I'll leave others to argue about that.

BTW, some things become obvious in my explanation of PAM/PCM that are otherwise not as apparent with the usual explanations of sampling. I don't have to say, explicitly, that you must ensure that the sample rate is greater than twice the highest frequency. That limitation is apparent—the "lower sideband" (to borrow a radio term) will step on the audio band if the source signal has frequencies above half. It requires no further explanation than pointing that out.

[stratum]

I don't think the mathematical notion of an impulse is any more difficult that of infinity, but I'll leave others to argue about that.

It requires assuming a vertical rectangle whose width is infinitely small, and its height is some unknown large number but its area is nevertheless taken to be the specific value that you would consider to be the value of the sample. Nothing other than calculus would express the same idea in a neat way therefore the idea of an ideal impulse necessarily belong to continous domains.

BTW, some things become obvious in my explanation of PAM/PCM that are otherwise not as apparent with the usual explanations of sampling. I don't have to say, explicitly, that you must ensure that the sample rate is greater than twice the highest frequency. That limitation is apparent—the "lower sideband" (to borrow a radio term) will step on the audio band if the source signal has frequencies above half. It requires no further explanation than pointing that out.

Pretty neat I think.

Returning back to DFT, admittedly it has a meaning even without assuming that samples were taken from an analog signal, namely a correlation with sinusoidal basis sin(nx), cos(nx) but.. the meaning of the time domain signal itself necessarily depends on sampling. i.e. the applicability of the vocabulary of a continous domain is taken to be granted anyway. It's call the sampling theory after all. People talk about leaky integrators for example, not leaky summation.

[Oden]

"once in the digital domain, samples represent perfect impulses"

Apparently that's in the very idea that has caused so much disagreement.
I consider it to be linguistic problem:)
What is a perfect or an ideal impulse? One cannot define that without resorting to calculus, but calculus is about continuous domains, not discrete. One can only fix that problem by imposing a mental picture(*) from the analog world into the digital, and indeed this is what earlevel does in that blog. Sort of, this is as best as the job could be done, there is no other way or you have to drop that word. Once you drop that idea though, what sense does it make to talk about the spectrum of the sampled digital signal and say that this specific interpretation is preferable? It's a totally arbitrary representation then, so there is no way around it. The math you find in text books would only say "DFT of it", with whatever further meaning you can assign is totally up to you. One can say DFT is periodic and therefore the spectra repeats just as its time domain does, but that's a requirement from the definition of DFT anyway. Where does DFT itself come from? Discrete analogous version of continuous Fourier transform, of course. Somehow it looks like there is no escaping from that analog mental picture, after all the whole story of DSP begins with sampling an analog signal.

(*) http://www.iep.utm.edu/intentio/

Well...not to quibble with your comment, it's fine, but maybe worth discussing: I didn't use the analog world in order to impose a mental picture of, or describe, the digital domain. Analog sampling came first, and that's what I represented. Digitizing is just a small detail (it ends up being an incredibly useful detail, because analog computing and storage is more difficult than digital) on top of that.

I wasn't there, but I believe this wasn't a case of mathematicians figuring things out and then we did it. We did it, then had to account for the interesting limitations observed. There is nothing new added by taking it digital. I don't think the mathematical notion of an impulse is any more difficult that of infinity, but I'll leave others to argue about that.

BTW, some things become obvious in my explanation of PAM/PCM that are otherwise not as apparent with the usual explanations of sampling. I don't have to say, explicitly, that you must ensure that the sample rate is greater than twice the highest frequency. That limitation is apparent—the "lower sideband" (to borrow a radio term) will step on the audio band if the source signal has frequencies above half. It requires no further explanation than pointing that out.

Again, it seems to me that you are engaging in philosophical discussion of a mathematical theorem. There is plenty that is "added" by thinking in discrete, most of the analytical basis or limits wouldn't apply as anything but approximations, the definition of an impulse would also be very easy in discrete.

If you want to teach people about the Nyquist Shannon sampling theorem, why not just label it as an explanation of the theorem and cut the philosophical notions? We have no proof that the world is continuous and we know that the impulses that come out of DACs aren't ideal impulses, from the engineering perspective none of it matters as long as the theorem works reasonably well.

[Oden]

delete

[Oden]

delete.

[earlevel]

If you want to teach people about the Nyquist Shannon sampling theorem, why not just label it as an explanation of the theorem and cut the philosophical notions? We have no proof that the world is continuous and we know that the impulses that come out of DACs aren't ideal impulses, from the engineering perspective none of it matters as long as the theorem works reasonably well.

You're thinking too academic here. My blog notes up in the upper right, "Practical digital audio signal processing". There are people who sit around concerned that there is no proof that the world is continuous, and those who code algorithms and make music. I didn't go into DACs at all—I didn't write a book, I wrote a short article—but for practical signal processing it doesn't matter a bit that you can't make ideal impulses. We are so lucky that our ear don't have a frequency response of DC to light. (As I noted here, we don't bother to go for impulses in DACs anyway, but we could make an impulse based DAC good enough for the audio range if we wanted to.)

PS—Sorry if I sound abrupt, just trying to get to the point, and time constrained. Thanks for your comment.

[Oden]

If you want to teach people about the Nyquist Shannon sampling theorem, why not just label it as an explanation of the theorem and cut the philosophical notions? We have no proof that the world is continuous and we know that the impulses that come out of DACs aren't ideal impulses, from the engineering perspective none of it matters as long as the theorem works reasonably well.

You're thinking too academic here. My blog notes up in the upper right, "Practical digital audio signal processing". There are people who sit around concerned that there is no proof that the world is continuous, and those who code algorithms and make music. I didn't go into DACs at all—I didn't write a book, I wrote a short article—but for practical signal processing it doesn't matter a bit that you can't make ideal impulses. We are so lucky that our ear don't have a frequency response of DC to light. (As I noted here, we don't bother to go for impulses in DACs anyway, but we could make an impulse based DAC good enough for the audio range if we wanted to.)

PS—Sorry if I sound abrupt, just trying to get to the point, and time constrained. Thanks for your comment.

But if you are going to write something "practical", the technical mumbo jumbo should be avoided. I find it a bit difficult to believe that you are given that the BIG SECRETS are not practice related but instead detail the sampling theorem (without stating it as such).

We could make an impulse based DAC, but would what came out be ideal impulses? Given that whatever the impulse size was I could always tell you that it's not slim enough... I think not.

[earlevel]

But if you are going to write something "practical", the technical mumbo jumbo should be avoided. I find it a bit difficult to believe that you are given that the BIG SECRETS are not practice related but instead detail the sampling theorem (without stating it as such).

We could make an impulse based DAC, but would what came out be ideal impulses? Given that whatever the impulse size was I could always tell you that it's not slim enough... I think not.

I think you are missing the point. You keep coming back to DACs. I know about DACs, I can talk about DACs—I'm an electrical engineer—but that's not what the article is about. The people I'm addressing buy converters. I'm addressing people who are writing audio DSP algorithms. The "BIG SECRETS" (I didn't call them that, you did—I even point out that "secrets" is an exaggeration, but apparently not understood by some) are that samples represent impulses, and there is zero in between (5,-2,1 cranked out at 1X produces the same spectrum as 5,0,-2,0,1,0 at 2x). The third is an important observation that follow from the first two points.

I don't see the point of turning my simple explanation into a complicated one. I'm sure you're not asking me to make these points because you don't understand, you're asking me to ensure that others understand. But, if they don't understand this, they aren't going to understand a more complicated explanation. I've had people suggest I add discussion of "Hilbert spaces, shift-invariance, Riesz representation theorem", and "shift-invariant spaces, eigenfunctions, harmonical analysis, and the rest of the cool stuff".

The "best explanation you've ever heard" does not mean "most complete" (this is not a book), it means that my goal is that someone, somewhere, will think (for the eventual video with more compelling graphics), "wow, this is the best explanation I've ever heard!".

[kryptonaut]

Thanks for posting your thoughts on digital sampling - it's always good to try to clarify these things, and an alternative perspective on a technical matter can be a big help in creating a more intuitive understanding.

However, if I can make a couple of (hopefully constructive) comments on your articles:

Firstly, I think there is possible confusion when talking about an 'audio sample' or a 'digital sample', as these terms are frequently used to mean a digital representation of a whole sound, rather than an individual value at a fixed point in time. It might be worth finding some terminology to avoid this potential confusion, especially if you are aiming your article at the novice. I think point 3 on your third post uses 'sample' to mean essentially 'digitised clip' whereas points 1 and 2 use 'sample' to mean an individual digital value.

Secondly - and you might disagree here - I think it's much clearer to think of a digital sample as simply a measurement of the analog signal at an instant in time, conveying no information about what values the analog signal took between this and the next or previous measurement. After all, that's what the word 'sample' means. Of course, by ensuring the analog signal had certain bandlimited properties before sampling, it is theoretically possible to reconstruct it accurately from these instantaneous measurements (together with the knowledge of the bandlimited properties) - and this reconstruction may involve generating an analog approximation to a scaled impulse from each digital sample, but I don't think it's really accurate to say that a digital sample is a scaled impulse any more than an array of numbers is a sound. So in your first post, where you say "conversion to discrete time adds high frequency components not in the original signal," I think it would be more correct to say "conversion from discrete time to an impulse stream adds high frequency components not in the original signal," and these components must subsequently be filtered out to reconstruct the analog signal.

I hope that makes some kind of sense - I think it's a great idea to cast some light on a theory that can sometimes seem a bit arcane, but I also think it's essential to avoid causing further confusion by using possibly ambiguous terminology.

[earlevel]

Thanks for posting your thoughts on digital sampling - it's always good to try to clarify these things, and an alternative perspective on a technical matter can be a big help in creating a more intuitive understanding.

However, if I can make a couple of (hopefully constructive) comments on your articles:

Firstly, I think there is possible confusion when talking about an 'audio sample' or a 'digital sample', as these terms are frequently used to mean a digital representation of a whole sound, rather than an individual value at a fixed point in time. It might be worth finding some terminology to avoid this potential confusion, especially if you are aiming your article at the novice. I think point 3 on your third post uses 'sample' to mean essentially 'digitised clip' whereas points 1 and 2 use 'sample' to mean an individual digital value.

Secondly - and you might disagree here - I think it's much clearer to think of a digital sample as simply a measurement of the analog signal at an instant in time, conveying no information about what values the analog signal took between this and the next or previous measurement. After all, that's what the word 'sample' means. Of course, by ensuring the analog signal had certain bandlimited properties before sampling, it is theoretically possible to reconstruct it accurately from these instantaneous measurements (together with the knowledge of the bandlimited properties) - and this reconstruction may involve generating an analog approximation to a scaled impulse from each digital sample, but I don't think it's really accurate to say that a digital sample is a scaled impulse any more than an array of numbers is a sound. So in your first post, where you say "conversion to discrete time adds high frequency components not in the original signal," I think it would be more correct to say "conversion from discrete time to an impulse stream adds high frequency components not in the original signal," and these components must subsequently be filtered out to reconstruct the analog signal.

I hope that makes some kind of sense - I think it's a great idea to cast some light on a theory that can sometimes seem a bit arcane, but I also think it's essential to avoid causing further confusion by using possibly ambiguous terminology.

Good points, don't disagree, but to elaborate:

Point #1: My #1 states "individual digital samples", so no ambiguity there. For point #2, I think it's clear that following #1 that when I say there are zeros between the samples, I mean each and not just confined to being between some individual samples and not others. If not, it becomes clearer later on (these are bullet points, after all). Yes, #3 (parsing, "...samples...don't represent...audio") necessarily refers to a group samples and not arbitrary individual ones. That's certainly the one that requires some unstated understanding of the concepts involved. But again, it's a bullet point, where brevity is key. Bullet points rarely explain everything you need to know—they are are there to be sticky for your brain. Anyway, agree completely on your point, just thinking it through here, and probably won't change it.

One your point #2, I do agree. It's more intuitive to explain capturing regular momentary levels—like motion under a strobe light, video frames, etc. But...The problem is that it's not convenient to move forward mathematically from there.

As I noted, I would rewrite the video script from scratch each time I would get back to it (a few months in between, it made more sense to start over that to wedge/mod other ideas into the previous draft). On most (all?) of these I started out with an explanation of taking measurement of regular points in time. But I want to get to the modulation in PCM, so I need to explain how it's equivalent to...This time I decided to skip that and go right to "sampling is equivalent to this...". The biggest change in the final draft (er, of the article, which is the first draft of the video) is that I added the history of analog sampling. Part of the logic there is I wanted to get the point across that "digital" is just a convenience. The magic is in the PAM, in the analog domain. PAM is very easy to explain mathematically, you need only accept how amplitude modulation works (the video will so a lot better here—I made the executive decision that I didn't have time to make a really nice AM-lab widget for the article). Veteran synthesists and various types of engineers will have an advantage there, certainly.

[earlevel]

After thinking about it, maybe the reason some suggest expansion far beyond what I intended (not referring to your suggestions, Krytonaut, but the Hilbert spaces, Riesz representation theorem, eigenfunctions suggestions I mentioned) is that they are perhaps looking at this like a dissertation? Therein, one proves he understands something fully, by encapsulating a collection of knowledge and observations about a particular subject. The primary readers are those who already know the subject, and critique how well the writer conveyed that breadth of knowledge. Just a thought...