?

Log in

No account? Create an account
A random question for people with science leanings: I need a… - The Mad Schemes of Dr. Tectonic [entries|archive|friends|userinfo]
Beemer

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

[Sep. 23rd, 2005|01:57 pm]
Beemer
A random question for people with science leanings:

I need a probability distribution, but its shape isn't really known. That inclines me to use a gaussian as a generic, because most things are gaussian.

However, it can't produce negative values. What would you use?

Truncate the gaussian below zero? Fold negative values back to positive? Something like a chi-square distribution that looks gaussian once the mean is far from zero?
LinkReply

Comments:
[User Picture]From: melted_snowball
2005-09-23 01:24 pm (UTC)
Is it continuous? Would the less-than-zero problem ever actually happen, or are you just annoyed that it does?

If it's continuous and "probably normal" because it's kind of like the sum of a bunch of observations, you probably want a gamma == sum of exponential variables.

If it's discrete and "probably normal" for the same reason, you probably want negative binomial == sum of geometrics or just plain old binomial == sum of Bernoullis.
(Reply) (Thread)
[User Picture]From: dr_tectonic
2005-09-23 02:46 pm (UTC)
It's continuous, and the problem is likely to happen.

Basically, we have an agent-based model where all our little software people are deciding whether or not to evacuate based on whether the number of warning signs they've seen has hit their panic threshold yet. The distribution of thresholds is what I'm concerned with here.

It doesn't really make sense for the threshold to be negative, but there's a fraction of the population will evacuate given any provocation at all, so their threshold is basically zero. It's "probably normal" on the grounds that, well, it's people. So it's really based on a jillion unobservables, but we're calling it (likely normal) random noise.

(Might need to use it for a couple other things, like response delay and attention paid to official announcements, both of which make no sense being negative.)

I was thinking about a gamma distribution, actually. I just have to remember what the hell ranges of coefficients gives you halfway-normal looking curves. Thanks!
(Reply) (Parent) (Thread)
[User Picture]From: melted_snowball
2005-09-23 03:03 pm (UTC)
This seems weird.

I realize I don't understand your way of thinking about this concept. But let me propose a way of thinking that makes more sense to me, at least:

Let x be the rv that is the number of times a person needs to be warned before they decide to evacuate. [I realize that in your model, probably warnings have weights, but for my sanity's sake, let's say all "warnings" have weight 1.] A person's threshold is the number of warnings that it takes before they run.

It seems to me that what you actually have is just an infinite-state Markov chain modeling the people: it has states x_0, x_1, ... and RAN, where x_i's indicator i is the number of warnings the person has previously seen, if they've not evacuated yet, while RAN is the catchall state for "I ran like hell." You want to know the distribution of the time of first occupancy of RAN.

I seems odd to assume that this distribution is quasi-normal. More likely quasi-geometric sounds right to me: that would suggest that every new warning convinces the same fraction of stragglers to run.
(Reply) (Parent) (Thread)
[User Picture]From: dr_tectonic
2005-09-23 03:38 pm (UTC)
So, the agents are integrating signals from a number of sources (weighted, yes indeed) to determine their general alarm level. And once they hit their threshold, they act. That seems to me to be equivalent to what you've said above, just continuous instead of discrete.

I don't follow once we get to the infinite-state Markov chain. How does it have any implications for the distribution? In other words, it seems like the markov chain representation is just a way of making a histogram of the agents' thresholds... what am I missing?

Note also that one of the signals the agents integrate is "who else has already evacuated", which would violate one of the assumptions for markov processes, no?
(Reply) (Parent) (Thread)
[User Picture]From: melted_snowball
2005-09-23 03:44 pm (UTC)
Oh, the chain is just to think about it in a different way. The point is that if the person is truly Markovian, the most no-information idea is to wrap all of the X_i states together into a single state, with probability p of "run like hell" and probability 1-p of "stay." In this context, the distribution of thresholds is geometric, and making it normal is the odd thing, not somehow a default. [Sorry--somehow I thought I'd written that, but I guess I just thought it...]

Yes, "who's already gone" would violate the Markovianness.
(Reply) (Parent) (Thread)
[User Picture]From: dr_tectonic
2005-09-23 04:00 pm (UTC)
Ah! Okay, gotcha. Yeah, I think people are definitely non-Markovian in their decision process.

I'll fool around with gammas some. (Hopefully, when we do some sensitivity analysis of the model, one of the things we'll find out is that the details of the distribution of thresholds is not especially important. That'd be convenient.)
(Reply) (Parent) (Thread)
[User Picture]From: melted_snowball
2005-09-23 04:55 pm (UTC)
There's a few decent handbooks on basic distributions that are a sensible thing to have on your shelf. I don't have any of them (I have a mathematical statistics book), but getting one of them is sensible.
(Reply) (Parent) (Thread)
[User Picture]From: dpolicar
2005-09-24 10:55 am (UTC)
Hm.
I think the assumption that the threshold is based on the number of warnings is flawed.
I suspect there's some fraction of the population that will run on first warning, and some that won't run at all, ever. But for the ones in between, the interesting threshold is probably the % of the population that already ran. As that happens, the odds of someone you know having run goes up sharply, and consequently your odds of doing so as well.
So I'd expect a small chunk at first, an accelerating rampup, and then a sudden plunge.
(Reply) (Parent) (Thread)
[User Picture]From: melted_snowball
2005-09-24 12:18 pm (UTC)
Well, I was already assuming (correctly, it appears) that his system was working based on some way of integrating who's already run into a number.
(Reply) (Parent) (Thread)
[User Picture]From: dr_tectonic
2005-09-24 12:47 pm (UTC)
I'm actually debating whether to do it using a global "fraction evacuated" number or to do it using random encounters...

That's separate from the number of people you know personally who have evacuated, for which we need to make a simple model social network.
(Reply) (Parent) (Thread)
[User Picture]From: dr_tectonic
2005-09-24 12:31 pm (UTC)
I suspect there's some fraction of the population that will run on first warning, and some that won't run at all, ever.

Absolutely correct. (Though there are some interesting questions about won't run versus can't run.)

But for the ones in between, the interesting threshold is probably the % of the population that already ran. As that happens, the odds of someone you know having run goes up sharply, and consequently your odds of doing so as well.

Yup. The really interesting part (I suspect) will be how the signal of other people deciding to act is filtered through people's social networks, because how your friends are acting has an entirely different kind of influence than how "everybody" is acting.

It'll probably usually be a sigmoid curve, but where it tops out and how it unfolds over time are really useful details we hope the model can provide.
(Reply) (Parent) (Thread)
From: detailbear
2005-09-24 05:09 am (UTC)

Human nature note.

attention paid to official announcements, both of which make no sense being negative

While it may be a really small proportion that wouldn't affect your program, I would note that in RL, people are perverse enough that they will pay negative attention to official announcements. Tell them that they should leave, and that't the impetus for them to stay.
(Reply) (Parent) (Thread)
[User Picture]From: dr_tectonic
2005-09-24 12:44 pm (UTC)

Re: Human nature note.

I think the number of people THAT perverse, especially with regard to life-threatening emergencies, is in fact negligible. I'll have to look into whether it's negligible when it comes to things like flu shots, though. (Another system we can study with the same model.)

There's an interesting phenomenon that some subpopulations (criminals, for example) may find it advantageous NOT to evacuate if everyone else is evacuating. But if we end up modeling that, I think the way to do it will probably be to give them an extra, negative signal representing the opportunity, because nromal meaning of the other signals (mortal danger) holds just as much for them as everyone else...
(Reply) (Parent) (Thread)
[User Picture]From: melted_snowball
2005-09-24 05:31 pm (UTC)

Re: Human nature note.

I'm unconvinced that many people hear from their friends that they've gotten their flu shots. I certainly never tell anyone except da_lj that he needs to get his. I started getting mine because a nurse practitioner yelled at me...
(Reply) (Parent) (Thread)
From: toosuto
2005-09-23 02:01 pm (UTC)
I really want to be able to understand everything in this post between the word probability and and the last zero.

Sadly I am incapable of absorbing new information as my brain has garglefilzered it's nuero-rhuemitizer.

Plus my science leanings are not at nearly acute enough anlge to count.
(Reply) (Thread)
[User Picture]From: dpolicar
2005-09-23 02:22 pm (UTC)
surely you know what negative values are!
(Reply) (Parent) (Thread)
From: toosuto
2005-09-23 02:32 pm (UTC)

OK, fine be all literall and stuff: A clarification? Correction? Some stuff anyway:

I really want to be able to understand everything in this post between the words probability and produce and then truncate and the last zero.

Sadly I am incapable parsing even the request in a reasobale manner because my schnitzel is sans noodles

Plus my science leanings are not at nearly obtuse enough angle to count.

Plus I probably am deficient in the maths. Are there maths involved?
Because I am bad at them. I blame the girl I was dating when I flunked those one maths.
(Reply) (Parent) (Thread)
[User Picture]From: dr_tectonic
2005-09-23 02:35 pm (UTC)

Re: OK, fine be all literall and stuff: A clarification? Correction? Some stuff anyway:

Oh, there's lots of maths.
It's nearly all math, in fact, with only a tenuous toehold on reality.
(Reply) (Parent) (Thread)
From: toosuto
2005-09-23 02:41 pm (UTC)

Re: OK, fine be all literal and stuff: A clarification? Correction? Some stuff anyway:

Well at least I have that in common with it today.

Incoherency tastes a lot like chartreuse.
(Reply) (Parent) (Thread)
[User Picture]From: bryree
2005-09-23 03:36 pm (UTC)

Re: OK, fine be all literal and stuff: A clarification? Correction? Some stuff anyway:

Mmmmmmm
(Reply) (Parent) (Thread)
From: toosuto
2005-09-23 02:42 pm (UTC)

Pony, Pony, Pony!

And my work has only a tenuous toehold on me as well.


Weeeeeeeeeeeeee!
(Reply) (Parent) (Thread)
[User Picture]From: bryree
2005-09-23 03:45 pm (UTC)

Re: OK, fine be all literall and stuff: A clarification? Correction? Some stuff anyway:

The good news is, I've got a great new verbal attack for those (like me) who don't know what it means:

Hey, why don't you go convolve it with itself in your mind's eye.
(Reply) (Parent) (Thread)
From: toosuto
2005-09-23 03:52 pm (UTC)

Re: OK, fine be all literall and stuff: A clarification? Correction? Some stuff anyway:

that sounds really painful.
(Reply) (Parent) (Thread)
[User Picture]From: melted_snowball
2005-09-23 04:53 pm (UTC)

Re: OK, fine be all literall and stuff: A clarification? Correction? Some stuff anyway:

Um. Hmm. Part of me wants to go off and be all "I'm Dr. Math" guy and bore you to tears explaining. But the rest of me knows that on a Friday evening, I should spend more time grinning.

The second part won.

So now I'll just drop that phrase into a conversation next week, and tell you how easily it went.
(Reply) (Parent) (Thread)
[User Picture]From: ocschwar
2005-09-23 02:10 pm (UTC)
When you convolve a probability distribution with itself, it converges towards a Guassian. When the distribution is constrained to only have non-zero probability density in the positive half, it looks more like the first derivative of the Gaussian. (Take a trncated Gaussian and convolve it with itself in your mind's eye.)
(Reply) (Thread)
[User Picture]From: melted_snowball
2005-09-23 02:24 pm (UTC)
Your second sentence contradicts the first, no? The central limit theorem applies to any distribution.

[I admit that I'm awkward with the style of language you're using here, but at least that's what I get when I look at what you've written...]
(Reply) (Parent) (Thread)
[User Picture]From: ocschwar
2005-09-23 03:00 pm (UTC)
After some coffee I'll rephrase that, if I can make those who read it not lose IQ points. *sigh* It's been a day.
(Reply) (Parent) (Thread)
[User Picture]From: melted_snowball
2005-09-23 03:07 pm (UTC)
Yes, it has. I hate teaching about finite automata...
(Reply) (Parent) (Thread)
From: nosato
2005-09-23 06:21 pm (UTC)
F-Distribution?
(Reply) (Thread)