Understanding Society User Support

PS In the FAQ on pooling cross-sectional data you say: "We strongly recommend that a non-zero value of the
weight variable is used to define the analysis base (see example below)." Removing individuals with a zero weight from my analytic sample produces the same result as specifying "if weight > 0" in the analysis itself. So, can I assume that removing individuals with a weight of zero from my analytic sample is okay?

Marie,

I am not sure about your question. If you use weights correctly people with 0 weights won't contribute to your estimate. If you drop them first (I would discourage from this and would just rely on weights to do the correct job for you) - the weighted estimate should be identical, including std and CI.

If you want to avoid many zero weights you may want to create a tailored weight.

Does this help?
Olena

Hi Olena,

Interesting! Let me give you an example that I just tested:

I have a sample of n = 2,671. Of these, 396 have a zero weight. Removing these from my analytic sample results in n = 2,275.

Let's say I want to test the effect of sex on SDQ total difficulties. I use:

svyset psu [pweight = weight], strata(strata)

svy: regress sdq_td sex

Three scenarios

1. I remove individuals with a zero weight from my analytic sample (n = 2,275) and run the above:
n = 2,275, df = 449, b = -0.024, SE = 0.360

2. I do not remove individuals with a zero weight from my analytic sample (n = 2,671) and run the above:
n = 2,671 , df = 496 , b = -0.024, SE = 0.352

3. I do not remove individuals with a zero weight from my analytic sample (n = 2,671) and run the above but specify "if weight > 0" in the regression:
n = 2,275, df = 449, b = -0.024, SE = 0.360

Scenario 1 and 3 result in exactly the same results. Scenario 2 results in the same b, but in a different SE (which I assume is due to the larger n). In scenario 3, I do what you say in the FAQ: "We strongly recommend that a non-zero value of the weight variable is used to define the analysis base (see example below)." And this is the same as removing individuals with zero weights from my analytic sample beforehand.

What am I missing?

Best wishes,
Marie

Marie,

Are all of your 3 scenarios weighted? And do you use the same svyset for them?

Thanks,
Olena

Hi Olena,

Yes, I use the same svyset (and therefore all 3 scenarios are weighted). I guess this is evident also in the coefficient which is exactly the same. The difference in scenario 2 is the n, df, SE, p, and CI. What I also noted: while the number of strata is the same in all three scenarios (because strata with no population members were omitted in scenario 2), this is not true for PSUs (there are more PSUs in scenario 2 than in scenarios 1 and 3).

Thank you!

Marie

Marie,

Yes, I see what you mean, and I was able to replicate your results. By expectation this should not happen. But Stata recognizes that this may happen in some circumstances:
"svy commands handle zero sampling weights properly. Standard commands ignore any observation with a weight of zero. Usually, this will yield the same standard errors, but sometimes
they will differ. Sampling weights of zero can arise from various postsampling adjustment
procedures. If the sum of weights for one or more PSUs is zero, svy and standard commands
will produce different standard errors, but usually this difference is very small."

from p. 104 https://www.stata.com/manuals/svy.pdf

Hope this helps,
Olena

Hi Olena,

Very helpful - thank you! Now I am confident that I did use the weights correctly.

I guess it is on me to decide whether or not to include youth with zero weights in my analytic sample. As some of my p-values are very close to .05, I'd rather take the more conservative approach: remove youth with zero weights from my analytic sample, so SEs are based on youth that are actually considered in my analysis.

For a better understanding, I wanted to check: if you typically assume that including individuals with zero weights will result in the same estimates, why does the FAQ say: "We strongly recommend that a non-zero value of the weight variable is used to define the analysis base (see example below)."

https://www.understandingsociety.ac.uk/sites/default/files/downloads/general/weighting_faqs.pdf (page 9)

Thank you very much for your help!

Best wishes,
Marie

Marie,

The FAQ is not related to the question you have. It relates to a pooled dataset where if you pool over many years some years (like BHPS years) will have a smaller sample size than waves from UKHLS - in a pooled analysis this will skew it towards later years. The FAQ talks about how to correct for this - by scaling totals of non-zero weights (non-zero weight totals will be sample sizes for each wave used in your analysis, so that's what needs scaling in that situation).

Best,
Olena

Hi Olena,

Thank you!

What confused me is this part:

svyset psu2011 [pw=weight2011], strata(strata2011) singleunit(centered)

svy: proportion jbstat2011 if weight2011>0

You seem to exclude individuals with a zero weight from the analysis. If I understand correctly, this part has nothing to do with the scaling itself, but is simply an example for a possible analysis. If weighting accounts for zero weights automatically (i.e., I do not need to remove them beforehand), why do you specify "if weight2011>0" here. Anyway, I may come back to the question about scaling in issue #1608 (after we clarified whether tailoring my own weight would be a good idea and what a suboptimal weight would be). [This issue can be marked as resolved, as the question about scaling does not really belong here.]

Best wishes,
Marie

Marie,

I can confirm that "if weight2011>0" is not necessary in this example and should result in the same estimates (by expectation) if this were not specified.

Thanks,
Olena

Thanks!