Efficiency and Completeness (again)

Finally I think I am doing this right. The log file with this code is here:
.../logs/110209log.pro

Below is an email to Adam explaining my problems, and newest findings
~~~~~~~
Adam,
For some reason when I wget the file, and the file is already in my directory, it doesn't overwrite it, but saves it as a different version:

BOSS_Quasars_3PCplus_v1.1_wcomp.fits.6

I didn't realize this was happening until just now, and so I kept re-downloading it, but then using the old version of the file (because BOSS_Quasars_3PCplus_v1.1_wcomp.fits was still the old version). Sorry about that!

I'm now getting the same numbers as you for the following:

print, n_elements(where(stripe82targets.like_ratio_core gt 0.543214))/45.93

20.03

spherematch, stripe82targets.ra, stripe82targets.dec, stripe82quasars.ra, stripe82quasars.dec, 2./3600, i1, i2, d12
print, n_elements(where(stripe82targets[i1].like_ratio_core gt 0.543214))/45.93

10.47

In terms of filling out your table I get the following numbers:

Here are the column headings (i've modified them slightly):
1) Spec Completeness
2) Area (deg^2)
3) Total fibers /deg^2 at like thresh > 0.543214
4) # z > 2.15 quasars confirmed / deg^2
5) # z < 2.15 quasars confirmed (low redshift) / deg^2
6) others that got a fiber (not quasars) / deg^2
7) not confirmed (didn't get fiber) / deg^2
8) Total known z > 2.15 quasars / deg^2
9) Total known z < 2.15 quasars / deg^2

(1) (2) (3) (4) (5) (6) (7) (8) (9)
0.5 45.93 20.03 10.47 0.348 2.24 6.97 24.21 18.05
0.6 37.80 20.45 11.24 0.397 2.27 6.53 25.61 15.77
0.7 26.61 21.12 11.01 0.526 2.56 7.03 25.82 11.56
0.8 14.02 22.40 10.70 0.856 3.21 7.63 24.05 6.065
0.9 2.124 18.36 8.473 0.941 1.41 7.53 20.24 5.178

Note, that as you say, col (4 + 5 + 6 + 7) = col 3

So I understand what you are saying that most of the targets that didn't get a fiber (column 7) are not quasars because they weren't found with the coadded data.

So it seems like you are saying that I should be defining the efficiency as:

col (4) / col (3) = 52% (@ spec completeness 0.5)

And the completeness as:
col (4) / col (8) = 43$ (@ spec completeness 0.5)


These numbers change the following way with spec competeness:

spec comp efficiency completeness
0.6 55% 44%
0.7 50% 43%
0.8 48% 44%
0.9 46% 42%

So which of these should I use? You had mentioned there was another step....
Jessica



Below is the code I used to generate these numbers:
; Read in the targeting file
adamfile = '/home/jessica/boss/chunk1_wcomp.fits'
targets = mrdfits(adamfile, 1)
stripe82targets1 = targets
newratio = stripe82targets1.LIKE_RATIO_CORE

; Read in the truthtable file
truthfile = '/home/jessica/boss/BOSS_Quasars_3PCplus_v1.1_wcomp.fits'
collateinfo1 = mrdfits(truthfile, 1)
stripe82data1 = collateinfo1
;High human confidence quasars (with z > 2.15)
confcut = where(stripe82data1.z_conf_person GE 2 and stripe82data1.Z_PERSON GE 2.15)
stripe82quasars1 = stripe82data1[confcut] ; with redshift > 2.15
lowqsocut = where(stripe82data1.z_conf_person GE 2 and stripe82data1.Z_PERSON GE 0.5 and stripe82data1.Z_PERSON LT 2.15)
stripe82lowquasars1 = stripe82data1[lowqsocut] ; with 0.5 < redshift < 2.15

compthresh = 0.9
wphot = where(stripe82targets1.completeness ge compthresh)
wspec = where(stripe82quasars1.completeness ge compthresh)
wdata = where(stripe82data1.completeness ge compthresh)
wlow = where(stripe82lowquasars1.completeness ge compthresh)

areas = mrdfits('/home/jessica/boss/area_wcomp.fits',1)
areacol = where(areas.comp_thresh ge compthresh)
thisArea = areas[areacol[0]].area

stripe82targets = stripe82targets1[wphot]
stripe82quasars = stripe82quasars1[wspec]
stripe82data = stripe82data1[wdata]
stripe82lowqso = stripe82lowquasars1[wlow]

area = 219.9 ; we know the exact area of the chunk1 file
tpsd = round(20.0*area)
sortRatio = targets[reverse(sort(targets.LIKE_RATIO_CORE))].LIKE_RATIO_CORE
threshold = sortRatio[tpsd-1]

;Comp threshold
print, compthresh

;Area
print, thisArea

;Like Threshold
print, threshold

;Likelihood Targets per square degree
thistpsd = n_elements(where(stripe82targets.like_ratio_core gt threshold))/thisArea
print, thistpsd

spherematch, stripe82targets.ra, stripe82targets.dec, stripe82quasars.ra, stripe82quasars.dec, 2./3600, i1, i2, d12
; Likelihood Quasars targeted per square degree
qpsd = n_elements(where(stripe82targets[i1].like_ratio_core gt threshold))/thisArea
print, qpsd


spherematch, stripe82targets.ra, stripe82targets.dec, stripe82lowqso.ra, stripe82lowqso.dec, 2./3600, i1, i2, d12
; Likelihood Low Redshift Quasars targeted per square degree
lqpsd = n_elements(where(stripe82targets[i1].like_ratio_core gt threshold))/thisArea
print, lqpsd

spherematch, stripe82targets.ra, stripe82targets.dec, stripe82data.ra, stripe82data.dec, 2./3600, i1, i2, d12
; Others targeted per square degree
otpsd = n_elements(where(stripe82targets[i1].like_ratio_core gt threshold))/thisArea
print, otpsd - (qpsd + lqpsd)

;Likelihood fibers not targeted
print, thistpsd-otpsd


;Total z > 2.15 qsos
totalqsos = n_elements(stripe82quasars)/thisArea
print, totalqsos

;Total 0.5 < z < 2.15 qsos
totallowqsos = n_elements(stripe82lowqso)/thisArea
print, totallowqsos

;Efficiency
print, qpsd/thistpsd

;Completeness
print, qpsd/totalqsos