Friday, July 30, 2010

HRH07 luminosity function

Joe,
I want to use the HRH07 luminosity function for the QSO Monte Carlo
for the likelihood. Back in May you told me that it would take you an
hour to get the code together and you would point me to it, but I
don't think you ever did.

I just need an equivalent file to:
/home/jhennawi/Projects/
QSOClustering/pro/k_correct/dNdz_qso_z0.5-5.0.dat

But using the HRH07 LF not the Jiang-Combo...

I know you are really busy, but I think you've already done this for
Extreme Convolution... and once I have this I can pretty much complete
likelihood testing and write this paper.

Thanks,
Jessica

Thursday, July 29, 2010

Continuing Likelihoods

I thought perhaps combining v1 and the new version would produce a better selection algorithm, but this doesn't improve things.

The field l_ratio is computed the following way:

;Calculate lratio
eps = 1e-30
bossqsolike = total(oldlikelihood.l_qso_z[2:18],1) + total(likelihood.L_QSO_Z[start:stop],1) ;quasars between redshift 2.1 and 3.5
qsolcut = where(alog10(bossqsolike) LT -9.0)
den = total(targets.L_EVERYTHING_ARRAY[0:4],1) + total(likelihood.L_QSO_Z[0:44],1) + total(oldlikelihood.l_qso_z,1) + eps
num = bossqsolike + eps
NEWRATIO = num/den
NEWRATIO[qsolcut] = 0 ; eliminate objects with low L_QSO value
likelihood.l_ratio = newratio

Targets/dec^2------v1 threshold----new threshold
-----0------------1.00000--------1.00000
-----10-----------0.749085-------0.784113
-----20-----------0.490932-------0.530647
-----30-----------0.347334-------0.401703
-----40-----------0.253020-------0.311692
-----50-----------0.198630-------0.247766
-----60-----------0.160074-------0.201979
-----70-----------0.128912-------0.171997
-----80-----------0.107913-------0.145289
-----90-----------0.0917588------0.126122


Targets/dec^2--# Targeted QSOs:----v1----new version
-----0------------------------1---------1
-----10----------------------443-------393
-----20----------------------656-------630
-----30----------------------781-------765
-----40----------------------897-------877
-----50----------------------965-------968
-----60---------------------1035-------1030
-----70---------------------1089-------1070
-----80---------------------1123-------1111
-----90---------------------1157-------1151


This doesn't improve things much, and performs worst at 20 targets per square degree (core).

If you look at color-color plots of what was targeted by both version (white), with version 1 (cyan) versus the new version (magenta):


We do seem to be better at targeting higher redshift QSOs with the new method, and missing more lower redshift QSOs, so perhaps this means we should use this method even though it doesn't improve our absolute numbers:

Monday, July 19, 2010

Likelihood (again)

Here is the final push to get the likelihood algorithm finalized for the rest of the survey....

Where we last left off, I had been trying to optimize the QSO Catalog, but had been setting the thresholds wrong. Once I fixed that, I tried again, and realized that something was going wrong with my attempts to change the inputs to the QSO Monte Carlo, such that it was failing. So now let's try again.

First step, reproduce old likehood results (v1):
../logs/100719_2log.pro

saved in the files:
../like/likeOldNewthresholds.fits
../like/qsolikeOldNewthresholds.fits

The field l_ratio is computed the following way:
eps = 1e-30
bossqsolike = total(likelihood.L_QSO_Z[2:18],1) ;quasars between redshift 2.1 and 3.5
qsolcut = where(alog10(bossqsolike) LT -9.0)
den = total(targets.L_EVERYTHING_ARRAY[0:4],1) + total(likelihood.L_QSO_Z[0:18],1) + eps
num = bossqsolike + eps
NEWRATIO = num/den
NEWRATIO[qsolcut] = 0 ; eliminate objects with low L_QSO value
likelihood.l_ratio = newratio

Here is the distribution of the likelihood ratios of v1 (white) and new (green) of the data used to set the thresholds:


Comparing the thresholds I find for v1 compared with Myers' they are within 10% of each other:

Targets/dec^2------My threshold----Myers Threshold
-----20-----------0.490932--------0.533
-----40-----------0.253020--------0.235

Now comparing v1 of the likelihood with a catalog with the same luminosity function and inputs, but a larger redshift range, we get the following results:

Targets/dec^2------v1 threshold----new threshold
-----0------------1.00000--------1.00000
-----10-----------0.749085-------0.625018
-----20-----------0.490932-------0.360219
-----30-----------0.347334-------0.249564
-----40-----------0.253020-------0.172273
-----50-----------0.198630-------0.134904
-----60-----------0.160074-------0.112677
-----70-----------0.128912-------0.0921011
-----80-----------0.107913-------0.0766008
-----90-----------0.0917588------0.0668184


Targets/dec^2--# Targeted QSOs:----v1----new version
-----0------------------------1---------1
-----10----------------------443-------408
-----20----------------------656-------644
-----30----------------------781-------781
-----40----------------------897-------915
-----50----------------------965-------977
-----60---------------------1035-------1031
-----70---------------------1089-------1080
-----80---------------------1123-------1117
-----90---------------------1157-------1160


Here is the distribution of the likelihood ratios of v1 (white) and new (green) of the quasars:


This doesn't improve things much, and performs worst at 20 targets per square degree (core). It might be because the statistics are now smaller in the BOSS redshift range (I kept the simulation at 10M objects, but now it is distributed over a larger redshift range).

If you look at color-color plots of what was targeted by both version (white), with version 1 (cyan) versus the new version (magenta):


The redshift distribution shows that the new method is finding some higher redshift QSOs however:

Thursday, July 15, 2010

Next Step?

I've created reconstructions that are similar to Alexia's configuration. They are in the following directories:

workingDir = 'run201077_1424'
workingDir = 'run2010712_1255'

And the following log files have the code I used to generate these:
../logs/100714log.py
../logs/100714_2log.py

Specs of the runs two runs
Run 1:
workingDir = 'run201077_1424'

#Angular Correlation
oversample = 5.
corrBins = 40.0 #should be one more than you want it to be
mincorr = 6.0
maxcorr = 12.0
convo = 180./pi
corrspace = (maxcorr - mincorr)/corrBins
corrarray = makecorrarray(mincorr, maxcorr, corrspace)

#Auto Correlation
nbins = 40
boxside = 1000.0 #in Mpc/h
xicorrmin = 5.0
xicorrmax = 50.0
nxicorr = 40

#make redshift bins
#boxside = 1000.0 #in Mpc/h
rbinmin = 0.0 #in Mpc/h
rbinspace = boxside/10 #in Mpc/h
rbinmax = boxside/2.0 + 1.0 #in Mpc/h
rbins = makeRbins(rbinmin,rbinmax,rbinspace)
nrbins = rbins.size - 1

#Catalog sizes
psubsize=100000
ssubsize=100000

Run 2 (larger):
workingDir = 'run2010712_1255'

#Angular Correlation
oversample = 5.
corrBins = 40.0 #should be one more than you want it to be
mincorr = 6.0
maxcorr = 12.0
convo = 180./pi

#Auto Correlation
boxside = 1000.0 #in Mpc/h
xicorrmin = 5.0
xicorrmax = 50.0
nxicorr = 40

#make redshift bins
#boxside = 1000.0 #in Mpc/h
rbinmin = 0.0 #in Mpc/h
rbinspace = boxside/20 #in Mpc/h
rbinmax = boxside/2.0 + 1.0 #in Mpc/h
rbins = makeRbins(rbinmin,rbinmax,rbinspace)
nrbins = rbins.size - 1

#Catalog sizes
psubsize=1000000
ssubsize=1000000

I've uploaded the run directories onto the repository for Alexia to look at, but I'm getting the same problems as before.

I've also updated the files for running and re-running the reconstruction. The primary change is that I now save all the run constants into the run directory, including the masks, so that everything can be recovered when you go back to an old run without looking at the initial run directory.

The code to run a full reconstruction in a new directory is here:
../pythonjess/jessreconstructionFull.py

The code to read in all the run constants from an old run (where the correlation functions have already been computed) and perform the reconstruction is here:
../pythonjess/jessreconstruction.py

Simply change first line of the code, to the working directory of the old run:
workingDir = 'run2010712_1255'

You can also change line 35 to skip redshift bins with too noisy correlation functions:
skip = 0

If all you want to do is rebin the reconstruction, use the following code:
../pythonjess/rebinjessrecontruction.py

And change the workingDir, npbins, rmin and rmax to be the appropriate run directory, number of reconstruction bins, min redshift bin, and max redshift bin.

Or you can fun the following function from correlationData:
reconstructionrebin(workingDir, npbins, rmin, rmax)

Alexia is moving and is still looking to see if she can figure out where the bug is, but it seems to me I am at the point where I need to possibly start from her working reconstruction, and step by step implement my changes to see where things break down.

In the mean time, I need to write the likelihood paper, so I'm going to work on that for now until Alexia is back online. This sucks.

Wednesday, July 7, 2010

Alexia Mimic Reconstruction

In order to try to pin down the differences between Alexia and my reconstruction, I've tried to mimic the conditions under which she did her reconstruction for her paper.

The binning is as follows:
#Angular Correlation
corrBins = 40.0 #should be one more than you want it to be
mincorr = 6.0 #in degrees
maxcorr = 12.0 #in degrees

#Auto Correlation
xicorrmin = 5.0
xicorrmax = 50.0
nxicorr = 40

#make redshift bins
rbinmin = 0.0 #in Mpc/h
rbinspace = 100.0 #in Mpc/h
rbinmax = 500.0 #in Mpc/h
nrbins = 5.0

Here are plots...

Correlation Functions

Distribution Functions and Data

Best Fit Reconstruction

A couple things I am noticing...
First: I still get a problem with changing the binning and the reconstruction going all weird.
Second: Even in this best fit scenario I am getting a mis-fit between the reconstruction and the data, could there be a problem with the lining up of my bins? Like perhaps something with left/right binning in the histogram?
Third: The reconstruction tends to go weird at the ends, even though both ends are contained within the redshift bins.

Some other reconstructions that aren't working as well:

The log file for this is here: ../logs/100707log.py
The run directory is here: ../pythonJess/run201077_1424